localairecipes launch configs for local inference

GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX layer split MTP3 393K prompt reuse (mannydelrio)

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by alexvnesta (server + ExLlamaV3 patches)↗, Flun (original 2x CMP 170HX layer-split recipe)↗, turboderp (ExLlamaV3, EXL3 quant)↗, cmpunlocker (amoghmunikote)↗ · listed by mannydelrio1

Initial version · updated 2 h ago · license CC-BY-4.0

chat code mtp-speculative long-context exl3 q8-kv layer-split prompt-cache

Everyday GLM-5.3-Flash setup on 2x CMP 170HX unlocked to 64 GB (PCIe Gen2 x4, 180 W each, 32 GB RAM, 850 W PSU): alexvnesta's ExLlamaV3 server in layer-split mode, MTP head drafting 3 tokens, Q8 KV, 392,960-token context. The prefix cache stays on, so a follow-up turn in a 48K chat starts in about 1 s instead of re-reading the chat. One request at a time. 1 user: 67.4 / 57.8 / 47.3 tok/s structured / code / prose, cold prefill 1,024 tok/s, TTFT 0.26 s. HumanEval 157/164, needle 3/3 up to 219K.

Launch command

python -u service/glm/glm_api.py -m models/GLM-5.3-Flash-exl3-3.05bpw -cq 8 -cs 458752 --context-cap 392960 -chunk_size 2048 -ambs 1 -gs 59,63 -lv --q8-staging --host 127.0.0.1 --port 8012 --recurrent-cache-gib 2 --q8-head-gpu0 --mtp -ndt 3 --fp16-draft-cache --recurrent_state_fp32

runtime

env: 'EXL3_BC_ATTN=1 EXL3_MLA_KIDX_RING=32 EXL3_INT8_GEMV=0 EXL3_GEMV=0 EXL3_DSA_QC_STAGE=1 EXL3_DSA_QC_STAGE_MIN_R=9 EXL3_AUTOSPLIT_MARGIN_MB=128 GLM53_PREFILL_CHUNK_SIZE=2048 GLM53_TP_ACTIVE_HOST=0 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false; PYTHONPATH=native:engine-src:service/glm; GLM53_NATIVE_EXT_SHA256 = sha256 of native/exllamav3_ext*.so; ulimit -n 65536'
notes: "Layer split (no -tp): GPU0 59 GB, GPU1 63 GB, output layer on GPU0 (--q8-head-gpu0; with it on GPU1, GPU1 ran out of memory after 5.5 h of traffic). MTP 3 draft tokens measured best: 2 = 59.8/52.8/49.6, 3 = 66.9/57.2/47.0, 4 = 63.2/56.0/42.1 tok/s. Why not TP2: +20% generation but -27% prefill, and on an 850 W PSU TP2's lock-step power swings cut the power, even with clocks capped at 1400 MHz. DFlash2 + hybrid is faster for single answers but has no prompt reuse in layer split."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-layer-split-mtp3-393k.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-layer-split-mtp3-393k.json

Engine

Install note: Source build as in the fork's README (setup_venv.sh, engine/apply_engine.sh, engine/build_native.sh for sm_80). One local change: in service/glm/glm_api.py the MTP guard only accepts -ndt 2; change `args.num_draft_tokens == 2` to `in (2, 3, 4)` to run -ndt 3. Launch with the fork's env exports (see runtime.env).

Model

Hardware

Context

  • max context: 392,960 tokens

KV pool

  • 448K tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
392,96011024 tok/s67.4 tok/s264 msreported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

Downloads

Revision history

Initial version.

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)