localairecipes launch configs for local inference

GLM-5.3-Flash EXL3 2.60bpw PP2 on 2x CMP 170HX DFlash2 k3

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by AlexBrandes (vLLM CMP170HX fork + plugin + this recipe + measurements)↗, turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 2.60bpw checkpoint)↗, incoai (GLM-5.3-Flash-DFlash2 drafter, CC BY-NC-ND 4.0)↗, Morrowmake (sm_80 GLM-5.3 code lineage via vllm-sm80)↗, Z.ai (GLM-5.3-Flash base model)↗ · listed by localai-recipes

Initial version · updated 5 h ago · license Apache-2.0

chat code vision pipeline-parallel vision dflash2 long-context exl3 speculative docker-build

GLM-5.3-Flash (turboderp's EXL3 2.60 bpw pack, sc_2.60bpw) served by AlexBrandes' vLLM fork at PP2/TP1 - VLLM_PP_LAYER_PARTITION=23,22, 23 target layers on card 1 and 22 plus the DFlash2 drafter (3 draft tokens) on card 2 - on two CMP 170HX 64 GB cards. sm_80 tensor-core EXL3 paths on for prefill and small-row decode; PP scheduler spreads decodes and never mixes prefill+decode tokens in one step; FULL_DECODE_ONLY CUDA graphs. 225,280-token context, up to 4 sequences, block size 4608. Image built locally from the repo Dockerfile at three pins (upstream vLLM a0d7356b98 + branch 2a4e1379 + plugin 947ef7c), 17 min, no GPU. Author-measured 2026-11-05: structured C1 83.0 / code 76.5 / prose 57.8 tok/s, aggregate C4 191.9, cold prefill 1,741-1,852 tok/s, HumanEval 142/164, GSM8K 1228/1319, needle 16/16, vision 7/7. Recipe, engine and measurements by AlexBrandes; checkpoint MIT, drafter CC BY-NC-ND 4.0.

Launch command

vllm serve /model --host 0.0.0.0 --port 8000 --served-model-name glm-5.3-flash-exl3-2.60bpw --quantization exl3 --load-format exl3 --dtype bfloat16 --pipeline-parallel-size 2 --tensor-parallel-size 1 --mm-encoder-tp-mode data --max-model-len 225280 --max-num-seqs 4 --max-num-batched-tokens 2048 --prefill-chunk-with-decodes 512 --block-size 4608 --gpu-memory-utilization 0.97 --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 --enable-prompt-tokens-details --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' --speculative-config '{"method": "dflash", "model": "/draft", "num_speculative_tokens": 3, "quantization": null, "draft_load_config": {"load_format": "safetensors"}}' --hf-overrides '{"text_config": {"aux_hidden_state_tensor": "stream_mean"}}'

runtime

env: "serve.sh exports (verbatim @ db1af3228): VLLM_CACHE_ROOT=/cache/vllm TRITON_CACHE_DIR=/cache/triton CUDA_CACHE_PATH=/cache/cuda VLLM_PP_LAYER_PARTITION=23,22 (23 target layers on card 1, 22 + drafter on card 2) VLLM_PP_SPREAD_DECODES=1 VLLM_PP_HOMOGENEOUS_STEPS=1 VLLM_SM80_KDA_PREFILL=1 VLLM_EXL3_MOE_PREFILL_TC=1 VLLM_EXL3_DENSE_PREFILL_TC=1 VLLM_EXL3_DENSE_PREFILL_ONE_PRODUCT=float VLLM_EXL3_MOE_DECODE_TC=1 VLLM_EXL3_MOE_DECODE_TC_MODE=full VLLM_EXL3_DENSE_DECODE_TC=1 VLLM_EXL3_DENSE_DECODE_TC_FAMILIES=all VLLM_EXL3_DENSE_DECODE_SPLITK=1 VLLM_EXL3_DECODE_VARIANT=off. Run (README step 3): docker build -t glm53-flash-exl3:947ef7c recipes/glm53-flash-exl3, then docker run --rm --gpus all --shm-size 16g --user $(id -u):$(id -g) -e HOME=/cache -e HF_HUB_OFFLINE=1 -e CUDA_VISIBLE_DEVICES=0,1 -p 127.0.0.1:8000:8000 -v $MODEL_DIR:/model:ro -v $DRAFT_DIR:/draft:ro -v $CACHE_DIR:/cache -v $PWD/recipes/glm53-flash-exl3/chat_template.jinja:/model/chat_template.jinja:ro glm53-flash-exl3:947ef7c (the jinja mount must replace the tokenizer's own template, not --chat-template: the GLM reasoning parser reads that one). Weights: hf download turboderp/GLM-5.3-Flash-exl3 --revision 24781209f2862a939cc21e01d256c943bd1a58a3; hf download incoai/GLM-5.3-Flash-DFlash2 --revision bf582e4eacc1810f76656d1811693ff6c6737d2a."
notes: 'Engine is three pins per the Dockerfile: upstream vLLM a0d7356b98e929c17051bac44f5a086e4117974d (prebuilt wheel from wheels.vllm.ai), AlexBrandes/vllm branch 2a4e1379093a84fd0852c197bb76c09a34a0473f (Python files over the wheel), vllm-cmp170hx plugin 947ef7c7e1a96135521dce27cf953178816da42e (EXL3 extension compiled at build time, TORCH_CUDA_ARCH_LIST=8.0). Measured image id sha256:ee218f38789644c2b42d8bcbc79b4dc18a5bb25cc7c33ca5491486dcd8d7156c (docker build --no-cache); rebuilds get their own image identity. serve.sh ends in exec vllm serve ... "$@"; extra argv (none used for the measured runs) would append here.'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-2-60bpw-pp2-on-2x-cmp-170hx-dflash2-k3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-2-60bpw-pp2-on-2x-cmp-170hx-dflash2-k3.json

Engine

Install note: NOT installable from a published image: build from a checkout of the fork's plugin branch - docker build -t glm53-flash-exl3:947ef7c recipes/glm53-flash-exl3 (base nvidia/cuda:13.0.3-devel-ubuntu24.04 pinned by digest; upstream vLLM prebuilt wheel for a0d7356b98 + the branch's Python files + the vllm-cmp170hx plugin with its EXL3 extension compiled against the image PyTorch, TORCH_CUDA_ARCH_LIST=8.0; every download sha256-checked; build needs network+CPU/RAM not a GPU, ~17 min). Engine = three pins: upstream a0d7356b98e929c17051bac44f5a086e4117974d, branch 2a4e1379093a84fd0852c197bb76c09a34a0473f, plugin 947ef7c7e1a96135521dce27cf953178816da42e. EXL3 quant/load-format and the PP/EXL3 kernels are registered by the plugin, not stock vLLM.

Model

Hardware

Context

  • max context: 220K tokens

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
1,0241—83 tok/s—reported
1,0242—122.5 tok/s—reported
1,0244—191.9 tok/s—reported
1,0248—161.1 tok/s—reported
4,096—1741.3 tok/s——reported
4,0961—59.1 tok/s—reported
8,192—1851.5 tok/s——reported
8,1921—59 tok/s—reported
11,2641—76.5 tok/s—reported
11,2642—109.9 tok/s—reported
11,2644—151.4 tok/s—reported
11,2648—137 tok/s—reported
16,384—1842.1 tok/s——reported
16,3841—60.8 tok/s—reported
21,5041—70.9 tok/s—reported
21,5042—113.2 tok/s—reported
21,5044—164.6 tok/s—reported
21,5048—156.5 tok/s—reported
31,7441—57.8 tok/s—reported
31,7442—92.4 tok/s—reported
31,7444—139.6 tok/s—reported
31,7448—123 tok/s—reported
32,768—1835.6 tok/s——reported
32,7681—58.1 tok/s—reported
65,536—1822.8 tok/s——reported
65,5361—56.1 tok/s—reported
131,072—1802.7 tok/s——reported
131,0721—59 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured 2026-11-05 on the rig (2x CMP 170HX, 74 SMs/card, 170 W/card, PCIe Gen2 x4, no P2P) with benchlib's SparkDash-compatible public-v1 battery (SparkDash 3e6d4ef: greedy, thinking off, 400 output tokens, engine-direct); values are medians of 3 complete runs (run IDs public-v1.20261105T192046Z.a455953f / T205324Z.696b161a / T210946Z.6cca7682). C2-C8 are aggregate across that many requests (max-num-seqs 4, so C8 queues). Structured C1 83.0 / code 76.5 / JSON 70.9 / prose 57.8 tok/s. Quality same session: HumanEval pass@1 142/164 chat (134/164 completion), GSM8K 1228/1319 chat (1190/1319 completion), needle 16/16 at 4K/16K/32K/64K, 225279+1 tokens accepted / 225280 refused, image+video 7/7. Full per-cell data in results.json @ db1af3228.

Downloads

Revision notes

Initial version.

Revision history

Initial version.

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)