GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX TP2 DFlash2 K7 hybrid drafting 393K (alexvnesta)

community Community submission — verify on your own hardware.

recipe by alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)↗, Flun (original 2-card layer-split server + recipe this fork starts from)↗, turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)↗, incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)↗, 0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)↗, Morrowmake (content-verified P2P check idea + benchmark prompt style)↗, bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)↗, Z.ai (GLM-5.3-Flash base model)↗ · listed by localai-recipes

fork of /r/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k

revision r4 · commit 4a4d9f8d79 · updated 21 h ago · license CC-BY-4.0

chat code vision tensor-parallel vision dflash2 long-context exl3 speculative q8-kv hybrid-drafting nccl-p2p concurrent-sessions

Tensor-parallel successor to Flun's 2x CMP 170HX layer-split recipe: ExLlamaV3 TP2 over both 64 GB cards with a 655,360-token Q8 KV pool (392,960-token request cap), DFlash2 K7 speculative decoding plus a NEW hybrid drafting mode (H=15 draft tokens, temperature-0 outputs byte-identical on/off), two concurrent sessions with fair scheduling, idle sessions parked in system RAM (resume 1-3 s), a cross-session 26.8K-prefix cache (repeat TTFT 20.8 s -> 0.7 s), and image input (base64 data URLs, bwrap-sandboxed decode). TP backend NCCL 2.30.7 over GPU P2P after content-verified p2p_check; beats the native host-RAM backend by +4.35/+6.48 % decode and +10 % prefill (serial n=10); TP_BACKEND=native is the fallback. Author A/B on one machine (2026-10-12): 123.6 / 44.3 tok/s easy/hard text vs Flun layer-split 96.2 / 34.2 on the SAME box (+28 %), hybrid drafting 134-147 tok/s on code-edit/JSON. OpenAI-compatible 127.0.0.1:8012/v1, no auth (loopback/proxy). STOCK cards only: PCIe Gen2 x16, 180 W.

Launch command

scripts/setup_venv.sh engine/apply_engine.sh engine/build_native.sh scripts/get_models.sh 'scripts/p2p_check.py   # optional gate: NCCL needs working P2P; on failure export TP_BACKEND=native' './scripts/run_tp.sh   # production profile: TP2 + DFlash2 K7 + hybrid + 2 sessions + NCCL'

runtime

env: 'run_tp.sh reads: PORT/HOST (8012 / 127.0.0.1), VISION=1 (0 = off, frees ~0.4 GB vision tower on GPU 0), HYBRID_DRAFT_TOKENS=15 (0 disables hybrid drafting; 15 measured faster than 8), RAM_TIER_GIB=8 (parked-session system RAM), MODELS_DIR, TP_BACKEND=nccl|native. Script exports (verbatim @ 0836b7dc): EXL3_BC_ATTN=1 EXL3_MLA_KIDX_RING=32 EXL3_INT8_GEMV=0 EXL3_GEMV=0 EXL3_DSA_QC_STAGE=1 EXL3_DSA_QC_STAGE_MIN_R=9 EXL3_TP_WORKER_TIMEOUT=300 EXL3_TP_SHARING_STRATEGY=file_descriptor EXL3_AUTOSPLIT_MARGIN_MB=128 GLM53_PREFILL_CHUNK_SIZE=2048 GLM53_DYNAMIC_DRAFT=0 GLM53_RECORD_DRAFT_STATS=1 GLM53_TP_ACTIVE_HOST=0 GLM53_K_HCFUSE=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false NCCL_SOCKET_IFNAME=lo (NCCL bootstrap stays on loopback); ulimit -n 65536.'
notes: "run_tp.sh glm_api.py argv (verbatim @ 0836b7dc): -m models/GLM-5.3-Flash-exl3-3.05bpw -dm models/GLM-5.3-Flash-DFlash2-exl3-6bpw -cq 8 -cs 655360 --context-cap 392960 -ndt 7 -chunk_size 2048 -ambs 2 -gs 62,63 -lv -tp --experimental-dflash2-tp -tpb $TP_BACKEND --q8-staging --q8-head-gpu0 --experimental-concurrent-generation --concurrent-max-sequences 2 --concurrent-fair-gen-rounds 4 --concurrent-fair-chunk-size 512 --host $HOST --port $PORT --experimental-b2-session-cache --b2-target-cpu-cache-gib $RAM_TIER_GIB --recurrent-cache-gib 4 --session-cache-gib 1 --session-cache-max-checkpoints 8 --experimental-b2-shared-prefix (plus vision sandbox args when VISION=1; bwrap optional - without it images still decode under memory/CPU limits). Startup checks engine pins and REFUSES vision/concurrency on a modified engine tree. First start: one-time fused-kernel compile + ~117 GiB weight load + CUDA graph capture = 2-3 min to healthy. VRAM ~62.9/62.3 GiB idle, ~64.4/63.0 GiB peaks with vision under load (65,536 MiB cards). Keep the vision tower on GPU 0 (GPU 1 placement showed one unexplained TP timeout). Session sharing: two active requests share the 655,360-token cache - ~320K + ~320K or 393K + ~255K fit, 330K + 330K does not; two streams total throughput ~9-30 % below one stream; large new prompts slow a decoding session to ~1/3 until prefill completes; rare early-token flips when two identical requests start the same instant (coherent outputs). NCCL troubleshooting: nvidia-smi topo -p2p r must show OK, else TP_BACKEND=native. DFlash2 drafter lineage incoai/GLM-5.3-Flash-DFlash2 is CC BY-NC-ND 4.0 (noncommercial) - converted locally, no converted redistribution. Quantization quality figures (~93 % top-1 agreement, KLD ~0.05 vs BF16 at 3.05 bpw) are turboderp's/Flun's reported numbers, not this fork's measurements."

upstream_drift

'Pinned to the last validated sha 0836b7dcb84e2816f2c59bdb8ffe117721a76eda; the published env and argv are verbatim at that revision. Upstream alexvnesta/glm53-flash-cmp170hx-tp has since advanced to db01c256a8003daf90c932ee25552526ede40f1f (38 commits, 329 files as of 2026-10-29), which this recipe does NOT validate — no boot, soak or capture has been run against it. The patch set and argv documented here apply to 0836b7dc only.'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid.json

Engine

exllamav3 · source↗ · 0836b7dc↗

Install note: NOT installable from an image: four-step source build on the rig - scripts/setup_venv.sh (Python 3.13 venv, torch==2.14.1+cu130 wheel index; brings triton 3.8.0 + nvidia-nccl-cu13 2.30.7), engine/apply_engine.sh (clones turboderp-org/exllamav3 at UPSTREAM.txt base_commit 3a8e2fb84cda5a87ec700f1e9935b641416b7e2b, applies 9 patches in filename order, verifies every file byte-identical against engine/engine.sha256 AND the service startup pin tables - hash mismatch = delete engine-src/ and re-apply), engine/build_native.sh (CUDA 13 nvcc, TORCH_CUDA_ARCH_LIST=8.0, ~25 min), scripts/get_models.sh (target + drafter at pinned revs, converts drafter once to EXL3 6bpw). Service runs glm_api.py with PYTHONPATH=native:engine-src:service/glm and JIT-compiles its own k101 fused kernel on first start (service/kernels/k_hcfuse.py; needs ninja + nvcc).

Model

GLM-5.3-Flash-EXL3-3.05bpw (exl3-3-05bpw) · turboderp/GLM-5.3-Flash-exl3↗
AA intelligence 42 (Artificial Analysis · measured)

Hardware

NVIDIA CMP 170HX 64GB ×2 (Ampere)

KV pool

640K tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
330,000	1	1267 tok/s	—	—	reported
392,960	1	1316 tok/s	123.6 tok/s	—	reported
392,960	2	—	115 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Author-measured on the rig 2026-10-12, methodology in docs/PERFORMANCE.md (decode: fixed-counting free-text prompts, serial n=10 per A/B arm; prefill: unique cold prompts, server-side timing, n=3 per size; hybrid: 7 workloads x 3 passes temp-0 with output-hash equality gates; soak: 912 mixed requests incl. tool calls + cancels + vision, 0 failures). Decode easy/structured 115-125 tok/s (123.6 NCCL build), hard reasoning ~40-45 (44.3); hybrid drafting +15-20 % on code-edit/JSON (134-147), unchanged on prose/counting controls, outputs byte-identical. Prefill 1,316 (8K) / 1,343 (32K) / 1,356 (96K) / 1,325 (225K) / 1,267 (330K) tok/s; TTFT ~6 s @8K. Cached 26.8K prefix repeat TTFT 20.8 s -> 0.7 s. NCCL-vs-native A/B: +4.35 % / +6.48 % decode (Welch p=3e-9 / 2e-9), +10 % prefill. Same-machine Flun layer-split baseline 96.2 / 34.2 tok/s. claimed_by=submitter T0 - methodology documented and stats-gated but no run file ingested. Headline: ~124 tok/s, 393K context, 2 concurrent sessions.

notes

Author-measured on the rig 2026-10-12, methodology in docs/PERFORMANCE.md (decode: fixed-counting free-text prompts, serial n=10 per A/B arm; prefill: unique cold prompts, server-side timing, n=3 per size; hybrid: 7 workloads x 3 passes temp-0 with output-hash equality gates; soak: 912 mixed requests incl. tool calls + cancels + vision, 0 failures). Decode easy/structured 115-125 tok/s (123.6 NCCL build), hard reasoning ~40-45 (44.3); hybrid drafting +15-20 % on code-edit/JSON (134-147), unchanged on prose/counting controls, outputs byte-identical. Prefill 1,316 (8K) / 1,343 (32K) / 1,356 (96K) / 1,325 (225K) / 1,267 (330K) tok/s; TTFT ~6 s @8K. Cached 26.8K prefix repeat TTFT 20.8 s -> 0.7 s. NCCL-vs-native A/B: +4.35 % / +6.48 % decode (Welch p=3e-9 / 2e-9), +10 % prefill. Same-machine Flun layer-split baseline 96.2 / 34.2 tok/s. claimed_by=submitter T0 - methodology documented and stats-gated but no run file ingested. Headline: ~124 tok/s, 393K context, 2 concurrent sessions.

Downloads

Weights https://huggingface.co/turboderp/GLM-5.3-Flash-exl3
332ab457b709b7ba30dd9a448be5de03b80a7ac9 - 3.05bpw branch tip (get_models.sh TARGET_REV; ~125 GB download / ~116.6 GiB resident; HF-resolved live 2026-10-13, 29 files)
Drafter https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2
bf582e4eacc1810f76656d1811693ff6c6737d2a - BF16 drafter (get_models.sh DRAFT_REV; license CC BY-NC-ND 4.0; converted locally to EXL3 6bpw via service/dflash2/convert_drafter.py; HF-resolved live 2026-10-13)
Engine image https://github.com/turboderp-org/exllamav3
3a8e2fb84cda5a87ec700f1e9935b641416b7e2b - UPSTREAM.txt base_commit (v1.6.0 + 42) + 9 hash-guarded patches (engine/engine.sha256); TP DFlash2, k_idx ring KV layout (655K tokens), fair scheduling, hybrid drafting hook
Reference https://github.com/Flun/glm53-flash-cmp170hx-exl3
5933112 @ HEAD - the layer-split original this fork builds on; server base, DFlash2 K7 + Q8 recipe, drafter conversion, ring design all Flun's (nine of his files unchanged in the fork per CREDITS.md)
Reference https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe
p2p_check.py idea provenance - content-verified GPU P2P before enabling NCCL (reimplemented, no code copied per CREDITS.md)

Revision notes

Community ingest 2026-10-13, owner-requested ('fork of the original flash 5.3'). Repo @ 0836b7dcb84e2816f2c59bdb8ffe117721a76eda (main HEAD verified live via ls-remote; 6 commits; MIT; CREDITS.md + NOTICE.md present). Launch argv + env token-verified against scripts/run_tp.sh at that HEAD; engine pin from engine/UPSTREAM.txt + engine/NOTES.md. Both HF pins resolved live via HF API. Performance numbers are the fork's own docs/PERFORMANCE.md tables (2026-10-12, serial n=10 A/B with Welch stats, output-hash gates on hybrid, 912-request soak) - claimed_by=submitter T0, no run file. NOTE: repo README carries the same numbers; one stale line in PERFORMANCE.md sec 1 ('NCCL not tried on this box') predates its own sec 5 NCCL A/B + roadmap 'Done 2026-10-12: NCCL default' - run_tp.sh default TP_BACKEND=nccl is authoritative. Sits BETWEEN the two existing 2-card-class GLM recipes: Flun layer-split (same 2x170HX silicon, 96/34 tok/s, 1 request, 384K) and Morrowmake 4x170HX vLLM W4A16 (weights don't fit on two cards). Runs on stock cards - no cap mod / SM unlock required (Gen2 x16, 180 W).

Revision history

4a4d9f8d79 recipe: update glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid (r4: /config/upstream_drift, /provenance/credit) 15 h ago diff
8ea0360ce4 recipe: update glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid (r3: /kv_pool_tokens) 10 d ago diff
5dc18df9eb recipe: update glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid (r2: /provenance/forked_from — fork lineage backfill (t_a22aebd6)) 18 d ago diff
91cfca9e7e recipe: create glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid 18 d ago

Fork

Export

recipe.yaml (raw)