{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX TP2 DFlash2 K7 hybrid drafting 393K (alexvnesta)","summary":"Tensor-parallel successor to Flun's 2x CMP 170HX layer-split recipe: ExLlamaV3 TP2 over both 64 GB cards with a 655,360-token Q8 KV pool (392,960-token request cap), DFlash2 K7 speculative decoding plus a NEW hybrid drafting mode (H=15 draft tokens, temperature-0 outputs byte-identical on/off), two concurrent sessions with fair scheduling, idle sessions parked in system RAM (resume 1-3 s), a cross-session 26.8K-prefix cache (repeat TTFT 20.8 s -> 0.7 s), and image input (base64 data URLs, bwrap-sandboxed decode). TP backend NCCL 2.30.7 over GPU P2P after content-verified p2p_check; beats the native host-RAM backend by +4.35/+6.48 % decode and +10 % prefill (serial n=10); TP_BACKEND=native is the fallback. Author A/B on one machine (2026-10-12): 123.6 / 44.3 tok/s easy/hard text vs Flun layer-split 96.2 / 34.2 on the SAME box (+28 %), hybrid drafting 134-147 tok/s on code-edit/JSON. OpenAI-compatible 127.0.0.1:8012/v1, no auth (loopback/proxy). STOCK cards only: PCIe Gen2 x16, 180 W.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":4,"commit_sha":"4a4d9f8d79ff001c6c845746708a86e4a3025350","content_hash":"32bd428f49f6a62760cd5c983a7324bfd38163eb","created_at":"2026-10-29 21:32:25.153466-06","notes":"Community ingest 2026-10-13, owner-requested ('fork of the original flash 5.3'). Repo @ 0836b7dcb84e2816f2c59bdb8ffe117721a76eda (main HEAD verified live via ls-remote; 6 commits; MIT; CREDITS.md + NOTICE.md present). Launch argv + env token-verified against scripts/run_tp.sh at that HEAD; engine pin from engine/UPSTREAM.txt + engine/NOTES.md. Both HF pins resolved live via HF API. Performance numbers are the fork's own docs/PERFORMANCE.md tables (2026-10-12, serial n=10 A/B with Welch stats, output-hash gates on hybrid, 912-request soak) - claimed_by=submitter T0, no run file. NOTE: repo README carries the same numbers; one stale line in PERFORMANCE.md sec 1 ('NCCL not tried on this box') predates its own sec 5 NCCL A/B + roadmap 'Done 2026-10-12: NCCL default' - run_tp.sh default TP_BACKEND=nccl is authoritative. Sits BETWEEN the two existing 2-card-class GLM recipes: Flun layer-split (same 2x170HX silicon, 96/34 tok/s, 1 request, 384K) and Morrowmake 4x170HX vLLM W4A16 (weights don't fit on two cards). Runs on stock cards - no cap mod / SM unlock required (Gen2 x16, 180 W)."},"engine_source":{"type":"fork","git_url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp","base_engine":"exllamav3","pip_package":null,"base_version":"ExLlamaV3 @ 3a8e2fb84c + 9 patches (hash-guarded), sm_80 native","image_digest":null,"install_note":"NOT installable from an image: four-step source build on the rig - scripts/setup_venv.sh (Python 3.13 venv, torch==2.14.1+cu130 wheel index; brings triton 3.8.0 + nvidia-nccl-cu13 2.30.7), engine/apply_engine.sh (clones turboderp-org/exllamav3 at UPSTREAM.txt base_commit 3a8e2fb84cda5a87ec700f1e9935b641416b7e2b, applies 9 patches in filename order, verifies every file byte-identical against engine/engine.sha256 AND the service startup pin tables - hash mismatch = delete engine-src/ and re-apply), engine/build_native.sh (CUDA 13 nvcc, TORCH_CUDA_ARCH_LIST=8.0, ~25 min), scripts/get_models.sh (target + drafter at pinned revs, converts drafter once to EXL3 6bpw). Service runs glm_api.py with PYTHONPATH=native:engine-src:service/glm and JIT-compiles its own k101 fused kernel on first start (service/kernels/k_hcfuse.py; needs ninja + nvcc).","pinned_commit":"0836b7dcb84e2816f2c59bdb8ffe117721a76eda"},"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["scripts/setup_venv.sh","engine/apply_engine.sh","engine/build_native.sh","scripts/get_models.sh","scripts/p2p_check.py   # optional gate: NCCL needs working P2P; on failure export TP_BACKEND=native","./scripts/run_tp.sh   # production profile: TP2 + DFlash2 K7 + hybrid + 2 sessions + NCCL"],"runtime":{"env":"run_tp.sh reads: PORT/HOST (8012 / 127.0.0.1), VISION=1 (0 = off, frees ~0.4 GB vision tower on GPU 0), HYBRID_DRAFT_TOKENS=15 (0 disables hybrid drafting; 15 measured faster than 8), RAM_TIER_GIB=8 (parked-session system RAM), MODELS_DIR, TP_BACKEND=nccl|native. Script exports (verbatim @ 0836b7dc): EXL3_BC_ATTN=1 EXL3_MLA_KIDX_RING=32 EXL3_INT8_GEMV=0 EXL3_GEMV=0 EXL3_DSA_QC_STAGE=1 EXL3_DSA_QC_STAGE_MIN_R=9 EXL3_TP_WORKER_TIMEOUT=300 EXL3_TP_SHARING_STRATEGY=file_descriptor EXL3_AUTOSPLIT_MARGIN_MB=128 GLM53_PREFILL_CHUNK_SIZE=2048 GLM53_DYNAMIC_DRAFT=0 GLM53_RECORD_DRAFT_STATS=1 GLM53_TP_ACTIVE_HOST=0 GLM53_K_HCFUSE=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false NCCL_SOCKET_IFNAME=lo (NCCL bootstrap stays on loopback); ulimit -n 65536.","notes":"run_tp.sh glm_api.py argv (verbatim @ 0836b7dc): -m models/GLM-5.3-Flash-exl3-3.05bpw -dm models/GLM-5.3-Flash-DFlash2-exl3-6bpw -cq 8 -cs 655360 --context-cap 392960 -ndt 7 -chunk_size 2048 -ambs 2 -gs 62,63 -lv -tp --experimental-dflash2-tp -tpb $TP_BACKEND --q8-staging --q8-head-gpu0 --experimental-concurrent-generation --concurrent-max-sequences 2 --concurrent-fair-gen-rounds 4 --concurrent-fair-chunk-size 512 --host $HOST --port $PORT --experimental-b2-session-cache --b2-target-cpu-cache-gib $RAM_TIER_GIB --recurrent-cache-gib 4 --session-cache-gib 1 --session-cache-max-checkpoints 8 --experimental-b2-shared-prefix (plus vision sandbox args when VISION=1; bwrap optional - without it images still decode under memory/CPU limits). Startup checks engine pins and REFUSES vision/concurrency on a modified engine tree. First start: one-time fused-kernel compile + ~117 GiB weight load + CUDA graph capture = 2-3 min to healthy. VRAM ~62.9/62.3 GiB idle, ~64.4/63.0 GiB peaks with vision under load (65,536 MiB cards). Keep the vision tower on GPU 0 (GPU 1 placement showed one unexplained TP timeout). Session sharing: two active requests share the 655,360-token cache - ~320K + ~320K or 393K + ~255K fit, 330K + 330K does not; two streams total throughput ~9-30 % below one stream; large new prompts slow a decoding session to ~1/3 until prefill completes; rare early-token flips when two identical requests start the same instant (coherent outputs). NCCL troubleshooting: nvidia-smi topo -p2p r must show OK, else TP_BACKEND=native. DFlash2 drafter lineage incoai/GLM-5.3-Flash-DFlash2 is CC BY-NC-ND 4.0 (noncommercial) - converted locally, no converted redistribution. Quantization quality figures (~93 % top-1 agreement, KLD ~0.05 vs BF16 at 3.05 bpw) are turboderp's/Flun's reported numbers, not this fork's measurements."},"kv_pool_tokens":655360,"upstream_drift":"Pinned to the last validated sha 0836b7dcb84e2816f2c59bdb8ffe117721a76eda; the published env and argv are verbatim at that revision. Upstream alexvnesta/glm53-flash-cmp170hx-tp has since advanced to db01c256a8003daf90c932ee25552526ede40f1f (38 commits, 329 files as of 2026-10-29), which this recipe does NOT validate \u2014 no boot, soak or capture has been run against it. The patch set and argv documented here apply to 0836b7dc only."},"models":[{"slug":"glm-5-3-flash-exl3-3-05bpw","name":"GLM-5.3-Flash-EXL3-3.05bpw","hf_repo":"turboderp/GLM-5.3-Flash-exl3","quant":"exl3-3-05bpw"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["concurrent-sessions","dflash2","exl3","hybrid-drafting","long-context","nccl-p2p","q8-kv","speculative","tensor-parallel","vision"],"tasks":["chat","code","vision"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured on the rig 2026-10-12, methodology in docs/PERFORMANCE.md (decode: fixed-counting free-text prompts, serial n=10 per A/B arm; prefill: unique cold prompts, server-side timing, n=3 per size; hybrid: 7 workloads x 3 passes temp-0 with output-hash equality gates; soak: 912 mixed requests incl. tool calls + cancels + vision, 0 failures). Decode easy/structured 115-125 tok/s (123.6 NCCL build), hard reasoning ~40-45 (44.3); hybrid drafting +15-20 % on code-edit/JSON (134-147), unchanged on prose/counting controls, outputs byte-identical. Prefill 1,316 (8K) / 1,343 (32K) / 1,356 (96K) / 1,325 (225K) / 1,267 (330K) tok/s; TTFT ~6 s @8K. Cached 26.8K prefix repeat TTFT 20.8 s -> 0.7 s. NCCL-vs-native A/B: +4.35 % / +6.48 % decode (Welch p=3e-9 / 2e-9), +10 % prefill. Same-machine Flun layer-split baseline 96.2 / 34.2 tok/s. claimed_by=submitter T0 - methodology documented and stats-gated but no run file ingested. Headline: ~124 tok/s, 393K context, 2 concurrent sessions.\", \"tok_s\": 124, \"points\": [{\"notes\": \"TP2 NCCL build, easy/structured text - docs/PERFORMANCE.md sec 2+5 @ 0836b7dc, submitter T0\", \"concurrency\": 1, \"context_len\": 392960, \"decode_tok_s\": 123.599999999999994315658113919198513031005859375, \"prefill_tok_s\": 1316}, {\"notes\": \"hard free-form reasoning text, same build/methodology\", \"concurrency\": 1, \"context_len\": 392960, \"decode_tok_s\": 44.2999999999999971578290569595992565155029296875}, {\"notes\": \"hybrid drafting (H=15) JSON workload, +20 % vs plain DFlash2; code edits 134-135 (+17 %); byte-identical outputs\", \"concurrency\": 1, \"context_len\": 392960, \"decode_tok_s\": 147}, {\"notes\": \"two concurrent sessions share the 655K pool (aggregate ~9-30 % below single-stream); third request queues; fair scheduling\", \"concurrency\": 2, \"context_len\": 392960, \"decode_tok_s\": 115}, {\"notes\": \"330K cold prefill 261 s; 225K: 1,325 tok/s / 170 s\", \"concurrency\": 1, \"context_len\": 330000, \"prefill_tok_s\": 1267}], \"context_len\": 392960}"}],"perf_points":[{"id":2051,"context_len":330000,"prefill_tok_s":1267,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"330K cold prefill 261 s; 225K: 1,325 tok/s / 170 s"},{"id":2052,"context_len":392960,"prefill_tok_s":1316,"decode_tok_s":123.599999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"TP2 NCCL build, easy/structured text - docs/PERFORMANCE.md sec 2+5 @ 0836b7dc, submitter T0"},{"id":2053,"context_len":392960,"prefill_tok_s":null,"decode_tok_s":115,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"two concurrent sessions share the 655K pool (aggregate ~9-30 % below single-stream); third request queues; fair scheduling"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3","pin":"332ab457b709b7ba30dd9a448be5de03b80a7ac9 - 3.05bpw branch tip (get_models.sh TARGET_REV; ~125 GB download / ~116.6 GiB resident; HF-resolved live 2026-10-13, 29 files)"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2","pin":"bf582e4eacc1810f76656d1811693ff6c6737d2a - BF16 drafter (get_models.sh DRAFT_REV; license CC BY-NC-ND 4.0; converted locally to EXL3 6bpw via service/dflash2/convert_drafter.py; HF-resolved live 2026-10-13)"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/turboderp-org/exllamav3","pin":"3a8e2fb84cda5a87ec700f1e9935b641416b7e2b - UPSTREAM.txt base_commit (v1.6.0 + 42) + 9 hash-guarded patches (engine/engine.sha256); TP DFlash2, k_idx ring KV layout (655K tokens), fair scheduling, hybrid drafting hook"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/Flun/glm53-flash-cmp170hx-exl3","pin":"5933112 @ HEAD - the layer-split original this fork builds on; server base, DFlash2 K7 + Q8 recipe, drafter conversion, ring design all Flun's (nine of his files unchanged in the fork per CREDITS.md)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe","pin":"p2p_check.py idea provenance - content-verified GPU P2P before enabling NCCL (reimplemented, no code copied per CREDITS.md)"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:25.153466-06","created_at":"2026-10-12 19:42:40.469693-06","speeds":{"decode_tok_s":123.599999999999994315658113919198513031005859375,"prefill_tok_s":1291.5,"leaderboard":"/api/v1/leaderboard.json"}}