{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX \u2014 DFlash2 K7 Q8 KV 384K (Flun)","summary":"First ExLlamaV3 serving recipe on 2x CMP 170HX: full-VRAM HBM-resident GLM-5.3-Flash EXL3 3.05bpw (~116.6 GiB across 2x64GB, no CPU/SSD tier) with DFlash2 K7 speculative decoding \u2014 drafter re-quantized locally BF16->EXL3 6bpw (~0.96 GiB) with an 8K-token GPU ring KV cache so a 384K request needs no 384K draft cache (bit-exact over ring wraps). Q8 quantized MLA latent KV, k_hcfuse fused decode kernels (SM80 JIT, shadow-check bit-exact), process-local monkeypatches only (engine tree unmodified), opt-in previous-request DFlash prefix reuse (8K repeat TTFT 7.9s->1.1s; general follow-up defect documented, not fixed), OpenAI-compatible API with native PP/TG + acceptance timings. Alternative stock MTP d2 profile (fp16 192K). Author-reported ~90 tok/s sustained, 110-120 at high acceptance (Reddit 1x0b1ws; no run file). Cards cap-modded PCIe x16 (Gen2 x8 vs stock x4), no P2P; single-request; Q8 changes numerics.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"f363025763f08eecd079592032038eb42acf00ae","content_hash":"f3d955e45e537ed75550188c286a6fed02089829","created_at":"2026-10-29 21:32:24.781602-06","notes":"Submitted 2026-10-08 from lar-scout ingest card t_eb5f504f. Repo @ 5933112fdb858c960dfa1a68f507c3a98a9f1201 (HEAD verified 2026-10-08, 2 commits, MIT license). Launch argv token-verified against scripts/serve.sh + scripts/setup.sh + glm/glm_api.py argparse gates at that sha (profile MEM_ARGS/SPEC_ARGS case blocks, DFlash2-vs-MTP parser guard, Q8-options-require-cq8 guard). HF pins verified live via HF API (target 332ab457 resolves 29 files; drafter bf582e4e matches serve.sh exactly; drafter license cc-by-nc-nd-4.0 confirmed on the model card). Reddit thread hard-blocked (403) from every access lane at ingest time \u2014 the ~90/110-120 tok/s figures are carried as submitter-T0 claims per the digest card's provenance, not re-verified; performance section says so. First exllamav3 engine row on the site (schema enum extended by the paired code commit; facet auto-creates per WAVE-3 SGLang precedent). Hardware profile cmp-170hx-x2 pre-existing (id 774); cap-mod PCIe note lives in config.runtime.notes + tags (cap-modded, no-p2p) per the DUFRelic modded-card precedent \u2014 same silicon profile, mod documented not profiled."},"engine_source":{"type":"fork","git_url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3","base_engine":"exllamav3","pip_package":"exllamav3==1.5.4","base_version":"ExLlamaV3 1.5.4 + k_hcfuse (SM80)","image_digest":null,"install_note":"NOT installable from an image: scripts/setup.sh builds a venv (torch cu130 + pip exllamav3==1.5.4 --no-build-isolation, TORCH_CUDA_ARCH_LIST=8.0 for SM80 native kernels) and the server runs glm/glm_api.py directly; ExLlamaV3 sources are NOT modified \u2014 every engine adjustment is a process-local monkeypatch active only while this server runs (k_hcfuse JIT-compiles on first decode via nvcc; GLM53_K_HCFUSE=1).","pinned_commit":"5933112fdb858c960dfa1a68f507c3a98a9f1201"},"credit":[{"name":"Flun (recipe, stack, profiles, drafter conversion)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused decode kernels)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"MiaAI-Lab (GLM-5.3-Flash-EXL3-2x-DGX-Sparks \u2014 conversion reference)","url":"https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["scripts/setup.sh","GLM53_K_HCFUSE=1 EXL3_AUTOSPLIT_MARGIN_MB=128 TORCH_CUDA_ARCH_LIST=8.0 HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false scripts/serve.sh --profile q8_384k"],"runtime":{"env":"GLM53_MODELS_DIR/GLM53_VENV/GLM53_PORT(8012)/GLM53_CUDA_ARCH(8.0)/GLM53_PREFIX_CACHE(1; =0 disables prefix reuse)/EXL3_AUTOSPLIT_MARGIN_MB(profile: 128 q8_384k, 192 q8_fast, 256 others)/GLM53_K_HCFUSE=1/HF_HUB_OFFLINE=1/TOKENIZERS_PARALLELISM=false (verbatim from scripts/serve.sh @ 5933112).","notes":"serve.sh --profile q8_384k resolves to: python -u glm/glm_api.py -m models/GLM-5.3-Flash-exl3-3.05bpw -cs 393216 -cq 8 -gs 62,63 --q8-staging --q8-head-gpu0 -dm models/GLM-5.3-Flash-DFlash2-exl3-6bpw -ndt 7 --dflash-prefix-cache -chunk_size 2048 --autosplit_max_batch_size 1 --port 8012. First run downloads turboderp/GLM-5.3-Flash-exl3 @ 332ab457 (3.05bpw branch tip, ~125 GB) + incoai/GLM-5.3-Flash-DFlash2 @ bf582e4e (BF16, ~2 GB) then converts the drafter once (dflash2/convert_drafter.py -b 6 --devices 0 + package_native.py; k_proj/v_proj stay BF16 for the fused context-KV weight; DFlash2DynConv.kernel_projection quantized via qmap; synthetic-Hessian calibration only, no target-model forwards; native quantization_config/tensor_storage metadata so ExLlamaV3 1.5.4 loads it directly). -gs is ExLlamaV3 per-component load budget, not a VRAM split; MTP profile effective split is 59,63 (MTP head loads on GPU 0 first). Preflight guards: both GPUs must have ~64 GB free (server refuses otherwise); ~140 GB disk for models + conversion workspace; CUDA toolkit with nvcc compute_80. Drafter ring: 2048-token sliding window over an 8,192-token physical page ring \u2014 absolute RoPE positions and attention window unchanged; verified bit-exact against the unbounded cache over repeated wraps. Prefix reuse pairs each target recurrent checkpoint with a ~43 MiB host-RAM snapshot of the last 2048 drafter tokens; on a hit 8K repeat TTFT 7.86->1.11 s (93.98% tokens reused), 32K repeat 28.49->1.12 s (98.42%); greedy repeated inputs match cold runs but longer real-text 8K/32K follow-ups did NOT fully reproduce full-reprocess outputs \u2014 author documents the defect rather than fixing it, treat reuse as validated for measured repeat cases only. API: 127.0.0.1:8012/v1, model id GLM-5.3-Flash, reasoning_content/content split, OpenAI tools via GLM XML translation, GET /health reports cache format + speculation mode + draft ring size + prompt_cache hits/misses/restored; timings expose native prompt_ms/predicted_ms + draft accepted/rejected + cached_tokens. Defaults reasoning_effort=high, temp 1.0, top_p 0.95, max_tokens 32768; one active request, others queue up to 120 s. HARDWARE: author cards are cap-modded PCIe x16 (physical x16 edge connector mod; link runs Gen2 x8 vs stock 170HX Gen2 x4), no P2P \u2014 cross-GPU traffic is host-bounce D2D; NOT 74-SM mods. Layout validated on Ryzen 5 5600X/DDR4 host; other hosts may need different margins. 448K/512K fail during final transformer-layer load with this checkpoint (not offered); 1M context remains open. DFlash2 drafter lineage incoai/GLM-5.3-Flash-DFlash2 is CC BY-NC-ND 4.0 (noncommercial) \u2014 the repo re-quantizes it locally and redistributes no weights. Credits per repo README: turboderp (engine MIT + 3.05bpw checkpoint), incoai (BF16 drafter), MiaAI-Lab 2x-DGX-Sparks repo (conversion reference), 0xSero (k_hcfuse kernels), Z.ai (GLM-5.3-Flash)."},"max_context_len":393216},"models":[{"slug":"glm-5-3-flash-exl3-3-05bpw","name":"GLM-5.3-Flash-EXL3-3.05bpw","hf_repo":"turboderp/GLM-5.3-Flash-exl3","quant":"exl3-3-05bpw"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["cap-modded","dflash2","exl3","long-context","no-p2p","q8-kv","speculative"],"tasks":["chat","code"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-reported typical sustained ~90 tok/s, 110-120 tok/s at high DFlash2 acceptance \u2014 via Reddit thread r/LocalLLaMA 1x0b1ws (author /u/Prudent_Appearance71, posted 2026-10-07). The repo explicitly does NOT ship the benchmark harness or raw measurement data ('out of scope' per README), so these are submitter-T0 claims without a run file; the repo DOES carry measured prefix-reuse TTFT data (8K repeat 7.86->1.11 s, 32K 28.49->1.12 s) and the ring-cache bit-exactness checks. 384K Q8 profile context 392,960 input+output (cache 393,216 tokens). [t_71325891: added the author's measured prefix-reuse TTFT points at 1K/8K/32K from README @ 5933112f. The stated 110-120 tok/s high-acceptance decode band cannot occupy a second point at (393216, 1) because a point is unique on (context_len, concurrency, claimed_by); it stays a note rather than inventing a context.]\", \"tok_s\": 90, \"context_len\": 393216}"}],"perf_points":[{"id":2048,"context_len":1024,"prefill_tok_s":null,"decode_tok_s":null,"ttft_ms":650,"concurrency":1,"claimed_by":"submitter","notes":"README.md @ 5933112f, prefix-KV reuse: repeated 1K prompt first-token 1.50 s cold -> 0.65 s cached (temperature 0, seed 42). TTFT only; the author states no decode at this context."},{"id":2049,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":null,"ttft_ms":1110,"concurrency":1,"claimed_by":"submitter","notes":"README.md @ 5933112f, prefix-KV reuse: 8K conversation follow-up first-token 7.86 s -> 1.11 s, 93.98% tokens reused (temperature 0, seed 42). TTFT only."},{"id":2050,"context_len":32768,"prefill_tok_s":null,"decode_tok_s":null,"ttft_ms":1120,"concurrency":1,"claimed_by":"submitter","notes":"README.md @ 5933112f, prefix-KV reuse: 32K follow-up first-token 28.49 s -> 1.12 s, 98.42% tokens reused (temperature 0, seed 42). TTFT only."},{"id":2047,"context_len":393216,"prefill_tok_s":null,"decode_tok_s":90,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"author-reported typical sustained decode, DFlash2 K7, Q8 KV \u2014 Reddit 1x0b1ws, submitter-T0 (no run file)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3","pin":"332ab457b709b7ba30dd9a448be5de03b80a7ac9 \u2014 3.05bpw branch tip (serve.sh TARGET_REV; ~125.2 GB download / ~116.6 GiB weights resident; base_model zai-org/GLM-5.3-Flash)"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2","pin":"bf582e4eacc1810f76656d1811693ff6c6737d2a \u2014 BF16 drafter (serve.sh DRAFT_BF16_REV; license CC BY-NC-ND 4.0; converted locally to EXL3 6bpw ~0.96 GiB via dflash2/convert_drafter.py + package_native.py @ repo HEAD)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/Flun/glm53-flash-cmp170hx-exl3","pin":"5933112 @ HEAD 'Add opt-in previous-request DFlash prefix cache reuse' \u2014 scripts/setup.sh + serve.sh are the verbatim argv/env source; glm/ monkeypatches + kernels/k_hcfuse.py + dflash2/ converter at this commit"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/sybil-solutions/glm-flash-lite","pin":"k_hcfuse fused mHC decode kernels source (verified exists 2026-10-08, HEAD 7f1ee89) \u2014 vendored as kernels/k_hcfuse.py; bit-exact w/ shadow-check GLM53_K_HCFUSE_CHECK"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:24.781602-06","created_at":"2026-10-07 19:52:54.674531-06","speeds":{"decode_tok_s":90,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}