GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX — DFlash2 K7 Q8 KV 384K (Flun)

scripts/setup.sh 'GLM53_K_HCFUSE=1 EXL3_AUTOSPLIT_MARGIN_MB=128 TORCH_CUDA_ARCH_LIST=8.0 HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false scripts/serve.sh --profile q8_384k'

env: 'GLM53_MODELS_DIR/GLM53_VENV/GLM53_PORT(8012)/GLM53_CUDA_ARCH(8.0)/GLM53_PREFIX_CACHE(1; =0 disables prefix reuse)/EXL3_AUTOSPLIT_MARGIN_MB(profile: 128 q8_384k, 192 q8_fast, 256 others)/GLM53_K_HCFUSE=1/HF_HUB_OFFLINE=1/TOKENIZERS_PARALLELISM=false (verbatim from scripts/serve.sh @ 5933112).'
notes: 'serve.sh --profile q8_384k resolves to: python -u glm/glm_api.py -m models/GLM-5.3-Flash-exl3-3.05bpw -cs 393216 -cq 8 -gs 62,63 --q8-staging --q8-head-gpu0 -dm models/GLM-5.3-Flash-DFlash2-exl3-6bpw -ndt 7 --dflash-prefix-cache -chunk_size 2048 --autosplit_max_batch_size 1 --port 8012. First run downloads turboderp/GLM-5.3-Flash-exl3 @ 332ab457 (3.05bpw branch tip, ~125 GB) + incoai/GLM-5.3-Flash-DFlash2 @ bf582e4e (BF16, ~2 GB) then converts the drafter once (dflash2/convert_drafter.py -b 6 --devices 0 + package_native.py; k_proj/v_proj stay BF16 for the fused context-KV weight; DFlash2DynConv.kernel_projection quantized via qmap; synthetic-Hessian calibration only, no target-model forwards; native quantization_config/tensor_storage metadata so ExLlamaV3 1.5.4 loads it directly). -gs is ExLlamaV3 per-component load budget, not a VRAM split; MTP profile effective split is 59,63 (MTP head loads on GPU 0 first). Preflight guards: both GPUs must have ~64 GB free (server refuses otherwise); ~140 GB disk for models + conversion workspace; CUDA toolkit with nvcc compute_80. Drafter ring: 2048-token sliding window over an 8,192-token physical page ring — absolute RoPE positions and attention window unchanged; verified bit-exact against the unbounded cache over repeated wraps. Prefix reuse pairs each target recurrent checkpoint with a ~43 MiB host-RAM snapshot of the last 2048 drafter tokens; on a hit 8K repeat TTFT 7.86->1.11 s (93.98% tokens reused), 32K repeat 28.49->1.12 s (98.42%); greedy repeated inputs match cold runs but longer real-text 8K/32K follow-ups did NOT fully reproduce full-reprocess outputs — author documents the defect rather than fixing it, treat reuse as validated for measured repeat cases only. API: 127.0.0.1:8012/v1, model id GLM-5.3-Flash, reasoning_content/content split, OpenAI tools via GLM XML translation, GET /health reports cache format + speculation mode + draft ring size + prompt_cache hits/misses/restored; timings expose native prompt_ms/predicted_ms + draft accepted/rejected + cached_tokens. Defaults reasoning_effort=high, temp 1.0, top_p 0.95, max_tokens 32768; one active request, others queue up to 120 s. HARDWARE: author cards are cap-modded PCIe x16 (physical x16 edge connector mod; link runs Gen2 x8 vs stock 170HX Gen2 x4), no P2P — cross-GPU traffic is host-bounce D2D; NOT 74-SM mods. Layout validated on Ryzen 5 5600X/DDR4 host; other hosts may need different margins. 448K/512K fail during final transformer-layer load with this checkpoint (not offered); 1M context remains open. DFlash2 drafter lineage incoai/GLM-5.3-Flash-DFlash2 is CC BY-NC-ND 4.0 (noncommercial) — the repo re-quantizes it locally and redistributes no weights. Credits per repo README: turboderp (engine MIT + 3.05bpw checkpoint), incoai (BF16 drafter), MiaAI-Lab 2x-DGX-Sparks repo (conversion reference), 0xSero (k_hcfuse kernels), Z.ai (GLM-5.3-Flash).'

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k.json