diff --git a/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k/recipe.yaml b/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k/recipe.yaml index 43202e53a..f3d955e45 100644 --- a/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k/recipe.yaml +++ b/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k/recipe.yaml @@ -41,21 +41,30 @@ config: notes: 'serve.sh --profile q8_384k resolves to: python -u glm/glm_api.py -m models/GLM-5.3-Flash-exl3-3.05bpw -cs 393216 -cq 8 -gs 62,63 --q8-staging --q8-head-gpu0 -dm models/GLM-5.3-Flash-DFlash2-exl3-6bpw -ndt 7 --dflash-prefix-cache -chunk_size 2048 --autosplit_max_batch_size 1 --port 8012. First run downloads turboderp/GLM-5.3-Flash-exl3 @ 332ab457 (3.05bpw branch tip, ~125 GB) + incoai/GLM-5.3-Flash-DFlash2 @ bf582e4e (BF16, ~2 GB) then converts the drafter once (dflash2/convert_drafter.py -b 6 --devices 0 + package_native.py; k_proj/v_proj stay BF16 for the fused context-KV weight; DFlash2DynConv.kernel_projection quantized via qmap; synthetic-Hessian calibration only, no target-model forwards; native quantization_config/tensor_storage metadata so ExLlamaV3 1.5.4 loads it directly). -gs is ExLlamaV3 per-component load budget, not a VRAM split; MTP profile effective split is 59,63 (MTP head loads on GPU 0 first). Preflight guards: both GPUs must have ~64 GB free (server refuses otherwise); ~140 GB disk for models + conversion workspace; CUDA toolkit with nvcc compute_80. Drafter ring: 2048-token sliding window over an 8,192-token physical page ring — absolute RoPE positions and attention window unchanged; verified bit-exact against the unbounded cache over repeated wraps. Prefix reuse pairs each target recurrent checkpoint with a ~43 MiB host-RAM snapshot of the last 2048 drafter tokens; on a hit 8K repeat TTFT 7.86->1.11 s (93.98% tokens reused), 32K repeat 28.49->1.12 s (98.42%); greedy repeated inputs match cold runs but longer real-text 8K/32K follow-ups did NOT fully reproduce full-reprocess outputs — author documents the defect rather than fixing it, treat reuse as validated for measured repeat cases only. API: 127.0.0.1:8012/v1, model id GLM-5.3-Flash, reasoning_content/content split, OpenAI tools via GLM XML translation, GET /health reports cache format + speculation mode + draft ring size + prompt_cache hits/misses/restored; timings expose native prompt_ms/predicted_ms + draft accepted/rejected + cached_tokens. Defaults reasoning_effort=high, temp 1.0, top_p 0.95, max_tokens 32768; one active request, others queue up to 120 s. HARDWARE: author cards are cap-modded PCIe x16 (physical x16 edge connector mod; link runs Gen2 x8 vs stock 170HX Gen2 x4), no P2P — cross-GPU traffic is host-bounce D2D; NOT 74-SM mods. Layout validated on Ryzen 5 5600X/DDR4 host; other hosts may need different margins. 448K/512K fail during final transformer-layer load with this checkpoint (not offered); 1M context remains open. DFlash2 drafter lineage incoai/GLM-5.3-Flash-DFlash2 is CC BY-NC-ND 4.0 (noncommercial) — the repo re-quantizes it locally and redistributes no weights. Credits per repo README: turboderp (engine MIT + 3.05bpw checkpoint), incoai (BF16 drafter), MiaAI-Lab 2x-DGX-Sparks repo (conversion reference), 0xSero (k_hcfuse kernels), Z.ai (GLM-5.3-Flash).' performance: claimed_by: submitter - metrics: - tok_s: 90 - context_len: 393216 - notes: "Author-reported typical sustained ~90 tok/s, 110-120 tok/s at high DFlash2 acceptance — via Reddit thread r/LocalLLaMA 1x0b1ws (author /u/Prudent_Appearance71, posted 2026-10-07). The repo explicitly does NOT ship the benchmark harness or raw measurement data ('out of scope' per README), so these are submitter-T0 claims without a run file; the repo DOES carry measured prefix-reuse TTFT data (8K repeat 7.86->1.11 s, 32K 28.49->1.12 s) and the ring-cache bit-exactness checks. 384K Q8 profile context 392,960 input+output (cache 393,216 tokens)." - points: - - - context_len: 393216 - decode_tok_s: 90 - concurrency: 1 - notes: 'author-reported typical sustained decode, DFlash2 K7, Q8 KV — Reddit 1x0b1ws, submitter-T0 (no run file)' - - - context_len: 393216 - decode_tok_s: 115 - concurrency: 1 - notes: 'author-reported 110-120 tok/s at high DFlash2 acceptance — Reddit 1x0b1ws, submitter-T0 (no run file)' + tok_s: 90 + context_len: 393216 + notes: "Author-reported typical sustained ~90 tok/s, 110-120 tok/s at high DFlash2 acceptance — via Reddit thread r/LocalLLaMA 1x0b1ws (author /u/Prudent_Appearance71, posted 2026-10-07). The repo explicitly does NOT ship the benchmark harness or raw measurement data ('out of scope' per README), so these are submitter-T0 claims without a run file; the repo DOES carry measured prefix-reuse TTFT data (8K repeat 7.86->1.11 s, 32K 28.49->1.12 s) and the ring-cache bit-exactness checks. 384K Q8 profile context 392,960 input+output (cache 393,216 tokens). [t_71325891: added the author's measured prefix-reuse TTFT points at 1K/8K/32K from README @ 5933112f. The stated 110-120 tok/s high-acceptance decode band cannot occupy a second point at (393216, 1) because a point is unique on (context_len, concurrency, claimed_by); it stays a note rather than inventing a context.]" + points: + - + context_len: 393216 + concurrency: 1 + decode_tok_s: 90 + notes: 'author-reported typical sustained decode, DFlash2 K7, Q8 KV — Reddit 1x0b1ws, submitter-T0 (no run file)' + - + context_len: 1024 + concurrency: 1 + ttft_ms: 650 + notes: 'README.md @ 5933112f, prefix-KV reuse: repeated 1K prompt first-token 1.50 s cold -> 0.65 s cached (temperature 0, seed 42). TTFT only; the author states no decode at this context.' + - + context_len: 8192 + concurrency: 1 + ttft_ms: 1110 + notes: 'README.md @ 5933112f, prefix-KV reuse: 8K conversation follow-up first-token 7.86 s -> 1.11 s, 93.98% tokens reused (temperature 0, seed 42). TTFT only.' + - + context_len: 32768 + concurrency: 1 + ttft_ms: 1120 + notes: 'README.md @ 5933112f, prefix-KV reuse: 32K follow-up first-token 28.49 s -> 1.12 s, 98.42% tokens reused (temperature 0, seed 42). TTFT only.' artifacts: - kind: model_weights @@ -75,7 +84,7 @@ artifacts: - kind: engine_image pointer_type: git_commit - pointer: 'https://github.com/0xSero/glm53-flash-offload' + pointer: 'https://github.com/sybil-solutions/glm-flash-lite' pin: 'k_hcfuse fused mHC decode kernels source (verified exists 2026-10-08, HEAD 7f1ee89) — vendored as kernels/k_hcfuse.py; bit-exact w/ shadow-check GLM53_K_HCFUSE_CHECK' provenance: credit: @@ -84,13 +93,13 @@ provenance: url: 'https://github.com/Flun/glm53-flash-cmp170hx-exl3' - name: 'turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint)' - url: 'https://github.com/turboderp/exllamav3' + url: 'https://github.com/turboderp-org/exllamav3' - name: 'incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)' url: 'https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2' - name: '0xSero (k_hcfuse fused decode kernels)' - url: 'https://github.com/0xSero/glm53-flash-offload' + url: 'https://github.com/sybil-solutions/glm-flash-lite' - name: 'MiaAI-Lab (GLM-5.3-Flash-EXL3-2x-DGX-Sparks — conversion reference)' url: 'https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks'