diff --git a/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid/recipe.yaml b/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid/recipe.yaml index 6f7f7fbea..32bd428f4 100644 --- a/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid/recipe.yaml +++ b/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid/recipe.yaml @@ -47,6 +47,7 @@ config: runtime: env: 'run_tp.sh reads: PORT/HOST (8012 / 127.0.0.1), VISION=1 (0 = off, frees ~0.4 GB vision tower on GPU 0), HYBRID_DRAFT_TOKENS=15 (0 disables hybrid drafting; 15 measured faster than 8), RAM_TIER_GIB=8 (parked-session system RAM), MODELS_DIR, TP_BACKEND=nccl|native. Script exports (verbatim @ 0836b7dc): EXL3_BC_ATTN=1 EXL3_MLA_KIDX_RING=32 EXL3_INT8_GEMV=0 EXL3_GEMV=0 EXL3_DSA_QC_STAGE=1 EXL3_DSA_QC_STAGE_MIN_R=9 EXL3_TP_WORKER_TIMEOUT=300 EXL3_TP_SHARING_STRATEGY=file_descriptor EXL3_AUTOSPLIT_MARGIN_MB=128 GLM53_PREFILL_CHUNK_SIZE=2048 GLM53_DYNAMIC_DRAFT=0 GLM53_RECORD_DRAFT_STATS=1 GLM53_TP_ACTIVE_HOST=0 GLM53_K_HCFUSE=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false NCCL_SOCKET_IFNAME=lo (NCCL bootstrap stays on loopback); ulimit -n 65536.' notes: "run_tp.sh glm_api.py argv (verbatim @ 0836b7dc): -m models/GLM-5.3-Flash-exl3-3.05bpw -dm models/GLM-5.3-Flash-DFlash2-exl3-6bpw -cq 8 -cs 655360 --context-cap 392960 -ndt 7 -chunk_size 2048 -ambs 2 -gs 62,63 -lv -tp --experimental-dflash2-tp -tpb $TP_BACKEND --q8-staging --q8-head-gpu0 --experimental-concurrent-generation --concurrent-max-sequences 2 --concurrent-fair-gen-rounds 4 --concurrent-fair-chunk-size 512 --host $HOST --port $PORT --experimental-b2-session-cache --b2-target-cpu-cache-gib $RAM_TIER_GIB --recurrent-cache-gib 4 --session-cache-gib 1 --session-cache-max-checkpoints 8 --experimental-b2-shared-prefix (plus vision sandbox args when VISION=1; bwrap optional - without it images still decode under memory/CPU limits). Startup checks engine pins and REFUSES vision/concurrency on a modified engine tree. First start: one-time fused-kernel compile + ~117 GiB weight load + CUDA graph capture = 2-3 min to healthy. VRAM ~62.9/62.3 GiB idle, ~64.4/63.0 GiB peaks with vision under load (65,536 MiB cards). Keep the vision tower on GPU 0 (GPU 1 placement showed one unexplained TP timeout). Session sharing: two active requests share the 655,360-token cache - ~320K + ~320K or 393K + ~255K fit, 330K + 330K does not; two streams total throughput ~9-30 % below one stream; large new prompts slow a decoding session to ~1/3 until prefill completes; rare early-token flips when two identical requests start the same instant (coherent outputs). NCCL troubleshooting: nvidia-smi topo -p2p r must show OK, else TP_BACKEND=native. DFlash2 drafter lineage incoai/GLM-5.3-Flash-DFlash2 is CC BY-NC-ND 4.0 (noncommercial) - converted locally, no converted redistribution. Quantization quality figures (~93 % top-1 agreement, KLD ~0.05 vs BF16 at 3.05 bpw) are turboderp's/Flun's reported numbers, not this fork's measurements." + upstream_drift: 'Pinned to the last validated sha 0836b7dcb84e2816f2c59bdb8ffe117721a76eda; the published env and argv are verbatim at that revision. Upstream alexvnesta/glm53-flash-cmp170hx-tp has since advanced to db01c256a8003daf90c932ee25552526ede40f1f (38 commits, 329 files as of 2026-10-29), which this recipe does NOT validate — no boot, soak or capture has been run against it. The patch set and argv documented here apply to 0836b7dc only.' performance: claimed_by: submitter metrics: @@ -122,7 +123,7 @@ provenance: url: 'https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2' - name: "0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)" - url: 'https://github.com/0xSero/glm53-flash-offload' + url: 'https://github.com/sybil-solutions/glm-flash-lite' - name: 'Morrowmake (content-verified P2P check idea + benchmark prompt style)' url: 'https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe'