{"slug":"glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141","title":"GLM-5.3-Flash on 4x CMP 170HX \u2014 Morrowmake recipe 1.8.0, TP4 DFlash2","summary":"PP4 (pipeline-parallel 4, TP1) as-run on a 4x CMP 170HX rig 2026-09-28 at --gpu-memory-utilization 0.93 after 0.95 OOM-killed warmup (a 72 MiB KDA prefill workspace did not fit in the ~70 MiB left on the last card). Author 1.8.0 measurements below are TP4 unless labeled PP4; KV pool 2,088,960 tokens under PP4 (author-published 1.8.0 figure). Recipe release 1.8.0 2026-10-16 (engine 819146023f, image engine-819146023f) \u2014 updated from 1.7.2; engine rebased on upstream vLLM 4e0681ed56, same context/batch/fair-prefill defaults; 21 engine-switch env names renamed (old names silently ignored at code defaults); container-only install; author re-measured throughput/KV tables at 1.8.0.","license":"CC-BY-4.0","tier":"curated","beta":false,"revision":{"no":10,"commit_sha":"6e47f035b02d04cee06466adb7fb69b21d9d5240","content_hash":"171456f9f739f5c140a9a08da526b95e056ea9fe","created_at":"2026-10-29 23:29:37.313564-06","notes":"Recipe by Morrowmake (github.com/Morrowmake/glm53-flash-cmp170hx-recipe, MIT licence), release 1.4.1 2026-09-27. Mirrored into localairecipes from the local checkout with every pin transcribed verbatim; engine+image+weights are digest/commit-pinned so this exact stack is reproducible. Updated 2026-10-06 from 1.4.1 to 1.7.2; updated 2026-10-28 from 1.7.2 to release 1.8.0 2026-10-16 (engine 819146023facb31c04f1f8512e52464835a10d40, image tag engine-819146023f, digest sha256:18a2f26207568128e1d792ce923e86a74e40def1e43e492db83132cab8cdbe14): engine rebased on upstream vLLM 4e0681ed56 (1,144 commits ahead of the 1.7.2 engine ab60b723ada; a rebase, not a linear descendant \u2014 merge-base e55d076f89 is the upstream base 1.7.0-1.7.2 sat on, and the 1.7.2 engine line is not an ancestor of 1.8.0), same features and same context/batch/fair-prefill defaults; 21 engine-switch env names renamed (old names silently ignored at code defaults \u2014 see the switch list in config.runtime.env); DFlash default is adaptive depth 7 with adaptive_k by_load [7,5,3]; container-only install (native/venv path removed; first boot compiles kernel caches); --host default is now 127.0.0.1 (the published 0.0.0.0 is the as-run box value only); author re-measured throughput/KV tables for 1.8.0 on 2026-10-16. FIELD REPORT now historical (2026-10-13, site owner): the 1.7.x \"off the rails\" reports were resolved in 1.7.2; superseded by 1.8.0. KEEPING 1.6.0 (image tag 1.6.0-3a2bf16dae) as the documented known-good fallback."},"engine_source":{"type":"docker_image","git_url":"https://github.com/Morrowmake/vllm-cmp170hx","base_engine":"vllm","pip_package":null,"base_version":"Morrowmake/vllm-cmp170hx @ 819146023f","image_digest":"ghcr.io/morrowmake/vllm-cmp170hx@sha256:18a2f26207568128e1d792ce923e86a74e40def1e43e492db83132cab8cdbe14","install_note":null,"pinned_commit":"819146023facb31c04f1f8512e52464835a10d40"},"credit":[{"name":"Morrowmake","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/models/GLM-5.3-Flash-W4A16-MTP","--served-model-name","glm-5.3-flash","--host","0.0.0.0","--port","8000","--pipeline-parallel-size","4","--max-model-len","262144","--max-num-seqs","8","--max-num-batched-tokens","2312","--long-prefill-token-threshold","0","--gpu-memory-utilization","0.93","--trust-remote-code","--reasoning-parser","glm47","--enable-auto-tool-choice","--tool-call-parser","glm47","--speculative-config","{\"method\":\"dflash\",\"model\":\"/models/GLM-5.3-Flash-DFlash2\",\"num_speculative_tokens\":7,\"adaptive_k\":{\"by_load\":[7,5,3],\"accept\":{\"costs\":[1.0,1.0894,1.1375,1.2001,1.2239],\"costs_multi\":[1.0,1.0889,1.1194,1.2001,1.2239],\"hysteresis\":0.03},\"log_interval\":0}}","--block-size","4608","--prefill-chunk-with-decodes","384","--max-num-partial-prefills","2"],"runtime":{"env":"AS-RUN 2026-09-28 (unchanged): LAYOUT=pp4 (PP4/TP1), GPU_UTIL=0.93 \u2014 0.95 filled the last 170HX to ~70 MiB free and a 72 MiB KDA prefill workspace OOM-killed warmup; 0.93 leaves ~1.3 GiB headroom the KV pool will not claim (deviation from serve.sh default GPU_UTIL=0.95). --host 0.0.0.0 is the as-run box value; the 1.8.0 default is HOST=127.0.0.1 (loopback) and the container publishes 127.0.0.1:8000 only. serve.sh v1.8.0 serving defaults behind this launch (documented here so a copy-paste into 1.8.0 keeps every feature): MAX_BATCHED=2312 (GLM_DEFAULT_MAX_BATCHED_PP4; TP4 default is 3463 = 3,456-token chunks + 7 draft slots), BLOCK_SIZE=4608 (PP4+dflash), FAIR_CHUNK=384 + FAIR_PARTIAL=2 (FAIR_PREFILL=1), PREFILL_CAP=0 -> --long-prefill-token-threshold 0 (keep 0; >0 costs -15.3% prefill / +16.5% TTFT on a 23K prompt and silently disables the prefill kernels), DRAFT_MODE=dflash adaptive depth num_speculative_tokens=7 + adaptive_k (SPEC_N is REFUSED in dflash mode at 1.8.0; another depth goes through EXTRA_ARGS --speculative-config), NCCL_P2P_LEVEL=SYS, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, VLLM_CUSTOM_ALLREDUCE_ALGO=2stage, VLLM_AMPERE_DECODE_KERNELS=1, sm_80 prefill kernels default on, VLLM_THIN_GEMM=1, VLLM_THIN_GEMM_CONCAT=1, VLLM_HOST_ALLREDUCE=1, VLLM_MOE_DECODE_ROUTE_V2=1, VLLM_MOE_DETERMINISTIC_ALIGN=1, VLLM_MOE_MASK_PADDING=1, VLLM_SHARED_EXPERT_REORDER=1, VLLM_FLA_PIN_AUTOTUNE=1, VLLM_AMPERE_MARLIN_DECODE_CUDA=1, VLLM_AMPERE_MARLIN_DECODE_V2=1 (TP4 only), VLLM_AMPERE_KDA_PREFILL_NARROW=1 + VLLM_AMPERE_KDA_PREFILL_WIDE=1, VLLM_AMPERE_KDA_FOLD=1, VLLM_AMPERE_KDA_SMALL_STACK=1, VLLM_AMPERE_MARLIN_PREFILL_NARROW=1, VLLM_AMPERE_MARLIN_PREFILL_WIDE=1, VLLM_AMPERE_SPARSE_MLA_PREFILL_WIDE=1, VLLM_INDEXER_DECODE_RAW_K=1, VLLM_TOOL_CHOICE_NONE_MASK=1, VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128 (128 at 1.8.0, was 512 in 1.7.2 docs), VLLM_DFLASH_BOUNDARY_CACHE=1, VLLM_STATE_INDEX_CHECK=1 (engine validation default), VLLM_CUSTOM_ALLREDUCE_PCIE_P2P=1 (default under TP4; under PP4 1.8.0 no longer exports it \u2014 TP=1 has no tensor-parallel all-reduce), VLLM_PP_DRAFT_TAIL_STAGE=2, VLLM_PP_SPREAD_DECODES=1, VLLM_PP_PACKED_HOP=1, VLLM_PP_HOP_NO_METADATA=1, VLLM_PP_SPLIT_DRAFT_EVENT=1. container.env: LAYOUT=pp4, P2P=auto, HF_HOME/FLASHINFER_WORKSPACE_BASE/TORCHINDUCTOR_CACHE_DIR under ./cache so the first-boot compile happens once and later boots reuse it. RENAMED at 1.8.0 \u2014 the engine logs \"... is retired and ignored\" once and keeps running at the CODE DEFAULT (many are OFF), so old names in .env/container.env lose features silently instead of erroring; rename them: VLLM_GLM5_DECODE_KERNELS -> VLLM_AMPERE_DECODE_KERNELS, VLLM_GLM5_DECODE_IDX_GLUE -> VLLM_AMPERE_DECODE_IDX_GLUE, VLLM_GLM5_DECODE_MOE_ROUTE_V2 -> VLLM_MOE_DECODE_ROUTE_V2, VLLM_GLM5_DETERMINISTIC_MOE_ALIGN -> VLLM_MOE_DETERMINISTIC_ALIGN, VLLM_GLM5_MOE_MASK_PADDING -> VLLM_MOE_MASK_PADDING, VLLM_GLM5_THIN_GEMM -> VLLM_THIN_GEMM, VLLM_GLM5_HOST_ALLREDUCE -> VLLM_HOST_ALLREDUCE, VLLM_GLM5_SHARED_EXPERT_REORDER -> VLLM_SHARED_EXPERT_REORDER, VLLM_GLM5_FLA_PIN_AUTOTUNE -> VLLM_FLA_PIN_AUTOTUNE, VLLM_GLM5_INDEXER_DECODE_RAW_K -> VLLM_INDEXER_DECODE_RAW_K, VLLM_GLM5_TOOL_CHOICE_NONE_MASK -> VLLM_TOOL_CHOICE_NONE_MASK, VLLM_GLM5_DFLASH_BOUNDARY_CACHE -> VLLM_DFLASH_BOUNDARY_CACHE, VLLM_GLM5_MARLIN_DECODE_CUDA -> VLLM_AMPERE_MARLIN_DECODE_CUDA, VLLM_GLM5_TP4_KDA_PREFILL -> VLLM_AMPERE_KDA_PREFILL_NARROW, VLLM_GLM5_TP4_MARLIN_PREFILL -> VLLM_AMPERE_MARLIN_PREFILL_NARROW, VLLM_GLM5_PP_KDA_PREFILL -> VLLM_AMPERE_KDA_PREFILL_WIDE, VLLM_GLM5_PP_SPARSE_MLA_PREFILL -> VLLM_AMPERE_SPARSE_MLA_PREFILL_WIDE, VLLM_GLM5_PP_MARLIN_PREFILL -> VLLM_AMPERE_MARLIN_PREFILL_WIDE, VLLM_GLM5_PP_MARLIN_PREFILL_COMPILED -> VLLM_AMPERE_MARLIN_PREFILL_WIDE_COMPILED, VLLM_ALLOW_PCIE_P2P_CUSTOM_ALLREDUCE -> VLLM_CUSTOM_ALLREDUCE_PCIE_P2P, VLLM_GLM5_STATE_INDEX_CHECK -> VLLM_STATE_INDEX_CHECK (VLLM_GLM5_TP4_MARLIN_PREFILL survives in 1.8.0 serve.sh only as a leftover comment \u2014 not authoritative). NOT READ at 1.8.0 (no warning; no-op at 1.8.0, setting them has no effect): VLLM_GLM5_PROLOGUE_FUSE, VLLM_GLM5_DECODE_KDA_V2, VLLM_GLM5_REPLICATED_EMBED, VLLM_GLM5_SPARSE_MLA_DECODE_LEGACY. VLLM_GLM5_LOCAL_LOGITS=0 became EXTRA_ARGS=--no-enable-batch-sharded-sampling; VLLM_GLM5_DFLASH_ADAPTIVE_K (+_K_DEPTHS/_K_ACCEPT/_K_COSTS/_K_COSTS_MULTI) and VLLM_GLM5_DFLASH_ADAPTIVE_DRAFT_WIDTH collapse into the single --speculative-config adaptive_k above; removed with no replacement (also silently ignored): VLLM_GLM5_PREFILL_OVERLAP (+_BACKEND/_SPLITS), VLLM_GLM5_MARLIN_DECODE_VARIANT, VLLM_GLM5_INDEXER_DECODE_ROWS, VLLM_GLM5_INDEXER_GATHER_CLAMP, VLLM_KV_SWA_INFLIGHT_SCRATCH, VLLM_GLM5_TP4_MARLIN_PREFILL_COMPILED, VLLM_GLM5_THIN_GEMM_V74, VLLM_GLM5_DECODE_MHC_V3, VLLM_GLM5_DECODE_MHC_V2_FN_BF16, VLLM_GLM5_PREFILL_PACK_BF16X2, VLLM_CUSTOM_ALLREDUCE_FLAGS_BUILD_DIR. INSTALL at 1.8.0: container-only \u2014 RUNTIME=native, the venv/native build, the install.sh venv path and the native docs are gone; ./start.sh or docker compose up does the whole first start; delete from .env: RUNTIME, VLLM_COMMIT, VLLM_REPO, VLLM_BRANCH, VLLM_PRECOMPILED_WHEEL_COMMIT, VENV, VLLM_SRC, CUDA_HOME, PYTHON_VERSION, BUILD_FROM_SOURCE, MAX_JOBS, NATIVE_INSTALL_RESUME, FLASHINFER_INSTALL_*, TORCH_CUDA_ARCH_LIST, TORCH_INDEX_URL, VLLM_BUILD_AMPERE_MARLIN (the last three named ones are warned about until removed; the image already carries the optional Marlin library).","notes":"AS-RUN PP4 box 2026-09-28: util cut 0.95 -> 0.93 (0.95 filled the last card to ~70 MiB free; a 72 MiB KDA prefill workspace OOM-killed warmup). PP4 layout: activations-only inter-card traffic, tuned for x4-link cards; start.sh checks link width and suggests LAYOUT=pp4 automatically. The image runs at native-install speed; 1.8.0 is container-only, so no native install exists. All sm_80 feature flags remain kill switches via env (VLLM_AMPERE_DECODE_KERNELS, FAIR_PREFILL etc.). FIELD REPORT now historical (2026-10-13, site owner): the 1.7.x \"off the rails\" reports were resolved in 1.7.2 and the 2026-10-08 reports resolved as of 1.7.2; superseded by release 1.8.0 (2026-10-16). KEEPING the 1.6.0 image (ghcr.io/morrowmake/vllm-cmp170hx:1.6.0-3a2bf16dae) as the documented known-good fallback: if output coherence degrades on your rig, trying 1.6.0 before debugging hardware remains reasonable.","container":"docker run -d --name glm53-flash --user \"$(id -u):$(id -g)\" --workdir /cache --gpus all --shm-size 16g -p 127.0.0.1:8000:8000 --env-file docker/container.env -v serve.sh:/recipe/serve.sh:ro -v boot_check.py:/recipe/boot_check.py:ro -v p2p_check.py:/recipe/p2p_check.py:ro -v p2p_check.sh:/recipe/p2p_check.sh:ro -v p2p_probe.cu:/recipe/p2p_probe.cu:ro -v models/GLM-5.3-Flash-W4A16-MTP:/models/GLM-5.3-Flash-W4A16-MTP:ro -v models/GLM-5.3-Flash-DFlash2:/models/GLM-5.3-Flash-DFlash2:ro -v cache:/cache --entrypoint /bin/bash ghcr.io/morrowmake/vllm-cmp170hx@sha256:18a2f26207568128e1d792ce923e86a74e40def1e43e492db83132cab8cdbe14 /recipe/serve.sh (1.8.0: one-command start is ./start.sh or docker compose up \u2014 a prepare service makes ./cache and models writable by the checkout owner and downloads the checkpoints; the server runs as the checkout owner, never root; HOST_UID/HOST_GID override; first boot compiles kernel caches into ./cache and is slower, later boots reuse them)"},"kv_pool_tokens":2088960,"max_context_len":262144},"models":[{"slug":"glm-5-3-flash-w4a16-mtp","name":"GLM-5.3-Flash-W4A16-MTP","hf_repo":"canada-quant/GLM-5.3-Flash-W4A16-MTP","quant":"w4a16-int4-g128"}],"hardware":[{"slug":"nvidia-cmp-170hxx4","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":4}],"generation":"ampere","tags":["dflash2","pipeline-parallel","tensor-parallel","vision"],"tasks":["chat","code","vision"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"T1 measured by the recipe author, release 1.8.0 (docs/results.md @ v1.8.0, measured 2026-10-16, engine 819146023f, 74 SMs/card, 180 W/card, PCIe x16, 262,144 ctx, temperature 0): 1-user decode = median of five streaming decode rates on ONE boot (1.7.x protocol was ten rates over two boots); scoring is on the answer after the reasoning (content) for both releases. TP4 1 user structured / code / prose 482.6 / 442.1 / 232.1 tok/s; 2-user aggregate 606.5 / 460.3 / 314.7; 4-user 717.7 / 618.7 / 501.4; 8-user 943.7 / 835.0 / 667.2; TP4 cold prefill 3,161 tok/s; KV pool 1,203,765 tokens at 262,144 (4.59 full-length). PP4 1 user 232.4 / 209.9 / 100.1; 2-user 343.8 / 273.4 / 195.5; 4-user 409.2 / 382.8 / 314.2; 8-user 664.1 / 608.8 / 469.2; PP4 cold prefill 7,419 tok/s; KV pool 2,088,960 tokens (7.97 full-length). Quality TP4: HumanEval 151/164, GSM8K 1,286/1,319 (1.7.2 re-scored the same way: 154/164, 1,284/1,319; neither suite significant, McNemar p=0.45 / 0.75). KL 1.8.0 vs 1.7.2: TP4 0.0158 short / 0.0083 long; PP4 0.0000 / 0.00014. The author publishes a vs-1.7.2-published column (1.7.2 published TP4 1-user 482.2 / 447.7 / 210.6; cold prefill 3,197; KV pool 1,199,570) \u2014 each column its own session, labelled published-vs-published, not a side-by-side run. Historical benchmark points on this recipe are 1.4.1/1.7.2-era measurements and keep their own attribution.\", \"tok_s\": 264.3999999999999772626324556767940521240234375, \"ttft_ms\": 63, \"context_len\": 262144}"}],"perf_points":[{"id":2185,"context_len":262144,"prefill_tok_s":6264,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"benchmark","notes":"PP4 cold prefill, 1.4.1 release measurement"},{"id":2183,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":264.3999999999999772626324556767940521240234375,"ttft_ms":63,"concurrency":1,"claimed_by":"submitter","notes":"As-run note 2026-09-28: util 0.95 -> 0.93 (0.95 filled the last card to ~70 MiB free; a 72 MiB KDA prefill workspace OOM-killed the engine at 09:35). Expect TP4-table decode to drop slightly; KV pool unchanged (pool is sized after the util cut). Carried from the r6 submitter point through the r7 performance replacement."},{"id":2184,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":264.3999999999999772626324556767940521240234375,"ttft_ms":63,"concurrency":1,"claimed_by":"benchmark","notes":"PP4 1.4.1 release measurement (author, median of 5): structured 142.2 / code 138.6 / prose 100.7 tok/s single-user; 8-user aggregate 556.3 / 503.5 / 376.8; TTFT 1.49 s at 6.2K / 3.79 s at 23.3K. TP4 single-user 264.4 (63 ms TTFT), 8-user 763.1. Cold prefill 2,670 (TP4) / 6,264 (PP4). KV pool 1,156,635 (TP4) / 2,320,328 (PP4). Quality: GSM8K 0.972, HumanEval 0.9634, PPL 3.2781."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/canada-quant/GLM-5.3-Flash-W4A16-MTP","pin":"INT4 weights, FP16 activations, group size 128 (~178 GiB)"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2","pin":"DFlash2 drafter ~2.2 GiB, adaptive depth at 1.8.0 (num_speculative_tokens 7 with adaptive_k by_load [7,5,3]; k=3 was the pre-1.8.0 fixed depth)"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://github.com/Morrowmake/vllm-cmp170hx","pin":"image ghcr.io/morrowmake/vllm-cmp170hx:engine-819146023f, digest sha256:18a2f26207568128e1d792ce923e86a74e40def1e43e492db83132cab8cdbe14 (recipe release 1.8.0). Image tags now name the engine source, independent of recipe release numbers. The 1.8.0 serve.sh launch differs from the as-run 1.7.2 command: DFlash is adaptive depth (num_speculative_tokens 7 + adaptive_k; SPEC_N refused in dflash mode) and --long-prefill-token-threshold 0 is explicit; PP4 batch/block/fair-prefill values are unchanged (2312 / 4608 / 384+2). The perf numbers below are the author 1.8.0 measurements (2026-10-16). (1.6.0-3a2bf16dae retained as the documented known-good fallback per field reports; the 1.7.x field report is historical \u2014 1.8.0 supersedes it.)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe","pin":"release 1.8.0 (engine 819146023f, tag glm53-recipe-1.8.0, rebased on upstream vLLM 4e0681ed56; image tags on GHCR include engine-819146023f and the older 1.7.2-ab60b723ad / 1.6.0-3a2bf16dae)"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json","alternate":"/r/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141","describedby":"/openapi.json"},"updated_at":"2026-10-29 23:29:37.313564-06","created_at":"2026-09-27 23:53:44.912006-06","speeds":{"decode_tok_s":264.3999999999999772626324556767940521240234375,"prefill_tok_s":6264,"leaderboard":"/api/v1/leaderboard.json"}}