GLM-5.3-Flash on 4x CMP 170HX — Morrowmake recipe 1.8.0, TP4 DFlash2

vllm serve /models/GLM-5.3-Flash-W4A16-MTP --served-model-name glm-5.3-flash --host 0.0.0.0 --port 8000 --pipeline-parallel-size 4 --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 2312 --long-prefill-token-threshold 0 --gpu-memory-utilization 0.93 --trust-remote-code --reasoning-parser glm47 --enable-auto-tool-choice --tool-call-parser glm47 --speculative-config '{"method":"dflash","model":"/models/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7,"adaptive_k":{"by_load":[7,5,3],"accept":{"costs":[1.0,1.0894,1.1375,1.2001,1.2239],"costs_multi":[1.0,1.0889,1.1194,1.2001,1.2239],"hysteresis":0.03},"log_interval":0}}' --block-size 4608 --prefill-chunk-with-decodes 384 --max-num-partial-prefills 2

env: 'AS-RUN 2026-09-28 (unchanged): LAYOUT=pp4 (PP4/TP1), GPU_UTIL=0.93 — 0.95 filled the last 170HX to ~70 MiB free and a 72 MiB KDA prefill workspace OOM-killed warmup; 0.93 leaves ~1.3 GiB headroom the KV pool will not claim (deviation from serve.sh default GPU_UTIL=0.95). --host 0.0.0.0 is the as-run box value; the 1.8.0 default is HOST=127.0.0.1 (loopback) and the container publishes 127.0.0.1:8000 only. serve.sh v1.8.0 serving defaults behind this launch (documented here so a copy-paste into 1.8.0 keeps every feature): MAX_BATCHED=2312 (GLM_DEFAULT_MAX_BATCHED_PP4; TP4 default is 3463 = 3,456-token chunks + 7 draft slots), BLOCK_SIZE=4608 (PP4+dflash), FAIR_CHUNK=384 + FAIR_PARTIAL=2 (FAIR_PREFILL=1), PREFILL_CAP=0 -> --long-prefill-token-threshold 0 (keep 0; >0 costs -15.3% prefill / +16.5% TTFT on a 23K prompt and silently disables the prefill kernels), DRAFT_MODE=dflash adaptive depth num_speculative_tokens=7 + adaptive_k (SPEC_N is REFUSED in dflash mode at 1.8.0; another depth goes through EXTRA_ARGS --speculative-config), NCCL_P2P_LEVEL=SYS, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, VLLM_CUSTOM_ALLREDUCE_ALGO=2stage, VLLM_AMPERE_DECODE_KERNELS=1, sm_80 prefill kernels default on, VLLM_THIN_GEMM=1, VLLM_THIN_GEMM_CONCAT=1, VLLM_HOST_ALLREDUCE=1, VLLM_MOE_DECODE_ROUTE_V2=1, VLLM_MOE_DETERMINISTIC_ALIGN=1, VLLM_MOE_MASK_PADDING=1, VLLM_SHARED_EXPERT_REORDER=1, VLLM_FLA_PIN_AUTOTUNE=1, VLLM_AMPERE_MARLIN_DECODE_CUDA=1, VLLM_AMPERE_MARLIN_DECODE_V2=1 (TP4 only), VLLM_AMPERE_KDA_PREFILL_NARROW=1 + VLLM_AMPERE_KDA_PREFILL_WIDE=1, VLLM_AMPERE_KDA_FOLD=1, VLLM_AMPERE_KDA_SMALL_STACK=1, VLLM_AMPERE_MARLIN_PREFILL_NARROW=1, VLLM_AMPERE_MARLIN_PREFILL_WIDE=1, VLLM_AMPERE_SPARSE_MLA_PREFILL_WIDE=1, VLLM_INDEXER_DECODE_RAW_K=1, VLLM_TOOL_CHOICE_NONE_MASK=1, VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128 (128 at 1.8.0, was 512 in 1.7.2 docs), VLLM_DFLASH_BOUNDARY_CACHE=1, VLLM_STATE_INDEX_CHECK=1 (engine validation default), VLLM_CUSTOM_ALLREDUCE_PCIE_P2P=1 (default under TP4; under PP4 1.8.0 no longer exports it — TP=1 has no tensor-parallel all-reduce), VLLM_PP_DRAFT_TAIL_STAGE=2, VLLM_PP_SPREAD_DECODES=1, VLLM_PP_PACKED_HOP=1, VLLM_PP_HOP_NO_METADATA=1, VLLM_PP_SPLIT_DRAFT_EVENT=1. container.env: LAYOUT=pp4, P2P=auto, HF_HOME/FLASHINFER_WORKSPACE_BASE/TORCHINDUCTOR_CACHE_DIR under ./cache so the first-boot compile happens once and later boots reuse it. RENAMED at 1.8.0 — the engine logs "... is retired and ignored" once and keeps running at the CODE DEFAULT (many are OFF), so old names in .env/container.env lose features silently instead of erroring; rename them: VLLM_GLM5_DECODE_KERNELS -> VLLM_AMPERE_DECODE_KERNELS, VLLM_GLM5_DECODE_IDX_GLUE -> VLLM_AMPERE_DECODE_IDX_GLUE, VLLM_GLM5_DECODE_MOE_ROUTE_V2 -> VLLM_MOE_DECODE_ROUTE_V2, VLLM_GLM5_DETERMINISTIC_MOE_ALIGN -> VLLM_MOE_DETERMINISTIC_ALIGN, VLLM_GLM5_MOE_MASK_PADDING -> VLLM_MOE_MASK_PADDING, VLLM_GLM5_THIN_GEMM -> VLLM_THIN_GEMM, VLLM_GLM5_HOST_ALLREDUCE -> VLLM_HOST_ALLREDUCE, VLLM_GLM5_SHARED_EXPERT_REORDER -> VLLM_SHARED_EXPERT_REORDER, VLLM_GLM5_FLA_PIN_AUTOTUNE -> VLLM_FLA_PIN_AUTOTUNE, VLLM_GLM5_INDEXER_DECODE_RAW_K -> VLLM_INDEXER_DECODE_RAW_K, VLLM_GLM5_TOOL_CHOICE_NONE_MASK -> VLLM_TOOL_CHOICE_NONE_MASK, VLLM_GLM5_DFLASH_BOUNDARY_CACHE -> VLLM_DFLASH_BOUNDARY_CACHE, VLLM_GLM5_MARLIN_DECODE_CUDA -> VLLM_AMPERE_MARLIN_DECODE_CUDA, VLLM_GLM5_TP4_KDA_PREFILL -> VLLM_AMPERE_KDA_PREFILL_NARROW, VLLM_GLM5_TP4_MARLIN_PREFILL -> VLLM_AMPERE_MARLIN_PREFILL_NARROW, VLLM_GLM5_PP_KDA_PREFILL -> VLLM_AMPERE_KDA_PREFILL_WIDE, VLLM_GLM5_PP_SPARSE_MLA_PREFILL -> VLLM_AMPERE_SPARSE_MLA_PREFILL_WIDE, VLLM_GLM5_PP_MARLIN_PREFILL -> VLLM_AMPERE_MARLIN_PREFILL_WIDE, VLLM_GLM5_PP_MARLIN_PREFILL_COMPILED -> VLLM_AMPERE_MARLIN_PREFILL_WIDE_COMPILED, VLLM_ALLOW_PCIE_P2P_CUSTOM_ALLREDUCE -> VLLM_CUSTOM_ALLREDUCE_PCIE_P2P, VLLM_GLM5_STATE_INDEX_CHECK -> VLLM_STATE_INDEX_CHECK (VLLM_GLM5_TP4_MARLIN_PREFILL survives in 1.8.0 serve.sh only as a leftover comment — not authoritative). NOT READ at 1.8.0 (no warning; no-op at 1.8.0, setting them has no effect): VLLM_GLM5_PROLOGUE_FUSE, VLLM_GLM5_DECODE_KDA_V2, VLLM_GLM5_REPLICATED_EMBED, VLLM_GLM5_SPARSE_MLA_DECODE_LEGACY. VLLM_GLM5_LOCAL_LOGITS=0 became EXTRA_ARGS=--no-enable-batch-sharded-sampling; VLLM_GLM5_DFLASH_ADAPTIVE_K (+_K_DEPTHS/_K_ACCEPT/_K_COSTS/_K_COSTS_MULTI) and VLLM_GLM5_DFLASH_ADAPTIVE_DRAFT_WIDTH collapse into the single --speculative-config adaptive_k above; removed with no replacement (also silently ignored): VLLM_GLM5_PREFILL_OVERLAP (+_BACKEND/_SPLITS), VLLM_GLM5_MARLIN_DECODE_VARIANT, VLLM_GLM5_INDEXER_DECODE_ROWS, VLLM_GLM5_INDEXER_GATHER_CLAMP, VLLM_KV_SWA_INFLIGHT_SCRATCH, VLLM_GLM5_TP4_MARLIN_PREFILL_COMPILED, VLLM_GLM5_THIN_GEMM_V74, VLLM_GLM5_DECODE_MHC_V3, VLLM_GLM5_DECODE_MHC_V2_FN_BF16, VLLM_GLM5_PREFILL_PACK_BF16X2, VLLM_CUSTOM_ALLREDUCE_FLAGS_BUILD_DIR. INSTALL at 1.8.0: container-only — RUNTIME=native, the venv/native build, the install.sh venv path and the native docs are gone; ./start.sh or docker compose up does the whole first start; delete from .env: RUNTIME, VLLM_COMMIT, VLLM_REPO, VLLM_BRANCH, VLLM_PRECOMPILED_WHEEL_COMMIT, VENV, VLLM_SRC, CUDA_HOME, PYTHON_VERSION, BUILD_FROM_SOURCE, MAX_JOBS, NATIVE_INSTALL_RESUME, FLASHINFER_INSTALL_*, TORCH_CUDA_ARCH_LIST, TORCH_INDEX_URL, VLLM_BUILD_AMPERE_MARLIN (the last three named ones are warned about until removed; the image already carries the optional Marlin library).'
notes: 'AS-RUN PP4 box 2026-09-28: util cut 0.95 -> 0.93 (0.95 filled the last card to ~70 MiB free; a 72 MiB KDA prefill workspace OOM-killed warmup). PP4 layout: activations-only inter-card traffic, tuned for x4-link cards; start.sh checks link width and suggests LAYOUT=pp4 automatically. The image runs at native-install speed; 1.8.0 is container-only, so no native install exists. All sm_80 feature flags remain kill switches via env (VLLM_AMPERE_DECODE_KERNELS, FAIR_PREFILL etc.). FIELD REPORT now historical (2026-10-13, site owner): the 1.7.x "off the rails" reports were resolved in 1.7.2 and the 2026-10-08 reports resolved as of 1.7.2; superseded by release 1.8.0 (2026-10-16). KEEPING the 1.6.0 image (ghcr.io/morrowmake/vllm-cmp170hx:1.6.0-3a2bf16dae) as the documented known-good fallback: if output coherence degrades on your rig, trying 1.6.0 before debugging hardware remains reasonable.'
container: 'docker run -d --name glm53-flash --user "$(id -u):$(id -g)" --workdir /cache --gpus all --shm-size 16g -p 127.0.0.1:8000:8000 --env-file docker/container.env -v serve.sh:/recipe/serve.sh:ro -v boot_check.py:/recipe/boot_check.py:ro -v p2p_check.py:/recipe/p2p_check.py:ro -v p2p_check.sh:/recipe/p2p_check.sh:ro -v p2p_probe.cu:/recipe/p2p_probe.cu:ro -v models/GLM-5.3-Flash-W4A16-MTP:/models/GLM-5.3-Flash-W4A16-MTP:ro -v models/GLM-5.3-Flash-DFlash2:/models/GLM-5.3-Flash-DFlash2:ro -v cache:/cache --entrypoint /bin/bash ghcr.io/morrowmake/vllm-cmp170hx@sha256:18a2f26207568128e1d792ce923e86a74e40def1e43e492db83132cab8cdbe14 /recipe/serve.sh (1.8.0: one-command start is ./start.sh or docker compose up — a prepare service makes ./cache and models writable by the checkout owner and downloads the checkpoints; the server runs as the checkout owner, never root; HOST_UID/HOST_GID override; first boot compiles kernel caches into ./cache and is slower, later boots reuse them)'

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json