Qwen3.8-Flash-Next-FP8 4x CMP 170HX — PP4 MTP-4 patched vLLM (g4bwy)

community Community submission — verify on your own hardware.

recipe by g4bwy↗ · listed by localai-recipes

revision r2 · commit 540d404493 · updated 10 d ago · license CC-BY-4.0

chat code pipeline-parallel mtp-speculative long-context engram speculative prefix-caching

Recipe by g4bwy. Official Qwen/Qwen3.8-Flash-Next-FP8 on 4x unlocked CMP 170HX (GA100 sm_80, 74 SMs, 64 GB) with PP=4 (layer partition 13,12,12,11) and MTP-4 speculative decoding on pinned vLLM mainline 183625d4f9 (2026-10-13, precompiled wheel 0.32.0rc1.dev34+g183625d4f) plus a 7-patch series applied with git am (tree hash must equal 18e0ec8942f61f82fd3a13bd9d9c6b9ce67ad371; 2 patches adopted from upstream PRs #56444/#58094) that unblocks PP+MTP+PLE on sm_80. 1M context at static YaRN 4.0, PLE n-gram table in pinned host RAM (--engram-config cpu_offload), Mamba align prefix caching with retention 16160 + match-unit 16 (99.8-99.9% hits), NCCL P2P disabled (host-staged wins at PP=4), expandable_segments, 200 W per-card caps, uv venv, no docker. Optional W8A16 int8-dense lane (checkpoint/) +15-20% single-stream.

Launch command

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 --served-model-name qwen3.8-flash-next-fp8 --port 8000 --pipeline-parallel-size 4 --engram-config '{"cpu_offload": true}' --moe-backend humming --enable-prefix-caching --mamba-cache-mode align --prefix-cache-retention-interval 16160 --block-size 1616 --prefix-match-unit 16 --gpu-memory-utilization 0.9453 --max-model-len 1000000 --hf-overrides '{"text_config":{"max_position_embeddings":1000000,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}}' --max-num-seqs 8 --speculative-config '{"method":"mtp","num_speculative_tokens":4}' -cc.cudagraph_mode=FULL_AND_PIECEWISE --no-enable-flashinfer-autotune --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3

runtime

env: 'CUDA_VISIBLE_DEVICES=0,1,2,3 | VLLM_PP_LAYER_PARTITION=13,12,12,11 | NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1 | PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | VLLM_SAFETENSORS_DROP_CACHE=1 (verbatim from serve-4x-1m.sh @ dcd5ba4).'
notes: "serve-4x-1m.sh is the service lane (PROF=1 arms a capped torch profiler). VLLM_PP_LAYER_PARTITION 13,12,12,11 is worth +2.13% pool at 1M against 12,12,12,12 (rank 0 binds: PLE layer at decoder index 1 costs it 0.80 GiB KV + 2.53 GiB weights; rank 3 also carries the MTP drafter). NCCL P2P measured 2-3% WORSE on prefill at PP=4 (host-staged wins; only 2-rank send/recv hops, TP=4 needs NCCL_P2P_LEVEL=SYS instead and was rejected at 1.74x pool). Startup checks: 'Using max model len 1000000' twice (target + Qwen4ExpMTP draft) and 'Maximum concurrency for 1000000 tokens per request' >= 1.0; proof line 'MTP drafter scores a 40000-token draft vocabulary' when checkpoint/ artifacts are present. Rank-0 shift costs rank 3 nothing (it gives the room back)."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy.json

Engine

vllm · source↗ · dcd5ba47↗

Install note: Clone vllm, checkout 183625d4f9f49968218ffbe43242877ead72d569, git am --keep-non-patch patches/00*.patch; git rev-parse 'HEAD^{tree}' must print 18e0ec8942f61f82fd3a13bd9d9c6b9ce67ad371. uv venv --python 3.12; VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_COMMIT=183625d4f9f49968218ffbe43242877ead72d569 uv pip install -e . --torch-backend=auto. Patches are pure Python (no rebuild on re-am).

Model

Qwen3.8-Flash-Next-FP8 (fp8)
AA intelligence 40 (Artificial Analysis · measured)

Hardware

NVIDIA CMP 170HX 64GB ×4 (Ampere)

Context

max context: 1,000,000 tokens

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
262,144	1	9181 tok/s	181.5 tok/s	21300 ms	reported
262,144	4	—	548 tok/s	56300 ms	reported
524,288	1	10038 tok/s	184.4 tok/s	43700 ms	reported
1,000,000	1	10138 tok/s	183.2 tok/s	91900 ms	reported
1,000,000	2	—	340 tok/s	140100 ms	reported
1,000,000	4	—	325 tok/s	239500 ms	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	All values g4bwy's own measurements on the 4x CMP 170HX box (200 W caps, 74 SMs, partition 13,12,12,11, draft-vocab head, decode 1024, repetition-task prompts; restored-silicon grid 2026-10-07 + rebase certification 2026-10-13 at base 183625d4f9, benchmark/results.md @ dcd5ba4). claimed_by=submitter T0 - no run file ingested; import via POST /api/v1/benchmarks to upgrade.

Downloads

Weights https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8
official FP8 checkpoint (173 GiB, 173 GiB boot read); MODEL=/local/path overrides the serve script default
Engine image https://github.com/vllm-project/vllm
mainline 183625d4f9f49968218ffbe43242877ead72d569 (2026-10-13) + patches/00*.patch (7-patch series, tree 18e0ec8942f61f82fd3a13bd9d9c6b9ce67ad371) + VLLM_PRECOMPILED_WHEEL_COMMIT=183625d4f9f49968218ffbe43242877ead72d569 (wheel 0.32.0rc1.dev34+g183625d4f)
Reference https://github.com/g4bwy/qwen3.8-flash-next-cmp-170hx
dcd5ba47c17f7c952f6b8d42b9508f3b4f4a4d34 - source recipe repo @ HEAD (serve-4x-1m.sh verbatim argv/env; README, benchmark/results.md, patches/, checkpoint/, archive/ all at this commit)

Revision notes

Community ingest of g4bwy's recipe repo @ dcd5ba4 (pushed 2026-10-14, 'series refresh: patch 4 method-gated ... new replay tree 18e0ec8942'). Launch argv transcribed verbatim from serve-4x-1m.sh @ dcd5ba4; env exports are part of the script (VLLM_PP_LAYER_PARTITION, NCCL_P2P_DISABLE, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, VLLM_SAFETENSORS_DROP_CACHE=1). Measured numbers are the author's benchmark/results.md tables: restored-silicon 4x-1M grid 2026-10-07 (74 SMs, bench-4x-1m-sm74-200w.json) + rebase certification on base 183625d4f9 2026-10-13 (logs/rebase-*.json) + decode step anatomy (212 tok/s stated as the single-stream roofline of this checkpoint on 4x170HX@200W at d4). Upstream (unpatched) vLLM refuses PP3+MTP+PLE on this checkpoint (drafter asserts on the last rank, PLE rejected across pipeline ranks, KV allocation dies with a bare StopIteration) - the 7 patches are load-bearing. Patch 5 (vocab-truncated MTP draft head, ~40k rows vs 248k) engages only when checkpoint/ artifacts exist and is worth ~9% single-stream; the int8-dense W8A16 lane (checkpoint/, patch 7) reads 216-226 tok/s (+15-20% over official FP8 at depth 4) with a 7.3% machine non-reproducibility floor measured and stated. Cards run the amoghmunikote/cmpunlocker module (SM compute + 64 GB HBM2e + Gen2 + BAR1); PCIe is Gen2 x16/x4 mixed, ~2 GB/s per pair, NODE distance; MTP acceptance 4.75-4.9 at depth 4 on pinned salt.

Revision history

14155efc25 recipe: update qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy (r2: /max_context_len) 10 d ago diff
c6f7b00485 recipe: create qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy 16 d ago

Fork

Export

recipe.yaml (raw)