{"slug":"qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy","title":"Qwen3.8-Flash-Next-FP8 4x CMP 170HX \u2014 PP4 MTP-4 patched vLLM (g4bwy)","summary":"Recipe by g4bwy. Official Qwen/Qwen3.8-Flash-Next-FP8 on 4x unlocked CMP 170HX (GA100 sm_80, 74 SMs, 64 GB) with PP=4 (layer partition 13,12,12,11) and MTP-4 speculative decoding on pinned vLLM mainline 183625d4f9 (2026-10-13, precompiled wheel 0.32.0rc1.dev34+g183625d4f) plus a 7-patch series applied with git am (tree hash must equal 18e0ec8942f61f82fd3a13bd9d9c6b9ce67ad371; 2 patches adopted from upstream PRs #56444/#58094) that unblocks PP+MTP+PLE on sm_80. 1M context at static YaRN 4.0, PLE n-gram table in pinned host RAM (--engram-config cpu_offload), Mamba align prefix caching with retention 16160 + match-unit 16 (99.8-99.9% hits), NCCL P2P disabled (host-staged wins at PP=4), expandable_segments, 200 W per-card caps, uv venv, no docker. Optional W8A16 int8-dense lane (checkpoint/) +15-20% single-stream.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"540d404493ab67413a71efa0f5e541d18d329313","content_hash":"ab27af4ddefebc602e00353e6d7b640e515f9c4d","created_at":"2026-10-20 21:52:36.721138-06","notes":"Community ingest of g4bwy's recipe repo @ dcd5ba4 (pushed 2026-10-14, 'series refresh: patch 4 method-gated ... new replay tree 18e0ec8942'). Launch argv transcribed verbatim from serve-4x-1m.sh @ dcd5ba4; env exports are part of the script (VLLM_PP_LAYER_PARTITION, NCCL_P2P_DISABLE, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, VLLM_SAFETENSORS_DROP_CACHE=1). Measured numbers are the author's benchmark/results.md tables: restored-silicon 4x-1M grid 2026-10-07 (74 SMs, bench-4x-1m-sm74-200w.json) + rebase certification on base 183625d4f9 2026-10-13 (logs/rebase-*.json) + decode step anatomy (212 tok/s stated as the single-stream roofline of this checkpoint on 4x170HX@200W at d4). Upstream (unpatched) vLLM refuses PP3+MTP+PLE on this checkpoint (drafter asserts on the last rank, PLE rejected across pipeline ranks, KV allocation dies with a bare StopIteration) - the 7 patches are load-bearing. Patch 5 (vocab-truncated MTP draft head, ~40k rows vs 248k) engages only when checkpoint/ artifacts exist and is worth ~9% single-stream; the int8-dense W8A16 lane (checkpoint/, patch 7) reads 216-226 tok/s (+15-20% over official FP8 at depth 4) with a 7.3% machine non-reproducibility floor measured and stated. Cards run the amoghmunikote/cmpunlocker module (SM compute + 64 GB HBM2e + Gen2 + BAR1); PCIe is Gen2 x16/x4 mixed, ~2 GB/s per pair, NODE distance; MTP acceptance 4.75-4.9 at depth 4 on pinned salt."},"engine_source":{"type":"fork","git_url":"https://github.com/g4bwy/qwen3.8-flash-next-cmp-170hx","base_engine":"vllm","pip_package":null,"base_version":"mainline 183625d4f9 + 7-patch series","image_digest":null,"install_note":"Clone vllm, checkout 183625d4f9f49968218ffbe43242877ead72d569, git am --keep-non-patch patches/00*.patch; git rev-parse 'HEAD^{tree}' must print 18e0ec8942f61f82fd3a13bd9d9c6b9ce67ad371. uv venv --python 3.12; VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_COMMIT=183625d4f9f49968218ffbe43242877ead72d569 uv pip install -e . --torch-backend=auto. Patches are pure Python (no rebuild on re-am).","pinned_commit":"dcd5ba47c17f7c952f6b8d42b9508f3b4f4a4d34"},"credit":[{"name":"g4bwy","url":"https://github.com/g4bwy/qwen3.8-flash-next-cmp-170hx"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","Qwen/Qwen3.8-Flash-Next-FP8","--served-model-name","qwen3.8-flash-next-fp8","--port","8000","--pipeline-parallel-size","4","--engram-config","{\"cpu_offload\": true}","--moe-backend","humming","--enable-prefix-caching","--mamba-cache-mode","align","--prefix-cache-retention-interval","16160","--block-size","1616","--prefix-match-unit","16","--gpu-memory-utilization","0.9453","--max-model-len","1000000","--hf-overrides","{\"text_config\":{\"max_position_embeddings\":1000000,\"rope_parameters\":{\"rope_type\":\"yarn\",\"factor\":4.0,\"original_max_position_embeddings\":262144}}}","--max-num-seqs","8","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":4}","-cc.cudagraph_mode=FULL_AND_PIECEWISE","--no-enable-flashinfer-autotune","--enable-auto-tool-choice","--tool-call-parser","qwen3_xml","--reasoning-parser","qwen3"],"layouts":["pp"],"runtime":{"env":"CUDA_VISIBLE_DEVICES=0,1,2,3 | VLLM_PP_LAYER_PARTITION=13,12,12,11 | NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1 | PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | VLLM_SAFETENSORS_DROP_CACHE=1 (verbatim from serve-4x-1m.sh @ dcd5ba4).","notes":"serve-4x-1m.sh is the service lane (PROF=1 arms a capped torch profiler). VLLM_PP_LAYER_PARTITION 13,12,12,11 is worth +2.13% pool at 1M against 12,12,12,12 (rank 0 binds: PLE layer at decoder index 1 costs it 0.80 GiB KV + 2.53 GiB weights; rank 3 also carries the MTP drafter). NCCL P2P measured 2-3% WORSE on prefill at PP=4 (host-staged wins; only 2-rank send/recv hops, TP=4 needs NCCL_P2P_LEVEL=SYS instead and was rejected at 1.74x pool). Startup checks: 'Using max model len 1000000' twice (target + Qwen4ExpMTP draft) and 'Maximum concurrency for 1000000 tokens per request' >= 1.0; proof line 'MTP drafter scores a 40000-token draft vocabulary' when checkpoint/ artifacts are present. Rank-0 shift costs rank 3 nothing (it gives the room back)."},"max_context_len":1000000},"models":[{"slug":"qwen3-8-flash-next-fp8","name":"Qwen3.8-Flash-Next-FP8","hf_repo":null,"quant":"fp8"}],"hardware":[{"slug":"nvidia-cmp-170hxx4","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":4}],"generation":"ampere","tags":["engram","long-context","mtp-speculative","pipeline-parallel","prefix-caching","speculative"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"All values g4bwy's own measurements on the 4x CMP 170HX box (200 W caps, 74 SMs, partition 13,12,12,11, draft-vocab head, decode 1024, repetition-task prompts; restored-silicon grid 2026-10-07 + rebase certification 2026-10-13 at base 183625d4f9, benchmark/results.md @ dcd5ba4). claimed_by=submitter T0 - no run file ingested; import via POST /api/v1/benchmarks to upgrade.\", \"tok_s\": 212, \"points\": [{\"notes\": \"1M YaRN 4.0 lane measured at 25% fill: TTFT 21.3 s, prefill 9,181 tok/s, decode 181.5 tok/s, acceptance 3.92 (bench-4x-1m-sm74-200w.json).\", \"ttft_ms\": 21300, \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 181.5, \"prefill_tok_s\": 9181}, {\"notes\": \"50% fill: TTFT 43.7 s, prefill 10,038 tok/s, decode 184.4 tok/s, acceptance 3.93.\", \"ttft_ms\": 43700, \"concurrency\": 1, \"context_len\": 524288, \"decode_tok_s\": 184.400000000000005684341886080801486968994140625, \"prefill_tok_s\": 10038}, {\"notes\": \"Full 1M window: TTFT 91.9 s, prefill 10,138 tok/s (peak 10.4k), decode 183.2 tok/s, acceptance 3.85. Rebase certification 2026-10-13 at base 183625d4f9 reads fixed-salt decode 209/212/211, acceptance 4.93/4.84/4.93, cold TTFT 91.8 s.\", \"ttft_ms\": 91900, \"concurrency\": 1, \"context_len\": 1000000, \"decode_tok_s\": 183.19999999999998863131622783839702606201171875, \"prefill_tok_s\": 10138}, {\"notes\": \"100% fill N=2: both streams resident (pool 2.42-2.60x one full request), win avg 95.3 tok/s/req, wall 193.5 s.\", \"ttft_ms\": 140100, \"concurrency\": 2, \"context_len\": 1000000, \"decode_tok_s\": 340}, {\"notes\": \"100% fill N=4: TTFT mean 239.5 s / p95 372.6 s, wall 392.9 s, zero preemptions (scheduler admission-gates; extra streams queue).\", \"ttft_ms\": 239500, \"concurrency\": 4, \"context_len\": 1000000, \"decode_tok_s\": 325}, {\"notes\": \"25% fill N=4: sust peak 548 agg (sust is a best-1.5-s sample that lands in the wind-down; use wall/out tok/s to compare - wall 98.7 s).\", \"ttft_ms\": 56300, \"concurrency\": 4, \"context_len\": 262144, \"decode_tok_s\": 548}], \"ttft_ms\": 91900, \"context_len\": 1000000}"}],"perf_points":[{"id":1686,"context_len":262144,"prefill_tok_s":9181,"decode_tok_s":181.5,"ttft_ms":21300,"concurrency":1,"claimed_by":"submitter","notes":"1M YaRN 4.0 lane measured at 25% fill: TTFT 21.3 s, prefill 9,181 tok/s, decode 181.5 tok/s, acceptance 3.92 (bench-4x-1m-sm74-200w.json)."},{"id":1687,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":548,"ttft_ms":56300,"concurrency":4,"claimed_by":"submitter","notes":"25% fill N=4: sust peak 548 agg (sust is a best-1.5-s sample that lands in the wind-down; use wall/out tok/s to compare - wall 98.7 s)."},{"id":1688,"context_len":524288,"prefill_tok_s":10038,"decode_tok_s":184.400000000000005684341886080801486968994140625,"ttft_ms":43700,"concurrency":1,"claimed_by":"submitter","notes":"50% fill: TTFT 43.7 s, prefill 10,038 tok/s, decode 184.4 tok/s, acceptance 3.93."},{"id":1689,"context_len":1000000,"prefill_tok_s":10138,"decode_tok_s":183.19999999999998863131622783839702606201171875,"ttft_ms":91900,"concurrency":1,"claimed_by":"submitter","notes":"Full 1M window: TTFT 91.9 s, prefill 10,138 tok/s (peak 10.4k), decode 183.2 tok/s, acceptance 3.85. Rebase certification 2026-10-13 at base 183625d4f9 reads fixed-salt decode 209/212/211, acceptance 4.93/4.84/4.93, cold TTFT 91.8 s."},{"id":1690,"context_len":1000000,"prefill_tok_s":null,"decode_tok_s":340,"ttft_ms":140100,"concurrency":2,"claimed_by":"submitter","notes":"100% fill N=2: both streams resident (pool 2.42-2.60x one full request), win avg 95.3 tok/s/req, wall 193.5 s."},{"id":1691,"context_len":1000000,"prefill_tok_s":null,"decode_tok_s":325,"ttft_ms":239500,"concurrency":4,"claimed_by":"submitter","notes":"100% fill N=4: TTFT mean 239.5 s / p95 372.6 s, wall 392.9 s, zero preemptions (scheduler admission-gates; extra streams queue)."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8","pin":"official FP8 checkpoint (173 GiB, 173 GiB boot read); MODEL=/local/path overrides the serve script default"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/vllm-project/vllm","pin":"mainline 183625d4f9f49968218ffbe43242877ead72d569 (2026-10-13) + patches/00*.patch (7-patch series, tree 18e0ec8942f61f82fd3a13bd9d9c6b9ce67ad371) + VLLM_PRECOMPILED_WHEEL_COMMIT=183625d4f9f49968218ffbe43242877ead72d569 (wheel 0.32.0rc1.dev34+g183625d4f)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/g4bwy/qwen3.8-flash-next-cmp-170hx","pin":"dcd5ba47c17f7c952f6b8d42b9508f3b4f4a4d34 - source recipe repo @ HEAD (serve-4x-1m.sh verbatim argv/env; README, benchmark/results.md, patches/, checkpoint/, archive/ all at this commit)"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy.json","alternate":"/r/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy","describedby":"/openapi.json"},"updated_at":"2026-10-20 21:52:36.721138-06","created_at":"2026-10-20 21:49:17.914882-06","speeds":{"decode_tok_s":183.19999999999998863131622783839702606201171875,"prefill_tok_s":10038,"leaderboard":"/api/v1/leaderboard.json"}}