{"slug":"qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes","title":"Qwen3.8-Flash-Next W4A16 2x CMP 170HX PP2 MTP5 PLE (AlexBrandes)","summary":"Recipe by AlexBrandes. Qwen3.8-Flash-Next (Intel/Qwen3.8-Flash-Next-W4A16-AutoRound @ 4c67bf68) on two NVIDIA CMP 170HX cards in PIPELINE parallel (PP2, layer split 26/22) with a vLLM v0.30.0 fork (AlexBrandes/vllm @ 02acf67c, 6 commits) in a digest-pinned container. Native MTP with 5 draft tokens (lm_head quantized to int4 by tools/quantize_lm_head.py, sha256 pinned), reasoning + tool calls, full 262,144-token window. The ~102 GB n-gram (PLE) table stays on NVMe read through mmap (VLLM_PLE_MMAP=1); BF16 KV, prefix caching, FP32 Mamba state. FIRST 2x170HX recipe with NO P2P or link modification needed - PP sends one activation per step, so it runs on stock PCIe Gen2 x4 links (all sibling rows need the unlock/P2P-mod prerequisite). Measured 2026-10-17 with sparkDash (3e6d4ef): 226 tok/s single-user code, 840 tok/s 8 users, 5,885-7,104 tok/s prefill 4k-128k; HumanEval 157/164, GSM8K 1,271/1,319, needle 12/12.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"44b6d37a739c1ee1ea463ffb4775b87382ef4c39","content_hash":"035586de62fecf1d4dec99de529bd4f53e8a3aa5","created_at":"2026-10-27 20:30:16.824026-06","notes":"Direct ingest of a 2026-10-17 GitHub-search find (lar-scout card t_67dee659). Recipe repo pinned at HEAD 3f0d60096b3548a5f5449ebca7ab4b0f353bc772 (verified via git ls-remote at ingest; single commit). Engine fork branch cmp170hx-qwen38-flash-next @ 02acf67c verified live (branch tip == pin at ingest). All launch argv transcribed from start.sh at that sha and cross-checked against results/2026-10-17.json launch.args (identical). Checkpoint revision 4c67bf68 + draft-head sha256 5c5461d7... resolved from versions.env as the card instructed. All perf numbers author-reported (T0 submitter) from results/2026-10-17.json three-run medians; no independent re-run. Distinctives vs the site's other 2x170HX QFN rows: pipeline parallel on STOCK PCIe links (no P2P/capacitor mod needed - clee PP3, Morrowmake TP2, Minachist PP2 and mannydelrio PP2 all require link/unlock work), NVMe-mmapped PLE via VLLM_PLE_MMAP (vs staged preadv / shard-direct variants), MTP-5 (siblings use 3-4), BF16 KV + FP32 Mamba state, 262k window (siblings stretch to 384k-1M). Host-tuning caveat: start.sh needs sudo (power limit + readahead + cache drop); HOST_TUNE=0 skips it at ~10% decode cost."},"engine_source":{"type":"fork","git_url":"https://github.com/AlexBrandes/vllm/tree/cmp170hx-qwen38-flash-next","base_engine":"vllm","pip_package":null,"base_version":"vLLM v0.30.0 fork (02acf67c)","image_digest":"https://ghcr.io/alexbrandes/cmp170hx-qwen38-flash-next@sha256:d6fc872d943b3e9e5cc9eb583b51d72af200e039fbbb7401700bb38d665f0957","install_note":"NOT a pip install: pull ghcr.io/alexbrandes/cmp170hx-qwen38-flash-next@sha256:d6fc872d...f0957 (official vllm/vllm-openai:v0.30.0 image + the fork's 18 changed Python files; docker/build.sh rebuilds from AlexBrandes/vllm @ 02acf67c). One command from a fresh clone: ./download.sh (~170 GB checkpoint at pinned revision + int4 MTP draft head built locally, sha256 pinned) then ./start.sh (runs host-tune.sh through sudo: 170 W power limit, 16 KB readahead on the model drive, drop caches + compact; HOST_TUNE=0 skips). Waits for /health.","pinned_commit":"02acf67ca10bb0c6efddd5454facb3bfe5dba7d9"},"credit":[{"name":"AlexBrandes (recipe, fork, image, measurements)","url":"https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next"},{"name":"AlexBrandes (vllm fork: PP + NVMe-mmapped PLE, Mamba PP state, draft lm_head, graph-safe PLE, bounded prefill)","url":"https://github.com/AlexBrandes/vllm/tree/cmp170hx-qwen38-flash-next"},{"name":"polyuij42-del (community CMP 170HX W4A16 work the fork builds on)","url":"https://github.com/polyuij42-del/qwen38-flash-next-w4a16-cmp170hx"},{"name":"nguyenthimy2022kg-alt (community SM80 170HX work the fork builds on)","url":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX"},{"name":"MiaAI-Lab (sparkDash benchmark protocol @ 3e6d4ef)","url":"https://github.com/MiaAI-Lab/sparkDash"},{"name":"Intel (INT4 W4A16 AutoRound quant)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["./download.sh","./start.sh","# ./start.sh (HOST_TUNE=1 default) runs host-tune.sh via sudo (170 W, 16 KB readahead, drop caches), then execs:","docker run -d --name cmp170hx-qwen38-flash-next --gpus \"device=0,1\" --shm-size 1g -p 127.0.0.1:8000:8000","-e VLLM_USE_BREAKABLE_CUDAGRAPH=1 -e VLLM_MTP_DRAFT_HEAD_PATH=/drafthead/head-w4a16.safetensors","-e VLLM_PLE_MMAP=1 -e VLLM_PLE_MMAP_RANDOM=1 -e VLLM_PLE_HOST_IDS=1 -e VLLM_MIXED_PREFILL_CHUNK=560","-e FLASHINFER_DISABLE_VERSION_CHECK=1 -e VLLM_PP_LAYER_PARTITION=26,22","-v ./models/Qwen3.8-Flash-Next-W4A16-AutoRound:/models/Qwen3.8-Flash-Next-W4A16-AutoRound:ro","-v ./models/draft-head-w4a16.safetensors:/drafthead/head-w4a16.safetensors:ro","ghcr.io/alexbrandes/cmp170hx-qwen38-flash-next@sha256:d6fc872d943b3e9e5cc9eb583b51d72af200e039fbbb7401700bb38d665f0957","vllm serve /models/Qwen3.8-Flash-Next-W4A16-AutoRound","--pipeline-parallel-size 2","--served-model-name qwen3.8-flash-next-w4a16","--max-model-len 262144","--block-size 1680","--mamba-cache-mode align","--mamba-ssm-cache-dtype float32","--max-num-seqs 8","--max-num-batched-tokens 2048","--gpu-memory-utilization 0.95","--max-parallel-loading-workers 1","--moe-backend auto","--enable-prefix-caching","--enable-prompt-tokens-details","--reasoning-parser qwen3","--enable-auto-tool-choice","--tool-call-parser qwen3_coder","--trust-remote-code","--no-enable-flashinfer-autotune","-cc {\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"splitting_ops\":[\"vllm::unified_attention_with_output\",\"vllm::mamba_mixer2\",\"vllm::mamba_mixer\",\"vllm::short_conv\",\"vllm::qwen4_exp_ple_short_conv\",\"vllm::qwen4_exp_qsa_with_output\",\"vllm::linear_attention\",\"vllm::qwen_gdn_attention_core\",\"vllm::ple_mmap_lookup\"]}","--speculative-config {\"method\":\"mtp\",\"num_speculative_tokens\":5}"],"layouts":["pp"],"max_context_len":262144,"storage_requirements":{"note":"~180 GB checkpoint + PLE table on NVMe (PLE read from it every forward pass); page cache over the PLE table (measured with 93 GB)","ram_gb":93,"nvme_gb":180}},"models":[{"slug":"qwen3-8-flash-next-w4a16-autoround","name":"Qwen3.8-Flash-Next-W4A16-AutoRound","hf_repo":"Intel/Qwen3.8-Flash-Next-W4A16-AutoRound","quant":"w4a16-autoround"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["long-context","mtp5","mtp-speculative","no-p2p","nvme-mmap","nvme-ple","pipeline-parallel","prefix-caching","sm80","sparkdash"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured 2026-10-17 (results/2026-10-17.json in-repo; sparkDash @ 3e6d4ef, each cell median of 3 runs, range in README): decode code 226.2 / json 198.0 / prose 157.4 / structured 250.8 tok/s @ c1; 8-user aggregate code 840.4 / json 873.2 / prose 566.8 / structured 999.5. Cold prefill 6,295 @4k to 7,104 @32k, 6,901 @64k, 5,885 @128k tok/s. Decode-at-depth 157.9 @4k to 141.8 @128k. Greedy, thinking off, output forced to 400 tokens, TTFT 67-231 ms @c1-c8. Quality: HumanEval 157/164, GSM8K 1,271/1,319, needle 12/12 @8k/32k/128k, 262,144-token window boundary verified. Chat-only model (raw completions EOS immediately).\", \"tok_s\": 226.19999999999998863131622783839702606201171875, \"context_len\": 262144}"}],"perf_points":[{"id":1878,"context_len":4096,"prefill_tok_s":6295,"decode_tok_s":null,"ttft_ms":740,"concurrency":null,"claimed_by":"submitter","notes":"cold single request, median of 3; prompt 4,133 tokens"},{"id":1879,"context_len":16384,"prefill_tok_s":7096,"decode_tok_s":null,"ttft_ms":2400,"concurrency":null,"claimed_by":"submitter","notes":"cold single request; 8k 6,893, 32k 7,104, 64k 6,901, 128k 5,885 tok/s in README table"},{"id":1880,"context_len":131072,"prefill_tok_s":null,"decode_tok_s":141.80000000000001136868377216160297393798828125,"ttft_ms":21430,"concurrency":null,"claimed_by":"submitter","notes":"decode-at-depth, single request prose prompt + filler, 400 tokens out; 4k 157.9, 8k 150.1, 16k 152.5, 32k 148.4, 64k 147.1"},{"id":1874,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":226.19999999999998863131622783839702606201171875,"ttft_ms":80,"concurrency":1,"claimed_by":"submitter","notes":"code prompts, median of 3 (223.3-228.0); json 198.0, prose 157.4, structured 250.8"},{"id":1875,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":362.8999999999999772626324556767940521240234375,"ttft_ms":109,"concurrency":2,"claimed_by":"submitter","notes":"code aggregate (359.7-369.4); json 320.3, prose 212.2, structured 406.1"},{"id":1876,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":560.200000000000045474735088646411895751953125,"ttft_ms":171,"concurrency":4,"claimed_by":"submitter","notes":"code aggregate (547.8-590.1); json 536.7, prose 384.8, structured 697.4"},{"id":1877,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":840.3999999999999772626324556767940521240234375,"ttft_ms":231,"concurrency":8,"claimed_by":"submitter","notes":"code aggregate (760.3-883.0); json 873.2, prose 566.8, structured 999.5"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound","pin":"4c67bf686b7f7fd386bae6b07ab59e8ff1d5b897"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://ghcr.io/alexbrandes/cmp170hx-qwen38-flash-next@sha256:d6fc872d943b3e9e5cc9eb583b51d72af200e039fbbb7401700bb38d665f0957","pin":"engine-02acf67ca1"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next","pin":"3f0d60096b3548a5f5449ebca7ab4b0f353bc772"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes.json","alternate":"/r/qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes","describedby":"/openapi.json"},"updated_at":"2026-10-27 20:30:16.824026-06","created_at":"2026-10-20 21:49:18.79508-06","speeds":{"decode_tok_s":226.19999999999998863131622783839702606201171875,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}