{"slug":"qwen3-8-flash-next-w4a16-3x-cmp-170hx-pp3-nvme-ple-fp8-kv-1m-cle","title":"Qwen3.8-Flash-Next W4A16 3x CMP 170HX PP3 NVMe PLE fp8 KV 1M (clee)","summary":"Working PP3 recipe for Qwen3.8-Flash-Next (Qwen4Exp 180B-A6B MoE, Intel INT4 W4A16 AutoRound) on 3x CMP 170HX (SM80, unlocked 64 GiB each): the 95.4 GiB BF16 PLE n-gram table streams live from NVMe (shard-direct preadv into a fixed GPU staging buffer, never fully loaded), fp8 e4m3 KV software-decoded on SM80, mamba SSM cache bf16, and 1,048,576-token context via YaRN x4 over the 262144-native checkpoint. KV pool 5.97M tokens (~5.7 concurrent 1M sessions). W4A16 experts run GPTQ-Marlin bit-exact; no GPU P2P on this card pairing, so PP stages through host RAM with NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce. Measured cold-cache temp=0: prefill 2389 tok/s @4k rising to 4690 @1M (QSA sparse crossover past ~64k), decode 73.2 -> 59.8 tok/s @1M, TTFT 223 s @1M; exact needle recall at all depths through 1M. Fork clee/vllm-sm80 on a PINNED nightly; newer nightlies not validated.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"fb3da33b3299da70eb24ce68a2c772de0dc1a356","content_hash":"0dea5f8b482a14a47e6cef76e85ad322d56f9821","created_at":"2026-10-20 15:34:25.555323-06","notes":"Submitted 2026-10-08 by localai-recipes: direct ingest order (owner, Oct 8) of github.com/clee/qwen3.8-flash-next-3x170hx \u2014 NOT a scout card. Repo @ 03458ade7a29fa22b750e273513ab16d79164243 (HEAD verified via git ls-remote 2026-10-08; 2 commits; young repo, 3 stars, owner-directed). Launch argv token-verified against run-qwen38fn.sh + configs/intel-1m-fp8-ssmbf16.yml at that sha: all serve flags live in the vLLM config file passed as `vllm serve --config /opt/stack.yml`, launcher env block transcribed verbatim (NCCL_P2P_DISABLE=1, VLLM_PP_LAYER_PARTITION=17,17,14, VLLM_PLE_STAGED=1, VLLM_ALLOW_LONG_MAX_MODEL_LEN=1, QWEN4EXP_PLE_MODEL_DIR=/models/qwen38fn-w4a16-autoround). Perf table is the repo's own bench-results-2026-10-04.md (harness tests/bench.py in-repo) carried as submitter-T0. Attribution: listed-by localai-recipes (poster); recipe author credit 'clee' via provenance.credit. Hardware profile cmp-170hx-x3 / 170hx-x3 pre-existing (the aebf24e deepseek 3x170hx ingest)."},"engine_source":{"type":"fork","git_url":"https://github.com/clee/vllm-sm80","base_engine":"vllm","pip_package":null,"base_version":"nightly-0cbac6cd1305 + clee/vllm-sm80 (q38fn/*)","image_digest":"docker.io/vllm/vllm-openai:nightly-0cbac6cd1305f710e12193596b27488397bcb205","install_note":"NOT installable from the stock image alone: the four q38fn/* patch branches in the clee/vllm-sm80 fork (units-fix, fp8-kv-sm80, ple-pp3, ple-nvme-staged) are pure-Python changes that must either be baked into an image (docker build -t vllm-sm80:q38fn . from the recipe repo's Dockerfile) or bind-mounted over the pinned nightly's vllm/ tree; the pinned image is vllm/vllm-openai:nightly-0cbac6cd1305f710e12193596b27488397bcb205 and NEWER NIGHTLIES ARE NOT VALIDATED against this stack. Fork branch pin: q38fn/base = 0cbac6cd1305.","pinned_commit":"0cbac6cd1305f710e12193596b27488397bcb205"},"credit":[{"name":"clee (recipe, fork patches, bench + needle harness)","url":"https://github.com/clee/qwen3.8-flash-next-3x170hx"},{"name":"clee (vllm-sm80 fork: units-fix, fp8-kv-sm80, ple-pp3, ple-nvme-staged)","url":"https://github.com/clee/vllm-sm80"},{"name":"Kai / 2Wild (Qwen38-Flash-Next-4x3090 \u2014 staged-PLE architecture inspiration)","url":"https://github.com/tonyd2wild/Qwen38-Flash-Next-4x3090"},{"name":"Intel (INT4 W4A16 AutoRound quant)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"froggeric (bugfixed Qwen3.8 chat templates)","url":"https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["./run-qwen38fn.sh IMAGE=vllm-sm80:q38fn"],"runtime":{"env":"NCCL_P2P_DISABLE=1 (mandatory: no GPU P2P on CMP170HX pairings across root buses) | NCCL_DEBUG=WARN | VLLM_PP_LAYER_PARTITION=17,17,14 (PLE layer idx 1 stays on rank 0) | VLLM_PLE_STAGED=1 (shard-direct preadv PLE; default in run-qwen38fn.sh; without it boot OOMs in allocate_embedding_weight) | VLLM_PLE_MMAP=0 | VLLM_PLE_ZERO=0 (1 = ablation knob that zeroes the PLE contribution) | QWEN4EXP_PLE_MODEL_DIR=/models/qwen38fn-w4a16-autoround (container path; host dir bind-mounted ro at /home/models) | VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 (1M > checkpoint's native 262144) | HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 | PYTORCH_NVML_BASED_CUDA_CHECK=1 VLLM_WORKER_MULTIPROC_METHOD=spawn (verbatim from run-qwen38fn.sh @ 03458ad)","notes":"All serve args live in configs/intel-1m-fp8-ssmbf16.yml passed as `vllm serve --config /opt/stack.yml` (run-qwen38fn.sh mounts it and execs docker run with --entrypoint vllm); the config resolves to: model /models/qwen38fn-w4a16-autoround, served-model-name qwen3.8, host 0.0.0.0 port 8000 (container) -> host :8080, distributed-executor-backend mp, tensor-parallel-size 1, pipeline-parallel-size 3, disable-custom-all-reduce true (mandatory on this hardware), dtype bfloat16, max-model-len 1048576, block-size 1632 (% 12 == 0 for the QSA ring; do NOT use 8192), max-num-seqs 8, max-num-batched-tokens 8192, gpu-memory-utilization 0.92, kv-cache-dtype fp8_e4m3 (software e4m3 decode on SM80 via the q38fn/fp8-kv-sm80 patch), enable-prefix-caching, enable-chunked-prefill, mamba-cache-mode align, mamba-ssm-cache-dtype bfloat16 (fp32 default eats the fp8 dividend), async-scheduling true, compilation-config cudagraph_mode FULL_AND_PIECEWISE, enable-flashinfer-autotune false, chat-template /opt/chat_template.jinja (froggeric's bugfixed Qwen3.8 templates, mounted by the launcher), trust-request-chat-template true, enable-auto-tool-choice true, tool-call-parser qwen3_xml, reasoning-parser qwen3, enable-prompt-tokens-details true, default-chat-template-kwargs {enable_thinking true, preserve_thinking true, reasoning_effort xhigh}, hf-overrides text_config.layer_types = [linear_attention x3, qwen_sparse_attention] x12 (transformers-5.18 bridge, canonical QSA vocabulary) and rope_parameters {rope_type yarn, factor 4.0, original_max_position_embeddings 262144, rope_theta 10000000, partial_rotary_factor 0.25, mrope_interleaved true, mrope_section [11,11,10]}. Quick start: (1) hf download Intel/Qwen3.8-Flash-Next-W4A16-AutoRound --local-dir /home/models/qwen38fn-w4a16-autoround (>=169 GiB free NVMe); (2) docker pull vllm/vllm-openai:nightly-0cbac6cd1305f710e12193596b27488397bcb205; (3) git clone https://github.com/clee/vllm-sm80 (branches q38fn/*); (4) docker build -t vllm-sm80:q38fn . && ./run-qwen38fn.sh IMAGE=vllm-sm80:q38fn. Healthy boot log check: 'Padding mamba page size by ~2%' \u2014 a large or negative percentage means the units-fix patch is not active. Validation: tests/needle.py \u2014 exact needle recall at all depths through 1M (d0.5/d0.97/d0.05) plus 17*23=391 deterministic x3, greedy/temperature=0. Troubleshooting: boot OOM in allocate_embedding_weight (cudaHostAlloc) -> VLLM_PLE_STAGED=1 missing; assert attn_page_size >= mamba_page_size -> units-fix patch not mounted; needles pass short-range but fail >=2k -> PLE rows poisoned (VLLM_PLE_ZERO=1 is the ablation knob; QSA attention still carries retrieval); TP2+ NCCL hang -> P2P, see hardware notes; fp8 boot with block 3264 -> units-fix cdiv half missing. Agent-use capability rows: W4A16 experts via GPTQ-Marlin bit-exact; 95.4 GiB BF16 PLE n-gram table streamed LIVE FROM NVMe (VLLM_PLE_STAGED=1, never fully loaded to RAM/VRAM); fp8 e4m3 KV software-decoded on SM80 (q38fn/fp8-kv-sm80); mamba SSM cache bf16 (fp32 default eats the fp8 dividend); ctx 1,048,576 via YaRN x4 (VLLM_ALLOW_LONG_MAX_MODEL_LEN=1; checkpoint native 262144); KV pool 5.97M tokens ~ 5.69 concurrent 1M sessions; froggeric fixed chat templates. Caveats: MTP speculative decoding NOT enabled (draft-layer KV would cost ~16% of the pool; fixes pending in the fork); pin image nightly-0cbac6cd... and fork branch q38fn/base \u2014 newer nightlies unvalidated."},"kv_pool_tokens":5970000,"max_context_len":1048576},"models":[{"slug":"qwen3-8-flash-next-w4a16-autoround","name":"Qwen3.8-Flash-Next-W4A16-AutoRound","hf_repo":"Intel/Qwen3.8-Flash-Next-W4A16-AutoRound","quant":"w4a16-autoround"}],"hardware":[{"slug":"nvidia-cmp-170hxx3","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":3}],"generation":"ampere","tags":["fp8-kv","long-context","mamba-hybrid","needle-validated","no-p2p","nvme-staged-ple","pipeline-parallel","sm80","yarn-1m"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured 2026-10-04 (tests/bench-results-2026-10-04.md @ 03458ad, tests/bench.py harness): cold-cache single stream temperature=0, unique random prompts so prefix caching cannot fake prefill. Prefill per-token rate IMPROVES past ~64k (QSA sparse-attention crossover: top-2048 budget, no quadratic term; cf. the Qwen tech report's 7.6x prefill claim at 1M). Decode drops 74 -> 60 tok/s at 1M: the SM80 fp8-KV software-dequant tax. 1M requests carry 809k prefill tokens in the needle suite. claimed_by=submitter T0 \u2014 no raw run file ingested, harness is in-repo; import via POST /api/v1/benchmarks to upgrade.\", \"points\": [{\"notes\": \"README bench table @ 03458ad (2026-10-04, cold-cache temp=0): TTFT 1.7 s, prefill 2389, decode 73.2\", \"ttft_ms\": 1700, \"concurrency\": 1, \"context_len\": 4096, \"decode_tok_s\": 73.2000000000000028421709430404007434844970703125, \"prefill_tok_s\": 2389}, {\"notes\": \"README bench table: TTFT 5.1 s, prefill 3210, decode 73.3\", \"ttft_ms\": 5100, \"concurrency\": 1, \"context_len\": 16384, \"decode_tok_s\": 73.2999999999999971578290569595992565155029296875, \"prefill_tok_s\": 3210}, {\"notes\": \"README bench table: TTFT 18.4 s, prefill 3563, decode 74.9\", \"ttft_ms\": 18400, \"concurrency\": 1, \"context_len\": 65536, \"decode_tok_s\": 74.900000000000005684341886080801486968994140625, \"prefill_tok_s\": 3563}, {\"notes\": \"README bench table: TTFT 78.4 s, prefill 3345, decode 61.9\", \"ttft_ms\": 78400, \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 61.89999999999999857891452847979962825775146484375, \"prefill_tok_s\": 3345}, {\"notes\": \"README bench table: TTFT 223 s, prefill 4690, decode 59.8 \u2014 1M ctx via YaRN x4\", \"ttft_ms\": 223000, \"concurrency\": 1, \"context_len\": 1048576, \"decode_tok_s\": 59.7999999999999971578290569595992565155029296875, \"prefill_tok_s\": 4690}], \"context_len\": 1048576}"}],"perf_points":[{"id":1203,"context_len":4096,"prefill_tok_s":2389,"decode_tok_s":73.2000000000000028421709430404007434844970703125,"ttft_ms":1700,"concurrency":1,"claimed_by":"submitter","notes":"README bench table @ 03458ad (2026-10-04, cold-cache temp=0): TTFT 1.7 s, prefill 2389, decode 73.2"},{"id":1204,"context_len":16384,"prefill_tok_s":3210,"decode_tok_s":73.2999999999999971578290569595992565155029296875,"ttft_ms":5100,"concurrency":1,"claimed_by":"submitter","notes":"README bench table: TTFT 5.1 s, prefill 3210, decode 73.3"},{"id":1205,"context_len":65536,"prefill_tok_s":3563,"decode_tok_s":74.900000000000005684341886080801486968994140625,"ttft_ms":18400,"concurrency":1,"claimed_by":"submitter","notes":"README bench table: TTFT 18.4 s, prefill 3563, decode 74.9"},{"id":1206,"context_len":262144,"prefill_tok_s":3345,"decode_tok_s":61.89999999999999857891452847979962825775146484375,"ttft_ms":78400,"concurrency":1,"claimed_by":"submitter","notes":"README bench table: TTFT 78.4 s, prefill 3345, decode 61.9"},{"id":1207,"context_len":1048576,"prefill_tok_s":4690,"decode_tok_s":59.7999999999999971578290569595992565155029296875,"ttft_ms":223000,"concurrency":1,"claimed_by":"submitter","notes":"README bench table: TTFT 223 s, prefill 4690, decode 59.8 \u2014 1M ctx via YaRN x4"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-3x-cmp-170hx-pp3-nvme-ple-fp8-kv-1m-cle.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-3x-cmp-170hx-pp3-nvme-ple-fp8-kv-1m-cle.json"},"requirements":null,"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/clee/qwen3.8-flash-next-3x170hx","pin":"03458ade @ HEAD 'benchmarks: cold-cache table + bench harness' \u2014 run-qwen38fn.sh + configs/intel-1m-fp8-ssmbf16.yml are the verbatim argv/env source; tests/needle.py + tests/bench.py at this commit"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/vllm/vllm-openai","pin":"vllm/vllm-openai:nightly-0cbac6cd1305f710e12193596b27488397bcb205 (base = v0.30.1rc1.dev558; PINNED \u2014 newer nightlies NOT validated against this stack)"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/clee/vllm-sm80","pin":"fork q38fn/* on q38fn/base = 0cbac6cd1305: units-fix (_align_hybrid_block_size), fp8-kv-sm80 (software decode in QSA gather), ple-pp3 (PLE under PP>1), ple-nvme-staged (preadv reader); pure Python, bind-mount or bake"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound","pin":"Intel INT4 W4A16 AutoRound quant of Qwen3.8-Flash-Next (Qwen4Exp 180B-A6B MoE); >=169 GiB free NVMe required for the local checkpoint dir"},{"kind":"reference","pointer_type":"url","pointer":"https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates","pin":"froggeric's extensively bugfixed Qwen3.8 chat templates \u2014 mounted by run-qwen38fn.sh as /opt/chat_template.jinja"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-3x-cmp-170hx-pp3-nvme-ple-fp8-kv-1m-cle.json","alternate":"/r/qwen3-8-flash-next-w4a16-3x-cmp-170hx-pp3-nvme-ple-fp8-kv-1m-cle","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:25.555323-06","created_at":"2026-10-08 17:11:47.217975-06","speeds":{"decode_tok_s":73.2000000000000028421709430404007434844970703125,"prefill_tok_s":3563,"leaderboard":"/api/v1/leaderboard.json"}}