{"slug":"qwen3-8-flash-next-w4a16-fp8ple-2x-cmp-170hx-pp2-mtp4-1m","title":"Qwen3.8-Flash-Next W4A16-FP8PLE 2x CMP 170HX PP2 MTP4 1M","summary":"Community 2x CMP 170HX recipe for Qwen3.8-Flash-Next (qwen4_exp 176B MoE, 6B active): community AutoRound W4A16 weights + the 48GB FP8 PLE n-gram table mmap'd from NVMe (never enters VRAM; hot rows in host page cache, thread-pool gather), vLLM 0.29 nightly with the repo's 3-file PLE-mmap patch set (MmapPleTable np.memmap over safetensors shards, hooks into Qwen4ExpNGramEmbedding, 5 env knobs), PP=2 (beats TP on PCIe Gen2 x4 \u2014 expert routing never leaves the GPU), BF16 KV (QSA layer rejects FP8), MTP spec=4 (hard cap: spec=5 breaks the QSA ring assert), 1,048,576-token context via YaRN factor 4.0 hf-overrides (orig 262144), cudagraph PIECEWISE. Author-measured 2026-09-01 (pp1m, C1, 1280-out cap, raw logs in bench/results/): prefill 6080 tok/s @80K down to 5480 @260K; decode C1 111.6 @80K, 62.8 @110K, 65.0 @140K, 69.8 @170K, 112.0 @200K, 61.6 @230K, 76.4 @260K (swings 62-112 with MTP acceptance 2.6-3.2 mean accepted len; step rate 31-33/s hardware-bound); TTFT 13.2s @80K / 47.4s @260K; C","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"b554e1569b5a9687fb0d0b4403e2136e4a89da23","content_hash":"c46ede747c21baba2b8b93836a3889ba31f96492","created_at":"2026-10-20 15:34:11.169336-06","notes":"Ingested from GitHub repo bluespace3/qwen38-flash-next-170hx @ 3c4c3978cd95b5d5ce341b882105a198bb75f8ba (wave 3, kanban t_21a3a083). README credits AutoRound community quantization + the vLLM project patch hooks; author states benchmarks are from one specific machine (PCIe topology matters), YMMV. Chinese deployment write-up in docs/deployment-notes.md (concurrency curves + pitfall list). License Apache-2.0. 1M ctx distinguishes this from the bozo 262144-context build already on site."},"engine_source":{"type":"fork","git_url":"https://github.com/bluespace3/qwen38-flash-next-170hx","base_engine":"vllm","pip_package":null,"base_version":"0.29 nightly (vllm==0.28.1rc1.dev93+cu133 verified)","image_digest":null,"install_note":"Stock vLLM nightly (requirements-vllm.txt: vllm>=0.28.1, verified 0.28.1rc1.dev93+cu133) + the repo's PLE-mmap patch set applied AFTER install: patches/ple_mmap.py copied to vllm/models/qwen4_exp/nvidia/, then patch -p1 < patches/0001-ple-layer-mmap-hook.patch (hooks Qwen4ExpNGramEmbedding init + load_weights to drop table tensors) and patches/0002-envs-ple-knobs.patch (registers VLLM_PLE_MMAP/WORKERS/CHUNK/PREWARM/READAHEAD in vllm/envs.py); patches/ple_layer.patched.py ships the full patched file for manual copy. No published image - source install per repo README quick start.","pinned_commit":"3c4c3978cd95b5d5ce341b882105a198bb75f8ba"},"credit":[{"name":"bluespace3","url":"https://github.com/bluespace3/qwen38-flash-next-170hx"},{"name":"AutoRound community quantization (W4A16 + FP8 PLE export)","url":"https://github.com/intel/auto-round"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["/root/vllm029-venv/bin/vllm","serve","/mnt/nvme0/llm_models/Qwen3.8-Flash-Next-W4A16-fp8ple","--pipeline-parallel-size","2","--served-model-name","qwen3.8-flash-next","Qwen3.8-27B-INT8","Qwen3.6-27B-INT8","--port","8000","--max-model-len","1048576","--max-num-seqs","8","--gpu-memory-utilization","0.97","--enable-prefix-caching","--reasoning-parser","qwen3","--enable-auto-tool-choice","--tool-call-parser","qwen3_coder","--trust-remote-code","--no-enable-flashinfer-autotune","-cc.cudagraph_mode=PIECEWISE","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":4}","--hf-overrides","{\"text_config\":{\"rope_parameters\":{\"mrope_interleaved\":true,\"mrope_section\":[11,11,10],\"rope_type\":\"yarn\",\"rope_theta\":10000000,\"partial_rotary_factor\":0.25,\"factor\":4.0,\"original_max_position_embeddings\":262144}}}"],"runtime":{"env":"CUDA_VISIBLE_DEVICES=0,1 | VLLM_PLE_MMAP=1 | VLLM_PLE_MMAP_WORKERS=32 | VLLM_PLE_MMAP_PREWARM=1 | VLLM_PLE_MMAP_READAHEAD=256 | VLLM_PLE_MMAP_CHUNK=2048 (512-2048 flat; 8192 breaks prefill 5300->2000) | VLLM_ENGINE_READY_TIMEOUT_S=5400 | VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 | VLLM_ATTENTION_BACKEND=FLASH_ATTN | TRITON_CACHE_DIR=/mnt/nvme0/triton-cache | VLLM_CACHE_ROOT=/mnt/nvme0/vllm-cache","notes":"Serve profile = scripts/start-flashnext.sh mode pp1m @ 3c4c3978 (defaults: chunk=2048 spec=4); transcribed token-for-token from the script. Launcher also ships pp (262144 ctx, util 0.90, MTP3), tp (210K legacy), vram (table-in-VRAM 6K ctx, --enforce-eager --kv-cache-dtype fp8_e4m3 --max-num-batched-tokens 4096, needs VLLM_PLE_FORCE_FP8=1) and single (1-GPU debug 32K) modes. Hard-won constraints from README 'lessons': MTP spec>=5 impossible (QSA ring must divide the 1616 attention block; capacity assert), KV must stay BF16 (fp8 KV assertion-blocked by QSA), cudagraph_mode=PIECEWISE required (full graph + PP + MTP recompiles forever), count usage.completion_tokens not SSE events (MTP emits 2-4 tokens/event), check lspci before blaming software (degraded x1 slot inflates the PP bubble). Checkpoint = community AutoRound export pattern Qwen3.8-Flash-Next-W4A16-fp8ple (17 shards; model-00016-of-00017.safetensors IS the 48GB FP8 PLE table - keep on fastest NVMe; ModelScope/HF search per docs/checkpoint.md, no single pinned repo)."},"max_context_len":1048576,"storage_requirements":{"ram_gb":60}},"models":[{"slug":"qwen3-8-flash-next-w4a16-fp8ple","name":"Qwen3.8-Flash-Next-W4A16-FP8PLE","hf_repo":"albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE","quant":"w4a16-fp8-ple"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["bf16-kv","bluespace3","long-context","mtp-speculative","nvme-mmap","pipeline-parallel","sm80","yarn-1m"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured 2026-09-01 (bench/results/qwen38flash_next_c1_0901_1333.log @ 3c4c3978, ws_bench.py C1, output cap 1280, MTP spec=4, seeds per stream to defeat prefix caching): prefill 6080 tok/s @80K, 5929 @110K, 5829 @140K, 5750 @170K, 5625 @200K, 5574 @230K, 5480 @260K; decode C1 111.6 @80K / 62.8 @110K / 65.0 @140K / 69.8 @170K / 112.0 @200K / 61.6 @230K / 76.4 @260K; TTFT 13.2s @80K - 47.4s @260K. claimed_by=submitter T0 - no run file ingested; import via POST /api/v1/benchmarks to upgrade.\", \"tok_s\": 111.599999999999994315658113919198513031005859375, \"points\": [{\"notes\": \"README bench table @ 3c4c3978 (2026-09-01, pp1m): TTFT 13.2s, prefill 6080, decode 111.6\", \"ttft_ms\": 13200, \"concurrency\": 1, \"context_len\": 80000, \"decode_tok_s\": 111.599999999999994315658113919198513031005859375, \"prefill_tok_s\": 6080}, {\"notes\": \"README bench table: TTFT 18.6s, prefill 5929, decode 62.8\", \"ttft_ms\": 18600, \"concurrency\": 1, \"context_len\": 110000, \"decode_tok_s\": 62.7999999999999971578290569595992565155029296875, \"prefill_tok_s\": 5929}, {\"notes\": \"README bench table: TTFT 24.0s, prefill 5829, decode 65.0\", \"ttft_ms\": 24000, \"concurrency\": 1, \"context_len\": 140000, \"decode_tok_s\": 65, \"prefill_tok_s\": 5829}, {\"notes\": \"README bench table: TTFT 29.6s, prefill 5750, decode 69.8\", \"ttft_ms\": 29600, \"concurrency\": 1, \"context_len\": 170000, \"decode_tok_s\": 69.7999999999999971578290569595992565155029296875, \"prefill_tok_s\": 5750}, {\"notes\": \"README bench table: TTFT 35.6s, prefill 5625, decode 112.0\", \"ttft_ms\": 35600, \"concurrency\": 1, \"context_len\": 200000, \"decode_tok_s\": 112, \"prefill_tok_s\": 5625}, {\"notes\": \"README bench table: TTFT 41.3s, prefill 5574, decode 61.6\", \"ttft_ms\": 41300, \"concurrency\": 1, \"context_len\": 230000, \"decode_tok_s\": 61.60000000000000142108547152020037174224853515625, \"prefill_tok_s\": 5574}, {\"notes\": \"README bench table: TTFT 47.4s, prefill 5480, decode 76.4\", \"ttft_ms\": 47400, \"concurrency\": 1, \"context_len\": 260000, \"decode_tok_s\": 76.400000000000005684341886080801486968994140625, \"prefill_tok_s\": 5480}], \"context_len\": 262144}"}],"perf_points":[{"id":1043,"context_len":80000,"prefill_tok_s":6080,"decode_tok_s":111.599999999999994315658113919198513031005859375,"ttft_ms":13200,"concurrency":1,"claimed_by":"submitter","notes":"README bench table @ 3c4c3978 (2026-09-01, pp1m): TTFT 13.2s, prefill 6080, decode 111.6"},{"id":1044,"context_len":110000,"prefill_tok_s":5929,"decode_tok_s":62.7999999999999971578290569595992565155029296875,"ttft_ms":18600,"concurrency":1,"claimed_by":"submitter","notes":"README bench table: TTFT 18.6s, prefill 5929, decode 62.8"},{"id":1045,"context_len":140000,"prefill_tok_s":5829,"decode_tok_s":65,"ttft_ms":24000,"concurrency":1,"claimed_by":"submitter","notes":"README bench table: TTFT 24.0s, prefill 5829, decode 65.0"},{"id":1046,"context_len":170000,"prefill_tok_s":5750,"decode_tok_s":69.7999999999999971578290569595992565155029296875,"ttft_ms":29600,"concurrency":1,"claimed_by":"submitter","notes":"README bench table: TTFT 29.6s, prefill 5750, decode 69.8"},{"id":1047,"context_len":200000,"prefill_tok_s":5625,"decode_tok_s":112,"ttft_ms":35600,"concurrency":1,"claimed_by":"submitter","notes":"README bench table: TTFT 35.6s, prefill 5625, decode 112.0"},{"id":1048,"context_len":230000,"prefill_tok_s":5574,"decode_tok_s":61.60000000000000142108547152020037174224853515625,"ttft_ms":41300,"concurrency":1,"claimed_by":"submitter","notes":"README bench table: TTFT 41.3s, prefill 5574, decode 61.6"},{"id":1049,"context_len":260000,"prefill_tok_s":5480,"decode_tok_s":76.400000000000005684341886080801486968994140625,"ttft_ms":47400,"concurrency":1,"claimed_by":"submitter","notes":"README bench table: TTFT 47.4s, prefill 5480, decode 76.4"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-2x-cmp-170hx-pp2-mtp4-1m.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-2x-cmp-170hx-pp2-mtp4-1m.json"},"requirements":null,"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/bluespace3/qwen38-flash-next-170hx","pin":"3c4c3978cd95b5d5ce341b882105a198bb75f8ba - source recipe repo @ HEAD (launcher scripts/start-flashnext.sh, patches/ple_mmap.py + 0001/0002 patches + ple_layer.patched.py, bench/ws_bench.py + raw logs, docs/checkpoint.md + deployment-notes.md + feasibility-"},{"kind":"model_weights","pointer_type":"url","pointer":"https://www.modelscope.cn/models?search=Qwen3.8-Flash-Next-W4A16-fp8ple","pin":"community AutoRound export, name pattern Qwen3.8-Flash-Next-W4A16-fp8ple (ModelScope/HF search per docs/checkpoint.md @ 3c4c3978) - 17 safetensors shards ~48GB total, shard model-00016-of-00017.safetensors IS the 48GB FP8 PLE n-gram table (mmap layer reads"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-2x-cmp-170hx-pp2-mtp4-1m.json","alternate":"/r/qwen3-8-flash-next-w4a16-fp8ple-2x-cmp-170hx-pp2-mtp4-1m","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:11.169336-06","created_at":"2026-10-07 11:30:07.173235-06","speeds":{"decode_tok_s":69.7999999999999971578290569595992565155029296875,"prefill_tok_s":5750,"leaderboard":"/api/v1/leaderboard.json"}}