Qwen3.8-Flash-Next W4A16-FP8PLE 2x CMP 170HX PP2 MTP4 1M

community Community submission — verify on your own hardware.

recipe by bluespace3↗, AutoRound community quantization (W4A16 + FP8 PLE export)↗ · listed by localai-recipes

revision r2 · commit b554e1569b · updated 10 d ago · license CC-BY-4.0

chat code pipeline-parallel mtp-speculative long-context bf16-kv bluespace3 nvme-mmap yarn-1m sm80

Community 2x CMP 170HX recipe for Qwen3.8-Flash-Next (qwen4_exp 176B MoE, 6B active): community AutoRound W4A16 weights + the 48GB FP8 PLE n-gram table mmap'd from NVMe (never enters VRAM; hot rows in host page cache, thread-pool gather), vLLM 0.29 nightly with the repo's 3-file PLE-mmap patch set (MmapPleTable np.memmap over safetensors shards, hooks into Qwen4ExpNGramEmbedding, 5 env knobs), PP=2 (beats TP on PCIe Gen2 x4 — expert routing never leaves the GPU), BF16 KV (QSA layer rejects FP8), MTP spec=4 (hard cap: spec=5 breaks the QSA ring assert), 1,048,576-token context via YaRN factor 4.0 hf-overrides (orig 262144), cudagraph PIECEWISE. Author-measured 2026-09-01 (pp1m, C1, 1280-out cap, raw logs in bench/results/): prefill 6080 tok/s @80K down to 5480 @260K; decode C1 111.6 @80K, 62.8 @110K, 65.0 @140K, 69.8 @170K, 112.0 @200K, 61.6 @230K, 76.4 @260K (swings 62-112 with MTP acceptance 2.6-3.2 mean accepted len; step rate 31-33/s hardware-bound); TTFT 13.2s @80K / 47.4s @260K; C

Launch command

/root/vllm029-venv/bin/vllm serve /mnt/nvme0/llm_models/Qwen3.8-Flash-Next-W4A16-fp8ple --pipeline-parallel-size 2 --served-model-name qwen3.8-flash-next Qwen3.8-27B-INT8 Qwen3.6-27B-INT8 --port 8000 --max-model-len 1048576 --max-num-seqs 8 --gpu-memory-utilization 0.97 --enable-prefix-caching --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --trust-remote-code --no-enable-flashinfer-autotune -cc.cudagraph_mode=PIECEWISE --speculative-config '{"method":"mtp","num_speculative_tokens":4}' --hf-overrides '{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144}}}'

runtime

env: 'CUDA_VISIBLE_DEVICES=0,1 | VLLM_PLE_MMAP=1 | VLLM_PLE_MMAP_WORKERS=32 | VLLM_PLE_MMAP_PREWARM=1 | VLLM_PLE_MMAP_READAHEAD=256 | VLLM_PLE_MMAP_CHUNK=2048 (512-2048 flat; 8192 breaks prefill 5300->2000) | VLLM_ENGINE_READY_TIMEOUT_S=5400 | VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 | VLLM_ATTENTION_BACKEND=FLASH_ATTN | TRITON_CACHE_DIR=/mnt/nvme0/triton-cache | VLLM_CACHE_ROOT=/mnt/nvme0/vllm-cache'
notes: "Serve profile = scripts/start-flashnext.sh mode pp1m @ 3c4c3978 (defaults: chunk=2048 spec=4); transcribed token-for-token from the script. Launcher also ships pp (262144 ctx, util 0.90, MTP3), tp (210K legacy), vram (table-in-VRAM 6K ctx, --enforce-eager --kv-cache-dtype fp8_e4m3 --max-num-batched-tokens 4096, needs VLLM_PLE_FORCE_FP8=1) and single (1-GPU debug 32K) modes. Hard-won constraints from README 'lessons': MTP spec>=5 impossible (QSA ring must divide the 1616 attention block; capacity assert), KV must stay BF16 (fp8 KV assertion-blocked by QSA), cudagraph_mode=PIECEWISE required (full graph + PP + MTP recompiles forever), count usage.completion_tokens not SSE events (MTP emits 2-4 tokens/event), check lspci before blaming software (degraded x1 slot inflates the PP bubble). Checkpoint = community AutoRound export pattern Qwen3.8-Flash-Next-W4A16-fp8ple (17 shards; model-00016-of-00017.safetensors IS the 48GB FP8 PLE table - keep on fastest NVMe; ModelScope/HF search per docs/checkpoint.md, no single pinned repo)."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-2x-cmp-170hx-pp2-mtp4-1m.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-2x-cmp-170hx-pp2-mtp4-1m.json

Engine

vllm · source↗ · 3c4c3978↗

Install note: Stock vLLM nightly (requirements-vllm.txt: vllm>=0.28.1, verified 0.28.1rc1.dev93+cu133) + the repo's PLE-mmap patch set applied AFTER install: patches/ple_mmap.py copied to vllm/models/qwen4_exp/nvidia/, then patch -p1 < patches/0001-ple-layer-mmap-hook.patch (hooks Qwen4ExpNGramEmbedding init + load_weights to drop table tensors) and patches/0002-envs-ple-knobs.patch (registers VLLM_PLE_MMAP/WORKERS/CHUNK/PREWARM/READAHEAD in vllm/envs.py); patches/ple_layer.patched.py ships the full patched file for manual copy. No published image - source install per repo README quick start.

Model

Qwen3.8-Flash-Next-W4A16-FP8PLE (w4a16-fp8-ple) · albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE↗
AA intelligence 40 (Artificial Analysis · measured)

Hardware

NVIDIA CMP 170HX 64GB ×2 (Ampere)

Context

max context: 1M tokens

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
80,000	1	6080 tok/s	111.6 tok/s	13200 ms	reported
110,000	1	5929 tok/s	62.8 tok/s	18600 ms	reported
140,000	1	5829 tok/s	65 tok/s	24000 ms	reported
170,000	1	5750 tok/s	69.8 tok/s	29600 ms	reported
200,000	1	5625 tok/s	112 tok/s	35600 ms	reported
230,000	1	5574 tok/s	61.6 tok/s	41300 ms	reported
260,000	1	5480 tok/s	76.4 tok/s	47400 ms	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Author-measured 2026-09-01 (bench/results/qwen38flash_next_c1_0901_1333.log @ 3c4c3978, ws_bench.py C1, output cap 1280, MTP spec=4, seeds per stream to defeat prefix caching): prefill 6080 tok/s @80K, 5929 @110K, 5829 @140K, 5750 @170K, 5625 @200K, 5574 @230K, 5480 @260K; decode C1 111.6 @80K / 62.8 @110K / 65.0 @140K / 69.8 @170K / 112.0 @200K / 61.6 @230K / 76.4 @260K; TTFT 13.2s @80K - 47.4s @260K. claimed_by=submitter T0 - no run file ingested; import via POST /api/v1/benchmarks to upgrade.

notes

Author-measured 2026-09-01 (bench/results/qwen38flash_next_c1_0901_1333.log @ 3c4c3978, ws_bench.py C1, output cap 1280, MTP spec=4, seeds per stream to defeat prefix caching): prefill 6080 tok/s @80K, 5929 @110K, 5829 @140K, 5750 @170K, 5625 @200K, 5574 @230K, 5480 @260K; decode C1 111.6 @80K / 62.8 @110K / 65.0 @140K / 69.8 @170K / 112.0 @200K / 61.6 @230K / 76.4 @260K; TTFT 13.2s @80K - 47.4s @260K. claimed_by=submitter T0 - no run file ingested; import via POST /api/v1/benchmarks to upgrade.

Downloads

Reference https://github.com/bluespace3/qwen38-flash-next-170hx
3c4c3978cd95b5d5ce341b882105a198bb75f8ba - source recipe repo @ HEAD (launcher scripts/start-flashnext.sh, patches/ple_mmap.py + 0001/0002 patches + ple_layer.patched.py, bench/ws_bench.py + raw logs, docs/checkpoint.md + deployment-notes.md + feasibility-
Weights https://www.modelscope.cn/models?search=Qwen3.8-Flash-Next-W4A16-fp8ple
community AutoRound export, name pattern Qwen3.8-Flash-Next-W4A16-fp8ple (ModelScope/HF search per docs/checkpoint.md @ 3c4c3978) - 17 safetensors shards ~48GB total, shard model-00016-of-00017.safetensors IS the 48GB FP8 PLE n-gram table (mmap layer reads

Revision notes

Ingested from GitHub repo bluespace3/qwen38-flash-next-170hx @ 3c4c3978cd95b5d5ce341b882105a198bb75f8ba (wave 3, kanban t_21a3a083). README credits AutoRound community quantization + the vLLM project patch hooks; author states benchmarks are from one specific machine (PCIe topology matters), YMMV. Chinese deployment write-up in docs/deployment-notes.md (concurrency curves + pitfall list). License Apache-2.0. 1M ctx distinguishes this from the bozo 262144-context build already on site.

Revision history

6dbc01bd3c recipe: update qwen3-8-flash-next-w4a16-fp8ple-2x-cmp-170hx-pp2-mtp4-1m (r2: /max_context_len, /storage_requirements) 10 d ago diff
0fcfa13e8b recipe: create qwen3-8-flash-next-w4a16-fp8ple-2x-cmp-170hx-pp2-mtp4-1m 23 d ago

Fork

Export

recipe.yaml (raw)