{"slug":"qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple","title":"Qwen3.8-Flash-Next W4A16 2x CMP 170HX TP2 DFlash2 host-RAM PLE","summary":"Qwen3.8-Flash-Next (125B MoE) INT4-expert W4A16 on two CMP 170HX cards, full 262,144-token context: 241/350/198 tok/s one user (structured/code/prose), 495 tok/s editing code, 895/987/770 across eight, up to 128 requests at once totaling 1,937 tok/s (DFlash 2; 1,966 MTP), 4,696 tok/s cold prefill, 1,475,836-token KV pool. Serves halt95's W4A16-Merlin checkpoint on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) with the 51 GB FP8 n-gram table in HOST RAM (cpu_offload; PLE_SHARED_DIR shares one copy across servers), the whole decode step one CUDA graph - built for a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3/2 schedule, 44% faster than MTP on code c1, 83% editing code) via engine-file patches plus RecoverSSM one-state GDN; optional KV offload restores an evicted 100k-token context in 0.89 s (21 s without). Measured 2026-10-18/23 from a fresh clone under Morrowmake's protocol (temp 0, 400-token pin, median-of-5 c1 / median-of-3 aggregates).","license":"MIT","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"d558ee307e9364af70f6a3c8317aa1db958be441","content_hash":"da435f81c05c69636771a5f32003ae1c413e0bf2","created_at":"2026-10-20 21:28:52.373948-06","notes":"Community ingest of ElectroHobo's recipe (MIT), made to the shape of Morrowmake's community-recipes guidelines, which state Morrowmake has not tested/endorsed it and neither has halt95. All perf numbers are the author's own measurements (claimed_by=submitter T0, bench harness + raw result files in-repo bench/results/release-0.1.0/) - import via POST /api/v1/benchmarks to upgrade. Measured on x16-capacitor-modded cards with P2P verified; PCIe-gen2-x8 no-P2P rigs will land near the club-3090 henrykrinkle01 numbers instead. Title shortened to land the slug inside the 64-char cut; ElectroHobo credited in provenance. Verified against repo HEAD ab437e0b18a8 (tag v0.1.0) at ingest 2026-10-21: all pins, env defaults and numbers match the README exactly. Sibling recipes stay: alexbrandes PP2-mtp5 and moriyasujapan TP2+EP mtp4-fp8 - different engine paths, checkpoints and drafter."},"engine_source":{"type":"docker_image","git_url":"https://github.com/Morrowmake/vllm-cmp170hx","base_engine":"vllm","pip_package":null,"base_version":"fork @ 18305948acba (tag qwen38-recipe-1.0.0)","image_digest":"ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79","install_note":"One command from fresh clone after one-time host setup: git clone https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe && cd qwen38-flash-next-w4a16-cmp170hx-recipe && ./start.sh (preflight checks -> engine image pull -> checkpoint download at pinned HF revision -> detached launch -> /health wait -> two-request boot check); docker compose up is the equivalent. First start compiles kernels (~17 min), later starts ~5 min. DFLASH mode bind-mounts two engine files from patches/ over the image's (drafter capture hook + speculation-method check, 48 lines, over /opt/vllm-src/vllm/models/qwen4_exp/nvidia/model.py and config.py); SPEC_MODE=mtp runs the checkpoint's MTP head unmodified. Host: 2x CMP 170HX 64 GiB PCIe x16 (capacitor mod), P2P verified at start, driver >=580 (measured 610.57.04), 96 GB host RAM MINIMUM (51 GB is the n-gram table; measured on 204 GiB), ~130 GiB disk.","pinned_commit":"18305948acba3f4c4d912cdd56f2202d2ebd94e8"},"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"halt95 (W4A16-Merlin checkpoint + companion 3090 engine; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound INT4 experts) + RadixArk (FP8 n-gram table) + DominikBucko (MTP-head packing)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/MiaAI-Lab"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["git clone https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe && cd qwen38-flash-next-w4a16-cmp170hx-recipe","./start.sh"],"layouts":["tp"],"runtime":{"env":"Defaults in .env: CUDA_VISIBLE_DEVICES=0,1 | HOST=127.0.0.1 PORT=8000 (API_KEY optional) | SERVED_MODEL_NAME=qwen3.8-flash-next | KV_OFFLOAD_GB=0 (64 held ~2.2M evicted tokens; KV_OFFLOAD_DIR adds a disk tier) | GRAPH_MODE=FULL_AND_PIECEWISE | P2P=auto | SPEC_MODE=dflash (dflash|mtp|none; none needs BOOT_CHECK=0) | SPEC_N=7 (dflash) / 3 (mtp) | DRAFTER_REPO=incoai/Qwen3.8-Flash-Next-DFlash2 @ a22a1744 (Apache-2.0, 1.1 GiB) | SPEC_SCHEDULE=[[1,2,7],[3,4,5],[5,8,3]] by request count (SPEC_SCHEDULE= for fixed depth) | MAX_SEQS=8 | MAX_BATCHED=3460 | parsers qwen3/qwen3_coder | BOOT_CHECK=1 | READY_TIMEOUT=1800 | engine env with kill switches: VLLM_QWEN4_QSA_TOPK_CANONICAL=1, VLLM_CUSTOM_ALLREDUCE_FLAGS=1, VLLM_THIN_GEMM=1 (+_GLUON=1), VLLM_FLA_AUTOTUNE_SKIP_SPILLING=1, VLLM_USE_BREAKABLE_CUDAGRAPH=1, VLLM_CUSTOM_ALLREDUCE_PCIE_P2P=1 + NCCL_P2P_LEVEL=SYS after the P2P check, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, CUDA_DEVICE_ORDER=PCI_BUS_ID | context 262144 + util 0.95 FIXED (serve.sh refuses others); FIRST_LIGHT=1 = 32K diagnostic; DRY=1 ./start.sh prints the exact argv","notes":"API on 127.0.0.1:8000 as qwen3.8-flash-next, no key by default; reasoning in message.reasoning, answer in content; thinking off via chat_template_kwargs.enable_thinking=false. Exact vLLM argv (DRY=1): --tensor-parallel-size 2 --pipeline-parallel-size 1 --enable-expert-parallel --dtype bfloat16 --kv-cache-dtype bfloat16 --load-format auto --mm-encoder-tp-mode data --attention-config {'indexer_kv_dtype':'bf16'} --engram-config {'cpu_offload':true} --max-model-len 262144 --gpu-memory-utilization 0.95 --max-num-seqs 8 --max-num-batched-tokens 3460 --long-prefill-token-threshold 0 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --prefix-match-unit 16 --block-size 1680 --compilation-config {'cudagraph_mode':'FULL_AND_PIECEWISE'} --speculative-config {'method':'dflash','model':'/drafter','num_speculative_tokens':7,'num_speculative_tokens_per_batch_size':[[1,2,7],[3,4,5],[5,8,3]]}; KV_OFFLOAD_GB adds --kv-transfer-config (OffloadingConnector/TieringOffloadingSpec) + disable_eagle_block_drop:true (vLLM #52735)."},"kv_pool_tokens":1398101,"max_context_len":262144},"models":[{"slug":"qwen3-8-flash-next-w4a16-merlin","name":"Qwen3.8-Flash-Next-W4A16-Merlin","hf_repo":"halt95/Qwen3.8-Flash-Next-W4A16-Merlin","quant":"w4a16-int4-experts-bf16-kv"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["community-recipe","cuda-graphs","dflash","host-ram-ple","morrowmake-fork","speculative","tp2","w4a16"],"tasks":["chat","code","vision"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"claimed_by=submitter T0 (author's own bench, Morrowmake protocol, measured 2026-10-19 from a fresh clone; 74 SMs/card, P2P on, 180 W/card, PCIe Gen2 x16, KV_OFFLOAD_GB=64, dual Xeon E5-2697A v4 Broadwell / 204 GiB). Single-user rows are median of 5, aggregates median of 3; context_len markers +10240/+20480 = code/prose rows. Quality (thinking on, reasoning_effort medium, temp 0.6, 16k budget): HumanEval 163/164, GSM8K 197/200. KV offload: 100k ctx evicted by 1.8M tokens of other traffic restored in 0.89 s (21-22 s without); 18.7k-token conversation second turn 0.30 s (first 4.0 s); NVMe tier behind 16 GiB RAM tier 1.85 s; 100k prompt read 21.9 s, decode after 177 tok/s. Comparators (their README columns, not re-run): Morrowmake EXL3 152.8/152.6/118.3 c1 and 588.2/674.7/489.9 c8, 2651 prefill; AlexBrandes 250.8 c1 / 999.5 c8 / 6300-7100 prefill; mannydelrio 192.4 c1 / 986.4 c8 / 7006 prefill.\", \"tok_s\": 253.19999999999998863131622783839702606201171875, \"points\": [{\"notes\": \"structured, median of 5\", \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 253.19999999999998863131622783839702606201171875}, {\"notes\": \"code; context_len is the tuple marker 262144+10240 (code row, not structured/prose)\", \"concurrency\": 1, \"context_len\": 272384, \"decode_tok_s\": 348.8999999999999772626324556767940521240234375}, {\"notes\": \"prose; context_len is the tuple marker 262144+20480 (prose row, not structured/code)\", \"concurrency\": 1, \"context_len\": 282624, \"decode_tok_s\": 203.19999999999998863131622783839702606201171875}, {\"notes\": \"aggregate structured, median of 3 (c2: 381.1, c4: 515.2)\", \"concurrency\": 8, \"context_len\": 262144, \"decode_tok_s\": 842.200000000000045474735088646411895751953125}, {\"notes\": \"aggregate code; context_len is the tuple marker 262144+10240 (c2: 483.8, c4: 688.6)\", \"concurrency\": 8, \"context_len\": 272384, \"decode_tok_s\": 992.1000000000000227373675443232059478759765625}, {\"notes\": \"aggregate prose; context_len is the tuple marker 262144+20480 (c2: 298.1, c4: 455.8)\", \"concurrency\": 8, \"context_len\": 282624, \"decode_tok_s\": 726.700000000000045474735088646411895751953125}, {\"notes\": \"cold prefill median, 21.9k/31.2k/34.9k-token prompts, no cache hits\", \"context_len\": 10, \"prefill_tok_s\": 4729}], \"context_len\": 262144}"}],"perf_points":[{"id":1235,"context_len":10,"prefill_tok_s":4729,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill median, 21.9k/31.2k/34.9k-token prompts, no cache hits"},{"id":1229,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":253.19999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"structured, median of 5"},{"id":1232,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":842.200000000000045474735088646411895751953125,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate structured, median of 3 (c2: 381.1, c4: 515.2)"},{"id":1230,"context_len":272384,"prefill_tok_s":null,"decode_tok_s":348.8999999999999772626324556767940521240234375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"code; context_len is the tuple marker 262144+10240 (code row, not structured/prose)"},{"id":1233,"context_len":272384,"prefill_tok_s":null,"decode_tok_s":992.1000000000000227373675443232059478759765625,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate code; context_len is the tuple marker 262144+10240 (c2: 483.8, c4: 688.6)"},{"id":1231,"context_len":282624,"prefill_tok_s":null,"decode_tok_s":203.19999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"prose; context_len is the tuple marker 262144+20480 (prose row, not structured/code)"},{"id":1234,"context_len":282624,"prefill_tok_s":null,"decode_tok_s":726.700000000000045474735088646411895751953125,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate prose; context_len is the tuple marker 262144+20480 (c2: 298.1, c4: 455.8)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin","pin":"cec53e9f90e5995b81691604ec0b32191debf797 - the revision the release was validated with (weights unchanged since 2026-09-08 per the card); 115.3 GiB checkpoint"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2","pin":"a22a1744e85c17364dcc54a6a420af7135d4c7f6 - DFlash2 drafter (Apache-2.0, 1.1 GiB), mounted at /drafter; capture patch shipped with it documents training data"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://github.com/Morrowmake/vllm-cmp170hx/pkgs/container/vllm-cmp170hx","pin":"ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79 on fork 18305948acba (tag qwen38-recipe-1.0.0) + two bind-mounted patch files in dflash mode"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe","pin":"release 0.1.0 (repo HEAD ab437e0b18a851caa055fa405d7d025dd9cbe058 at ingest 2026-10-21) - start.sh/serve.sh, bench/results/release-0.1.0/, docs/results.md, patches/"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple.json","alternate":"/r/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple","describedby":"/openapi.json"},"updated_at":"2026-10-29 20:26:39.946771-06","created_at":"2026-10-20 21:28:52.373948-06","speeds":{"decode_tok_s":253.19999999999998863131622783839702606201171875,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}