Qwen3.8-Flash-Next W4A16 2x CMP 170HX TP2 DFlash2 host-RAM PLE

community Community submission — verify on your own hardware.

recipe by ElectroHobo↗, Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)↗, halt95 (W4A16-Merlin checkpoint + companion 3090 engine; not involved in this recipe)↗, inco.ai (DFlash2 drafter, Apache-2.0)↗, Intel (AutoRound INT4 experts) + RadixArk (FP8 n-gram table) + DominikBucko (MTP-head packing)↗, Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)↗, MiaAI-Lab (benchmark protocol + category prompts, used as data)↗ · listed by localai-recipes

Initial version · updated 22 h ago · license MIT

chat code vision speculative cuda-graphs w4a16 community-recipe dflash morrowmake-fork host-ram-ple tp2

Qwen3.8-Flash-Next (125B MoE) INT4-expert W4A16 on two CMP 170HX cards, full 262,144-token context: 241/350/198 tok/s one user (structured/code/prose), 495 tok/s editing code, 895/987/770 across eight, up to 128 requests at once totaling 1,937 tok/s (DFlash 2; 1,966 MTP), 4,696 tok/s cold prefill, 1,475,836-token KV pool. Serves halt95's W4A16-Merlin checkpoint on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) with the 51 GB FP8 n-gram table in HOST RAM (cpu_offload; PLE_SHARED_DIR shares one copy across servers), the whole decode step one CUDA graph - built for a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3/2 schedule, 44% faster than MTP on code c1, 83% editing code) via engine-file patches plus RecoverSSM one-state GDN; optional KV offload restores an evicted 100k-token context in 0.89 s (21 s without). Measured 2026-10-18/23 from a fresh clone under Morrowmake's protocol (temp 0, 400-token pin, median-of-5 c1 / median-of-3 aggregates).

Launch command

'git clone https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe && cd qwen38-flash-next-w4a16-cmp170hx-recipe' ./start.sh

runtime

env: 'Defaults in .env: CUDA_VISIBLE_DEVICES=0,1 | HOST=127.0.0.1 PORT=8000 (API_KEY optional) | SERVED_MODEL_NAME=qwen3.8-flash-next | KV_OFFLOAD_GB=0 (64 held ~2.2M evicted tokens; KV_OFFLOAD_DIR adds a disk tier) | GRAPH_MODE=FULL_AND_PIECEWISE | P2P=auto | SPEC_MODE=dflash (dflash|mtp|none; none needs BOOT_CHECK=0) | SPEC_N=7 (dflash) / 3 (mtp) | DRAFTER_REPO=incoai/Qwen3.8-Flash-Next-DFlash2 @ a22a1744 (Apache-2.0, 1.1 GiB) | SPEC_SCHEDULE=[[1,2,7],[3,4,5],[5,8,3]] by request count (SPEC_SCHEDULE= for fixed depth) | MAX_SEQS=8 | MAX_BATCHED=3460 | parsers qwen3/qwen3_coder | BOOT_CHECK=1 | READY_TIMEOUT=1800 | engine env with kill switches: VLLM_QWEN4_QSA_TOPK_CANONICAL=1, VLLM_CUSTOM_ALLREDUCE_FLAGS=1, VLLM_THIN_GEMM=1 (+_GLUON=1), VLLM_FLA_AUTOTUNE_SKIP_SPILLING=1, VLLM_USE_BREAKABLE_CUDAGRAPH=1, VLLM_CUSTOM_ALLREDUCE_PCIE_P2P=1 + NCCL_P2P_LEVEL=SYS after the P2P check, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, CUDA_DEVICE_ORDER=PCI_BUS_ID | context 262144 + util 0.95 FIXED (serve.sh refuses others); FIRST_LIGHT=1 = 32K diagnostic; DRY=1 ./start.sh prints the exact argv'
notes: "API on 127.0.0.1:8000 as qwen3.8-flash-next, no key by default; reasoning in message.reasoning, answer in content; thinking off via chat_template_kwargs.enable_thinking=false. Exact vLLM argv (DRY=1): --tensor-parallel-size 2 --pipeline-parallel-size 1 --enable-expert-parallel --dtype bfloat16 --kv-cache-dtype bfloat16 --load-format auto --mm-encoder-tp-mode data --attention-config {'indexer_kv_dtype':'bf16'} --engram-config {'cpu_offload':true} --max-model-len 262144 --gpu-memory-utilization 0.95 --max-num-seqs 8 --max-num-batched-tokens 3460 --long-prefill-token-threshold 0 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --prefix-match-unit 16 --block-size 1680 --compilation-config {'cudagraph_mode':'FULL_AND_PIECEWISE'} --speculative-config {'method':'dflash','model':'/drafter','num_speculative_tokens':7,'num_speculative_tokens_per_batch_size':[[1,2,7],[3,4,5],[5,8,3]]}; KV_OFFLOAD_GB adds --kv-transfer-config (OffloadingConnector/TieringOffloadingSpec) + disable_eagle_block_drop:true (vLLM #52735)."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple.json

Engine

vllm · source↗ · 18305948↗ · ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79↗

Install note: One command from fresh clone after one-time host setup: git clone https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe && cd qwen38-flash-next-w4a16-cmp170hx-recipe && ./start.sh (preflight checks -> engine image pull -> checkpoint download at pinned HF revision -> detached launch -> /health wait -> two-request boot check); docker compose up is the equivalent. First start compiles kernels (~17 min), later starts ~5 min. DFLASH mode bind-mounts two engine files from patches/ over the image's (drafter capture hook + speculation-method check, 48 lines, over /opt/vllm-src/vllm/models/qwen4_exp/nvidia/model.py and config.py); SPEC_MODE=mtp runs the checkpoint's MTP head unmodified. Host: 2x CMP 170HX 64 GiB PCIe x16 (capacitor mod), P2P verified at start, driver >=580 (measured 610.57.04), 96 GB host RAM MINIMUM (51 GB is the n-gram table; measured on 204 GiB), ~130 GiB disk.

Model

Qwen3.8-Flash-Next-W4A16-Merlin (w4a16-int4-experts-bf16-kv) · halt95/Qwen3.8-Flash-Next-W4A16-Merlin↗
AA intelligence 40 (Artificial Analysis · measured)

Hardware

NVIDIA CMP 170HX 64GB ×2 (Ampere)

Context

max context: 256K tokens

KV pool

1,398,101 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
10	—	4729 tok/s	—	—	reported
262,144	1	—	253.2 tok/s	—	reported
262,144	8	—	842.2 tok/s	—	reported
272,384	1	—	348.9 tok/s	—	reported
272,384	8	—	992.1 tok/s	—	reported
282,624	1	—	203.2 tok/s	—	reported
282,624	8	—	726.7 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	claimed_by=submitter T0 (author's own bench, Morrowmake protocol, measured 2026-10-19 from a fresh clone; 74 SMs/card, P2P on, 180 W/card, PCIe Gen2 x16, KV_OFFLOAD_GB=64, dual Xeon E5-2697A v4 Broadwell / 204 GiB). Single-user rows are median of 5, aggregates median of 3; context_len markers +10240/+20480 = code/prose rows. Quality (thinking on, reasoning_effort medium, temp 0.6, 16k budget): HumanEval 163/164, GSM8K 197/200. KV offload: 100k ctx evicted by 1.8M tokens of other traffic restored in 0.89 s (21-22 s without); 18.7k-token conversation second turn 0.30 s (first 4.0 s); NVMe tier behind 16 GiB RAM tier 1.85 s; 100k prompt read 21.9 s, decode after 177 tok/s. Comparators (their README columns, not re-run): Morrowmake EXL3 152.8/152.6/118.3 c1 and 588.2/674.7/489.9 c8, 2651 prefill; AlexBrandes 250.8 c1 / 999.5 c8 / 6300-7100 prefill; mannydelrio 192.4 c1 / 986.4 c8 / 7006 prefill.

notes

claimed_by=submitter T0 (author's own bench, Morrowmake protocol, measured 2026-10-19 from a fresh clone; 74 SMs/card, P2P on, 180 W/card, PCIe Gen2 x16, KV_OFFLOAD_GB=64, dual Xeon E5-2697A v4 Broadwell / 204 GiB). Single-user rows are median of 5, aggregates median of 3; context_len markers +10240/+20480 = code/prose rows. Quality (thinking on, reasoning_effort medium, temp 0.6, 16k budget): HumanEval 163/164, GSM8K 197/200. KV offload: 100k ctx evicted by 1.8M tokens of other traffic restored in 0.89 s (21-22 s without); 18.7k-token conversation second turn 0.30 s (first 4.0 s); NVMe tier behind 16 GiB RAM tier 1.85 s; 100k prompt read 21.9 s, decode after 177 tok/s. Comparators (their README columns, not re-run): Morrowmake EXL3 152.8/152.6/118.3 c1 and 588.2/674.7/489.9 c8, 2651 prefill; AlexBrandes 250.8 c1 / 999.5 c8 / 6300-7100 prefill; mannydelrio 192.4 c1 / 986.4 c8 / 7006 prefill.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin
cec53e9f90e5995b81691604ec0b32191debf797 - the revision the release was validated with (weights unchanged since 2026-09-08 per the card); 115.3 GiB checkpoint
Weights https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2
a22a1744e85c17364dcc54a6a420af7135d4c7f6 - DFlash2 drafter (Apache-2.0, 1.1 GiB), mounted at /drafter; capture patch shipped with it documents training data
Engine image https://github.com/Morrowmake/vllm-cmp170hx/pkgs/container/vllm-cmp170hx
ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79 on fork 18305948acba (tag qwen38-recipe-1.0.0) + two bind-mounted patch files in dflash mode
Reference https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe
release 0.1.0 (repo HEAD ab437e0b18a851caa055fa405d7d025dd9cbe058 at ingest 2026-10-21) - start.sh/serve.sh, bench/results/release-0.1.0/, docs/results.md, patches/

Revision notes

Initial version.

Revision history

Initial version.

Fork

Export

recipe.yaml (raw)