Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM

community Community submission — verify on your own hardware.

recipe by ElectroHobo↗, Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)↗, klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)↗, halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)↗, inco.ai (DFlash2 drafter, Apache-2.0)↗, Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)↗, Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)↗, MiaAI-Lab (benchmark protocol + category prompts, used as data)↗ · listed by localai-recipes

revision r2 · commit af7d7dde12 · updated 21 h ago · license MIT

chat code vision single-gpu bf16-kv speculative cuda-graphs autoround community-recipe dflash morrowmake-fork host-ram-ple tp1 recover-ssm 3bpw

Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.

Launch command

'git clone https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe && cd qwen38-flash-next-3bpw-1x-cmp170hx-recipe' ./start.sh

runtime

env: 'Defaults in .env: CUDA_VISIBLE_DEVICES=0 (exactly one card) | HOST=127.0.0.1 PORT=8000 (API_KEY optional) | SERVED_MODEL_NAME=qwen3.8-flash-next | GRAPH_MODE=FULL_AND_PIECEWISE | SPEC_MODE=dflash (dflash|mtp|none; none needs BOOT_CHECK=0) | SPEC_N=7 (dflash) / 3 (mtp) | DRAFTER_REPO=incoai/Qwen3.8-Flash-Next-DFlash2 @ a22a1744 (Apache-2.0, 1.1 GiB) | SPEC_SCHEDULE=[[1,2,7],[3,4,5],[5,48,3]] by request count (SPEC_SCHEDULE= for fixed depth) | MAX_SEQS=48 (48 fill the KV pool) | MAX_BATCHED=3460 | BLOCK_SIZE=912 (1680 with float32 SSM states) | RECOVER_SSM=1 | SSM_DTYPE=bfloat16 | REASONING_PARSER/TOOL_PARSER=qwen3/qwen3_coder | KV_OFFLOAD_GB=0 | PLE_SHARED_DIR unset (set it to share one 48 GiB table across servers) | BOOT_CHECK=1 | READY_TIMEOUT=1800 | engine env: VLLM_GDN_RECOVER=1, VLLM_QWEN4_QSA_TOPK_CANONICAL=1, VLLM_THIN_GEMM=1 (+_GLUON=1), VLLM_FLA_AUTOTUNE_SKIP_SPILLING=1, VLLM_USE_BREAKABLE_CUDAGRAPH=1, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, CUDA_DEVICE_ORDER=PCI_BUS_ID, NCCL_P2P_DISABLE=1 (one card) | context 262144 + util 0.95 FIXED (serve.sh refuses others); FIRST_LIGHT=1 = 32K diagnostic; DRY=1 ./start.sh prints the exact argv'
notes: "API on 127.0.0.1:8000 as qwen3.8-flash-next, no key by default; reasoning in message.reasoning, answer in content; thinking off via chat_template_kwargs.enable_thinking=false. Exact vLLM argv (DRY=1): vllm serve /models/Qwen3.8-Flash-Next-3bpw-fp8ple --tensor-parallel-size 1 --pipeline-parallel-size 1 --dtype bfloat16 --kv-cache-dtype bfloat16 --load-format auto --mm-encoder-tp-mode data --attention-config {'indexer_kv_dtype':'bf16'} --engram-config {'cpu_offload':true} --max-model-len 262144 --gpu-memory-utilization 0.95 --max-num-seqs 48 --max-num-batched-tokens 3460 --long-prefill-token-threshold 0 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --prefix-match-unit 16 --block-size 912 --mamba-ssm-cache-dtype bfloat16 --compilation-config {'cudagraph_mode':'FULL_AND_PIECEWISE'} --speculative-config {'method':'dflash','model':'/drafter','num_speculative_tokens':7,'num_speculative_tokens_per_batch_size':[[1,2,7],[3,4,5],[5,48,3]]}; KV_OFFLOAD_GB adds --kv-transfer-config (OffloadingConnector/TieringOffloadingSpec) + disable_eagle_block_drop:true (vLLM #52735)."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json

Engine

vllm · source↗ · 18305948↗ · ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79↗

Install note: One command from fresh clone after one-time host setup: git clone https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe && cd qwen38-flash-next-3bpw-1x-cmp170hx-recipe && ./start.sh (preflight checks -> engine image pull -> checkpoint downloads at pinned HF revisions + hard-linked FP8-ple build -> detached launch -> /health wait -> two-request boot check); docker compose up is the equivalent. First start compiles kernels and RecoverSSM's verify (~12 min), later starts ~4 min from SSD. Patches ride over the image read-only (patches/ mounted, never baked): 9 files covering DFlash2 capture+method check, RecoverSSM (GDN+PLE verify/commit), AutoRound INC fixes (expert bits, MTP plan, n-gram table), shared-table loader, chunked CoW. Host: 1x CMP 170HX 64 GiB PCIe x16, driver >=580 (measured 610.57.04), 96 GB host RAM MINIMUM (48 GiB is the FP8 n-gram table; measured on 204 GiB), ~100 GiB disk.

Model

Qwen3.8-Flash-Next-3bpw-AutoRound-MTP (ar-3bpw-fp8-ple-bf16-kv) · klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP↗
AA intelligence 40 (Artificial Analysis · measured)

Hardware

NVIDIA CMP 170HX 64GB ×1 (Ampere)

Context

max context: 256K tokens

KV pool

292,784 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
10	—	4243 tok/s	—	—	reported
262,144	1	—	197.8 tok/s	—	reported
262,144	8	—	799.5 tok/s	—	reported
272,384	1	—	296.4 tok/s	—	reported
272,384	8	—	791.3 tok/s	—	reported
272,394	48	—	1384 tok/s	—	reported
282,624	1	—	135.5 tok/s	—	reported
282,624	8	—	586.4 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	claimed_by=submitter T0 (author's own bench, MiaAI-Lab protocol: 400-token replies temp 0 thinking off, measured 2026-10-23 from a fresh clone; 74 SMs, 180 W, PCIe Gen2 x16, dual Xeon E5-2697A v4 Broadwell / 204 GiB). Single-user rows median of 5, aggregates median of 3; context_len markers +10240/+20480 = code/prose rows. 48-user totals from the 700-token concurrency sweep (DFlash2 1,384.2; MTP 1,402.9; MTP leads from ~24 users on this sweep, 3-15% on a mixed dev load). Code editing 371.0 tok/s one user (215.2 MTP); TTFT follow-up 0.42 s (0.24 MTP); KV pool 292,784 tokens @ 262144 ctx (304,893 MTP). Quality (thinking on, temp 0.6): HumanEval 161/164, GSM8K 195/200 (DFlash2). Comparators, same host (their README columns, not re-run): the author's own 2x W4A16 recipe 241/350/198 c1 and 1,937 at 128 users.

notes

claimed_by=submitter T0 (author's own bench, MiaAI-Lab protocol: 400-token replies temp 0 thinking off, measured 2026-10-23 from a fresh clone; 74 SMs, 180 W, PCIe Gen2 x16, dual Xeon E5-2697A v4 Broadwell / 204 GiB). Single-user rows median of 5, aggregates median of 3; context_len markers +10240/+20480 = code/prose rows. 48-user totals from the 700-token concurrency sweep (DFlash2 1,384.2; MTP 1,402.9; MTP leads from ~24 users on this sweep, 3-15% on a mixed dev load). Code editing 371.0 tok/s one user (215.2 MTP); TTFT follow-up 0.42 s (0.24 MTP); KV pool 292,784 tokens @ 262144 ctx (304,893 MTP). Quality (thinking on, temp 0.6): HumanEval 161/164, GSM8K 195/200 (DFlash2). Comparators, same host (their README columns, not re-run): the author's own 2x W4A16 recipe 241/350/198 c1 and 1,937 at 128 users.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP
ce0e0b94083895bd836b916f29bf105c40a8162a - the revision the release was validated with; experts 3-bit g128 layers 0-11/36-47 + 2-bit g64 layers 12-35, MTP head 4/8-bit; downloaded WITHOUT its 95 GiB BF16 n-gram table shard
Weights https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin
cec53e9f90e5995b81691604ec0b32191debf797 - only the FP8 n-gram table shards (model-plefp8-*, ~48 GiB, RadixArk quantisation) are downloaded and hard-linked into the built checkpoint
Drafter https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2
a22a1744e85c17364dcc54a6a420af7135d4c7f6 - DFlash2 drafter (Apache-2.0, 1.1 GiB), mounted at /drafter
Engine image https://github.com/Morrowmake/vllm-cmp170hx/pkgs/container/vllm-cmp170hx
ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79 on fork 18305948acba + patches/ mounted read-only over the image at start (image used as released)
Reference https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe
release 0.1.0 (repo HEAD 735adb80734b4c7e3839246a430e4ed638b3e355 at ingest 2026-10-25) - start.sh/serve.sh, bench/results/release-0.1.0/, docs/results.md, patches/

Revision notes

Community ingest of ElectroHobo's recipe (MIT), made to the shape of Morrowmake's community-recipes guidelines, which state Morrowmake has not tested/endorsed it and neither have klee100 or halt95. All perf numbers are the author's own measurements (claimed_by=submitter T0, harness + raw result files in-repo bench/results/release-0.1.0/) - import via POST /api/v1/benchmarks to upgrade. First single-card CMP 170HX recipe with the full 262,144 context + DFlash2 (the iIIusi0n PLE-on-SSD AIO build is the only other 1x lane). Cross-recipe checkpoint combination: klee100 3bpw weights + halt95 FP8 n-gram table hard-linked at build (docker/build_checkpoint.py). Verified against repo HEAD 735adb80734b (tag v0.1.0) at ingest 2026-10-25: pins, serve defaults and every quoted number match README + docs/results.md at that sha. Distinct from the author's 2x W4A16 recipe qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple (two cards, INT4, 128 requests).

Revision history

af7d7dde12 recipe: update qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm (r2: /provenance/credit) 15 h ago diff
e96b2e5b08 recipe: create qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm 6 d ago

Fork

Export

recipe.yaml (raw)