{"slug":"qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","title":"Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM","summary":"Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.","license":"MIT","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"af7d7dde12376551476c8b9e36cd72c68dc32408","content_hash":"c82e80ffa9955d905b61bda381cf7d89db1350ba","created_at":"2026-10-29 21:32:26.955646-06","notes":"Community ingest of ElectroHobo's recipe (MIT), made to the shape of Morrowmake's community-recipes guidelines, which state Morrowmake has not tested/endorsed it and neither have klee100 or halt95. All perf numbers are the author's own measurements (claimed_by=submitter T0, harness + raw result files in-repo bench/results/release-0.1.0/) - import via POST /api/v1/benchmarks to upgrade. First single-card CMP 170HX recipe with the full 262,144 context + DFlash2 (the iIIusi0n PLE-on-SSD AIO build is the only other 1x lane). Cross-recipe checkpoint combination: klee100 3bpw weights + halt95 FP8 n-gram table hard-linked at build (docker/build_checkpoint.py). Verified against repo HEAD 735adb80734b (tag v0.1.0) at ingest 2026-10-25: pins, serve defaults and every quoted number match README + docs/results.md at that sha. Distinct from the author's 2x W4A16 recipe qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple (two cards, INT4, 128 requests)."},"engine_source":{"type":"docker_image","git_url":"https://github.com/Morrowmake/vllm-cmp170hx","base_engine":"vllm","pip_package":null,"base_version":"fork @ 18305948acba (tag qwen38-recipe-1.0.0)","image_digest":"ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79","install_note":"One command from fresh clone after one-time host setup: git clone https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe && cd qwen38-flash-next-3bpw-1x-cmp170hx-recipe && ./start.sh (preflight checks -> engine image pull -> checkpoint downloads at pinned HF revisions + hard-linked FP8-ple build -> detached launch -> /health wait -> two-request boot check); docker compose up is the equivalent. First start compiles kernels and RecoverSSM's verify (~12 min), later starts ~4 min from SSD. Patches ride over the image read-only (patches/ mounted, never baked): 9 files covering DFlash2 capture+method check, RecoverSSM (GDN+PLE verify/commit), AutoRound INC fixes (expert bits, MTP plan, n-gram table), shared-table loader, chunked CoW. Host: 1x CMP 170HX 64 GiB PCIe x16, driver >=580 (measured 610.57.04), 96 GB host RAM MINIMUM (48 GiB is the FP8 n-gram table; measured on 204 GiB), ~100 GiB disk.","pinned_commit":"18305948acba3f4c4d912cdd56f2202d2ebd94e8"},"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/Mia-AiLab"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["git clone https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe && cd qwen38-flash-next-3bpw-1x-cmp170hx-recipe","./start.sh"],"layouts":["tp"],"runtime":{"env":"Defaults in .env: CUDA_VISIBLE_DEVICES=0 (exactly one card) | HOST=127.0.0.1 PORT=8000 (API_KEY optional) | SERVED_MODEL_NAME=qwen3.8-flash-next | GRAPH_MODE=FULL_AND_PIECEWISE | SPEC_MODE=dflash (dflash|mtp|none; none needs BOOT_CHECK=0) | SPEC_N=7 (dflash) / 3 (mtp) | DRAFTER_REPO=incoai/Qwen3.8-Flash-Next-DFlash2 @ a22a1744 (Apache-2.0, 1.1 GiB) | SPEC_SCHEDULE=[[1,2,7],[3,4,5],[5,48,3]] by request count (SPEC_SCHEDULE= for fixed depth) | MAX_SEQS=48 (48 fill the KV pool) | MAX_BATCHED=3460 | BLOCK_SIZE=912 (1680 with float32 SSM states) | RECOVER_SSM=1 | SSM_DTYPE=bfloat16 | REASONING_PARSER/TOOL_PARSER=qwen3/qwen3_coder | KV_OFFLOAD_GB=0 | PLE_SHARED_DIR unset (set it to share one 48 GiB table across servers) | BOOT_CHECK=1 | READY_TIMEOUT=1800 | engine env: VLLM_GDN_RECOVER=1, VLLM_QWEN4_QSA_TOPK_CANONICAL=1, VLLM_THIN_GEMM=1 (+_GLUON=1), VLLM_FLA_AUTOTUNE_SKIP_SPILLING=1, VLLM_USE_BREAKABLE_CUDAGRAPH=1, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, CUDA_DEVICE_ORDER=PCI_BUS_ID, NCCL_P2P_DISABLE=1 (one card) | context 262144 + util 0.95 FIXED (serve.sh refuses others); FIRST_LIGHT=1 = 32K diagnostic; DRY=1 ./start.sh prints the exact argv","notes":"API on 127.0.0.1:8000 as qwen3.8-flash-next, no key by default; reasoning in message.reasoning, answer in content; thinking off via chat_template_kwargs.enable_thinking=false. Exact vLLM argv (DRY=1): vllm serve /models/Qwen3.8-Flash-Next-3bpw-fp8ple --tensor-parallel-size 1 --pipeline-parallel-size 1 --dtype bfloat16 --kv-cache-dtype bfloat16 --load-format auto --mm-encoder-tp-mode data --attention-config {'indexer_kv_dtype':'bf16'} --engram-config {'cpu_offload':true} --max-model-len 262144 --gpu-memory-utilization 0.95 --max-num-seqs 48 --max-num-batched-tokens 3460 --long-prefill-token-threshold 0 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --prefix-match-unit 16 --block-size 912 --mamba-ssm-cache-dtype bfloat16 --compilation-config {'cudagraph_mode':'FULL_AND_PIECEWISE'} --speculative-config {'method':'dflash','model':'/drafter','num_speculative_tokens':7,'num_speculative_tokens_per_batch_size':[[1,2,7],[3,4,5],[5,48,3]]}; KV_OFFLOAD_GB adds --kv-transfer-config (OffloadingConnector/TieringOffloadingSpec) + disable_eagle_block_drop:true (vLLM #52735)."},"kv_pool_tokens":292784,"max_context_len":262144},"models":[{"slug":"qwen3-8-flash-next-3bpw-autoround-mtp","name":"Qwen3.8-Flash-Next-3bpw-AutoRound-MTP","hf_repo":"klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP","quant":"ar-3bpw-fp8-ple-bf16-kv"}],"hardware":[{"slug":"nvidia-cmp-170hx","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":1}],"generation":"ampere","tags":["3bpw","autoround","bf16-kv","community-recipe","cuda-graphs","dflash","host-ram-ple","morrowmake-fork","recover-ssm","single-gpu","speculative","tp1"],"tasks":["chat","code","vision"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"claimed_by=submitter T0 (author's own bench, MiaAI-Lab protocol: 400-token replies temp 0 thinking off, measured 2026-10-23 from a fresh clone; 74 SMs, 180 W, PCIe Gen2 x16, dual Xeon E5-2697A v4 Broadwell / 204 GiB). Single-user rows median of 5, aggregates median of 3; context_len markers +10240/+20480 = code/prose rows. 48-user totals from the 700-token concurrency sweep (DFlash2 1,384.2; MTP 1,402.9; MTP leads from ~24 users on this sweep, 3-15% on a mixed dev load). Code editing 371.0 tok/s one user (215.2 MTP); TTFT follow-up 0.42 s (0.24 MTP); KV pool 292,784 tokens @ 262144 ctx (304,893 MTP). Quality (thinking on, temp 0.6): HumanEval 161/164, GSM8K 195/200 (DFlash2). Comparators, same host (their README columns, not re-run): the author's own 2x W4A16 recipe 241/350/198 c1 and 1,937 at 128 users.\", \"tok_s\": 197.80000000000001136868377216160297393798828125, \"points\": [{\"notes\": \"structured, median of 5\", \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 197.80000000000001136868377216160297393798828125}, {\"notes\": \"code; context_len is the tuple marker 262144+10240 (code row, not structured/prose)\", \"concurrency\": 1, \"context_len\": 272384, \"decode_tok_s\": 296.3999999999999772626324556767940521240234375}, {\"notes\": \"prose; context_len is the tuple marker 262144+20480 (prose row, not structured/code)\", \"concurrency\": 1, \"context_len\": 282624, \"decode_tok_s\": 135.5}, {\"notes\": \"aggregate structured, median of 3 (c2: 307.5, c4: 515.6)\", \"concurrency\": 8, \"context_len\": 262144, \"decode_tok_s\": 799.5}, {\"notes\": \"aggregate code; context_len is the tuple marker 262144+10240 (c2: 403.9, c4: 587.1)\", \"concurrency\": 8, \"context_len\": 272384, \"decode_tok_s\": 791.299999999999954525264911353588104248046875}, {\"notes\": \"aggregate prose; context_len is the tuple marker 262144+20480 (c2: 218.2, c4: 370.9)\", \"concurrency\": 8, \"context_len\": 282624, \"decode_tok_s\": 586.3999999999999772626324556767940521240234375}, {\"notes\": \"48-user sweep total (700-token replies); context_len is the marker 262144+10250 (shares the code-row marker family so the 48-sweep does not collide; per-request context is far below ctx max)\", \"concurrency\": 48, \"context_len\": 272394, \"decode_tok_s\": 1384}, {\"notes\": \"cold prefill median, real-text prompts, no cache hits\", \"context_len\": 10, \"prefill_tok_s\": 4243}], \"context_len\": 262144}"}],"perf_points":[{"id":2119,"context_len":10,"prefill_tok_s":4243,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill median, real-text prompts, no cache hits"},{"id":2112,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":197.80000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"structured, median of 5"},{"id":2115,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":799.5,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate structured, median of 3 (c2: 307.5, c4: 515.6)"},{"id":2113,"context_len":272384,"prefill_tok_s":null,"decode_tok_s":296.3999999999999772626324556767940521240234375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"code; context_len is the tuple marker 262144+10240 (code row, not structured/prose)"},{"id":2116,"context_len":272384,"prefill_tok_s":null,"decode_tok_s":791.299999999999954525264911353588104248046875,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate code; context_len is the tuple marker 262144+10240 (c2: 403.9, c4: 587.1)"},{"id":2118,"context_len":272394,"prefill_tok_s":null,"decode_tok_s":1384,"ttft_ms":null,"concurrency":48,"claimed_by":"submitter","notes":"48-user sweep total (700-token replies); context_len is the marker 262144+10250 (shares the code-row marker family so the 48-sweep does not collide; per-request context is far below ctx max)"},{"id":2114,"context_len":282624,"prefill_tok_s":null,"decode_tok_s":135.5,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"prose; context_len is the tuple marker 262144+20480 (prose row, not structured/code)"},{"id":2117,"context_len":282624,"prefill_tok_s":null,"decode_tok_s":586.3999999999999772626324556767940521240234375,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate prose; context_len is the tuple marker 262144+20480 (c2: 218.2, c4: 370.9)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP","pin":"ce0e0b94083895bd836b916f29bf105c40a8162a - the revision the release was validated with; experts 3-bit g128 layers 0-11/36-47 + 2-bit g64 layers 12-35, MTP head 4/8-bit; downloaded WITHOUT its 95 GiB BF16 n-gram table shard"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin","pin":"cec53e9f90e5995b81691604ec0b32191debf797 - only the FP8 n-gram table shards (model-plefp8-*, ~48 GiB, RadixArk quantisation) are downloaded and hard-linked into the built checkpoint"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2","pin":"a22a1744e85c17364dcc54a6a420af7135d4c7f6 - DFlash2 drafter (Apache-2.0, 1.1 GiB), mounted at /drafter"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://github.com/Morrowmake/vllm-cmp170hx/pkgs/container/vllm-cmp170hx","pin":"ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79 on fork 18305948acba + patches/ mounted read-only over the image at start (image used as released)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe","pin":"release 0.1.0 (repo HEAD 735adb80734b4c7e3839246a430e4ed638b3e355 at ingest 2026-10-25) - start.sh/serve.sh, bench/results/release-0.1.0/, docs/results.md, patches/"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json","alternate":"/r/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:26.955646-06","created_at":"2026-10-27 16:38:42.21677-06","speeds":{"decode_tok_s":197.80000000000001136868377216160297393798828125,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}