localairecipes launch configs for local inference

Qwen3.8-Flash-Next W4A16 2x CMP 170HX PP2 MTP5 PLE (AlexBrandes)

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by AlexBrandes (recipe, fork, image, measurements)↗, AlexBrandes (vllm fork: PP + NVMe-mmapped PLE, Mamba PP state, draft lm_head, graph-safe PLE, bounded prefill)↗, polyuij42-del (community CMP 170HX W4A16 work the fork builds on)↗, nguyenthimy2022kg-alt (community SM80 170HX work the fork builds on)↗, MiaAI-Lab (sparkDash benchmark protocol @ 3e6d4ef)↗, Intel (INT4 W4A16 AutoRound quant)↗ · listed by localai-recipes

revision r3 · commit 44b6d37a73 · updated 3 d ago · license CC-BY-4.0

chat code pipeline-parallel mtp-speculative long-context nvme-mmap sm80 nvme-ple no-p2p prefix-caching mtp5 sparkdash

Recipe by AlexBrandes. Qwen3.8-Flash-Next (Intel/Qwen3.8-Flash-Next-W4A16-AutoRound @ 4c67bf68) on two NVIDIA CMP 170HX cards in PIPELINE parallel (PP2, layer split 26/22) with a vLLM v0.30.0 fork (AlexBrandes/vllm @ 02acf67c, 6 commits) in a digest-pinned container. Native MTP with 5 draft tokens (lm_head quantized to int4 by tools/quantize_lm_head.py, sha256 pinned), reasoning + tool calls, full 262,144-token window. The ~102 GB n-gram (PLE) table stays on NVMe read through mmap (VLLM_PLE_MMAP=1); BF16 KV, prefix caching, FP32 Mamba state. FIRST 2x170HX recipe with NO P2P or link modification needed - PP sends one activation per step, so it runs on stock PCIe Gen2 x4 links (all sibling rows need the unlock/P2P-mod prerequisite). Measured 2026-10-17 with sparkDash (3e6d4ef): 226 tok/s single-user code, 840 tok/s 8 users, 5,885-7,104 tok/s prefill 4k-128k; HumanEval 157/164, GSM8K 1,271/1,319, needle 12/12.

Launch command

./download.sh ./start.sh '# ./start.sh (HOST_TUNE=1 default) runs host-tune.sh via sudo (170 W, 16 KB readahead, drop caches), then execs:' 'docker run -d --name cmp170hx-qwen38-flash-next --gpus "device=0,1" --shm-size 1g -p 127.0.0.1:8000:8000' '-e VLLM_USE_BREAKABLE_CUDAGRAPH=1 -e VLLM_MTP_DRAFT_HEAD_PATH=/drafthead/head-w4a16.safetensors' '-e VLLM_PLE_MMAP=1 -e VLLM_PLE_MMAP_RANDOM=1 -e VLLM_PLE_HOST_IDS=1 -e VLLM_MIXED_PREFILL_CHUNK=560' '-e FLASHINFER_DISABLE_VERSION_CHECK=1 -e VLLM_PP_LAYER_PARTITION=26,22' '-v ./models/Qwen3.8-Flash-Next-W4A16-AutoRound:/models/Qwen3.8-Flash-Next-W4A16-AutoRound:ro' '-v ./models/draft-head-w4a16.safetensors:/drafthead/head-w4a16.safetensors:ro' ghcr.io/alexbrandes/cmp170hx-qwen38-flash-next@sha256:d6fc872d943b3e9e5cc9eb583b51d72af200e039fbbb7401700bb38d665f0957 'vllm serve /models/Qwen3.8-Flash-Next-W4A16-AutoRound' '--pipeline-parallel-size 2' '--served-model-name qwen3.8-flash-next-w4a16' '--max-model-len 262144' '--block-size 1680' '--mamba-cache-mode align' '--mamba-ssm-cache-dtype float32' '--max-num-seqs 8' '--max-num-batched-tokens 2048' '--gpu-memory-utilization 0.95' '--max-parallel-loading-workers 1' '--moe-backend auto' --enable-prefix-caching --enable-prompt-tokens-details '--reasoning-parser qwen3' --enable-auto-tool-choice '--tool-call-parser qwen3_coder' --trust-remote-code --no-enable-flashinfer-autotune '-cc {"cudagraph_mode":"FULL_AND_PIECEWISE","splitting_ops":["vllm::unified_attention_with_output","vllm::mamba_mixer2","vllm::mamba_mixer","vllm::short_conv","vllm::qwen4_exp_ple_short_conv","vllm::qwen4_exp_qsa_with_output","vllm::linear_attention","vllm::qwen_gdn_attention_core","vllm::ple_mmap_lookup"]}' '--speculative-config {"method":"mtp","num_speculative_tokens":5}'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes.json

Engine

Install note: NOT a pip install: pull ghcr.io/alexbrandes/cmp170hx-qwen38-flash-next@sha256:d6fc872d...f0957 (official vllm/vllm-openai:v0.30.0 image + the fork's 18 changed Python files; docker/build.sh rebuilds from AlexBrandes/vllm @ 02acf67c). One command from a fresh clone: ./download.sh (~170 GB checkpoint at pinned revision + int4 MTP draft head built locally, sha256 pinned) then ./start.sh (runs host-tune.sh through sudo: 170 W power limit, 16 KB readahead on the model drive, drop caches + compact; HOST_TUNE=0 skips). Waits for /health.

Model

Hardware

Context

  • max context: 256K tokens

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
4,096—6295 tok/s—740 msreported
16,384—7096 tok/s—2400 msreported
131,072——141.8 tok/s21430 msreported
262,1441—226.2 tok/s80 msreported
262,1442—362.9 tok/s109 msreported
262,1444—560.2 tok/s171 msreported
262,1448—840.4 tok/s231 msreported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured 2026-10-17 (results/2026-10-17.json in-repo; sparkDash @ 3e6d4ef, each cell median of 3 runs, range in README): decode code 226.2 / json 198.0 / prose 157.4 / structured 250.8 tok/s @ c1; 8-user aggregate code 840.4 / json 873.2 / prose 566.8 / structured 999.5. Cold prefill 6,295 @4k to 7,104 @32k, 6,901 @64k, 5,885 @128k tok/s. Decode-at-depth 157.9 @4k to 141.8 @128k. Greedy, thinking off, output forced to 400 tokens, TTFT 67-231 ms @c1-c8. Quality: HumanEval 157/164, GSM8K 1,271/1,319, needle 12/12 @8k/32k/128k, 262,144-token window boundary verified. Chat-only model (raw completions EOS immediately).

Downloads

Revision notes

Direct ingest of a 2026-10-17 GitHub-search find (lar-scout card t_67dee659). Recipe repo pinned at HEAD 3f0d60096b3548a5f5449ebca7ab4b0f353bc772 (verified via git ls-remote at ingest; single commit). Engine fork branch cmp170hx-qwen38-flash-next @ 02acf67c verified live (branch tip == pin at ingest). All launch argv transcribed from start.sh at that sha and cross-checked against results/2026-10-17.json launch.args (identical). Checkpoint revision 4c67bf68 + draft-head sha256 5c5461d7... resolved from versions.env as the card instructed. All perf numbers author-reported (T0 submitter) from results/2026-10-17.json three-run medians; no independent re-run. Distinctives vs the site's other 2x170HX QFN rows: pipeline parallel on STOCK PCIe links (no P2P/capacitor mod needed - clee PP3, Morrowmake TP2, Minachist PP2 and mannydelrio PP2 all require link/unlock work), NVMe-mmapped PLE via VLLM_PLE_MMAP (vs staged preadv / shard-direct variants), MTP-5 (siblings use 3-4), BF16 KV + FP32 Mamba state, 262k window (siblings stretch to 384k-1M). Host-tuning caveat: start.sh needs sudo (power limit + readahead + cache drop); HOST_TUNE=0 skips it at ~10% decode cost.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)