localairecipes launch configs for local inference

Qwen3.8-Flash-Next EXL3 5.05bpw 2x CMP 170HX TP2 MTP3 (Morrowmake 1.0.0)

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by Morrowmake↗ · listed by localai-recipes

revision r2 · commit 12a77349c7 · updated 10 d ago · license CC-BY-4.0

chat code vision mtp-speculative tensor-parallel vision mtp3 ple-ssd long-context bf16-kv exl3 sm80

First EXL3 row for Qwen3.8-Flash-Next and first Morrowmake QFN lane: turboderp/Qwen3.8-Flash-Next-exl3 (EXL3 5.05 bpw) on two CMP 170HX cards (TP2, whole routed experts per card, PCIe x16 links) with the Morrowmake sm_80 vLLM fork @ 18305948ac, container pinned by digest. Native MTP speculative decoding k=3, full 262,144-token context, BF16 KV cache, n-gram (PLE) table read from the checkpoint SSD during decode, vision + reasoning + tool calls, every weight on-GPU except the PLE table. One-command repro (./start.sh or docker compose up); checkpoint pinned to HF revision 7cef615f. Measured 2026-10-15: 152.8 tok/s single-user, 674.7 tok/s at 8 users (code), 2,651 tok/s cold prefill, 1,044,325-token KV pool; HumanEval 157/164, GSM8K 1,289/1,319.

Launch command

vllm serve /models/Qwen3.8-Flash-Next-exl3 --served-model-name qwen3.8-flash-next --host 0.0.0.0 --tensor-parallel-size 2 --pipeline-parallel-size 1 --enable-expert-parallel --dtype bfloat16 --kv-cache-dtype bfloat16 --load-format exl3 --mm-encoder-tp-mode data --attention-config '{"indexer_kv_dtype":"bf16"}' --engram-config '{"cpu_offload":false}' --max-model-len 262144 --gpu-memory-utilization 0.95 --max-num-seqs 8 --max-num-batched-tokens 3460 --long-prefill-token-threshold 0 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --port 8000 --prefix-match-unit 64 --compilation-config '{"cudagraph_mode":"PIECEWISE"}' --speculative-config '{"method":"mtp","num_speculative_tokens":3,"disable_eagle_block_drop":true}'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-exl3-5-05bpw-2x-cmp-170hx-tp2-mtp3-morrowmake.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-exl3-5-05bpw-2x-cmp-170hx-tp2-mtp3-morrowmake.json

Engine

Install note: Engine is Morrowmake/vllm-cmp170hx @ 18305948acba3f4c4d912cdd56f2202d2ebd94e8 (sm_80 Ampere EXL3 extension on vLLM). Runs only in the digest-pinned container ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79 (recipe release 1.0.0); not a pip install. One command from a fresh clone: ./start.sh (or docker compose up) - preflight, engine install, checkpoint download at the pinned revision, launch, /health wait. First start compiles cold; later starts reuse ./cache. Measured on PCIe x16 capacitor-mod links; x4 links not measured (start.sh reports link width and warns).

Model

Hardware

Context

  • max context: 256K tokens

KV pool

  • 1,044,325 tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
262,144—2651 tok/s——reported
262,1441—152.8 tok/s—reported
262,1442—265 tok/s—reported
262,1444—458.6 tok/s—reported
262,1448—674.7 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesT0 submitter-reported (recipe release notes / README, measured by Morrowmake 2026-10-15 on 2x CMP 170HX, 180 W/card, P2P on, PCIe x16, 262,144 ctx, temperature 0, thinking off, 400-token generations): single-user structured 152.8 / code 152.6 / prose 118.3 tok/s; 2-user totals 236.5 / 265.0 / 204.6; 4-user totals 357.1 / 458.6 / 327.2; 8-user totals 588.2 / 674.7 / 489.9 (structured totals lower because replies finish early: median 346/273/279 tokens vs 400). Cold prefill 2,651 tok/s (median of 9). KV pool 1,044,325 tokens. Quality: HumanEval 157/164, GSM8K 1,289/1,319; 262,144-window boundary tests pass; needle retrieval 9/9 at 8K/32K/128K; vision 31/31.

Downloads

Revision notes

Recipe by Morrowmake, release v1.0.0 (2026-10-16), repo pinned at 207867e80430701f85f5e99157b03f705ff79ef0 = tag qwen38-recipe-1.0.0; ingested by localai-recipes from the Morrowmake engine-repo watch. Benchmarks upstream-measured 2026-10-15 (MiaAI-Lab protocol; 1 user = median of 10 runs over two boots, 2/4/8 users = median aggregate of 3 runs on one boot; prompt reading = median of 9 cold prefills). Serving defaults are locked by serve.sh (MAX_LEN=262144, GPU_UTIL=0.95, LAYOUT=tp2; FIRST_LIGHT=1 is a 32K/eager/no-MTP diagnostic, not serving). PLE placement ssd is the engine default; ram/device are comparison placements. EXL3 sm_80 kernel switches ship on by default with six one-variable kill switches (off: +18% single-user decode, +43-56% at 2-8 users). P2P=auto verifies peer content at every start. Recipe MIT; engine Apache-2.0; checkpoint under its own model-card licence.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)