localairecipes launch configs for local inference

Qwen3.8-Flash-Next W4A16 INT4PLE on 2x modded CMP 170HX — PP2 MTP3 (DUFRelic)

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by DUFRelic↗, zankich (INT4-PLE repackage + plugin)↗, halt95 (W4A16-Merlin base checkpoint)↗, michael-lgtm (public overlay rebuild)↗, amoghmunikote (cmpunlocker SM unlock + BAR1 P2P)↗ · listed by localai-recipes

revision r3 · commit c26a7fd62d · updated 21 h ago · license CC-BY-4.0

chat code pipeline-parallel mtp-speculative int4-ple pinned-host-memory modded-hardware sm-unlock p2p speedsuite dufrelic

Production 2x CMP 170HX recipe for Qwen3.8-Flash-Next: zankich INT4PLE repackage of halt95 Merlin W4A16 (94.4 GiB, 51B-param PLE table repacked INT4) on stock vLLM 0.30.0 + a 6-layer hash-guarded overlay stack (PP2/PLE #56444, KV filter #54793, MTP-over-PP #54709, slot statefix #55506, SpeedSuite 5-PR overlay, batched-P2P). PP2/TP1, MTP 3-token draft (measured 52.27% accept). Hardware is the recipe's own mod stack: 74-SM unlock (cmpunlocker), VBIOS 92.00.6D.00.0A, 170tune 1420 MHz SM cap + HBM 1728 MHz undervolt, BAR1 P2P driver patches, 140 W/card. Author-measured 2026-10-03 (median of 3, isolation audit): prefill C1 6213.7 tok/s @16k, decode C1 133.4 tok/s (SD 22.8 reported), decode C4 agg 245.2 tok/s; 55/56 C at 160 W peaks, no Xid/OOM. INT4 PLE table (~26.8 GiB) pinned in HOST RAM; KV BF16 (pinned build's QSA path rejects FP8). Driver mods optional for function but required for the published throughput.

Launch command

vllm serve /models/qwen38-flash-next-w4a16-merlin-int4ple-fp8kv --pipeline-parallel-size 2 --tensor-parallel-size 1 --distributed-executor-backend mp --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 2048 --enable-chunked-prefill --compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,4,8,16,32]}' --gpu-memory-utilization 0.90 --kv-cache-dtype auto --reasoning-parser qwen3 --tool-call-parser qwen3_xml --chat-template /templates/qwen-fixed-chat-template-v22.5.jinja --host 127.0.0.1 --port 8000

runtime

env: "NCCL_P2P_DISABLE=0 | NCCL_P2P_LEVEL=SYS | VLLM_COMPUTE_NANS_IN_LOGITS=1 (diagnostic; per README section 4). KV cache stays BF16 (default) — the pinned build's Qwen4Exp QSA path rejects an FP8 cache, so --kv-cache-dtype auto is load-bearing. Host-RAM guard: the launcher refuses to start when MemAvailable < 34 GiB (26.8 GiB INT4 PLE table pinned via UVA gather)."
notes: "Serve profile transcribed from DUFRelic/qwen38-flash-next-recipe README section 4 @ 25f97f3c (profile p-prefill2048-mtp3-speedsuite) — this repo documents the profile as a table rather than shipping a serve script. Stack: Caddy/TLS → LiteLLM → vLLM (loopback-only) on a Lenovo ThinkStation P520, Docker. Parallelism PP2/TP1 with the mp executor; NOTE from the author: --max-num-seqs 8 under PP2 does NOT create 8 independent in-flight forwards per rank — the V1 runner caps concurrent batches at the pipeline size. Multimodal: up to 4 images + 1 video (2 frames) per prompt. Chat template: froggeric/Qwen-Fixed-Chat-Templates v22.5 @ 855bffc49448e299789730ff92c9b8d834d6cc14 (chat_template.jinja sha256 e57684ba...), mounted READ-ONLY. Hardware mod stack (verified live, rollback in author's private ops runbooks): VBIOS 92.00.6D.00.0A flashed both cards (max customer boost 1695 MHz, 300 W ceiling); 170tune persistence re-applied every boot (GPC V/F undervolt +250/+275, SM ceiling 1420 MHz via nvidia-smi -lgc 210,1420, HBM NDIV 64 = 1728 MHz, relaxed HBM refresh ~8.3 us); SM unlock 70→74 via amoghmunikote/cmpunlocker @ 6c442eeb (0016-sm-reconfig.patch, RECONFIG_PLM + GPC override; verified by CUDA device attributes + %smid across all 74 SMs, 3,145,728 error-free computed values per GPU); BAR1 P2P driver patches (cmpunlocker + Bayley 0011/0013/0015 @ 5a7bb4b7) enabling NCCL P2P SYS. Power: 140 W/card persistent with a 200 W observed-draw abort guard; benchmark telemetry showed 160 W peaks / 55-56 C. Driver mods are OPTIONAL for function (runs with 70 SMs + NCCL_P2P_DISABLE=1) but REQUIRED to match the published throughput. Model files ship pre-packed (model-pleint4-*.safetensors + ple-int4-manifest.json) — no repacking step. Checkpoint config: Qwen4Exp, ple_layer_ids=[2], ple_embedding_dtype=int4, split_ngram_parts=128, ngram_size=3, heads_per_ngram=8, ple_embed_dim=2560."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3.json

Engine

Install note: Build: start from vllm/vllm-openai:v0.30.0 @ manifest sha256:5f5e5352..., apply the 6 hash-guarded overlay layers in order (#56444 PP2/PLE, #54793 KV filter, #54709 MTP-over-PP, #55506 statefix @ 5f71d6fa, SpeedSuite #58762/#58763/#58449/#58114/#57097 — #58449 was OPEN at pin, re-check upstream — and the batched-P2P rewrite of 3 isend/irecv sites in parallel_state.py); every layer fails closed on hash mismatch. INT4-PLE plugin: zankich/vllm branch v0.29.0z @ 6609bfef (wheel overlay w/ source-hash guard). PUBLIC REBUILD (author's duf/ image name is private): github.com/michael-lgtm/qwen38-flash-next-cmp-overlay apply_overlay.py over the stock image.

Model

Hardware

Context

  • max context: 256K tokens

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
16,38416213.7 tok/s133.4 tok/s—reported
16,38446639.3 tok/s245.2 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured 2026-10-03 (README section 5 @ 25f97f3c, harness llama-benchy on loopback, --pp 16384 --tg 2048 --concurrency 1 4 --runs 3 --exact-tg --no-cache --no-warmup; full-shape warmup excluded from scoring; isolation audit on the scored window: exactly 15 successful requests, 30,720 output tokens, zero prefix-cache hits / preemptions / errors; telemetry every 2 s; stats = mean ± population SD over 3 scored runs). Prefill C1 @16k 6213.66 ± 99.27; decode C1 (2k out) 133.39 ± 22.81 (min-max 108.10-163.37 — dispersion reported, not best-run); prefill C4 6639.25 ± 34.79; decode C4 aggregate 245.17 ± 1.17. MTP depth 3 active, draft accept 52.27% over the whole benchmark. 140 W limits held, peaks 159.66/162.77 W, 55/56 C, no Xid/OOM/EngineDeadError. claimed_by=submitter T0 — no run file ingested; import via POST /api/v1/benchmarks to upgrade.

Downloads

Revision notes

Community ingest of DUFRelic/qwen38-flash-next-recipe @ 25f97f3c (lar-scout GitHub sweep Oct 5, queued as t_dd357cb3; repo created 2026-10-05, 5 commits, README-licensed for reuse; patches and models remain with their upstream authors). Serve profile transcribed from README section 4 @ the pinned commit; benchmark numbers are the author's section 5 tables (median-of-3 methodology with isolation audit), logged claimed_by=submitter T0 — no run file. Hardware is 'modded CMP 170HX' (74-SM unlock + VBIOS + undervolt + BAR1 P2P) — mod stack documented in runtime notes and tags (modded-hardware, sm-unlock); profile linked is the existing cmp-170hx-x2 (same silicon, 64 GB x2). First 170hx recipe whose hardware mod stack is part of the recipe. KV cache runs BF16 despite the FP8KV suffix (pinned build rejects FP8 for Qwen4Exp QSA). Model license: qwen-community-1.0.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)