localairecipes launch configs for local inference

Qwen3.8-27B INT8 MTP3 prefix-cache on 1x CMP 170HX Hynix

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by ahnguyen17 (recipe, patches, probes, A/B benchmarks)↗, lued (Qwen3.8-27B-INT8-W8A16-MTP checkpoint)↗, vllm-project (vLLM upstream; #50021 GDN spec bounds, #45238 align-mode cache retention)↗ · listed by localai-recipes

revision r2 · commit ad3f55dab0 · updated 10 d ago · license CC-BY-4.0

chat code fp8-kv spec-decode mtp-speculative single-gpu prefix-cache mtp3 long-context cuda-graphs gdn-fusion int8

Qwen3.8-27B at 65-67 tok/s decode (74 structured) with MTP k=3 speculative decoding AND prefix caching simultaneously on a single NVIDIA CMP 170HX 64GB (Hynix HBM2e, GA100/sm80), 262,144-token context on fp8_e4m3 KV, 24/7 production on pinned upstream vLLM nightly 0.26.1rc1.dev1024+g1fe3a1571 — the first build where prefix caching and MTP spec decode coexist (GDN mamba-align). Cached TTFT on a 7K shared prefix drops ~4.0 s -> 1.44 s (33.5% hit rate) while decode gains 7-9% over the 0.27.1 fork fallback — the 'MTP vs prefix cache, pick one' tradeoff is gone. Load-bearing trio --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 (default chunked-prefill batch = 0.0% hit rate while looking healthy); --prefix-match-unit 400 halves cached TTFT further (must divide the 1600-token GDN block); batch budget 32000 OOMs and hard-wedges the card. Samsung-40GB side recipe (int8 + DFlash2, 97.8 tok/s) with an sm80 int8-repack CPU-fallback patch.

Launch command

vllm serve /path/to/Qwen3.8-27B-INT8-W8A16-MTP --quantization gptq --dtype bfloat16 --api-key '$API_KEY' --max-model-len 262144 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8_e4m3 --mamba-ssm-cache-dtype auto --enable-prefix-caching --enable-chunked-prefill --prefix-match-unit 400 --async-scheduling --max-num-batched-tokens 8192 --max-num-scheduled-tokens 8192 --speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}' --compilation-config '{"max_cudagraph_capture_size":128,"custom_ops":["+rms_norm","+silu_and_mul"]}' --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --served-model-name qwen3.8-27b --host 0.0.0.0 --port 8012

runtime

env: 'PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (prevents fragmentation OOM on the 64GB HBM2e across long uptimes — Samsung sm80 bins instead wedge with it and want :False); CUDA_DEVICE_ORDER=PCI_BUS_ID + CUDA_VISIBLE_DEVICES pinned (FASTEST_FIRST hands the server the wrong GPU in mixed rigs); VLLM_USE_FLASHINFER_SAMPLER=1 (verbatim serve-qwen38-nightly.sh @ 192cc33).'
notes: "THE CRITICAL FLAG TRIO (verbatim script comments): --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 — with the default chunked-prefill batch size the prefix cache silently runs at 0.0% hit rate while everything looks healthy: the mamba-align split logic needs single-shot prefills that end exactly on the 1600-token GDN hash boundary to register cacheable blocks. Do NOT raise max-num-batched-tokens above 8192: GDN prefill intermediates scale with scheduled tokens — 32000 OOM'd in chunk_fwd_o and hard-wedged the GA100 card (reboot-only recovery). --prefix-match-unit 400 must divide the 1600-token GDN block (512 boot-fails with a ValueError). MTP k>3 is a measured dead end (acceptance saturates ~3.2/8 at k=7; k=4 and k=7 both net -27..45% tok/s). --enable-prefix-caching auto-enables mamba-cache-mode align. Samsung-memory 40GB 170HX fallback inside the repo (serve-g3-int8df2.sh, port 18020): int8 + DFlash2 via the qwen38-27b-rtx3090 fork (SPEC=dflash2 CTX=fast), 97.8 tok/s single-stream / 275 tok/s aggregate x8, with patches/sm80-int8-repack-cpu-fallback.patch (Xid-31 wedge during gptq_marlin_repack at production occupancy; CPU-side repack is bit-exact, ~3 extra boot minutes) and expandable_segments:False — opposite allocator setting from the Hynix recipe; Samsung bins still throw Xid-79 under sustained load."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json

Engine

Install note: Install from the upstream nightly wheel index: uv pip install --extra-index-url https://wheels.vllm.ai/nightly vllm==0.26.1rc1.dev1024 (commit-pinned build; nightly is a moving target — pin exactly this). Validated on torch 2.13.0+cu130, driver 610.43.03. No patches needed on this build (the GDN rework supersedes vllm-project/vllm#50021); the repo's patches/vllm-pr50021-gdn-spec-bounds.patch is required ONLY for the vLLM 0.27.1 fallback recipe (serve-qwen38-mtp.sh) — without it, unbounded GDN state reads under MTP spec decode crash the server.

Model

Hardware

Context

  • max context: 256K tokens

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
6,0001—79 tok/s—reported
7,0001——1440 msreported
7,2001—65.4 tok/s—reported
40,0001—68.7 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesSame-day A/B nightly-vs-0.27.1-fork on identical harness + hardware (300-token single-stream gens, 3-run medians, tokens counted via usage.completion_tokens — SSE delta-counting reads a false ~30 tok/s because MTP emits ~3.5 tok/delta), from README.md @ repo HEAD: decode prose greedy 65.4 vs 60.0 tok/s, prose temp 0.7 66.8 vs 62.5, structured greedy 74.0 vs 74.1; MTP acceptance 3.02 vs 3.12 of 4; TTFT on a 7K shared prefix: cold ~6.7 s nightly / ~4.0 s fork (no cache), CACHED 1.44 s at 33.5% hit rate vs ~4.0 s flat on the fork (which cannot combine prefix caching with spec decode). Deep-dive doc: decode flat 69-79 tok/s 6K->40K ctx (79.0 @6K, 68.7 @40K usage-counted); cold prefill collapses superlinearly (17.3k tok/s @6K -> 0.5k @80K, Triton GDN chunk kernels); turn-2 cache miss root-caused to upstream vllm#45238 (align-mode single-checkpoint retention, ~3 sightings to hit), mitigated not eliminated by --prefix-match-unit 400 (20K turn-3 1.67->0.70 s, 40K 5.56->2.79 s).

Downloads

Revision notes

Submitted 2026-10-09 by localai-recipes from lar-scout ingest card t_35407999 (sweep QUEUE #3, 2026-10-09). Repo ahnguyen17/cmp-170hx-vllm @ 192cc330d9434e915bab05a2263944ae5fd12361 (HEAD via git ls-remote 2026-10-09; created 2026-08-20, 11 commits, actively maintained). Launch argv token-verified verbatim against serve-qwen38-nightly.sh at that sha; flag rationale cross-checked against script comments + README flag-by-flag table + docs/gdn-prefix-caching-2026-08-23.md; performance figures carried from the README A/B table and deep-dive probes as submitter-T0 with in-repo receipts (same-day A/B, 3-run medians, usage-counted). Engine pins: upstream nightly wheel vllm==0.26.1rc1.dev1024 from wheels.vllm.ai/nightly (commit 1fe3a1571ac67581478a11743e55a306de1d136f), torch 2.13.0+cu130, driver 610.43.03; model HF rev 7c12373712d1363e2b76655cb3332c9c124627d7. FIRST single-170HX recipe on the site (buckets were x2/x3/x4 only, verified live 2026-10-09); first GPTQ-int8/W8A16 lane on any 170HX recipe (site's other 170HX entries are W4A16/FP8/EXL3); first upstream-nightly vLLM 170HX entry (existing ones use forks). Documents the Hynix-vs-Samsung memory-bin split with working recipes for both. Hardware resolves to the pre-existing cmp-170hx single-card profile via the 0041 seed-slug convention (no orphan bucket).

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)