localairecipes launch configs for local inference

Qwen3.8-Flash-Next Swift15 AWQ 2x CMP 170HX PP2 MTP4 PLE Asgard

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by inordic (recipe, patch series, measurements), 2dameneko (base vLLM build vllm-Qwen3.8-Flash-Next-2xCMP170HX, commit 56e1347), iIIusi0n (PLE-SSD offload recipe and ple_ssd_io.so helper), ukisai (Swift 1.5 W4A16-AWQ checkpoint), doth4580 (DotH MTP head retrained on agent traffic), leonbollerup (Qwen Asgard chat template), AlexBrandes (int4 MTP draft head patch 08, quantize_lm_head.py), vLLM upstream PRs ported: #55601, #60210, #56734, #60615, #58784, #54912, #62344, #65470 · listed by localai-recipes

Initial version · updated 3 h ago · license MIT

chat code long-context pp2 bf16-kv ple-nvme speculative-decoding sm80 no-p2p prefix-caching mtp4 asgard-template

Qwen3.8-Flash-Next (Swift 1.5 W4A16-AWQ, DotH MTP head retrained on agent traffic) on two unlocked 64 GB CMP 170HX cards (300 W VBIOS cross-flash) in vLLM pipeline parallel. PP2 (24,24 layers), MTP4 with n-gram lookup and an int4 draft lm_head, --prefix-match-unit 16, BF16 KV, the ~95 GiB PLE n-gram table streamed from NVMe via Linux AIO (16 GiB row cache), 262,144-token max-model-len, 24 seqs, Qwen Asgard chat template with reasoning_effort medium. Author-measured (night 6, 2026-11-05): 199.6 tok/s median code-gen decode at default clocks with 3.62 mean acceptance, 304 tok/s code editing at 1440 MHz, 8,403 tok/s prefill on a 128K prompt, 511 tok/s aggregate at 16 users, BFCL v4 80.3% (night-3 suite; night-6 re-run 80.0%). recipe by inordic.

Launch command

/opt/vllm-qfn-n6k/.venv/bin/vllm serve /opt/vllm-qfn/models/swift15-doth --served-model-name qwen3.8-flash-next --host 0.0.0.0 --port 8080 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 24 --max-num-batched-tokens 4096 --gpu-memory-utilization 0.90 --safetensors-load-strategy lazy --mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml --additional-config '{"ple_ssd_offload":true,"ple_ssd_workers":32,"ple_ssd_cache_mb":16384,"ple_ssd_native_library":"/opt/vllm-qfn/optimization/ple_ssd_io.so","ple_ssd_io_depth":512,"ple_ssd_prefetch_tokens":131072}' --pipeline-parallel-size 2 --enable-expert-parallel --async-scheduling --prefix-match-unit 16 --enable-mamba-fine-grained-prefix-cache --served-model-name qwen3.8-flash-next qwen3.8-flash-next-terse swift-1.5-awq Qwen3.8-Flash-Next-AWQ-INT4 swift-1.5-iq3_xxs --chat-template /opt/vllm-qfn/templates/qwen-asgard-v1.jinja --default-chat-template-kwargs '{"reasoning_effort":"medium","tool_prompt":"chromix"}' --speculative-config '{"method":"mtp","num_speculative_tokens":4,"ngram_lookup":true}' --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,18,20,22,24,26,28,30,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,128,192,256,512,1024,2048],"mode":1}' --api-key '<from /root/llm-api-key>'

env

- VLLM_VENV=/opt/vllm-qfn-n6k/.venv
- MODEL=/opt/vllm-qfn/models/swift15-doth
- SERVED_NAME=qwen3.8-flash-next
- PORT=8080
- 'CUDA_DEVICES=1,0'
- MTP=0
- PP_SIZE=2
- 'VLLM_PP_LAYER_PARTITION=24,24'
- MAX_MODEL_LEN=262144
- MAX_NUM_SEQS=24
- MAX_BATCH_TOKENS=4096
- GPU_MEMORY=0.90
- PLE_SSD_OFFLOAD=1
- PLE_SSD_WORKERS=32
- PLE_SSD_CACHE_MB=16384
- PLE_SSD_IO_DEPTH=512
- PLE_SSD_PREFETCH=131072
- PLE_SSD_LIB=/opt/vllm-qfn/optimization/ple_ssd_io.so
- COMPILE_MODE=1
- GRAPH_MODE=FULL_AND_PIECEWISE
- VLLM_USE_BREAKABLE_CUDAGRAPH=1
- VLLM_PLE_NONPLE_LOADER=instanttensor
- QWEN38_ABLITERATION=
- 'CAPTURE_SIZES=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,18,20,22,24,26,28,30,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,128,192,256,512,1024,2048]'
- VLLM_MTP_DRAFT_HEAD_PATH=/opt/vllm-qfn/models/draft-heads/swift15-awq-lm_head-w4a16.safetensors

runtime

notes: 'Card order matters: CUDA_DEVICES=1,0 puts the slower (250 W VBIOS) card as PP stage 0: +9% decode, prefill equal. --prefix-match-unit 16 because with 4 draft tokens the attention block is 1616 tokens, which 32 does not divide. reasoning_effort medium is the measured optimum (xhigh costs -5.2 BFCL points). Production profile config/qfn-prod.env (night 6, 2026-11-05) is the argv authority; systemd unit config/vllm-qfn.service. INT8 KV (patch 10/K1) is validated but NOT in production (BF16 KV kept).'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-swift15-awq-2x-cmp-170hx-pp2-mtp4-ple-asgard.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-swift15-awq-2x-cmp-170hx-pp2-mtp4-ple-asgard.json

Engine

Install note: Not a pip package: a venv at /opt/vllm-qfn-n6k built by patches/make-venv.sh. Base = upstream vLLM a5a30471ff2bb7f0824f2da10e358af98d304472 (0.29.1rc1.dev402) + iIIusi0n's PLE-SSD recipe (ple_ssd_io.so helper) + 2dameneko's build at commit 56e1347, then this repo's patches 01-18 and night-6 set 19-23 (terse alias, QSA ring, QSA indexer sm80 kernels; ports of vllm#55601 #60210 #56734 #60615 #58784 #54912 #62344 #65470). Launch goes through 2dameneko's serve/serve.sh with RUN_ENV=config/qfn-prod.env (systemd unit config/vllm-qfn.service); the argv here is serve.sh's exec line assembled from that profile. Required artifacts: optimization/ple_ssd_io.so, templates/qwen-asgard-v1.jinja (leonbollerup/qwen-asgard-jinja), models/draft-heads/swift15-awq-lm_head-w4a16.safetensors (tools/models/quantize_lm_head.py), and the DotH MTP head grafted from doth4580/Qwen3.8-Flash-Next-DotH-MTP-BF16 (tools/models/doth.md). Abliteration patch 14 is not redistributed and is OFF. CUDA arch sm80 (GA100).

Model

Hardware

Context

  • max context: 256K tokens

KV pool

  • 838,024 tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
1,0248—400.2 tok/s—reported
1,02416—511 tok/s—reported
9,2161—199.6 tok/s—reported
19,4561—304 tok/s—reported
32,768—7847 tok/s——reported
131,072—8403 tok/s——reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured, single box (i7-8700K, 64 GB RAM, NVMe, unprivileged LXC, PCIe Gen2 x8, no P2P). Production checks at default clocks (cards hold ~1455-1485 MHz under load); A/B rows locked at 1440 MHz. Decode 199.6 = median of decode-N6-prod.json (9,216-token code-gen prompt, T=0.6, acceptance 3.62, results/logs/night6-NIGHT-LOG.md 01:03:55 UTC). Code-edit 304.0 at 1440 MHz (night-4/5 standard edit-D4p16.txt/edit-R.txt; night-6 re-check 304.3); not yet re-measured on the night-6 standard at default clocks. Concurrency rows from results/concurrency/night6-conc.jsonl (final build C, 1K prompts, 150 s, 0 errors; aggregate output tok/s). context_len on the code-edit row is the tuple marker 9216+10240. Quality: BFCL v4 80.3% (2,080 items), HumanEval+ 95.7%, MBPP+ 82.5%, LiveCodeBench v6 58.3%, long-context needles 32K/64K/110K = 100/96.7/95.0% - nights 2-3 suite (results/quality/); night-6 re-run BFCL 80.0% within noise. KV pool 838,024 tokens (night-6 startup log).

Downloads

Revision notes

Initial version.

Revision history

Initial version.

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)