localairecipes launch configs for local inference

Qwen3.8-Flash-Next W4A16-AutoRound DGX Spark vLLM MTP4 Iter7g

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by kyaky↗ · listed by localai-recipes

revision r4 · commit dc8982acab · updated 21 h ago · license CC-BY-4.0

chat code vision mtp-speculative nvme-ple dgx-spark gb10 sm121 vllm autoround block-verification deterministic-floor kyaky w4a16

Recipe by kyaky. Qwen3.8-Flash-Next W4A16-AutoRound hybrid (INT4 experts + FP8 side layers + INT8 head; 51B-token n-gram PLE table on NVMe mmap) on one DGX Spark (GB10, 121 GB unified memory) on a verified-deterministic vLLM floor (image qwen38-flash-dgx:iter7g-20261015 on the digest-pinned vllm/vllm-openai:qwen38-flash-next base): MoE align arrival-order shim, drafter-Marlin atomicAdd fix, pinned triton config, compile-cache mount. Measured line v19a-w5 = MTP=4 window-5 block verification: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 streams (+9.2% N1 / +4.8% N8 over MTP=3, two boots per arm, same image). Promoted serving line v20-fp8: fp8 KV cache on a 24 GB pool at native 262,144 context, 1,291,120-token capacity. Chinese +31.3% via a 25,862-id (39.5%) Han draft-vocab rebalance (prose-zhen 34.8 to 45.7 tok/s). Stochastic-correctness gate 37/37 vs bit-exact fp64 oracle; per-round raw data + scripts ship in-repo; copy-agentic-1 failure disclosed.

Launch command

podman run -d --name qwen38-flash --device /dev/nvidia0 --device /dev/nvidiactl --device /dev/nvidia-uvm --userns=keep-id -v '$HOME/gb10-podman-root:/root' --ipc=host -p 8000:8000 -v '$HOME/models/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-mtpdense-g32:/model:ro' -v '$HOME/models/ple-table-fp8:/ple-table:ro' -v '$HOME/.cache/qwen38-v16b/draft-vocab-ids-K65536.txt:/draft-vocab/ids.txt:ro' -v '$HOME/.cache/qwen38-compile:/root/.cache' -e 'TRITON_LIBCUDA_PATH=<driver lib dir>' -e VLLM_PLE_MMAP=1 -e VLLM_PLE_MMAP_WORKERS=32 -e VLLM_PLE_MMAP_PREWARM=1 -e VLLM_PLE_MMAP_MADV_RANDOM=1 -e VLLM_MOE_ALIGN_STABLESORT=1 -e VLLM_MARLIN_USE_ATOMIC_ADD=0 -e VLLM_TRITON_FORCE_FIRST_CONFIG=1 -e VLLM_FP8_HYBRID=1 -e VLLM_USE_DEEP_GEMM=0 -e VLLM_USE_FLASHINFER_SAMPLER=1 -e CUDA_LAUNCH_BLOCKING=0 -e VLLM_MTP_DRAFT_VOCAB=/draft-vocab/ids.txt -e VLLM_DRAFTER_EXPERTS_FP8=1 -e VLLM_LOAD_CLONE=1 -e VLLM_MOE_NAME_INDEX=1 -e VLLM_LOAD_EMBED_CHUNK=1 -e VLLM_QSA_DET_TOPK=1 qwen38-flash-dgx:fp8kv-promo-20261024 /model --served-model-name qwen --host 0.0.0.0 --port 8000 --load-format fastsafetensors --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.01 --enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 8192 -cc.cudagraph_mode=PIECEWISE '-cc.splitting_ops=["vllm::unified_attention_with_output","vllm::unified_mla_attention_with_output","vllm::mamba_mixer2","vllm::mamba_mixer","vllm::short_conv","vllm::qwen3_8_flash_next_ple_short_conv","vllm::qwen3_8_flash_next_qsa_with_output","vllm::linear_attention","vllm::qwen_gdn_attention_core","vllm::qwen_gdn_attention_core_fused_norm_packed","vllm::sparse_attn_indexer","vllm::ple_mmap_lookup"]' --no-enable-flashinfer-autotune --kv-cache-dtype fp8 --no-async-scheduling --kv-cache-memory-bytes 24g --enable-prompt-tokens-details --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --speculative-config '{"method":"mtp","num_speculative_tokens":4,"rejection_sample_method":"block","draft_sample_method":"probabilistic"}'

runtime

env: 'launch.sh sources recipe/config/v20-fp8/env under env -i (cleared environment); edit the env file, do not export'
notes: 'Promoted serving line v20-fp8 transcribed verbatim from recipe/config/v20-fp8/{env,launch.sh,serve.sh,image} @ 2ddeb883fe692e3eee1840655a3df7de8eddf87d: the v19a-w5 promoted script with EXACTLY two functional edits (recorded upstream in runs/2026-10-17-qsa-fp8-kv/od4/config-diff.md) -- --kv-cache-dtype auto->fp8 and --no-async-scheduling added (D-021 pin-hygiene check-6 closure; the pinned engine otherwise auto-resolves async_scheduling TRUE) -- plus KV_BYTES 16g->24g and image qwen38-flash-dgx:fp8kv-promo-20261024 (local podman tag; not built by the repo build chain). The fp8 pool normalizes to 1,291,120 KV tokens at native 262,144. launch.sh inspects the image via podman (docker cannot see the local tag, 2a3cde5); the whole serving launch path is rootless podman (65b39aa). Defaults when serve.sh is inspected directly: MTP=3, PORT 18300, TOOL_PARSER qwen3_coder - the promoted env overrides to MTP=4, port 8000, qwen3_xml. Deterministic floor unchanged: MOE_ALIGN_STABLESORT=1 + MARLIN_ATOMIC_ADD=0 + TRITON_FORCE_FIRST_CONFIG=1 + the persistent ~/.cache/qwen38-compile mount. Driver-lib set must include libnvidia-ptxjitcompiler.so.1 or engine init fails on first JIT compile (mounted by launch.sh). Requires ~130 GB NVMe for checkpoint + FP8 PLE table (separate Saren/Qwen3.8-Flash-Next-ple-table-fp8 @ 50511b0a41aa1d34b8beb7e5d4bb06a0b650dc14, served read-only at /ple-table) plus ~4.8 GiB drafter shard data. podman<5 + nvidia-ctk>=1.20 CDI additionalGids strip workaround documented in repo Troubleshooting. Published throughput numbers remain the v19a-w5 bf16-line measurement (image qwen38-flash-dgx:iter7g-20261015); the v20-fp8 stream-curve re-measurement is pending upstream.'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json

Engine

Install note: Rootless podman 4.x/5 on ARM64 GB10 with NVIDIA CDI; CUDA 13.0-pinned driver. Build chain: recipe/build/image/build.sh (iter6c->iter6d) then build-iter7g.sh on top, then recipe/build/model/build.sh T80 dense-MTP g32 drafter directory (about 4.8 GiB). Measured-line image tag qwen38-flash-dgx:iter7g-20261015; the promoted v20-fp8 serving line runs the local podman tag qwen38-flash-dgx:fp8kv-promo-20261024 (fp8 KV read path + #59528 null-block backport; not built by the repo build chain). launch.sh enforces a fixed image tag and inspects the image via podman. Deterministic floor needs the env knobs MOE_ALIGN_STABLESORT=1, MARLIN_ATOMIC_ADD=0, TRITON_FORCE_FIRST_CONFIG=1 + the persistent ~/.cache/qwen38-compile mount, all sourced from recipe/config/v20-fp8/env (measured line: recipe/config/v19a-w5/env) under a cleared environment by launch.sh.

Model

Hardware

Context

  • max context: 256K tokens

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
262,1441—76.8 tok/s—reported
262,1442—117.7 tok/s—reported
262,1444—179.9 tok/s—reported
262,1446—224.1 tok/s—reported
262,1448—261.2 tok/s—reported
272,3841—45.7 tok/s—reported
272,3844—62.1 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured 2026-10-15 (docs/benchmarks/2026-10-15-w54-ab/*.json @ 0d346970, structured records + boot fingerprints): copy-heavy workload, 9,600-token shared cached prefix, peak = fastest of 3 rounds per stream count, completion tokens/s over the all-streams-decoding window. v19a-w5 peak N1 76.8 (boot-matched median 76.7); aggregate N8 peak 265.9, boot-matched median 261.2 (+4.79% boot-matched / +8.43% boot2-matched / +6.58% median-of-medians vs stock; cross-boot N1 spread <0.3%). Two boots per arm (w5, w4) on the same iter7g image; stock v16b anchor on iter6d. Decode step 52.33 ms agent-shaped; warm TTFT 0.57 s.

Downloads

Revision notes

Ingested from GitHub repo kyaky/qwen3.8-flash-next-gb10-ultrafast @ 0d3469706696015992959a12b059b02b75100744 (lar-scout ingest, kanban t_296a1f5d, 2026-10-16). Recipe Apache-2.0. Attribution: recipe by kyaky; submitter/listed_by localai-recipes. One recorded failure disclosed by the author: copy-agentic-1 fails at rep 2 on three independent W5 boots (attribution unresolved, reproduction queued; W4 0/30). Quality on same boots: code-edit 25/25 verified, MGSM 16/16 both arms, fixed-suite 93.1-93.3% on the v16b line. First live run of the owner-shaped tool-loop suite scored 29/50 (author-baselined observation, not an acceptance gate). perf note: N8 headline reported in all three accountings per README. Pin refreshed 2026-10-28 to 2ddeb883fe692e3eee1840655a3df7de8eddf87d (verified live origin/main HEAD via git ls-remote; kanban t_c3370f04, owner go t_eab4d745, preprod only). Pin honesty: the previous pin 0d3469706696015992959a12b059b02b75100744 was ORPHANED by an upstream history rewrite (sha still reachable via the GitHub API with message 'benchmarks: the stateful reproduction ran...'; absent from all refs after a full fetch), so no ancestry diff exists old-pin..new-pin - the delta was built via GitHub API per-file compare, not git log. Launch re-transcribed from recipe/config/v20-fp8/{env,launch.sh,serve.sh,image} @ 2ddeb883fe: serving line promoted v19a-w5->v20-fp8 (55199a4/2ddeb88) - image qwen38-flash-dgx:fp8kv-promo-20261024, --kv-cache-dtype auto->fp8, --kv-cache-memory-bytes 16g->24g (fp8 KV native 262k, 1,291,120-token capacity), +--no-async-scheduling (D-021 check-6 closure); launch path is rootless podman (65b39aa; launcher inspects via podman, 2a3cde5). Throughput record stays attributed to the measured v19a-w5 bf16 line; all 7 perf points preserved 7/7 unchanged, still grounded in docs/benchmarks/2026-10-15-w54-ab/*.json (unchanged at the pin) and the 37/37 gate claim in README + recipe/build/image/tests/blockverifier_gate_results.json (verdict gate_passed; results file unchanged at the pin, test file touched only by privacy-scan commits). Privacy: the site publishes no privacy/telemetry claim for this slug, so the upstream removal of the CI privacy grep step (98bc1c2, owner completed privacy review, donor-pattern false positives) removes no site claim. Hero: repo now ships docs/assets/hero-banner-v8.png + .license (Apache-2.0, 26f3d32); the site embeds no hero image for this slug - recorded, no site change. CI-only files (.shellcheckrc, .github/workflows/checks.yml, AGENTS.md): no site impact.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)