{"slug":"qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g","title":"Qwen3.8-Flash-Next W4A16-AutoRound DGX Spark vLLM MTP4 Iter7g","summary":"Recipe by kyaky. Qwen3.8-Flash-Next W4A16-AutoRound hybrid (INT4 experts + FP8 side layers + INT8 head; 51B-token n-gram PLE table on NVMe mmap) on one DGX Spark (GB10, 121 GB unified memory) on a verified-deterministic vLLM floor (image qwen38-flash-dgx:iter7g-20261015 on the digest-pinned vllm/vllm-openai:qwen38-flash-next base): MoE align arrival-order shim, drafter-Marlin atomicAdd fix, pinned triton config, compile-cache mount. Measured line v19a-w5 = MTP=4 window-5 block verification: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 streams (+9.2% N1 / +4.8% N8 over MTP=3, two boots per arm, same image). Promoted serving line v20-fp8: fp8 KV cache on a 24 GB pool at native 262,144 context, 1,291,120-token capacity. Chinese +31.3% via a 25,862-id (39.5%) Han draft-vocab rebalance (prose-zhen 34.8 to 45.7 tok/s). Stochastic-correctness gate 37/37 vs bit-exact fp64 oracle; per-round raw data + scripts ship in-repo; copy-agentic-1 failure disclosed.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":4,"commit_sha":"dc8982acabf5e6b6417f4871ad5149d61f01755d","content_hash":"e6a2cfdf4903c7ca1b80df3d0f496d3ca3f13840","created_at":"2026-10-29 21:32:28.107205-06","notes":"Ingested from GitHub repo kyaky/qwen3.8-flash-next-gb10-ultrafast @ 0d3469706696015992959a12b059b02b75100744 (lar-scout ingest, kanban t_296a1f5d, 2026-10-16). Recipe Apache-2.0. Attribution: recipe by kyaky; submitter/listed_by localai-recipes. One recorded failure disclosed by the author: copy-agentic-1 fails at rep 2 on three independent W5 boots (attribution unresolved, reproduction queued; W4 0/30). Quality on same boots: code-edit 25/25 verified, MGSM 16/16 both arms, fixed-suite 93.1-93.3% on the v16b line. First live run of the owner-shaped tool-loop suite scored 29/50 (author-baselined observation, not an acceptance gate). perf note: N8 headline reported in all three accountings per README. Pin refreshed 2026-10-28 to 2ddeb883fe692e3eee1840655a3df7de8eddf87d (verified live origin/main HEAD via git ls-remote; kanban t_c3370f04, owner go t_eab4d745, preprod only). Pin honesty: the previous pin 0d3469706696015992959a12b059b02b75100744 was ORPHANED by an upstream history rewrite (sha still reachable via the GitHub API with message 'benchmarks: the stateful reproduction ran...'; absent from all refs after a full fetch), so no ancestry diff exists old-pin..new-pin - the delta was built via GitHub API per-file compare, not git log. Launch re-transcribed from recipe/config/v20-fp8/{env,launch.sh,serve.sh,image} @ 2ddeb883fe: serving line promoted v19a-w5->v20-fp8 (55199a4/2ddeb88) - image qwen38-flash-dgx:fp8kv-promo-20261024, --kv-cache-dtype auto->fp8, --kv-cache-memory-bytes 16g->24g (fp8 KV native 262k, 1,291,120-token capacity), +--no-async-scheduling (D-021 check-6 closure); launch path is rootless podman (65b39aa; launcher inspects via podman, 2a3cde5). Throughput record stays attributed to the measured v19a-w5 bf16 line; all 7 perf points preserved 7/7 unchanged, still grounded in docs/benchmarks/2026-10-15-w54-ab/*.json (unchanged at the pin) and the 37/37 gate claim in README + recipe/build/image/tests/blockverifier_gate_results.json (verdict gate_passed; results file unchanged at the pin, test file touched only by privacy-scan commits). Privacy: the site publishes no privacy/telemetry claim for this slug, so the upstream removal of the CI privacy grep step (98bc1c2, owner completed privacy review, donor-pattern false positives) removes no site claim. Hero: repo now ships docs/assets/hero-banner-v8.png + .license (Apache-2.0, 26f3d32); the site embeds no hero image for this slug - recorded, no site change. CI-only files (.shellcheckrc, .github/workflows/checks.yml, AGENTS.md): no site impact."},"engine_source":{"type":"fork","git_url":"https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast","base_engine":"vllm","pip_package":null,"base_version":"0.1.dev20073 (qwen38-flash-next, iter7g floor)","image_digest":"docker.io/vllm/vllm-openai@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8","install_note":"Rootless podman 4.x/5 on ARM64 GB10 with NVIDIA CDI; CUDA 13.0-pinned driver. Build chain: recipe/build/image/build.sh (iter6c->iter6d) then build-iter7g.sh on top, then recipe/build/model/build.sh T80 dense-MTP g32 drafter directory (about 4.8 GiB). Measured-line image tag qwen38-flash-dgx:iter7g-20261015; the promoted v20-fp8 serving line runs the local podman tag qwen38-flash-dgx:fp8kv-promo-20261024 (fp8 KV read path + #59528 null-block backport; not built by the repo build chain). launch.sh enforces a fixed image tag and inspects the image via podman. Deterministic floor needs the env knobs MOE_ALIGN_STABLESORT=1, MARLIN_ATOMIC_ADD=0, TRITON_FORCE_FIRST_CONFIG=1 + the persistent ~/.cache/qwen38-compile mount, all sourced from recipe/config/v20-fp8/env (measured line: recipe/config/v19a-w5/env) under a cleared environment by launch.sh.","pinned_commit":"2ddeb883fe692e3eee1840655a3df7de8eddf87d"},"credit":[{"name":"kyaky","url":"https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["podman","run","-d","--name","qwen38-flash","--device","/dev/nvidia0","--device","/dev/nvidiactl","--device","/dev/nvidia-uvm","--userns=keep-id","-v","$HOME/gb10-podman-root:/root","--ipc=host","-p","8000:8000","-v","$HOME/models/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-mtpdense-g32:/model:ro","-v","$HOME/models/ple-table-fp8:/ple-table:ro","-v","$HOME/.cache/qwen38-v16b/draft-vocab-ids-K65536.txt:/draft-vocab/ids.txt:ro","-v","$HOME/.cache/qwen38-compile:/root/.cache","-e","TRITON_LIBCUDA_PATH=<driver lib dir>","-e","VLLM_PLE_MMAP=1","-e","VLLM_PLE_MMAP_WORKERS=32","-e","VLLM_PLE_MMAP_PREWARM=1","-e","VLLM_PLE_MMAP_MADV_RANDOM=1","-e","VLLM_MOE_ALIGN_STABLESORT=1","-e","VLLM_MARLIN_USE_ATOMIC_ADD=0","-e","VLLM_TRITON_FORCE_FIRST_CONFIG=1","-e","VLLM_FP8_HYBRID=1","-e","VLLM_USE_DEEP_GEMM=0","-e","VLLM_USE_FLASHINFER_SAMPLER=1","-e","CUDA_LAUNCH_BLOCKING=0","-e","VLLM_MTP_DRAFT_VOCAB=/draft-vocab/ids.txt","-e","VLLM_DRAFTER_EXPERTS_FP8=1","-e","VLLM_LOAD_CLONE=1","-e","VLLM_MOE_NAME_INDEX=1","-e","VLLM_LOAD_EMBED_CHUNK=1","-e","VLLM_QSA_DET_TOPK=1","qwen38-flash-dgx:fp8kv-promo-20261024","/model","--served-model-name","qwen","--host","0.0.0.0","--port","8000","--load-format","fastsafetensors","--max-model-len","262144","--max-num-seqs","8","--gpu-memory-utilization","0.01","--enable-prefix-caching","--enable-chunked-prefill","--max-num-batched-tokens","8192","-cc.cudagraph_mode=PIECEWISE","-cc.splitting_ops=[\"vllm::unified_attention_with_output\",\"vllm::unified_mla_attention_with_output\",\"vllm::mamba_mixer2\",\"vllm::mamba_mixer\",\"vllm::short_conv\",\"vllm::qwen3_8_flash_next_ple_short_conv\",\"vllm::qwen3_8_flash_next_qsa_with_output\",\"vllm::linear_attention\",\"vllm::qwen_gdn_attention_core\",\"vllm::qwen_gdn_attention_core_fused_norm_packed\",\"vllm::sparse_attn_indexer\",\"vllm::ple_mmap_lookup\"]","--no-enable-flashinfer-autotune","--kv-cache-dtype","fp8","--no-async-scheduling","--kv-cache-memory-bytes","24g","--enable-prompt-tokens-details","--enable-auto-tool-choice","--tool-call-parser","qwen3_xml","--reasoning-parser","qwen3","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":4,\"rejection_sample_method\":\"block\",\"draft_sample_method\":\"probabilistic\"}"],"runtime":{"env":"launch.sh sources recipe/config/v20-fp8/env under env -i (cleared environment); edit the env file, do not export","notes":"Promoted serving line v20-fp8 transcribed verbatim from recipe/config/v20-fp8/{env,launch.sh,serve.sh,image} @ 2ddeb883fe692e3eee1840655a3df7de8eddf87d: the v19a-w5 promoted script with EXACTLY two functional edits (recorded upstream in runs/2026-10-17-qsa-fp8-kv/od4/config-diff.md) -- --kv-cache-dtype auto->fp8 and --no-async-scheduling added (D-021 pin-hygiene check-6 closure; the pinned engine otherwise auto-resolves async_scheduling TRUE) -- plus KV_BYTES 16g->24g and image qwen38-flash-dgx:fp8kv-promo-20261024 (local podman tag; not built by the repo build chain). The fp8 pool normalizes to 1,291,120 KV tokens at native 262,144. launch.sh inspects the image via podman (docker cannot see the local tag, 2a3cde5); the whole serving launch path is rootless podman (65b39aa). Defaults when serve.sh is inspected directly: MTP=3, PORT 18300, TOOL_PARSER qwen3_coder - the promoted env overrides to MTP=4, port 8000, qwen3_xml. Deterministic floor unchanged: MOE_ALIGN_STABLESORT=1 + MARLIN_ATOMIC_ADD=0 + TRITON_FORCE_FIRST_CONFIG=1 + the persistent ~/.cache/qwen38-compile mount. Driver-lib set must include libnvidia-ptxjitcompiler.so.1 or engine init fails on first JIT compile (mounted by launch.sh). Requires ~130 GB NVMe for checkpoint + FP8 PLE table (separate Saren/Qwen3.8-Flash-Next-ple-table-fp8 @ 50511b0a41aa1d34b8beb7e5d4bb06a0b650dc14, served read-only at /ple-table) plus ~4.8 GiB drafter shard data. podman<5 + nvidia-ctk>=1.20 CDI additionalGids strip workaround documented in repo Troubleshooting. Published throughput numbers remain the v19a-w5 bf16-line measurement (image qwen38-flash-dgx:iter7g-20261015); the v20-fp8 stream-curve re-measurement is pending upstream."},"max_context_len":262144,"storage_requirements":{"note":"~130 GB NVMe: checkpoint + FP8 PLE table plus ~4.8 GiB drafter shard data (recipe notes state the requirement)","nvme_gb":130}},"models":[{"slug":"qwen3-8-flash-next-w4a16-autoround-hybrid","name":"Qwen3.8-Flash-Next-W4A16-AutoRound-Hybrid","hf_repo":"Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid","quant":"w4a16-autoround"}],"hardware":[{"slug":"dgx-spark","vendor":"nvidia","card":"DGX Spark (GB10, unified memory)","card_family":"dgx-spark","vram_gb":128,"device_count":1}],"generation":"blackwell","tags":["autoround","block-verification","deterministic-floor","dgx-spark","gb10","kyaky","mtp-speculative","nvme-ple","sm121","vllm","w4a16"],"tasks":["chat","code","vision"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured 2026-10-15 (docs/benchmarks/2026-10-15-w54-ab/*.json @ 0d346970, structured records + boot fingerprints): copy-heavy workload, 9,600-token shared cached prefix, peak = fastest of 3 rounds per stream count, completion tokens/s over the all-streams-decoding window. v19a-w5 peak N1 76.8 (boot-matched median 76.7); aggregate N8 peak 265.9, boot-matched median 261.2 (+4.79% boot-matched / +8.43% boot2-matched / +6.58% median-of-medians vs stock; cross-boot N1 spread <0.3%). Two boots per arm (w5, w4) on the same iter7g image; stock v16b anchor on iter6d. Decode step 52.33 ms agent-shaped; warm TTFT 0.57 s.\", \"tok_s\": 76.7999999999999971578290569595992565155029296875, \"points\": [{\"notes\": \"w54-ab copy-v19a-w5: peak 76.8 (median 76.7) single-stream; +9.2% vs MTP=3 (70.2) same floor\", \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 76.7999999999999971578290569595992565155029296875}, {\"notes\": \"w54-ab copy-v19a-w5: boot-matched median 261.2 aggregate at 8 streams (peak round 265.9); +4.8% vs MTP=3 (249.3)\", \"concurrency\": 8, \"context_len\": 262144, \"decode_tok_s\": 261.19999999999998863131622783839702606201171875}, {\"notes\": \"w54-ab copy-v19a-w5: peak 117.7 (median 114.8)\", \"concurrency\": 2, \"context_len\": 262144, \"decode_tok_s\": 117.7000000000000028421709430404007434844970703125}, {\"notes\": \"w54-ab copy-v19a-w5: peak 179.9 (median 175.2)\", \"concurrency\": 4, \"context_len\": 262144, \"decode_tok_s\": 179.900000000000005684341886080801486968994140625}, {\"notes\": \"w54-ab copy-v19a-w5: peak 224.1 (median 218.5)\", \"concurrency\": 6, \"context_len\": 262144, \"decode_tok_s\": 224.099999999999994315658113919198513031005859375}, {\"notes\": \"real-code-edit suite: 25/25 verified file-edit tasks, median decode 62.1 tok/s (agentic shape, distinct workload - ctx 262144 + 10240 marker offset, not a second ctx setting)\", \"concurrency\": 4, \"context_len\": 272384, \"decode_tok_s\": 62.10000000000000142108547152020037174224853515625}, {\"notes\": \"real-prose-zhen, W4 window + mixed 39.5% Han draft vocab: 45.7 tok/s (+31.3% over stock-vocab 34.8) at equal verified quality (MGSM zh 16/16); ctx 262144 + 10240 tuple marker for the zh lane\", \"concurrency\": 1, \"context_len\": 272384, \"decode_tok_s\": 45.7000000000000028421709430404007434844970703125}], \"context_len\": 262144}"}],"perf_points":[{"id":2131,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":76.7999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"w54-ab copy-v19a-w5: peak 76.8 (median 76.7) single-stream; +9.2% vs MTP=3 (70.2) same floor"},{"id":2132,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":117.7000000000000028421709430404007434844970703125,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"w54-ab copy-v19a-w5: peak 117.7 (median 114.8)"},{"id":2133,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":179.900000000000005684341886080801486968994140625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"w54-ab copy-v19a-w5: peak 179.9 (median 175.2)"},{"id":2134,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":224.099999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":6,"claimed_by":"submitter","notes":"w54-ab copy-v19a-w5: peak 224.1 (median 218.5)"},{"id":2135,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":261.19999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"w54-ab copy-v19a-w5: boot-matched median 261.2 aggregate at 8 streams (peak round 265.9); +4.8% vs MTP=3 (249.3)"},{"id":2136,"context_len":272384,"prefill_tok_s":null,"decode_tok_s":45.7000000000000028421709430404007434844970703125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"real-prose-zhen, W4 window + mixed 39.5% Han draft vocab: 45.7 tok/s (+31.3% over stock-vocab 34.8) at equal verified quality (MGSM zh 16/16); ctx 262144 + 10240 tuple marker for the zh lane"},{"id":2137,"context_len":272384,"prefill_tok_s":null,"decode_tok_s":62.10000000000000142108547152020037174224853515625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"real-code-edit suite: 25/25 verified file-edit tasks, median decode 62.1 tok/s (agentic shape, distinct workload - ctx 262144 + 10240 marker offset, not a second ctx setting)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid","pin":"revision 8b82f0b7abe3d1150a7827d298c75e86267636ae (W4A16-AutoRound hybrid: INT4 experts + FP8 side layers + INT8 head)"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/Saren/Qwen3.8-Flash-Next-ple-table-fp8","pin":"revision 50511b0a41aa1d34b8beb7e5d4bb06a0b650dc14 (FP8 PLE table, 51B-token n-gram index, NVMe mmap at /ple-table)"},{"kind":"drafter","pointer_type":"url","pointer":"https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast/blob/main/recipe/build/model/build.sh","pin":"T80 dense-MTP g32 drafter directory built by recipe/build/model/build.sh (nine bf16 drafter modules at group size 32, ~4.8 GiB new bytes, reference report in repo)"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/vllm/vllm-openai","pin":"base vllm/vllm-openai @ sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8 -> measured image qwen38-flash-dgx:iter7g-20261015; promoted v20-fp8 image qwen38-flash-dgx:fp8kv-promo-20261024"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast","pin":"2ddeb883fe... - recipe/config/v20-fp8 promoted fp8 line + v19a-w5 measured line, build chain iter6c..iter7g, docs/benchmarks/2026-10-15-w54-ab raw JSON + boot fingerprints"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json","alternate":"/r/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:28.107205-06","created_at":"2026-10-20 21:49:19.081118-06","speeds":{"decode_tok_s":61.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}