diff --git a/recipes/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data/recipe.yaml b/recipes/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data/recipe.yaml index 68211b741..7e5daf390 100644 --- a/recipes/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data/recipe.yaml +++ b/recipes/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data/recipe.yaml @@ -2,13 +2,13 @@ slug: qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data title: 'Qwen3.8-Flash-Next NVFP4 1x DGX Spark vLLM 71.3 c1 (salient-data)' engine: name: vllm - version: 'nightly build 2026-10-14-77154ca6 (vLLM 77154ca6 + 14 patches)' + version: 'nightly build 2026-10-22-8e2beaa3 (vLLM 8e2beaa3 + 14 patches)' source_type: docker_image git_url: 'https://github.com/salient-data/qwen38-next-dgx-recipe' - pinned_commit: f309019de10d55774a579d42f71be67b2636ce8e - image_digest: 'ghcr.io/salient-data/qwen38-next-dgx@sha256:3d67df7b7b7d25a5da64f62c0f87f6109062584261fe77157f6d37de178d10b2' - install_note: 'git clone https://github.com/salient-data/qwen38-next-dgx-recipe && cd qwen38-next-dgx-recipe && ./recipe/preflight.sh (read-only host checks; --verify hashes every checkpoint file against the hub) && ./recipe/download.sh (hf download at the pinned revision + model-files.sha256 verify) && ./recipe/build.sh (pulls the digest-pinned base vllm/vllm-openai@sha256:a7deff2c..., ~10 GB DL / ~25 GB disk, applies the 14 patches with patch --fuzz=0) && ./recipe/launch.sh (--print shows the exact docker command; foreground load ~5-6 min). ./recipe/qfn up = prod path (pulls PROD_IMAGE by registry digest, cosign-verified if installed); ./recipe/qfn dev = rebuild from the published patches and serve in-terminal (AST-equivalence proof in results//dev-path-equivalence.json). Base image pin: vllm/vllm-openai@sha256:a7deff2c216def98c193352aca50fb03a2ee29231355b3033dc3789009f6d23e (vLLM 0.32.0rc1.dev100+g77154ca61, torch 2.13.0+cu130, FlashInfer 0.7.0.post1).' -summary: "Qwen3.8-Flash-Next (~180B multimodal MoE, NVFP4) on one NVIDIA DGX Spark (GB10, 121 GiB unified memory) with a pinned vLLM nightly image + 14 provenance-tracked patches (patch --fuzz=0) + one docker run, fitting the 51.2 GB PLE n-gram table by never making it resident (rows read per forward-pass step from the checkpoint file). 71.3 tok/s single-stream median decode (runs 69.2/71.3/71.5), 42.6 prose median, 84.4 tok/s 6-stream aggregate (one run), GSM8K 99/100 on a fixed 100-problem subset (temp 0.6 top-p 0.95 thinking on), 12/12 arithmetic + 131K needle pass + prefix-cache canary pass, prefix caching ON. Measured with tonyd2wild's bench_categories.py @ 6ad1c8f unchanged; re-measured 2026-10-15 on the serving build (71.8/42.8/84.0). Prod/dev split: prod image digest-pinned + cosign-signed + SPDX SBOM; dev rebuilds the same serving code from published patches with an AST-equivalence proof. Outside the model card's support statement (B200/B300 listed, GB10 unlisted)." + pinned_commit: 7432036e7d549ed266d16832c43ea960f2cb7d55 + image_digest: 'ghcr.io/salient-data/qwen38-next-dgx@sha256:d55a2740651064f6c8aba20bc9ada104ae9df0a793cc87d0c170d312aa052b4d' + install_note: 'git clone https://github.com/salient-data/qwen38-next-dgx-recipe && cd qwen38-next-dgx-recipe && ./recipe/preflight.sh (read-only host checks; --verify hashes every checkpoint file against the hub) && ./recipe/download.sh (hf download at the pinned revision + model-files.sha256 verify) && ./recipe/build.sh (pulls the digest-pinned base vllm/vllm-openai@sha256:167ccd86..., ~10 GB DL / ~25 GB disk, applies the 14 patches with patch --fuzz=0) && ./recipe/launch.sh (--print shows the exact docker command; foreground load ~6 min, 350-371 s on the prod boots). ./recipe/qfn up = prod path (pulls PROD_IMAGE by registry digest, cosign-verified if installed); ./recipe/qfn dev = rebuild from the published patches and serve in-terminal (AST-equivalence proof in results//dev-path-equivalence.json). Base image pin: vllm/vllm-openai@sha256:167ccd8696bc09f1b63301185134e04b883623f8adcbd8e3a0ec3140a193abcf (vLLM 0.32.1rc1.dev197+g8e2beaa3b, torch 2.14.0+cu130, FlashInfer 0.7.2).' +summary: "Qwen3.8-Flash-Next (~180B multimodal MoE, NVFP4) on one NVIDIA DGX Spark (GB10, 121 GiB unified memory) with a pinned vLLM nightly image + 14 provenance-tracked patches (patch --fuzz=0) + one docker run, fitting the 51.2 GB PLE n-gram table by never making it resident (rows read per forward-pass step from the checkpoint file). 69.3 tok/s single-stream median decode (runs 66.8/69.5/69.3), 41.0 prose median, 91.9 tok/s 6-stream aggregate (1 run), GSM8K 99/100 on a fixed 100-problem subset (temp 0.6 top-p 0.95 thinking on), 12/12 arithmetic + 131K needle pass + prefix-cache canary pass, prefix caching ON. Measured with tonyd2wild's bench_categories.py @ 6ad1c8f unchanged on build 2026-10-22-8e2beaa3; clean first-time run 2026-10-25 re-measured 69.8/41.5/82.6. Prod/dev: prod image digest-pinned + cosign-signed + SPDX SBOM; dev rebuilds the same serving code from published patches with an AST-equivalence proof. Outside the model card's support statement (B200/B300 listed, GB10 unlisted)." license: Apache-2.0 model: name: Qwen3.8-Flash-Next-NVFP4 @@ -45,7 +45,7 @@ storage_requirements: layouts: - tp max_context_len: 262144 -kv_pool_tokens: 981604 +kv_pool_tokens: 1002985 config: launch: - docker @@ -83,14 +83,14 @@ config: - '-e' - VLLM_PLE_FILE_GATHER_WORKERS=64 - '-e' - - QWEN4EXP_DRAFT_VOCAB=0 + - QWEN4EXP_DRAFT_VOCAB=65536 - '-e' - VLLM_USE_V2_MODEL_RUNNER=1 - '-e' - 'QWEN38_FP8_DENSE=gdn,attn,shared,mtp,lm_head' - '--entrypoint' - vllm - - 'ghcr.io/salient-data/qwen38-next-dgx@sha256:3d67df7b7b7d25a5da64f62c0f87f6109062584261fe77157f6d37de178d10b2' + - 'ghcr.io/salient-data/qwen38-next-dgx@sha256:d55a2740651064f6c8aba20bc9ada104ae9df0a793cc87d0c170d312aa052b4d' - serve - /hfrepo/snapshots/fc694b54fb0174e0913e6adf86691ef85a4ead47 - '--host' @@ -133,27 +133,27 @@ config: - bfloat16 - '--enable-prefix-caching' - '--speculative-config' - - '{"method": "mtp", "num_speculative_tokens": 6, "draft_confidence_threshold": 0.7, "draft_confidence_fallback_depth": 3, "draft_token_map": "/root/.cache/draft-token-maps/d22ff5232d12f6935add32541a2a447976349217e5b125669fab0e92e3e2a20b.json", "draft_token_map_dynamic_rows": 16384, "draft_token_map_dynamic_rank": 256, "draft_token_map_quantization": "nvfp4"}' + - '{"method": "mtp", "num_speculative_tokens": 6, "draft_lm_head_quantization": "nvfp4", "draft_confidence_threshold": 0.7, "draft_confidence_fallback_depth": 3, "draft_token_map": "/root/.cache/draft-token-maps/d22ff5232d12f6935add32541a2a447976349217e5b125669fab0e92e3e2a20b.json", "draft_token_map_dynamic_rows": 16384, "draft_token_map_dynamic_rank": 256, "draft_token_map_quantization": "nvfp4", "draft_token_map_max_rows": 1}' runtime: env: 'Same vllm serve args (recipe/serve.args + recipe/serve.env) for the prod and dev paths; launch.sh adds --host/--port from BIND/PORT env (127.0.0.1:8000 default, BIND=0.0.0.0 exposes unauthenticated unless VLLM_API_KEY is set). Drafting: up to 6 tokens single-stream with early stop when the drafter top-1 probability falls below 0.7 (draft_confidence_fallback_depth 3 with 2-6 concurrent requests). Dense BF16 layers cast to FP8 block-128 (197 linears, QWEN38_FP8_DENSE=gdn,attn,shared,mtp,lm_head). Draft head: context-aware sparse head (32,766 fixed token ids + 16,384 rows chosen per context, NVFP4; token map d22ff523... copied into the cache mount by launch.sh). MTP-layer KV BF16. Prefix caching ON (shortens TTFT; author estimates at most 1-2% of the 6-stream aggregate). --max-num-seqs 6.' - notes: "Launch argv transcribed from recipe/serve.args + serve.env + launch.sh @ f309019de1 (identity.json in results/2026-10-14-77154ca6/ records the measured run's exact args/env). launch.sh also: warms the server with two known-answer requests so first real answers are not slowed by kernel compilation (WARMUP=0 skips), runs a memory watch sampling every 10 s that warns when any other process grows 2 GiB+ (MEMWATCH=0 skips), and verifies after load (./recipe/verify.sh; known-answer checks battery). Thinking off by default; per-request enable_thinking=true. -- overrides: ./recipe/launch.sh -- --flag value changes one run's args. API is a standard vllm serve OpenAI endpoint. Results dir per build results/2026-10-14-77154ca6/ (harness raw + quality records); tools for the fixed checks not yet published." + notes: "Launch argv transcribed from recipe/serve.args + serve.env + launch.sh @ 7432036e7d (identity.json in results/2026-10-22-8e2beaa3/ records the measured run's exact args/env). launch.sh also: warms the server with two known-answer requests so first real answers are not slowed by kernel compilation (WARMUP=0 skips), runs a memory watch sampling every 10 s that warns when any other process grows 2 GiB+ (MEMWATCH=0 skips), and verifies after load (./recipe/verify.sh; known-answer checks battery). Thinking off by default; per-request enable_thinking=true. -- overrides: ./recipe/launch.sh -- --flag value changes one run's args. API is a standard vllm serve OpenAI endpoint. Results dir per build results/2026-10-22-8e2beaa3/ (harness raw + quality records, incl. harness-confirm/ and the 2026-10-25 harness-clean/ first-time run with gsm8k.json and boots.txt); tools for the fixed checks not yet published." performance: claimed_by: submitter metrics: - tok_s: 71.3 + tok_s: 69.3 context_len: 131072 - notes: "claimed_by=submitter T0 (author's bench, tonyd2wild bench_categories.py @ 6ad1c8f unchanged, sha256-checked; build 2026-10-14-77154ca6, measured 2026-10-14): 1-stream median decode 71.3 tok/s over 40 prompts (69.2/71.3/71.5), prose median 42.6 (41.4/42.6/42.6), 6-stream median batch aggregate 84.4 (ONE run), auto-score 0.824/0.842 (1/6), GSM8K 99/100 fixed subset (temp 0.6 top-p 0.95 thinking on), 12/12 arithmetic, 9/10 ten-tasks, ~131K needle pass, prefix-cache canary pass. Re-measured 2026-10-15: 71.8 (69.8/71.8/71.9) / 42.8 / 84.0. Comparator rows are tonyd2wild's PUBLISHED figures (different Spark/date, revision unstated): 43.9 c1 / 29.0 prose / 68.8 c6. Declared differences: PLE rows read per-step vs staged gather; draft 6 w/ early-stop vs 3; dense BF16->FP8 cast; sparse draft head vs theirs; prefix caching on vs off. Author reran tonyd2wild's recipe 2026-10-03: 42.6/28.4/64.2. Bench prompts are short; 131072 is the needle-verified depth (server accepts 262144)." + notes: "claimed_by=submitter T0 (author's bench, tonyd2wild bench_categories.py @ 6ad1c8f unchanged, sha256-checked; build 2026-10-22-8e2beaa3, measured 2026-10-22 on one DGX Spark, server bound to localhost, nothing else running): 1-stream median decode 69.3 tok/s over 40 prompts (runs 66.8/69.5/69.3), prose median 41.0 (39.7/41.1/41.0), 6-stream median batch aggregate 91.9 (ONE run), 6-stream total output over the run 85.1, harness auto-score 0.876/0.86 (1/6 streams), GSM8K 99/100 fixed subset (temp 0.6 top-p 0.95 thinking on, Wilson 95% 94.55-99.82), 12/12 arithmetic, 9/10 ten-tasks, ~131K needle pass (20/20 depths), prefix-cache canary pass (16/16 residues, 0 failures). Re-measured 2026-10-25 fresh boot same image: 69.3 (67.2/69.4/69.3) / 40.5 / 92.4 / 87.9; clean first-time run 2026-10-25 (host rebooted, fresh clone): 69.8 (68.5/69.8/70.0) / 41.5 / 82.6 / 81.4; three more 6-stream runs on a second boot that evening: 88.2/83.5, 90.3/85.7, 90.3/85.3. Raw records: results/2026-10-22-8e2beaa3/ (harness/ byte-identical to the run, lane field 'nightly'; harness-confirm/ second measurement; harness-clean/ clean run + gsm8k.json 99/100 + boots.txt; quality.json + quality/ raw records; identity.json build identity). KV pool on the 2026-10-22 boot: 1,088,467 tokens (aggregate capacity, not one request's context). First 6-stream run after a server start measured slower both times (clean-run batch 1 76.6 vs 96-101 warm; headline fell 10%, total output 4%). Comparator rows are tonyd2wild's PUBLISHED figures (different Spark/date, revision unstated): 43.9 c1 / 29.0 prose / 68.8 c6. Declared differences: PLE rows read per-step vs staged gather; draft 6 w/ early-stop vs 3; dense BF16->FP8 cast; sparse draft head vs theirs; prefix caching on vs off. Author reran tonyd2wild's recipe 2026-10-03: 42.6/28.4/64.2. Bench prompts are short; 131072 is the needle-verified depth (server accepts 262144)." points: - context_len: 131072 concurrency: 1 - decode_tok_s: 71.3 - notes: 'median decode over 40 prompts, 3 runs 69.2/71.3/71.5; prose median 42.6 (41.4/42.6/42.6)' + decode_tok_s: 69.3 + notes: 'median decode over 40 prompts, 3 runs 66.8/69.5/69.3; prose median 41.0 (39.7/41.1/41.0); build 2026-10-22-8e2beaa3 measured 2026-10-22' - context_len: 131072 concurrency: 6 - decode_tok_s: 84.4 - notes: 'median batch aggregate, single run (no noise floor stated); includes TTFT' + decode_tok_s: 91.9 + notes: 'median batch aggregate, single run (no noise floor stated); includes TTFT; total output over the run 85.1; build 2026-10-22-8e2beaa3' artifacts: - kind: model_weights @@ -163,13 +163,13 @@ artifacts: - kind: engine_image pointer_type: docker_digest - pointer: 'https://ghcr.io/salient-data/qwen38-next-dgx@sha256:3d67df7b7b7d25a5da64f62c0f87f6109062584261fe77157f6d37de178d10b2' - pin: 'PROD_IMAGE in recipe/pins.env (registry digest, cosign-signed, SPDX SBOM); image id sha256:320ab0f7dff536d4422a0bfdecc49b45d41d9a06cbccc7dcd1b1067f703d2dd4' + pointer: 'https://ghcr.io/salient-data/qwen38-next-dgx@sha256:d55a2740651064f6c8aba20bc9ada104ae9df0a793cc87d0c170d312aa052b4d' + pin: 'PROD_IMAGE in recipe/pins.env @ 7432036e7d (registry digest, cosign-signed, SPDX SBOM); image id sha256:3eda66c8163064d664465a803ca09fb66ab626814b6bba7e58a6fd97a3cde282' - kind: reference pointer_type: git_commit pointer: 'https://github.com/salient-data/qwen38-next-dgx-recipe' - pin: 'HEAD f309019de10d55774a579d42f71be67b2636ce8e (main, pushed 2026-10-16) at sweep 2026-10-22; tag build-2026-10-08-81198e97; README verified-date 2026-10-14; 14 patches in recipe/patches/PROVENANCE.md' + pin: 'HEAD 7432036e7d549ed266d16832c43ea960f2cb7d55 (main, 2026-10-25) at revision 2026-10-28; tag build-2026-10-22-8e2beaa3 @ eed3f7a47f97; 14 patches in recipe/patches/PROVENANCE.md' - kind: reference pointer_type: git_commit @@ -197,4 +197,4 @@ provenance: - name: 'Sigstore/cosign (image signing + SBOM timestamping)' url: 'https://docs.sigstore.dev' - notes: "Community ingest of salient-data's recipe (Apache-2.0), lar-scout ingest t_5d21dce4 2026-10-22. Attribution: submitter/listed_by localai-recipes; recipe by Jeremy Newhouse / salient-data; announced in local-inference-lab #qwen-38-flash-next 2026-10-21 17:48Z. All perf numbers author-measured T0 with a named third-party harness and published raw data per build (results/); the only non-author numbers are tonyd2wild's published comparator rows (flagged in-line, different host/date, revision unstated). The recipe is explicitly outside the model card's support statement (B200/B300 only; GB10 unlisted). WATCH: author says a private inference engine beats this vLLM build and may publish findings/PRs after Flare testing (17:52Z follow-up). corrections_to_card: the bench-harness repo is tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark (card artifact named the DeepSeek-V4.1-Flash repo - that is the comparator-stack home; the harness lives in the QFN repo @ 6ad1c8f); serve --max-model-len is 262144 (card prose used 131072 - that is the needle-verified depth, kept for perf points); driver pin is 580.178.04 open kernel module; README carries a generated verified-date block (re-check the generated:verified date + CHANGELOG on every drift pass)." + notes: "Community ingest of salient-data's recipe (Apache-2.0), lar-scout ingest t_5d21dce4 2026-10-22. Attribution: submitter/listed_by localai-recipes; recipe by Jeremy Newhouse / salient-data; announced in local-inference-lab #qwen-38-flash-next 2026-10-21 17:48Z. All perf numbers author-measured T0 with a named third-party harness and published raw data per build (results/); the only non-author numbers are tonyd2wild's published comparator rows (flagged in-line, different host/date, revision unstated). The recipe is explicitly outside the model card's support statement (B200/B300 only; GB10 unlisted). WATCH: author says a private inference engine beats this vLLM build and may publish findings/PRs after Flare testing (17:52Z follow-up). corrections_to_card: the bench-harness repo is tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark (card artifact named the DeepSeek-V4.1-Flash repo - that is the comparator-stack home; the harness lives in the QFN repo @ 6ad1c8f); serve --max-model-len is 262144 (card prose used 131072 - that is the needle-verified depth, kept for perf points); driver pin is 580.178.04 open kernel module; README carries a generated verified-date block (re-check the generated:verified date + CHANGELOG on every drift pass). BUILD REVISION (kanban t_976fa730, owner go 2026-10-28 Casey 'for 1 we should update' via t_b21a5b73): recipe re-pinned f309019de1..7432036e7d (build 2026-10-22-8e2beaa3, commit #7 2026-10-25, live origin HEAD verified 2026-10-28). Patch series renumbered to the 14-patch set at the pin: DELETED 0005-vllm58737-fuse-the-mamba-align-block-table + 0006-vllm59260-warm-up-top-k-top-p-and-greedy; NEW 0013-vllm62841-avoid-re-marking-committed-prompt-tail-boundaries + 0014-draft-head-per-batch; RENUMBERED 0005-quiet-boot 0006-mtp-shard-filter 0007-vllm54912-widen-the-qsa-raw-key-ring 0008-vllm60068-confidence-stop-for-autoregressive-drafting 0009-skinny-gemm-rows 0010-vllm58863-recoverssm-for-qwen-gdn-and 0011-vllm59740-context-aware-sparse-lm-head 0012-moe-batched-load (patches.sha256 byte-verified 14/14; PROVENANCE.md applies at vLLM 8e2beaa3bff595577a069d49d96bde9074294ec2). serve.args/serve.env re-transcribed verbatim @ 7432036e7d: QWEN4EXP_DRAFT_VOCAB 0->65536; speculative-config adds draft_lm_head_quantization=nvfp4 + draft_token_map_max_rows=1 (launch argv delta 3 of 86 tokens, positions 35/42/85). PROD_IMAGE d55a27406510 (image id 3eda66c8...), base digest 167ccd86; engine vLLM 0.32.1rc1.dev197+g8e2beaa3b / torch 2.14.0+cu130 / FlashInfer 0.7.2. Measured table republished from results/2026-10-22-8e2beaa3/summary.md (69.3 c1 / 41.0 prose / 91.9 c6; 2/2 perf points preserved, values re-measured on the new build); quality set unchanged vs quality.json at the pin (GSM8K 99/100, arith 12/12, ten-task 9/10, needle 20/20, canary 0 failures). memory-budget.md changed at the pin: weights 73.51->73.57 GiB, KV pool 981,604-1,039,008 -> 1,002,985-1,094,166 tokens on four boots -> kv_pool_tokens 981604->1002985. HF pins resolve: nvidia/Qwen3.8-Flash-Next-NVFP4 @ fc694b54 HTTP 200 (2026-10-28); draft token map d22ff523... self-hash sha256==filename verified. Title/slug unchanged (legacy 71.3 in the slug is a historical name, not a current claim)."