diff --git a/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g/recipe.yaml b/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g/recipe.yaml index 7467ef2d8..5b9f6c3ca 100644 --- a/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g/recipe.yaml +++ b/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g/recipe.yaml @@ -5,10 +5,10 @@ engine: version: '0.1.dev20073 (qwen38-flash-next, iter7g floor)' source_type: fork git_url: 'https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast' - pinned_commit: 0d3469706696015992959a12b059b02b75100744 + pinned_commit: 2ddeb883fe692e3eee1840655a3df7de8eddf87d image_digest: 'https://hub.docker.com/r/vllm/vllm-openai@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8' - install_note: 'Rootless podman 4.x/5 on ARM64 GB10 with NVIDIA CDI; CUDA 13.0-pinned driver. Build chain: recipe/build/image/build.sh (iter6c->iter6d) then build-iter7g.sh on top, then recipe/build/model/build.sh T80 dense-MTP g32 drafter directory (about 4.8 GiB). Serve image tag qwen38-flash-dgx:iter7g-20261015 (fixed tag enforced by launch.sh). Deterministic floor needs the env knobs MOE_ALIGN_STABLESORT=1, MARLIN_ATOMIC_ADD=0, TRITON_FORCE_FIRST_CONFIG=1 + the persistent ~/.cache/qwen38-compile mount, all sourced from recipe/config/v19a-w5/env under a cleared environment by launch.sh.' -summary: 'Recipe by kyaky. Qwen3.8-Flash-Next W4A16-AutoRound hybrid (INT4 experts + FP8 side layers + INT8 head; 51B-token n-gram PLE table on NVMe mmap) on one DGX Spark (GB10, 121 GB unified memory) on a verified-deterministic vLLM floor (image qwen38-flash-dgx:iter7g-20261015 on the digest-pinned vllm/vllm-openai:qwen38-flash-next base): MoE align arrival-order shim, drafter-Marlin atomicAdd fix, pinned triton config, compile-cache mount. Promoted v19a-w5 = MTP=4 window-5 block verification: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 streams (+9.2% N1 / +4.8% N8 over MTP=3, two boots per arm, same image). 262,144-token context, 16 GB KV pool. Chinese +31.3% via a 25,862-id (39.5%) Han draft-vocab rebalance (prose-zhen 34.8 to 45.7 tok/s). Stochastic-correctness gate 37/37 vs bit-exact fp64 oracle; per-round raw data + scripts ship in-repo; copy-agentic-1 failure disclosed.' + install_note: 'Rootless podman 4.x/5 on ARM64 GB10 with NVIDIA CDI; CUDA 13.0-pinned driver. Build chain: recipe/build/image/build.sh (iter6c->iter6d) then build-iter7g.sh on top, then recipe/build/model/build.sh T80 dense-MTP g32 drafter directory (about 4.8 GiB). Measured-line image tag qwen38-flash-dgx:iter7g-20261015; the promoted v20-fp8 serving line runs the local podman tag qwen38-flash-dgx:fp8kv-promo-20261024 (fp8 KV read path + #59528 null-block backport; not built by the repo build chain). launch.sh enforces a fixed image tag and inspects the image via podman. Deterministic floor needs the env knobs MOE_ALIGN_STABLESORT=1, MARLIN_ATOMIC_ADD=0, TRITON_FORCE_FIRST_CONFIG=1 + the persistent ~/.cache/qwen38-compile mount, all sourced from recipe/config/v20-fp8/env (measured line: recipe/config/v19a-w5/env) under a cleared environment by launch.sh.' +summary: 'Recipe by kyaky. Qwen3.8-Flash-Next W4A16-AutoRound hybrid (INT4 experts + FP8 side layers + INT8 head; 51B-token n-gram PLE table on NVMe mmap) on one DGX Spark (GB10, 121 GB unified memory) on a verified-deterministic vLLM floor (image qwen38-flash-dgx:iter7g-20261015 on the digest-pinned vllm/vllm-openai:qwen38-flash-next base): MoE align arrival-order shim, drafter-Marlin atomicAdd fix, pinned triton config, compile-cache mount. Measured line v19a-w5 = MTP=4 window-5 block verification: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 streams (+9.2% N1 / +4.8% N8 over MTP=3, two boots per arm, same image). Promoted serving line v20-fp8: fp8 KV cache on a 24 GB pool at native 262,144 context, 1,291,120-token capacity. Chinese +31.3% via a 25,862-id (39.5%) Han draft-vocab rebalance (prose-zhen 34.8 to 45.7 tok/s). Stochastic-correctness gate 37/37 vs bit-exact fp64 oracle; per-round raw data + scripts ship in-repo; copy-agentic-1 failure disclosed.' license: CC-BY-4.0 model: name: Qwen3.8-Flash-Next-W4A16-AutoRound-Hybrid @@ -103,7 +103,7 @@ config: - VLLM_LOAD_EMBED_CHUNK=1 - '-e' - VLLM_QSA_DET_TOPK=1 - - 'qwen38-flash-dgx:iter7g-20261015' + - 'qwen38-flash-dgx:fp8kv-promo-20261024' - /model - '--served-model-name' - qwen @@ -127,9 +127,10 @@ config: - '-cc.splitting_ops=["vllm::unified_attention_with_output","vllm::unified_mla_attention_with_output","vllm::mamba_mixer2","vllm::mamba_mixer","vllm::short_conv","vllm::qwen3_8_flash_next_ple_short_conv","vllm::qwen3_8_flash_next_qsa_with_output","vllm::linear_attention","vllm::qwen_gdn_attention_core","vllm::qwen_gdn_attention_core_fused_norm_packed","vllm::sparse_attn_indexer","vllm::ple_mmap_lookup"]' - '--no-enable-flashinfer-autotune' - '--kv-cache-dtype' - - auto + - fp8 + - '--no-async-scheduling' - '--kv-cache-memory-bytes' - - 16g + - 24g - '--enable-prompt-tokens-details' - '--enable-auto-tool-choice' - '--tool-call-parser' @@ -139,8 +140,8 @@ config: - '--speculative-config' - '{"method":"mtp","num_speculative_tokens":4,"rejection_sample_method":"block","draft_sample_method":"probabilistic"}' runtime: - env: 'launch.sh sources recipe/config/v19a-w5/env under env -i (cleared environment); edit the env file, do not export' - notes: 'Promoted v19a-w5 profile transcribed verbatim from recipe/config/v19a-w5/{env,launch.sh,serve.sh,image} @ 0d3469706696015992959a12b059b02b75100744. Defaults when env is inspected directly: serve.sh falls back MTP=3, PORT 18300, TOOL_PARSER qwen3_coder - the promoted env overrides to MTP=4, port 8000, qwen3_xml. Image pin: recipe/config/v19a-w5/image = qwen38-flash-dgx:iter7g-20261015; the deterministic floor is tied to the pinned triton config (TRITON_FORCE_FIRST_CONFIG=1) + the persistent compile-cache mount. Driver-lib set must include libnvidia-ptxjitcompiler.so.1 or engine init fails on first JIT compile (mounted by launch.sh). Requires ~130 GB NVMe for checkpoint + FP8 PLE table (separate Saren/Qwen3.8-Flash-Next-ple-table-fp8 @ 50511b0a41aa1d34b8beb7e5d4bb06a0b650dc14, served read-only at /ple-table) plus ~4.8 GiB drafter shard data. podman<5 + nvidia-ctk>=1.20 CDI additionalGids strip workaround documented in repo Troubleshooting.' + env: 'launch.sh sources recipe/config/v20-fp8/env under env -i (cleared environment); edit the env file, do not export' + notes: 'Promoted serving line v20-fp8 transcribed verbatim from recipe/config/v20-fp8/{env,launch.sh,serve.sh,image} @ 2ddeb883fe692e3eee1840655a3df7de8eddf87d: the v19a-w5 promoted script with EXACTLY two functional edits (recorded upstream in runs/2026-10-17-qsa-fp8-kv/od4/config-diff.md) -- --kv-cache-dtype auto->fp8 and --no-async-scheduling added (D-021 pin-hygiene check-6 closure; the pinned engine otherwise auto-resolves async_scheduling TRUE) -- plus KV_BYTES 16g->24g and image qwen38-flash-dgx:fp8kv-promo-20261024 (local podman tag; not built by the repo build chain). The fp8 pool normalizes to 1,291,120 KV tokens at native 262,144. launch.sh inspects the image via podman (docker cannot see the local tag, 2a3cde5); the whole serving launch path is rootless podman (65b39aa). Defaults when serve.sh is inspected directly: MTP=3, PORT 18300, TOOL_PARSER qwen3_coder - the promoted env overrides to MTP=4, port 8000, qwen3_xml. Deterministic floor unchanged: MOE_ALIGN_STABLESORT=1 + MARLIN_ATOMIC_ADD=0 + TRITON_FORCE_FIRST_CONFIG=1 + the persistent ~/.cache/qwen38-compile mount. Driver-lib set must include libnvidia-ptxjitcompiler.so.1 or engine init fails on first JIT compile (mounted by launch.sh). Requires ~130 GB NVMe for checkpoint + FP8 PLE table (separate Saren/Qwen3.8-Flash-Next-ple-table-fp8 @ 50511b0a41aa1d34b8beb7e5d4bb06a0b650dc14, served read-only at /ple-table) plus ~4.8 GiB drafter shard data. podman<5 + nvidia-ctk>=1.20 CDI additionalGids strip workaround documented in repo Troubleshooting. Published throughput numbers remain the v19a-w5 bf16-line measurement (image qwen38-flash-dgx:iter7g-20261015); the v20-fp8 stream-curve re-measurement is pending upstream.' performance: claimed_by: submitter metrics: @@ -203,12 +204,12 @@ artifacts: kind: engine_image pointer_type: docker_digest pointer: 'https://hub.docker.com/r/vllm/vllm-openai' - pin: 'base vllm/vllm-openai:qwen38-flash-next @ sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8 -> repo build chain -> serving image qwen38-flash-dgx:iter7g-20261015 (fixed tag enforced; BUILD.md records measured image IDs)' + pin: 'base vllm/vllm-openai @ sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8 -> measured image qwen38-flash-dgx:iter7g-20261015; promoted v20-fp8 image qwen38-flash-dgx:fp8kv-promo-20261024' - kind: reference pointer_type: git_commit pointer: 'https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast' - pin: '0d3469706696015992959a12b059b02b75100744 - recipe/config/v19a-w5 (env/launch/serve/image + 65,536-id draft vocab), build chain iter6c..iter7g, benchmarks scripts, docs/benchmarks/2026-10-15-w54-ab raw per-round JSON + boot fingerprints' + pin: '2ddeb883fe... - recipe/config/v20-fp8 promoted fp8 line + v19a-w5 measured line, build chain iter6c..iter7g, docs/benchmarks/2026-10-15-w54-ab raw JSON + boot fingerprints' provenance: source_urls: - 'https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast' @@ -219,7 +220,7 @@ provenance: - name: kyaky url: 'https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast' - notes: 'Ingested from GitHub repo kyaky/qwen3.8-flash-next-gb10-ultrafast @ 0d3469706696015992959a12b059b02b75100744 (lar-scout ingest, kanban t_296a1f5d, 2026-10-16). Recipe Apache-2.0. Attribution: recipe by kyaky; submitter/listed_by localai-recipes. One recorded failure disclosed by the author: copy-agentic-1 fails at rep 2 on three independent W5 boots (attribution unresolved, reproduction queued; W4 0/30). Quality on same boots: code-edit 25/25 verified, MGSM 16/16 both arms, fixed-suite 93.1-93.3% on the v16b line. First live run of the owner-shaped tool-loop suite scored 29/50 (author-baselined observation, not an acceptance gate). perf note: N8 headline reported in all three accountings per README.' + notes: "Ingested from GitHub repo kyaky/qwen3.8-flash-next-gb10-ultrafast @ 0d3469706696015992959a12b059b02b75100744 (lar-scout ingest, kanban t_296a1f5d, 2026-10-16). Recipe Apache-2.0. Attribution: recipe by kyaky; submitter/listed_by localai-recipes. One recorded failure disclosed by the author: copy-agentic-1 fails at rep 2 on three independent W5 boots (attribution unresolved, reproduction queued; W4 0/30). Quality on same boots: code-edit 25/25 verified, MGSM 16/16 both arms, fixed-suite 93.1-93.3% on the v16b line. First live run of the owner-shaped tool-loop suite scored 29/50 (author-baselined observation, not an acceptance gate). perf note: N8 headline reported in all three accountings per README. Pin refreshed 2026-10-28 to 2ddeb883fe692e3eee1840655a3df7de8eddf87d (verified live origin/main HEAD via git ls-remote; kanban t_c3370f04, owner go t_eab4d745, preprod only). Pin honesty: the previous pin 0d3469706696015992959a12b059b02b75100744 was ORPHANED by an upstream history rewrite (sha still reachable via the GitHub API with message 'benchmarks: the stateful reproduction ran...'; absent from all refs after a full fetch), so no ancestry diff exists old-pin..new-pin - the delta was built via GitHub API per-file compare, not git log. Launch re-transcribed from recipe/config/v20-fp8/{env,launch.sh,serve.sh,image} @ 2ddeb883fe: serving line promoted v19a-w5->v20-fp8 (55199a4/2ddeb88) - image qwen38-flash-dgx:fp8kv-promo-20261024, --kv-cache-dtype auto->fp8, --kv-cache-memory-bytes 16g->24g (fp8 KV native 262k, 1,291,120-token capacity), +--no-async-scheduling (D-021 check-6 closure); launch path is rootless podman (65b39aa; launcher inspects via podman, 2a3cde5). Throughput record stays attributed to the measured v19a-w5 bf16 line; all 7 perf points preserved 7/7 unchanged, still grounded in docs/benchmarks/2026-10-15-w54-ab/*.json (unchanged at the pin) and the 37/37 gate claim in README + recipe/build/image/tests/blockverifier_gate_results.json (verdict gate_passed; results file unchanged at the pin, test file touched only by privacy-scan commits). Privacy: the site publishes no privacy/telemetry claim for this slug, so the upstream removal of the CI privacy grep step (98bc1c2, owner completed privacy review, donor-pattern false positives) removes no site claim. Hero: repo now ships docs/assets/hero-banner-v8.png + .license (Apache-2.0, 26f3d32); the site embeds no hero image for this slug - recorded, no site change. CI-only files (.shellcheckrc, .github/workflows/checks.yml, AGENTS.md): no site impact." max_context_len: 262144 storage_requirements: nvme_gb: 130.0