diff --git a/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang/recipe.yaml b/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang/recipe.yaml index 7a7959e95..5fd4ef10f 100644 --- a/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang/recipe.yaml +++ b/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang/recipe.yaml @@ -7,7 +7,7 @@ engine: git_url: 'https://github.com/bill-h-lin/sglang-exact-spec-decode' pinned_commit: e0deb2658e311387f8103ba27d52ef932bdc543e install_note: 'Built from source: SGLang main at 542a043f1b68796d269ef1ee11be7865f8d4a5f1 (2026-09-23) with patches 1-sglang-exact-verify + 2-sglang-small-m-route applied (git apply inside the sglang tree, then pip install -e python); DeepGEMM 0.2.0 patched with 3-deepgemm-sm120-fence (patch -p1 -d $SP). Patch 3 is required for run-to-run repeatability and DeepGEMM 0.2.2/0.2.3rc0 still ship the unfenced header. Point SGLANG_DG_CACHE_DIR at an empty directory on first launch (kernels compile at runtime and a stale cache can serve the old kernel); a pip reinstall/upgrade of sgl-deep-gemm drops patch 3. SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON=16 gates the small-M Triton route (sm120/121 only); deterministic decode is 18.0 t/s without it.' -summary: 'Recipe by bill-h-lin. Qwen3.8-27B BF16 (BF16 KV, temperature 0) on one RTX PRO 6000 Blackwell Max-Q (SM120) with SGLang made bit-exact under speculative decoding: verify GDN recurrence routed through the decode kernel (the old verify kernel tiled the state differently and kept beta in fp32 where decode rounds bf16), verify attention run as decode rows, a DeepGEMM sm120 bf16 small-M GEMM fence for run-to-run repeatability, and upstream sgl-project/sglang#42145 routing small-M bf16 GEMMs to a bit-identical Triton tile to escape the DeepGEMM sm120 bandwidth cliff. Base: SGLang main 542a043 (2026-09-23), DeepGEMM 0.2.0. DFlash drafter (incoai/Qwen3.8-27B-DFlash2) at 8 draft tokens: 106.6 tok/s exact vs 109.8 non-exact vLLM DFlash2 k=9 on short prompts, ahead at 2.2k (73.9 vs 67.4), 42.6 vs 56.1 at 128k. Receipts: 24/24 prompt token-id+logprob parity vs no-speculation (forced-reject, 8 in-flight, 128k) and 0/1200 run-to-run divergences with the fence.' +summary: 'Recipe by bill-h-lin. Qwen3.8-27B BF16 (BF16 KV, temperature 0) on one RTX PRO 6000 Blackwell Max-Q (SM120) with SGLang made bit-exact under speculative decoding: verify GDN recurrence routed through the decode kernel (old verify kernel tiled state differently, kept beta fp32), verify attention run as decode rows, a DeepGEMM sm120 bf16 small-M GEMM fence for repeatability (upstream from 0.2.3), and upstream sgl-project/sglang#42145 routing small-M bf16 GEMMs to a bit-identical Triton tile to escape the DeepGEMM sm120 bandwidth cliff. Base: SGLang main 542a043 (2026-09-23), DeepGEMM 0.2.0. DFlash drafter (incoai/Qwen3.8-27B-DFlash2) at 8 draft tokens: 106.6 tok/s exact vs 109.8 non-exact vLLM short, 73.9 vs 67.4 at 2.2k, 42.6 vs 56.1 at 128k. Receipts: 24/24 token-id+logprob parity vs no-speculation (forced-reject, 8 in-flight, 128k); 0/1200 run-to-run divergences with the fence; divergences cluster where top-1/top-2 logits are within 0.25 (0.125 for backend/--max-num-seqs).' license: Apache-2.0 model: name: Qwen3.8-27B @@ -66,41 +66,40 @@ config: - '--disable-radix-cache' runtime: env: - SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON: '16' SGLANG_DG_CACHE_DIR: /path/to/empty/dir + SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON: '16' + notes: "Launch argv transcribed verbatim from the README Launch block and verified byte-identical at e0deb26 and 566d10a (the 566d10a commits change only the Apply prose, the divergence-margin wording and the licence note - no argv change). Why-notes in that block: patch 3 unchanged only on the recipe's DeepGEMM 0.2.0 base; DeepGEMM#93 is released in 0.2.3 (2026-10-16) and SGLang main pins it since 2026-10-18, so on 0.2.3+ skip patch 3 - it will not apply. Apply: cd sglang && git checkout 542a043; git apply $P/1 $P/2; pip install -e python; patch -p1 -d $SP < $P/3. Patch 1+3 are active under --enable-deterministic-inference; patch 2 only with SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON set. Author-documented control: --max-num-seqs 1 vs 32 alone moved 21/24 outputs (cause unisolated); attention backend choice moved 20/24." + base_ref: 'SGLang main 542a043f1b68796d269ef1ee11be7865f8d4a5f1 (2026-09-23) + DeepGEMM 0.2.0' + upstream: + - 'sgl-project/sglang#42145 - patch 2 rebased upstream; patches/2 is the measured version' + - 'sgl-project/DeepGEMM#93 - the fence, merged to dev 2026-10-03; released in 0.2.3 (2026-10-16), pinned by SGLang main since 2026-10-18' + source_pin: 'https://github.com/bill-h-lin/sglang-exact-spec-decode @ e0deb2658e311387f8103ba27d52ef932bdc543e (measured) ; HEAD 566d10a3b840dbfc7e629763d0299e75440187a8 (docs-only drift, 2026-10-26)' patches_applied: - patches/1-sglang-exact-verify.patch - patches/2-sglang-small-m-route.patch - patches/3-deepgemm-sm120-fence.patch - base_ref: 'SGLang main 542a043f1b68796d269ef1ee11be7865f8d4a5f1 (2026-09-23) + DeepGEMM 0.2.0' - upstream: - - 'sgl-project/sglang#42145 - patch 2 rebased upstream; patches/2 is the measured version' - - 'sgl-project/DeepGEMM#93 - the fence, merged to dev 2026-10-03; 0.2.2/0.2.3rc0 still unfenced' - source_pin: 'https://github.com/bill-h-lin/sglang-exact-spec-decode @ e0deb2658e311387f8103ba27d52ef932bdc543e' - notes: 'Launch argv transcribed verbatim from the README Launch block at e0deb26; env exports from the same block. Apply: cd sglang && git checkout 542a043; git apply patches/1 patches/2; pip install -e python; patch -p1 -d $SP < patches/3. Patch 1+3 are active under --enable-deterministic-inference; patch 2 only with SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON set. Author-documented control: --max-num-seqs 1 vs 32 alone moved 21/24 outputs (cause unisolated); attention backend choice moved 20/24.' performance: claimed_by: submitter - metrics: - tok_s: 106.6 - context_len: 640 - notes: 'All rows single-stream temp 0 on one RTX PRO 6000 Blackwell Max-Q, README table at e0deb26. SGLang no-speculation baseline 26.4/26.6/23.2 (short/2.2k/128k); deterministic mode alone is 18.0 t/s without #42145 (DeepGEMM sm120 small-M bandwidth cliff). vLLM DFlash2 k=9+XQA (not exact): 109.8/67.4/56.1. Exact is 3% behind vLLM on short prompts, ahead at 2.2k, 24% behind at 128k; 128k TTFT 68.5 s vs 44.3 s on vLLM.' - points: - - - context_len: 640 - concurrency: 1 - decode_tok_s: 106.6 - notes: 'README short-prompts column, exact mode (DFlash 8 + patches + #42145). context_len is a bucket marker (battery prompts 560-740 tokens, exact length unspecified). vLLM DFlash2 k=9 non-exact ref 109.8.' - - - context_len: 2183 - concurrency: 1 - decode_tok_s: 73.9 - notes: '2.2k prompts (parity battery used 2,183 tokens); exact 73.9 vs vLLM non-exact 67.4 - exact ahead here.' - - - context_len: 127735 - concurrency: 1 - decode_tok_s: 42.6 - ttft_ms: 68500 - notes: '128k prompts (parity battery 127,735 tokens); exact 42.6 vs vLLM 56.1; TTFT 68.5 s vs 44.3 s on vLLM.' + tok_s: 106.6 + context_len: 640 + notes: 'All rows single-stream temp 0 on one RTX PRO 6000 Blackwell Max-Q, README table at e0deb26 (numbers unchanged at 566d10a - the newer commits are README wording only). SGLang no-speculation baseline 26.4/26.6/23.2 (short/2.2k/128k); deterministic mode alone is 18.0 t/s without #42145 (DeepGEMM sm120 small-M bandwidth cliff). vLLM DFlash2 k=9+XQA (not exact): 109.8/67.4/56.1. Exact is 3% behind vLLM on short prompts, ahead at 2.2k, 24% behind at 128k; 128k TTFT 68.5 s vs 44.3 s on vLLM. Updated 2026-10-27 against repo HEAD 566d10a3b840 (README: divergence margins within 0.25 / 0.125 for backend and max-num-seqs; DeepGEMM#93 released in 0.2.3 and pinned by SGLang main since 10-18 -> skip patch 3 on 0.2.3+; DeepGEMM is MIT; apply paths via $P) - tables-only refresh, no launch-argv change.' + points: + - + context_len: 640 + concurrency: 1 + decode_tok_s: 106.6 + notes: 'README short-prompts column, exact mode (DFlash 8 + patches + #42145). context_len is a bucket marker (battery prompts 560-740 tokens, exact length unspecified). vLLM DFlash2 k=9 non-exact ref 109.8.' + - + context_len: 2183 + concurrency: 1 + decode_tok_s: 73.9 + notes: '2.2k prompts (parity battery used 2,183 tokens); exact 73.9 vs vLLM non-exact 67.4 - exact ahead here.' + - + context_len: 127735 + concurrency: 1 + decode_tok_s: 42.6 + ttft_ms: 68500 + notes: '128k prompts (parity battery 127,735 tokens); exact 42.6 vs vLLM 56.1; TTFT 68.5 s vs 44.3 s on vLLM.' artifacts: - kind: reference