{"slug":"qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang","title":"Qwen3.8-27B Bit-Exact Spec-Decode 1x RTX PRO 6000 SGLang","summary":"Recipe by bill-h-lin. Qwen3.8-27B BF16 (BF16 KV, temperature 0) on one RTX PRO 6000 Blackwell Max-Q (SM120) with SGLang made bit-exact under speculative decoding: verify GDN recurrence routed through the decode kernel (old verify kernel tiled state differently, kept beta fp32), verify attention run as decode rows, a DeepGEMM sm120 bf16 small-M GEMM fence for repeatability (upstream from 0.2.3), and upstream sgl-project/sglang#42145 routing small-M bf16 GEMMs to a bit-identical Triton tile to escape the DeepGEMM sm120 bandwidth cliff. Base: SGLang main 542a043 (2026-09-23), DeepGEMM 0.2.0. DFlash drafter (incoai/Qwen3.8-27B-DFlash2) at 8 draft tokens: 106.6 tok/s exact vs 109.8 non-exact vLLM short, 73.9 vs 67.4 at 2.2k, 42.6 vs 56.1 at 128k. Receipts: 24/24 token-id+logprob parity vs no-speculation (forced-reject, 8 in-flight, 128k); 0/1200 run-to-run divergences with the fence; divergences cluster where top-1/top-2 logits are within 0.25 (0.125 for backend/--max-num-seqs).","license":"Apache-2.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"d47db89f9ef31a0a96530e29d246e7f562509ee6","content_hash":"5fd4ef10f7e92f70e8b5831e5c33c58cfa0d635c","created_at":"2026-10-27 20:30:16.519777-06","notes":"Ingested from repo HEAD e0deb2658e311387f8103ba27d52ef932bdc543e (2026-10-15) by localai-recipes; the repo was rewritten after the scout snapshot (card pinned 952d88e4, which no longer resolves) - history db00c95/cbfed1e/e0deb26, same three patches, e0deb26 adds the DeepGEMM 0.2.2/0.2.3rc0 still-unfenced note. Every launch flag verified against SGLang arg_groups fields at 542a043f1b68796d269ef1ee11be7865f8d4a5f1 (enable_deterministic_inference, mamba_ssm_dtype choices incl float32, DFLASH builtin, --model-path / --speculative-draft-model-path spellings); SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON verified against patch 2's own sm120/121 gate. HF pins resolved live: Qwen/Qwen3.8-27B @ 1d4bf0f2, incoai/Qwen3.8-27B-DFlash2 @ 015e7956. Card corrections: drafter is incoai (card said z-lab); DeepGEMM upgrade caveat added by e0deb26. Numbers are author-measured T0 from the README table at e0deb26; the r/LocalLLaMA thread (1x3sx3d) not re-verified (403 lane). context_len 640 on the first point is a bucket marker for the README short-prompts column (repeatability battery prompts ran 560-740 tokens); 2183/127735 are the parity battery's measured prompt lengths."},"engine_source":{"type":"release","git_url":"https://github.com/bill-h-lin/sglang-exact-spec-decode","base_engine":"sglang","pip_package":null,"base_version":"main 542a043 (2026-09-23) + 2 patches + DeepGEMM 0.2.0 fence","image_digest":null,"install_note":"Built from source: SGLang main at 542a043f1b68796d269ef1ee11be7865f8d4a5f1 (2026-09-23) with patches 1-sglang-exact-verify + 2-sglang-small-m-route applied (git apply inside the sglang tree, then pip install -e python); DeepGEMM 0.2.0 patched with 3-deepgemm-sm120-fence (patch -p1 -d $SP). Patch 3 is required for run-to-run repeatability and DeepGEMM 0.2.2/0.2.3rc0 still ship the unfenced header. Point SGLANG_DG_CACHE_DIR at an empty directory on first launch (kernels compile at runtime and a stale cache can serve the old kernel); a pip reinstall/upgrade of sgl-deep-gemm drops patch 3. SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON=16 gates the small-M Triton route (sm120/121 only); deterministic decode is 18.0 t/s without it.","pinned_commit":"e0deb2658e311387f8103ba27d52ef932bdc543e"},"credit":[{"name":"bill-h-lin","url":"https://github.com/bill-h-lin/sglang-exact-spec-decode"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["python","-m","sglang.launch_server","--model-path","Qwen/Qwen3.8-27B","--enable-deterministic-inference","--attention-backend","triton","--speculative-algorithm","DFLASH","--speculative-draft-model-path","incoai/Qwen3.8-27B-DFlash2","--speculative-num-draft-tokens","8","--mamba-ssm-dtype","float32","--max-running-requests","8","--chunked-prefill-size","8192","--mem-fraction-static","0.88","--context-length","262144","--disable-radix-cache"],"runtime":{"env":{"SGLANG_DG_CACHE_DIR":"/path/to/empty/dir","SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON":"16"},"notes":"Launch argv transcribed verbatim from the README Launch block and verified byte-identical at e0deb26 and 566d10a (the 566d10a commits change only the Apply prose, the divergence-margin wording and the licence note - no argv change). Why-notes in that block: patch 3 unchanged only on the recipe's DeepGEMM 0.2.0 base; DeepGEMM#93 is released in 0.2.3 (2026-10-16) and SGLang main pins it since 2026-10-18, so on 0.2.3+ skip patch 3 - it will not apply. Apply: cd sglang && git checkout 542a043; git apply $P/1 $P/2; pip install -e python; patch -p1 -d $SP < $P/3. Patch 1+3 are active under --enable-deterministic-inference; patch 2 only with SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON set. Author-documented control: --max-num-seqs 1 vs 32 alone moved 21/24 outputs (cause unisolated); attention backend choice moved 20/24.","base_ref":"SGLang main 542a043f1b68796d269ef1ee11be7865f8d4a5f1 (2026-09-23) + DeepGEMM 0.2.0","upstream":["sgl-project/sglang#42145 - patch 2 rebased upstream; patches/2 is the measured version","sgl-project/DeepGEMM#93 - the fence, merged to dev 2026-10-03; released in 0.2.3 (2026-10-16), pinned by SGLang main since 2026-10-18"],"source_pin":"https://github.com/bill-h-lin/sglang-exact-spec-decode @ e0deb2658e311387f8103ba27d52ef932bdc543e (measured) ; HEAD 566d10a3b840dbfc7e629763d0299e75440187a8 (docs-only drift, 2026-10-26)","patches_applied":["patches/1-sglang-exact-verify.patch","patches/2-sglang-small-m-route.patch","patches/3-deepgemm-sm120-fence.patch"]},"max_context_len":262144},"models":[{"slug":"qwen3-8-27b","name":"Qwen3.8-27B","hf_repo":"RadixArk/Qwen3.8-27B-NVFP4","quant":"bf16"}],"hardware":[{"slug":"nvidia-rtx-pro-6000-blackwell","vendor":"nvidia","card":"rtx-pro-6000-blackwell","card_family":"rtx-pro-6000-blackwell","vram_gb":96,"device_count":1}],"generation":"blackwell","tags":["bf16-kv","bit-exact","blackwell","deepgemm","deterministic-decode","dflash2","gdn-fusion","long-context","sglang","single-gpu","sm120","spec-decode"],"tasks":["chat"],"aa":{"aa_index":34,"aa_provenance":"published","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-07 18:00:00-06","aa_family_slug":"qwen-3-8-27b","aa_resolved_slug":"qwen-3-8-27b"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"All rows single-stream temp 0 on one RTX PRO 6000 Blackwell Max-Q, README table at e0deb26 (numbers unchanged at 566d10a - the newer commits are README wording only). SGLang no-speculation baseline 26.4/26.6/23.2 (short/2.2k/128k); deterministic mode alone is 18.0 t/s without #42145 (DeepGEMM sm120 small-M bandwidth cliff). vLLM DFlash2 k=9+XQA (not exact): 109.8/67.4/56.1. Exact is 3% behind vLLM on short prompts, ahead at 2.2k, 24% behind at 128k; 128k TTFT 68.5 s vs 44.3 s on vLLM. Updated 2026-10-27 against repo HEAD 566d10a3b840 (README: divergence margins within 0.25 / 0.125 for backend and max-num-seqs; DeepGEMM#93 released in 0.2.3 and pinned by SGLang main since 10-18 -> skip patch 3 on 0.2.3+; DeepGEMM is MIT; apply paths via $P) - tables-only refresh, no launch-argv change.\", \"tok_s\": 106.599999999999994315658113919198513031005859375, \"context_len\": 640}"}],"perf_points":[{"id":1871,"context_len":640,"prefill_tok_s":null,"decode_tok_s":106.599999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"README short-prompts column, exact mode (DFlash 8 + patches + #42145). context_len is a bucket marker (battery prompts 560-740 tokens, exact length unspecified). vLLM DFlash2 k=9 non-exact ref 109.8."},{"id":1872,"context_len":2183,"prefill_tok_s":null,"decode_tok_s":73.900000000000005684341886080801486968994140625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"2.2k prompts (parity battery used 2,183 tokens); exact 73.9 vs vLLM non-exact 67.4 - exact ahead here."},{"id":1873,"context_len":127735,"prefill_tok_s":null,"decode_tok_s":42.60000000000000142108547152020037174224853515625,"ttft_ms":68500,"concurrency":1,"claimed_by":"submitter","notes":"128k prompts (parity battery 127,735 tokens); exact 42.6 vs vLLM 56.1; TTFT 68.5 s vs 44.3 s on vLLM."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang.json"},"requirements":null,"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/bill-h-lin/sglang-exact-spec-decode","pin":"e0deb2658e311387f8103ba27d52ef932bdc543e"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/Qwen/Qwen3.8-27B","pin":"1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/incoai/Qwen3.8-27B-DFlash2","pin":"015e795645c74b1a0eeef3b570031fb62e769bc5"},{"kind":"reference","pointer_type":"url","pointer":"https://github.com/sgl-project/sglang/pull/42145","pin":null}],"links":{"self":"/api/v1/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang.json","alternate":"/r/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang","describedby":"/openapi.json"},"updated_at":"2026-10-27 20:30:16.519777-06","created_at":"2026-10-20 21:49:16.178917-06","speeds":{"decode_tok_s":73.900000000000005684341886080801486968994140625,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}