localairecipes launch configs for local inference

Qwen3.8-27B Bit-Exact Spec-Decode 1x RTX PRO 6000 SGLang

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by bill-h-lin↗ · listed by localai-recipes

revision r3 · commit d47db89f9e · updated 3 d ago · license Apache-2.0

chat spec-decode single-gpu dflash2 long-context bf16-kv sglang sm120 gdn-fusion deterministic-decode bit-exact blackwell deepgemm

Recipe by bill-h-lin. Qwen3.8-27B BF16 (BF16 KV, temperature 0) on one RTX PRO 6000 Blackwell Max-Q (SM120) with SGLang made bit-exact under speculative decoding: verify GDN recurrence routed through the decode kernel (old verify kernel tiled state differently, kept beta fp32), verify attention run as decode rows, a DeepGEMM sm120 bf16 small-M GEMM fence for repeatability (upstream from 0.2.3), and upstream sgl-project/sglang#42145 routing small-M bf16 GEMMs to a bit-identical Triton tile to escape the DeepGEMM sm120 bandwidth cliff. Base: SGLang main 542a043 (2026-09-23), DeepGEMM 0.2.0. DFlash drafter (incoai/Qwen3.8-27B-DFlash2) at 8 draft tokens: 106.6 tok/s exact vs 109.8 non-exact vLLM short, 73.9 vs 67.4 at 2.2k, 42.6 vs 56.1 at 128k. Receipts: 24/24 token-id+logprob parity vs no-speculation (forced-reject, 8 in-flight, 128k); 0/1200 run-to-run divergences with the fence; divergences cluster where top-1/top-2 logits are within 0.25 (0.125 for backend/--max-num-seqs).

Launch command

python -m sglang.launch_server --model-path Qwen/Qwen3.8-27B --enable-deterministic-inference --attention-backend triton --speculative-algorithm DFLASH --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8 --mamba-ssm-dtype float32 --max-running-requests 8 --chunked-prefill-size 8192 --mem-fraction-static 0.88 --context-length 262144 --disable-radix-cache

runtime

env:
  SGLANG_DG_CACHE_DIR: /path/to/empty/dir
  SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON: '16'
notes: "Launch argv transcribed verbatim from the README Launch block and verified byte-identical at e0deb26 and 566d10a (the 566d10a commits change only the Apply prose, the divergence-margin wording and the licence note - no argv change). Why-notes in that block: patch 3 unchanged only on the recipe's DeepGEMM 0.2.0 base; DeepGEMM#93 is released in 0.2.3 (2026-10-16) and SGLang main pins it since 2026-10-18, so on 0.2.3+ skip patch 3 - it will not apply. Apply: cd sglang && git checkout 542a043; git apply $P/1 $P/2; pip install -e python; patch -p1 -d $SP < $P/3. Patch 1+3 are active under --enable-deterministic-inference; patch 2 only with SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON set. Author-documented control: --max-num-seqs 1 vs 32 alone moved 21/24 outputs (cause unisolated); attention backend choice moved 20/24."
base_ref: 'SGLang main 542a043f1b68796d269ef1ee11be7865f8d4a5f1 (2026-09-23) + DeepGEMM 0.2.0'
upstream:
  - 'sgl-project/sglang#42145 - patch 2 rebased upstream; patches/2 is the measured version'
  - 'sgl-project/DeepGEMM#93 - the fence, merged to dev 2026-10-03; released in 0.2.3 (2026-10-16), pinned by SGLang main since 2026-10-18'
source_pin: 'https://github.com/bill-h-lin/sglang-exact-spec-decode @ e0deb2658e311387f8103ba27d52ef932bdc543e (measured) ; HEAD 566d10a3b840dbfc7e629763d0299e75440187a8 (docs-only drift, 2026-10-26)'
patches_applied:
  - patches/1-sglang-exact-verify.patch
  - patches/2-sglang-small-m-route.patch
  - patches/3-deepgemm-sm120-fence.patch

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang.json

Engine

Install note: Built from source: SGLang main at 542a043f1b68796d269ef1ee11be7865f8d4a5f1 (2026-09-23) with patches 1-sglang-exact-verify + 2-sglang-small-m-route applied (git apply inside the sglang tree, then pip install -e python); DeepGEMM 0.2.0 patched with 3-deepgemm-sm120-fence (patch -p1 -d $SP). Patch 3 is required for run-to-run repeatability and DeepGEMM 0.2.2/0.2.3rc0 still ship the unfenced header. Point SGLANG_DG_CACHE_DIR at an empty directory on first launch (kernels compile at runtime and a stale cache can serve the old kernel); a pip reinstall/upgrade of sgl-deep-gemm drops patch 3. SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON=16 gates the small-M Triton route (sm120/121 only); deterministic decode is 18.0 t/s without it.

Model

Hardware

Context

  • max context: 256K tokens

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
6401—106.6 tok/s—reported
2,1831—73.9 tok/s—reported
127,7351—42.6 tok/s68500 msreported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAll rows single-stream temp 0 on one RTX PRO 6000 Blackwell Max-Q, README table at e0deb26 (numbers unchanged at 566d10a - the newer commits are README wording only). SGLang no-speculation baseline 26.4/26.6/23.2 (short/2.2k/128k); deterministic mode alone is 18.0 t/s without #42145 (DeepGEMM sm120 small-M bandwidth cliff). vLLM DFlash2 k=9+XQA (not exact): 109.8/67.4/56.1. Exact is 3% behind vLLM on short prompts, ahead at 2.2k, 24% behind at 128k; 128k TTFT 68.5 s vs 44.3 s on vLLM. Updated 2026-10-27 against repo HEAD 566d10a3b840 (README: divergence margins within 0.25 / 0.125 for backend and max-num-seqs; DeepGEMM#93 released in 0.2.3 and pinned by SGLang main since 10-18 -> skip patch 3 on 0.2.3+; DeepGEMM is MIT; apply paths via $P) - tables-only refresh, no launch-argv change.

Downloads

Revision notes

Ingested from repo HEAD e0deb2658e311387f8103ba27d52ef932bdc543e (2026-10-15) by localai-recipes; the repo was rewritten after the scout snapshot (card pinned 952d88e4, which no longer resolves) - history db00c95/cbfed1e/e0deb26, same three patches, e0deb26 adds the DeepGEMM 0.2.2/0.2.3rc0 still-unfenced note. Every launch flag verified against SGLang arg_groups fields at 542a043f1b68796d269ef1ee11be7865f8d4a5f1 (enable_deterministic_inference, mamba_ssm_dtype choices incl float32, DFLASH builtin, --model-path / --speculative-draft-model-path spellings); SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON verified against patch 2's own sm120/121 gate. HF pins resolved live: Qwen/Qwen3.8-27B @ 1d4bf0f2, incoai/Qwen3.8-27B-DFlash2 @ 015e7956. Card corrections: drafter is incoai (card said z-lab); DeepGEMM upgrade caveat added by e0deb26. Numbers are author-measured T0 from the README table at e0deb26; the r/LocalLLaMA thread (1x3sx3d) not re-verified (403 lane). context_len 640 on the first point is a bucket marker for the README short-prompts column (repeatability battery prompts ran 560-740 tokens); 2183/127735 are the parity battery's measured prompt lengths.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)