localairecipes launch configs for local inference

Qwen3.8-27B NVFP4A4-AWQ 1x RTX PRO 6000 SGLang Fast Decode

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by AppNetDeveloper↗, SGLang 0.5.22 port + MTP5/FR-Spec real-text results @ c00cfec525 (published upstream 2026-10-21)↗ · listed by localai-recipes

revision r5 · commit 607c8d8bab · updated 2 d ago · license CC-BY-4.0

chat code fp8-kv awq speculative sglang sm120 mtp nvfp4 blackwell fp8 fr-spec hicache hot-token-map kda nextn w4a4

Recipe by AppNetDeveloper. ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ on 1x RTX PRO 6000 Blackwell (SM120) with stock SGLang 0.5.22 plus engine overlays, no fork: FR-Spec pruned draft lm_head for a quantized FP8 head (slices weight AND per-row scale along vocab, bit-identical kept rows); KDA NVFP4 SM120 GEMM allowlist extension (KN=5120x34816,17408x5120) 85-89% DRAM bandwidth at M<=16 vs 50-70% default; tuned triton_scaled_mm fp8_w8a8_channelwise JSONs. On 0.5.22 the overlays land as ONE patch, patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1). NEXTN/MTP, FP8_e4m3 KV cache, HiCache, 40k hot-token draft list; hybrid GDN+MTP needs --enable-linear-replayssm-spec. Author bench 2026-10-14 (400 output tokens, unique prefixes): 131.7 tok/s 1req 45k ctx vs 98.5 (+34%); 4req@45k +13.5%; 8req@27k +3%; 1req@141k +24%; same quality, no extra memory. MTP5 + FR-Spec is the author production default; measured on real agent text: 149 vs 140 @1req, 117 vs 103 @2req, 77 vs 69 @4req over MTP4.

Launch command

python3 -m sglang.launch_server --model bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-token-map tokens/hot_tokens_qwen38_es_en_code_40000.pt --kv-cache-dtype fp8_e4m3 --enable-hierarchical-cache --enable-linear-replayssm-spec

runtime

env:
  SGLANG_KDEV_KDA_NVFP4_KN: '5120x34816,17408x5120'
  SGLANG_KDEV_KDA_NVFP4_MAX_M: '16'
  SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE: '1'
tokens: 'tokens/hot_tokens_qwen38_es_en_code_40000.pt (40000 ids, multiple of 64)'
not_helped:
  - '--linear-attn-prefill-backend flashinfer'
  - '--enable-torch-compile'
  - '--bf16-gemm-backend gemv (requires SM90)'
  - '--fp8-gemm-backend (block-wise FP8 only)'
source_pin: 'https://github.com/AppNetDeveloper/sglang-sm120-fast-decode @ c00cfec525efb20e9c4185ca6c86e61a3235dbc1 (SGLang 0.5.22 port; v0.5.22 = baa9befa381511cd4d56f83980951e5a45fffc6d)'
base_args_note: "README elides base serve args (python -m sglang.launch_server ...). Shown flags are the ones the repo states at 937460a; KV/HiCache flags rendered from stock 0.5.21 (e00930c5489053f26d86b179cee0d087f846acbb) spellings for the README's 'FP8 KV cache, HiCache'."
patches_applied:
  - 'patches/frspec_fp8_lmhead.patch (0.5.21 lane, patch -p0)'
  - 'patches/kda_nvfp4_allowlist.patch (0.5.21 lane, patch -p0)'
  - 'patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (live 0.5.22 lane, patch -p1)'
verify_log_lines:
  - 'FR-Spec: draft lm_head pruned to 40000 rows (quantized, weight ...)'
  - 'Using configuration from .../N=...json'
  - 'KDA NVFP4 SM120 path (m, k, n) = ...'
fp8_configs_installed: 'fp8_configs/*.json -> <sglang>/kernels/ops/gemm/configs/ (valid only for NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition at these N,K)'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode.json

Engine

Install note: Stock SGLang 0.5.22 + ONE combined overlay patch, no fork: apply patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch with patch -p1 inside the copied sglang package (README @ c00cfec525; verified to apply cleanly against sglang==0.5.22). The 0.5.21 lane (stock SGLang 0.5.21) instead applies patches/frspec_fp8_lmhead.patch and patches/kda_nvfp4_allowlist.patch with patch -p0. Copy the 7 fp8_configs/*.json (N=5120..248320, K=5120/17408, device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition, dtype=fp8_w8a8_channelwise; unchanged for 0.5.22) into sglang/kernels/ops/gemm/configs/, and stage tokens/hot_tokens_qwen38_es_en_code_40000.pt. Launch env: SGLANG_KDEV_KDA_NVFP4_KN and SGLANG_KDEV_KDA_NVFP4_MAX_M=16 (from the patch) plus SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE=1 (README). 0.5.22 + hybrid (Gated DeltaNet) models + MTP requires --enable-linear-replayssm-spec or KV-cache sizing fails ("Loaded weights leave no GPU memory for the KV cache"). JSONs are per (N,K,GPU). Token list length must stay a multiple of 64 or the pruned FP8 head falls back to an untuned kernel.

Model

Hardware

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
27,6488—40 tok/s—reported
46,0801—131.7 tok/s—reported
46,0804—52.1 tok/s—reported
74,7521—149 tok/s—reported
74,7522—117 tok/s—reported
74,7524—77 tok/s—reported
144,3841—116.6 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured 2026-10-14 (README results table @ 937460a; 'same server, same bench, 400 output tokens, unique prefixes'): 1req@45k-ctx 131.7 tok/s vs 98.5 baseline (+34%), FR-Spec alone 102.0; 4req@45k 52.1 vs 45.9 (+13.5%); 8req@27k 40.0 vs 38.7 (+3%); 1req@141k 116.6 vs 93.9 (+24%). +22-38% decode over 1h production traffic (gateway logs, responses>=300 tokens, bucketed by ctx), 0 server errors. Quality same both arms: reasoning 48/48 (3x16), tool calls 24/24, needle 4/4, MTP accept ~3.0. context_len on points = the ctx sizes named in the table (45k/27k/141k); bench prompt build not further specified in the repo. | MTP5 update (author-measured 2026-10-21, README @ c00cfec525; 'measured on real agent text': multi-turn coding sessions with ~73k-token prompts, 4 turns each; same server 1x RTX PRO 6000 Workstation power-capped at 450 W; EAGLE/MTP top-k 1; output tokens/s per request): MTP5 + FR-Spec vs MTP4 + FR-Spec 1req 149 vs 140 (+6%), 2req 117 vs 103 (+13%), 4req 77 vs 69 (+12%). With stock SGLang (no FR-Spec) MTP5 lost: 1req 96 vs 107 (-10%), 10req 31 vs 34 (-9%); real acceptance only 2.90 -> 3.02 tokens/step. In production MTP5 + FR-Spec is now the author default (--speculative-num-steps 5 --speculative-num-draft-tokens 6 --speculative-token-map <40k list>). Lesson (verbatim): decide draft settings on real text only - a synthetic bench reported +42% for MTP5 with mean acceptance 4.19; on real traffic acceptance was ~3.0. Points context_len 74752 = 73*1024 per the repo's k->1024 convention.

Downloads

Revision notes

Scout ingest (localai-recipes) of AppNetDeveloper/sglang-sm120-fast-decode @ 937460a718819c7b646b48b8ce67290896f96b97 (HEAD 2026-10-14, created+measured same day). All launch/env/patch/config facts transcribed from that tree: README.md results table + how-to-apply env exports, both patch files, 7 fp8_configs JSONs (device-locked N,K names), tokens/ 40000-id hot list. Repo shows no full serve script; base serve args elided in README, so config.launch carries the stated flags and 0.5.21 spellings for FP8-KV/HiCache. Hot-token list built only from public HyperQwen + MiaAI-Lab es_en_code_65k lists per README. Model card: 23.4 GB on disk; license:other on HF (thinking-cap model card); recipe yaml licensed CC-BY-4.0. Not re-benchmarked by us; T0 submitter-provenance throughout. 2026-10-28 (t_ad75e28d, owner Tier-2 go 2026-10-28): SGLang base bumped 0.5.21->0.5.22 per upstream c00cfec525efb20e9c4185ca6c86e61a3235dbc1 (2026-10-21) - the FR-Spec + KDA overlays land as ONE combined patch patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1 inside the copied sglang package; fp8_configs unchanged; patch -p1 --dry-run verified clean vs sglang==0.5.22 wheel). 0.5.22 hybrid (Gated DeltaNet) + MTP requires --enable-linear-replayssm-spec or KV-cache sizing fails. Published the author MTP5+FR-Spec table measured on real agent text (verbatim README @ c00cfec): MTP5 vs MTP4 both with FR-Spec: 149 vs 140 @1req (+6%), 117 vs 103 @2req (+13%), 77 vs 69 @4req (+12%); stock-SGLang MTP5 lost (96 vs 107 @1req -10%, 31 vs 34 @10req -9%; acceptance 2.90->3.02); synthetic bench (+42%, accept 4.19) rejected - decide draft settings on real text only. git/artifact/credit pins -> c00cfec525ef; HF model pin 32c92c6fd22f unchanged (idle since 2026-09-25 per t_8d91ff6e). Still author-measured T0, not re-benchmarked by us.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)