{"slug":"qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode","title":"Qwen3.8-27B NVFP4A4-AWQ 1x RTX PRO 6000 SGLang Fast Decode","summary":"Recipe by AppNetDeveloper. ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ on 1x RTX PRO 6000 Blackwell (SM120) with stock SGLang 0.5.22 plus engine overlays, no fork: FR-Spec pruned draft lm_head for a quantized FP8 head (slices weight AND per-row scale along vocab, bit-identical kept rows); KDA NVFP4 SM120 GEMM allowlist extension (KN=5120x34816,17408x5120) 85-89% DRAM bandwidth at M<=16 vs 50-70% default; tuned triton_scaled_mm fp8_w8a8_channelwise JSONs. On 0.5.22 the overlays land as ONE patch, patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1). NEXTN/MTP, FP8_e4m3 KV cache, HiCache, 40k hot-token draft list; hybrid GDN+MTP needs --enable-linear-replayssm-spec. Author bench 2026-10-14 (400 output tokens, unique prefixes): 131.7 tok/s 1req 45k ctx vs 98.5 (+34%); 4req@45k +13.5%; 8req@27k +3%; 1req@141k +24%; same quality, no extra memory. MTP5 + FR-Spec is the author production default; measured on real agent text: 149 vs 140 @1req, 117 vs 103 @2req, 77 vs 69 @4req over MTP4.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":5,"commit_sha":"607c8d8babf1cd4237a56128e0614aceb2176332","content_hash":"ba21205503ad8f52405ab4becff180d4e348b45f","created_at":"2026-10-28 16:25:30.123672-06","notes":"Scout ingest (localai-recipes) of AppNetDeveloper/sglang-sm120-fast-decode @ 937460a718819c7b646b48b8ce67290896f96b97 (HEAD 2026-10-14, created+measured same day). All launch/env/patch/config facts transcribed from that tree: README.md results table + how-to-apply env exports, both patch files, 7 fp8_configs JSONs (device-locked N,K names), tokens/ 40000-id hot list. Repo shows no full serve script; base serve args elided in README, so config.launch carries the stated flags and 0.5.21 spellings for FP8-KV/HiCache. Hot-token list built only from public HyperQwen + MiaAI-Lab es_en_code_65k lists per README. Model card: 23.4 GB on disk; license:other on HF (thinking-cap model card); recipe yaml licensed CC-BY-4.0. Not re-benchmarked by us; T0 submitter-provenance throughout. 2026-10-28 (t_ad75e28d, owner Tier-2 go 2026-10-28): SGLang base bumped 0.5.21->0.5.22 per upstream c00cfec525efb20e9c4185ca6c86e61a3235dbc1 (2026-10-21) - the FR-Spec + KDA overlays land as ONE combined patch patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1 inside the copied sglang package; fp8_configs unchanged; patch -p1 --dry-run verified clean vs sglang==0.5.22 wheel). 0.5.22 hybrid (Gated DeltaNet) + MTP requires --enable-linear-replayssm-spec or KV-cache sizing fails. Published the author MTP5+FR-Spec table measured on real agent text (verbatim README @ c00cfec): MTP5 vs MTP4 both with FR-Spec: 149 vs 140 @1req (+6%), 117 vs 103 @2req (+13%), 77 vs 69 @4req (+12%); stock-SGLang MTP5 lost (96 vs 107 @1req -10%, 31 vs 34 @10req -9%; acceptance 2.90->3.02); synthetic bench (+42%, accept 4.19) rejected - decide draft settings on real text only. git/artifact/credit pins -> c00cfec525ef; HF model pin 32c92c6fd22f unchanged (idle since 2026-09-25 per t_8d91ff6e). Still author-measured T0, not re-benchmarked by us."},"engine_source":{"type":"release","git_url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode/tree/c00cfec525efb20e9c4185ca6c86e61a3235dbc1","base_engine":"sglang","pip_package":null,"base_version":"0.5.22","image_digest":null,"install_note":"Stock SGLang 0.5.22 + ONE combined overlay patch, no fork: apply patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch with patch -p1 inside the copied sglang package (README @ c00cfec525; verified to apply cleanly against sglang==0.5.22). The 0.5.21 lane (stock SGLang 0.5.21) instead applies patches/frspec_fp8_lmhead.patch and patches/kda_nvfp4_allowlist.patch with patch -p0. Copy the 7 fp8_configs/*.json (N=5120..248320, K=5120/17408, device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition, dtype=fp8_w8a8_channelwise; unchanged for 0.5.22) into sglang/kernels/ops/gemm/configs/, and stage tokens/hot_tokens_qwen38_es_en_code_40000.pt. Launch env: SGLANG_KDEV_KDA_NVFP4_KN and SGLANG_KDEV_KDA_NVFP4_MAX_M=16 (from the patch) plus SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE=1 (README). 0.5.22 + hybrid (Gated DeltaNet) models + MTP requires --enable-linear-replayssm-spec or KV-cache sizing fails (\"Loaded weights leave no GPU memory for the KV cache\"). JSONs are per (N,K,GPU). Token list length must stay a multiple of 64 or the pruned FP8 head falls back to an untuned kernel.","pinned_commit":"c00cfec525efb20e9c4185ca6c86e61a3235dbc1"},"credit":[{"name":"AppNetDeveloper","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode"},{"name":"SGLang 0.5.22 port + MTP5/FR-Spec real-text results @ c00cfec525 (published upstream 2026-10-21)","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode/commit/c00cfec525efb20e9c4185ca6c86e61a3235dbc1"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["python3","-m","sglang.launch_server","--model","bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ","--speculative-algorithm","NEXTN","--speculative-num-steps","3","--speculative-eagle-topk","1","--speculative-num-draft-tokens","4","--speculative-token-map","tokens/hot_tokens_qwen38_es_en_code_40000.pt","--kv-cache-dtype","fp8_e4m3","--enable-hierarchical-cache","--enable-linear-replayssm-spec"],"runtime":{"env":{"SGLANG_KDEV_KDA_NVFP4_KN":"5120x34816,17408x5120","SGLANG_KDEV_KDA_NVFP4_MAX_M":"16","SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE":"1"},"tokens":"tokens/hot_tokens_qwen38_es_en_code_40000.pt (40000 ids, multiple of 64)","not_helped":["--linear-attn-prefill-backend flashinfer","--enable-torch-compile","--bf16-gemm-backend gemv (requires SM90)","--fp8-gemm-backend (block-wise FP8 only)"],"source_pin":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode @ c00cfec525efb20e9c4185ca6c86e61a3235dbc1 (SGLang 0.5.22 port; v0.5.22 = baa9befa381511cd4d56f83980951e5a45fffc6d)","base_args_note":"README elides base serve args (python -m sglang.launch_server ...). Shown flags are the ones the repo states at 937460a; KV/HiCache flags rendered from stock 0.5.21 (e00930c5489053f26d86b179cee0d087f846acbb) spellings for the README's 'FP8 KV cache, HiCache'.","patches_applied":["patches/frspec_fp8_lmhead.patch (0.5.21 lane, patch -p0)","patches/kda_nvfp4_allowlist.patch (0.5.21 lane, patch -p0)","patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (live 0.5.22 lane, patch -p1)"],"verify_log_lines":["FR-Spec: draft lm_head pruned to 40000 rows (quantized, weight ...)","Using configuration from .../N=...json","KDA NVFP4 SM120 path (m, k, n) = ..."],"fp8_configs_installed":"fp8_configs/*.json -> <sglang>/kernels/ops/gemm/configs/ (valid only for NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition at these N,K)"}},"models":[{"slug":"qwen3-8-27b-nvfp4a4-awq","name":"Qwen3.8-27B-NVFP4A4-AWQ","hf_repo":"bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ","quant":"nvfp4"}],"hardware":[{"slug":"nvidia-rtx-pro-6000-blackwell","vendor":"nvidia","card":"rtx-pro-6000-blackwell","card_family":"rtx-pro-6000-blackwell","vram_gb":96,"device_count":1}],"generation":"blackwell","tags":["awq","blackwell","fp8","fp8-kv","fr-spec","hicache","hot-token-map","kda","mtp","nextn","nvfp4","sglang","sm120","speculative","w4a4"],"tasks":["chat","code"],"aa":{"aa_index":34,"aa_provenance":"published","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-07 18:00:00-06","aa_family_slug":"qwen-3-8-27b","aa_resolved_slug":"qwen-3-8-27b"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured 2026-10-14 (README results table @ 937460a; 'same server, same bench, 400 output tokens, unique prefixes'): 1req@45k-ctx 131.7 tok/s vs 98.5 baseline (+34%), FR-Spec alone 102.0; 4req@45k 52.1 vs 45.9 (+13.5%); 8req@27k 40.0 vs 38.7 (+3%); 1req@141k 116.6 vs 93.9 (+24%). +22-38% decode over 1h production traffic (gateway logs, responses>=300 tokens, bucketed by ctx), 0 server errors. Quality same both arms: reasoning 48/48 (3x16), tool calls 24/24, needle 4/4, MTP accept ~3.0. context_len on points = the ctx sizes named in the table (45k/27k/141k); bench prompt build not further specified in the repo. | MTP5 update (author-measured 2026-10-21, README @ c00cfec525; 'measured on real agent text': multi-turn coding sessions with ~73k-token prompts, 4 turns each; same server 1x RTX PRO 6000 Workstation power-capped at 450 W; EAGLE/MTP top-k 1; output tokens/s per request): MTP5 + FR-Spec vs MTP4 + FR-Spec 1req 149 vs 140 (+6%), 2req 117 vs 103 (+13%), 4req 77 vs 69 (+12%). With stock SGLang (no FR-Spec) MTP5 lost: 1req 96 vs 107 (-10%), 10req 31 vs 34 (-9%); real acceptance only 2.90 -> 3.02 tokens/step. In production MTP5 + FR-Spec is now the author default (--speculative-num-steps 5 --speculative-num-draft-tokens 6 --speculative-token-map <40k list>). Lesson (verbatim): decide draft settings on real text only - a synthetic bench reported +42% for MTP5 with mean acceptance 4.19; on real traffic acceptance was ~3.0. Points context_len 74752 = 73*1024 per the repo's k->1024 convention.\", \"tok_s\": 131.69999999999998863131622783839702606201171875, \"context_len\": 46080}"}],"perf_points":[{"id":1943,"context_len":27648,"prefill_tok_s":null,"decode_tok_s":40,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"27k ctx; baseline 38.7 (+3%). FR-Spec only: 38.9. Attention dominates at 8+ concurrent."},{"id":1941,"context_len":46080,"prefill_tok_s":null,"decode_tok_s":131.69999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"45k ctx, patched all-three overlay; baseline 98.5 (+34%). FR-Spec only: 102.0."},{"id":1942,"context_len":46080,"prefill_tok_s":null,"decode_tok_s":52.10000000000000142108547152020037174224853515625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"45k ctx; baseline 45.9 (+13.5%). FR-Spec only: 49.7."},{"id":1945,"context_len":74752,"prefill_tok_s":null,"decode_tok_s":149,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"MTP5+FR-Spec 149 vs MTP4+FR-Spec 140 (+6%), measured on real agent text (multi-turn coding sessions, ~73k-token prompts, 4 turns each, 450 W cap; README @ c00cfec, verbatim). Stock SGLang MTP5 lost: 96 vs 107 (-10%); real acceptance 2.90->3.02. In production MTP5 + FR-Spec is now our default (--speculative-num-steps 5 --speculative-num-draft-tokens 6 --speculative-token-map <40k list>). Synthetic bench (+42%, mean accept 4.19) rejected vs real text."},{"id":1946,"context_len":74752,"prefill_tok_s":null,"decode_tok_s":117,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"MTP5+FR-Spec 117 vs MTP4+FR-Spec 103 (+13%), same real-agent-text bench as the 1-req row (README @ c00cfec, verbatim)."},{"id":1947,"context_len":74752,"prefill_tok_s":null,"decode_tok_s":77,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"MTP5+FR-Spec 77 vs MTP4+FR-Spec 69 (+12%), same real-agent-text bench as the 1-req row (README @ c00cfec, verbatim). Stock-SGLang negative 10req: 31 vs 34 (-9%)."},{"id":1944,"context_len":144384,"prefill_tok_s":null,"decode_tok_s":116.599999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"141k ctx; baseline 93.9 (+24%). FR-Spec only: 104.7."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode.json"},"requirements":null,"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode","pin":"c00cfec525efb20e9c4185ca6c86e61a3235dbc1"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ","pin":"32c92c6fd22fe633bf382609839050147e23f98a"}],"links":{"self":"/api/v1/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode.json","alternate":"/r/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode","describedby":"/openapi.json"},"updated_at":"2026-10-28 16:25:30.123672-06","created_at":"2026-10-20 21:49:16.469378-06","speeds":{"decode_tok_s":131.69999999999998863131622783839702606201171875,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}