diff --git a/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode/recipe.yaml b/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode/recipe.yaml index 99fdc9ecc..ba2120550 100644 --- a/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode/recipe.yaml +++ b/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode/recipe.yaml @@ -2,12 +2,12 @@ slug: qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode title: 'Qwen3.8-27B NVFP4A4-AWQ 1x RTX PRO 6000 SGLang Fast Decode' engine: name: sglang - version: 0.5.21 + version: '0.5.22' source_type: release git_url: 'https://github.com/AppNetDeveloper/sglang-sm120-fast-decode' - pinned_commit: 937460a718819c7b646b48b8ce67290896f96b97 - install_note: 'Stock SGLang 0.5.21 + patch overlays, no fork: apply patches/frspec_fp8_lmhead.patch and patches/kda_nvfp4_allowlist.patch with patch -p0 inside the sglang tree, copy the 7 fp8_configs/*.json (N=5120..248320, K=5120/17408, device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition, dtype=fp8_w8a8_channelwise) into sglang/kernels/ops/gemm/configs/, and stage tokens/hot_tokens_qwen38_es_en_code_40000.pt. Launch env: SGLANG_KDEV_KDA_NVFP4_KN and SGLANG_KDEV_KDA_NVFP4_MAX_M=16 (from the patch) plus SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE=1 (README). Overlays qualify on SGLang 0.5.21 only; JSONs are per (N,K,GPU). Token list length must stay a multiple of 64 or the pruned FP8 head falls back to an untuned kernel.' -summary: 'Recipe by AppNetDeveloper. ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ (compressed-tensors: NVFP4 W4A4 MLPs, FP8 attention/GDN/lm_head, bf16 MTP head) served on one RTX PRO 6000 Blackwell (SM120) with stock SGLang 0.5.21 plus three engine-side overlays, no fork: FR-Spec pruned draft lm_head for a quantized FP8 head (slices weight AND per-row scale along the vocab axis, bit-identical kept rows); KDA NVFP4 SM120 GEMM shape allowlist extension (KN=5120x34816,17408x5120, MAX_M 16) reaching 85-89% DRAM bandwidth at M<=16 vs 50-70% default; tuned triton_scaled_mm fp8_w8a8_channelwise JSONs (16x32x256; null from M=24 so prefill keeps CUTLASS). NEXTN/MTP 3/1/4, FP8_e4m3 KV cache, HiCache, 40k hot-token draft list. Measured 2026-10-14 by the author (400 output tokens, unique prefixes): 131.7 tok/s single-stream at 45k ctx vs 98.5 baseline (+34%); 4req@45k +13.5%; 8req@27k +3%; 1req@141k +24%. Same quality both arms and no extra memory; 1h production traffic +15..38%, 0 errors.' + pinned_commit: c00cfec525efb20e9c4185ca6c86e61a3235dbc1 + install_note: 'Stock SGLang 0.5.22 + ONE combined overlay patch, no fork: apply patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch with patch -p1 inside the copied sglang package (README @ c00cfec525; verified to apply cleanly against sglang==0.5.22). The 0.5.21 lane (stock SGLang 0.5.21) instead applies patches/frspec_fp8_lmhead.patch and patches/kda_nvfp4_allowlist.patch with patch -p0. Copy the 7 fp8_configs/*.json (N=5120..248320, K=5120/17408, device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition, dtype=fp8_w8a8_channelwise; unchanged for 0.5.22) into sglang/kernels/ops/gemm/configs/, and stage tokens/hot_tokens_qwen38_es_en_code_40000.pt. Launch env: SGLANG_KDEV_KDA_NVFP4_KN and SGLANG_KDEV_KDA_NVFP4_MAX_M=16 (from the patch) plus SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE=1 (README). 0.5.22 + hybrid (Gated DeltaNet) models + MTP requires --enable-linear-replayssm-spec or KV-cache sizing fails ("Loaded weights leave no GPU memory for the KV cache"). JSONs are per (N,K,GPU). Token list length must stay a multiple of 64 or the pruned FP8 head falls back to an untuned kernel.' +summary: 'Recipe by AppNetDeveloper. ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ on 1x RTX PRO 6000 Blackwell (SM120) with stock SGLang 0.5.22 plus engine overlays, no fork: FR-Spec pruned draft lm_head for a quantized FP8 head (slices weight AND per-row scale along vocab, bit-identical kept rows); KDA NVFP4 SM120 GEMM allowlist extension (KN=5120x34816,17408x5120) 85-89% DRAM bandwidth at M<=16 vs 50-70% default; tuned triton_scaled_mm fp8_w8a8_channelwise JSONs. On 0.5.22 the overlays land as ONE patch, patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1). NEXTN/MTP, FP8_e4m3 KV cache, HiCache, 40k hot-token draft list; hybrid GDN+MTP needs --enable-linear-replayssm-spec. Author bench 2026-10-14 (400 output tokens, unique prefixes): 131.7 tok/s 1req 45k ctx vs 98.5 (+34%); 4req@45k +13.5%; 8req@27k +3%; 1req@141k +24%; same quality, no extra memory. MTP5 + FR-Spec is the author production default; measured on real agent text: 149 vs 140 @1req, 117 vs 103 @2req, 77 vs 69 @4req over MTP4.' license: CC-BY-4.0 model: name: Qwen3.8-27B-NVFP4A4-AWQ @@ -62,6 +62,7 @@ config: - '--kv-cache-dtype' - fp8_e4m3 - '--enable-hierarchical-cache' + - '--enable-linear-replayssm-spec' runtime: env: SGLANG_KDEV_KDA_NVFP4_KN: '5120x34816,17408x5120' @@ -69,8 +70,9 @@ config: SGLANG_ENABLE_FP8_GEMM_CONFIG_TUNE: '1' base_args_note: "README elides base serve args (python -m sglang.launch_server ...). Shown flags are the ones the repo states at 937460a; KV/HiCache flags rendered from stock 0.5.21 (e00930c5489053f26d86b179cee0d087f846acbb) spellings for the README's 'FP8 KV cache, HiCache'." patches_applied: - - patches/frspec_fp8_lmhead.patch - - patches/kda_nvfp4_allowlist.patch + - 'patches/frspec_fp8_lmhead.patch (0.5.21 lane, patch -p0)' + - 'patches/kda_nvfp4_allowlist.patch (0.5.21 lane, patch -p0)' + - 'patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (live 0.5.22 lane, patch -p1)' fp8_configs_installed: 'fp8_configs/*.json -> /kernels/ops/gemm/configs/ (valid only for NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition at these N,K)' tokens: 'tokens/hot_tokens_qwen38_es_en_code_40000.pt (40000 ids, multiple of 64)' verify_log_lines: @@ -82,40 +84,27 @@ config: - '--enable-torch-compile' - '--bf16-gemm-backend gemv (requires SM90)' - '--fp8-gemm-backend (block-wise FP8 only)' - source_pin: 'https://github.com/AppNetDeveloper/sglang-sm120-fast-decode @ 937460a718819c7b646b48b8ce67290896f96b97' + source_pin: 'https://github.com/AppNetDeveloper/sglang-sm120-fast-decode @ c00cfec525efb20e9c4185ca6c86e61a3235dbc1 (SGLang 0.5.22 port; v0.5.22 = baa9befa381511cd4d56f83980951e5a45fffc6d)' performance: claimed_by: submitter metrics: tok_s: 131.7 context_len: 46080 - notes: "Author-measured 2026-10-14 (README results table @ 937460a; 'same server, same bench, 400 output tokens, unique prefixes'): 1req@45k-ctx 131.7 tok/s vs 98.5 baseline (+34%), FR-Spec alone 102.0; 4req@45k 52.1 vs 45.9 (+13.5%); 8req@27k 40.0 vs 38.7 (+3%); 1req@141k 116.6 vs 93.9 (+24%). +22-38% decode over 1h production traffic (gateway logs, responses>=300 tokens, bucketed by ctx), 0 server errors. Quality same both arms: reasoning 48/48 (3x16), tool calls 24/24, needle 4/4, MTP accept ~3.0. context_len on points = the ctx sizes named in the table (45k/27k/141k); bench prompt build not further specified in the repo." + notes: "Author-measured 2026-10-14 (README results table @ 937460a; 'same server, same bench, 400 output tokens, unique prefixes'): 1req@45k-ctx 131.7 tok/s vs 98.5 baseline (+34%), FR-Spec alone 102.0; 4req@45k 52.1 vs 45.9 (+13.5%); 8req@27k 40.0 vs 38.7 (+3%); 1req@141k 116.6 vs 93.9 (+24%). +22-38% decode over 1h production traffic (gateway logs, responses>=300 tokens, bucketed by ctx), 0 server errors. Quality same both arms: reasoning 48/48 (3x16), tool calls 24/24, needle 4/4, MTP accept ~3.0. context_len on points = the ctx sizes named in the table (45k/27k/141k); bench prompt build not further specified in the repo. | MTP5 update (author-measured 2026-10-21, README @ c00cfec525; 'measured on real agent text': multi-turn coding sessions with ~73k-token prompts, 4 turns each; same server 1x RTX PRO 6000 Workstation power-capped at 450 W; EAGLE/MTP top-k 1; output tokens/s per request): MTP5 + FR-Spec vs MTP4 + FR-Spec 1req 149 vs 140 (+6%), 2req 117 vs 103 (+13%), 4req 77 vs 69 (+12%). With stock SGLang (no FR-Spec) MTP5 lost: 1req 96 vs 107 (-10%), 10req 31 vs 34 (-9%); real acceptance only 2.90 -> 3.02 tokens/step. In production MTP5 + FR-Spec is now the author default (--speculative-num-steps 5 --speculative-num-draft-tokens 6 --speculative-token-map <40k list>). Lesson (verbatim): decide draft settings on real text only - a synthetic bench reported +42% for MTP5 with mean acceptance 4.19; on real traffic acceptance was ~3.0. Points context_len 74752 = 73*1024 per the repo's k->1024 convention." points: - - - context_len: 46080 - concurrency: 1 - decode_tok_s: 131.7 - notes: '45k ctx, patched all-three overlay; baseline 98.5 (+34%). FR-Spec only: 102.0.' - - - context_len: 46080 - concurrency: 4 - decode_tok_s: 52.1 - notes: '45k ctx; baseline 45.9 (+13.5%). FR-Spec only: 49.7.' - - - context_len: 27648 - concurrency: 8 - decode_tok_s: 40.0 - notes: '27k ctx; baseline 38.7 (+3%). FR-Spec only: 38.9. Attention dominates at 8+ concurrent.' - - - context_len: 144384 - concurrency: 1 - decode_tok_s: 116.6 - notes: '141k ctx; baseline 93.9 (+24%). FR-Spec only: 104.7.' + - { context_len: 46080, concurrency: 1, decode_tok_s: 131.7, notes: '45k ctx, patched all-three overlay; baseline 98.5 (+34%). FR-Spec only: 102.0.' } + - { context_len: 46080, concurrency: 4, decode_tok_s: 52.1, notes: '45k ctx; baseline 45.9 (+13.5%). FR-Spec only: 49.7.' } + - { context_len: 27648, concurrency: 8, decode_tok_s: 40.0, notes: '27k ctx; baseline 38.7 (+3%). FR-Spec only: 38.9. Attention dominates at 8+ concurrent.' } + - { context_len: 144384, concurrency: 1, decode_tok_s: 116.6, notes: '141k ctx; baseline 93.9 (+24%). FR-Spec only: 104.7.' } + - { context_len: 74752, concurrency: 1, decode_tok_s: 149.0, notes: 'MTP5+FR-Spec 149 vs MTP4+FR-Spec 140 (+6%), measured on real agent text (multi-turn coding sessions, ~73k-token prompts, 4 turns each, 450 W cap; README @ c00cfec, verbatim). Stock SGLang MTP5 lost: 96 vs 107 (-10%); real acceptance 2.90->3.02. In production MTP5 + FR-Spec is now our default (--speculative-num-steps 5 --speculative-num-draft-tokens 6 --speculative-token-map <40k list>). Synthetic bench (+42%, mean accept 4.19) rejected vs real text.' } + - { context_len: 74752, concurrency: 2, decode_tok_s: 117.0, notes: 'MTP5+FR-Spec 117 vs MTP4+FR-Spec 103 (+13%), same real-agent-text bench as the 1-req row (README @ c00cfec, verbatim).' } + - { context_len: 74752, concurrency: 4, decode_tok_s: 77.0, notes: 'MTP5+FR-Spec 77 vs MTP4+FR-Spec 69 (+12%), same real-agent-text bench as the 1-req row (README @ c00cfec, verbatim). Stock-SGLang negative 10req: 31 vs 34 (-9%).' } artifacts: - kind: reference pointer_type: git_commit pointer: 'https://github.com/AppNetDeveloper/sglang-sm120-fast-decode' - pin: 937460a718819c7b646b48b8ce67290896f96b97 + pin: c00cfec525efb20e9c4185ca6c86e61a3235dbc1 - kind: model_weights pointer_type: hf_model @@ -127,12 +116,12 @@ provenance: name: AppNetDeveloper url: 'https://github.com/AppNetDeveloper/sglang-sm120-fast-decode' - - name: 'SGLang 0.5.22 port + MTP5+FR-Spec results published upstream 2026-10-21 (same 0.5.21 build): patches/sglang-0.5.22/frspe' - url: 'https://github.com/AppNetDeveloper/sglang-sm120-fast-decode/tree/main/patches' + name: 'SGLang 0.5.22 port + MTP5/FR-Spec real-text results @ c00cfec525 (published upstream 2026-10-21)' + url: 'https://github.com/AppNetDeveloper/sglang-sm120-fast-decode/commit/c00cfec525efb20e9c4185ca6c86e61a3235dbc1' source_urls: - 'https://github.com/AppNetDeveloper/sglang-sm120-fast-decode' - 'https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ' - notes: 'Scout ingest (localai-recipes) of AppNetDeveloper/sglang-sm120-fast-decode @ 937460a718819c7b646b48b8ce67290896f96b97 (HEAD 2026-10-14, created+measured same day). All launch/env/patch/config facts transcribed from that tree: README.md results table + how-to-apply env exports, both patch files, 7 fp8_configs JSONs (device-locked N,K names), tokens/ 40000-id hot list. Repo shows no full serve script; base serve args elided in README, so config.launch carries the stated flags and 0.5.21 spellings for FP8-KV/HiCache. Hot-token list built only from public HyperQwen + MiaAI-Lab es_en_code_65k lists per README. Model card: 23.4 GB on disk; license:other on HF (thinking-cap model card); recipe yaml licensed CC-BY-4.0. Not re-benchmarked by us; T0 submitter-provenance throughout.' + notes: 'Scout ingest (localai-recipes) of AppNetDeveloper/sglang-sm120-fast-decode @ 937460a718819c7b646b48b8ce67290896f96b97 (HEAD 2026-10-14, created+measured same day). All launch/env/patch/config facts transcribed from that tree: README.md results table + how-to-apply env exports, both patch files, 7 fp8_configs JSONs (device-locked N,K names), tokens/ 40000-id hot list. Repo shows no full serve script; base serve args elided in README, so config.launch carries the stated flags and 0.5.21 spellings for FP8-KV/HiCache. Hot-token list built only from public HyperQwen + MiaAI-Lab es_en_code_65k lists per README. Model card: 23.4 GB on disk; license:other on HF (thinking-cap model card); recipe yaml licensed CC-BY-4.0. Not re-benchmarked by us; T0 submitter-provenance throughout. 2026-10-28 (t_ad75e28d, owner Tier-2 go 2026-10-28): SGLang base bumped 0.5.21->0.5.22 per upstream c00cfec525efb20e9c4185ca6c86e61a3235dbc1 (2026-10-21) - the FR-Spec + KDA overlays land as ONE combined patch patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1 inside the copied sglang package; fp8_configs unchanged; patch -p1 --dry-run verified clean vs sglang==0.5.22 wheel). 0.5.22 hybrid (Gated DeltaNet) + MTP requires --enable-linear-replayssm-spec or KV-cache sizing fails. Published the author MTP5+FR-Spec table measured on real agent text (verbatim README @ c00cfec): MTP5 vs MTP4 both with FR-Spec: 149 vs 140 @1req (+6%), 117 vs 103 @2req (+13%), 77 vs 69 @4req (+12%); stock-SGLang MTP5 lost (96 vs 107 @1req -10%, 31 vs 34 @10req -9%; acceptance 2.90->3.02); synthetic bench (+42%, accept 4.19) rejected - decide draft settings on real text only. git/artifact/credit pins -> c00cfec525ef; HF model pin 32c92c6fd22f unchanged (idle since 2026-09-25 per t_8d91ff6e). Still author-measured T0, not re-benchmarked by us.' storage_requirements: ssd_gb: 23.4 note: 'Model card: 23.4 GB on disk (recipe provenance notes state the size); download-size statement only, no RAM requirement stated'