Qwen3.8-Flash-Next NVFP4 RTX PRO 6000 Blackwell SM120 SGLang

community Community submission — verify on your own hardware.

recipe by yhfgyyf↗, sgl-project (SGLang upstream + PRs 36497/36567/36556)↗ · listed by localai-recipes

revision r2 · commit 3b38bdadad · updated 10 d ago · license CC-BY-4.0

chat code mtp-speculative sglang nvme-ple io-uring breakable-graphs sm120 yhfgyyf gdn-fusion qsa-sparse-decode

Experimental SGLang fork for Qwen3.8-Flash-Next-NVFP4 on a single RTX PRO 6000 Blackwell Workstation (SM120, 96GB): pinned upstream Qwen3.8 (PR #36497 @ 73a255206f), NVMe-backed PLE (PR #36567 @ e14d1c3cb6/405e494199/2179ad9bcb/d4477bd298), SM120 QSA sparse decode (PR #36556, snapshot 93fd20af6f - extends the sparse GQA decode path from SM121 DOWN to SM120 so no incompatible fallback fires) + repo additions: (1) breakable-CUDA-graph dataclass outputs (recursive alloc/copy/slice so a 1-request speculative target-verify graph safely holds 4 rows of verify logits + hidden states; CPU unit coverage), (2) hot-PLE row gathering (all-hit LRU fast path avoiding page plans + duplicate-row reuse for repetitive batches; commit 7df4aea02b), (3) GDN projection + Conv1D decode fusion (commit b2f0b02763, SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV=1 default), (4) QSA indexer query-tile tuning to 8 rows for SM120 w/ 4 index heads dim 128 + text RoPE bound from known CPU seq lens (commit 2b56a39e70). Meas

Launch command

python -m sglang.launch_server --model-path '$MODEL_PATH' --quantization modelopt_fp4 --fp4-gemm-backend flashinfer_cutlass --tp-size 1 --dtype bfloat16 --page-size 64 --mamba-radix-cache-strategy extra_buffer --mamba-track-interval 64 --linear-attn-backend triton --chunked-prefill-size 8192 --max-prefill-tokens 16384 --max-running-requests 1 --context-length 262144 --max-total-tokens 262144 --mem-fraction-static 0.98 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --cuda-graph-backend-decode breakable --cuda-graph-backend-prefill tc_piecewise --cuda-graph-max-bs-decode 1 --cuda-graph-bs-decode 1 --cuda-graph-max-bs-prefill 8192 --cuda-graph-bs-prefill 8192 --host 127.0.0.1 --port 30000

runtime

env: 'CUDA_VISIBLE_DEVICES=0 | CUDA_HOME/CUDA_PATH/CUDACXX=<cuda 13.2 root> | FLASHINFER_CUDA_ARCH_LIST=12.0f | PYTORCH_ALLOC_CONF=expandable_segments:True | SGLANG_QWEN4_PLE_NVME_PATH=$MODEL_PATH | SGLANG_QWEN4_PLE_NVME_BACKEND=io_uring | SGLANG_QWEN4_PLE_NVME_QUEUE_DEPTH=512 | SGLANG_QWEN4_PLE_NVME_MAX_BATCH_PAGES=4096 | SGLANG_QWEN4_PLE_NVME_CACHE_PAGES=5242880 (20GiB LRU payload) | SGLANG_QWEN4_PLE_NVME_LOG_INTERVAL=0 | SGLANG_QWEN4_PLE_NVME_NVTX=1 | SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV=1'
notes: "Launch = README 'Launch: default 256K context' @ 4cacd4e5 verbatim (env block + argv transcribed token-for-token). With this EAGLE + FP4/MoE combination the correctness safeguard auto-disables target-prefill tc_piecewise replay: actual ordinary prefill was EAGER in all reported A/B arms - do not bypass or interpret the configured prefill graph size as proof replay was active (repo's own callout). Total input+output must not exceed --max-total-tokens; PLE path requires --tp-size 1. Optional 512MiB exact-row cache (SGLANG_QWEN4_PLE_NVME_CACHE_MODE=row + CACHE_BYTES=536870912 + CACHE_PAGES=0): byte budget counts retained payload + Python containers, NOT process RSS; mmap backend does not support bounded caching. Multi-turn checks use --reasoning-parser qwen3 --tool-call-parser qwen3_coder. NVTX ranges (disk_read_pages/read_rows/future_wait/stage_and_copy) are profiling-only; future_wait includes CPU work - do not sum across overlap."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang.json

Engine

sglang · source↗ · 4cacd4e5↗

Install note: Source install to reproduce the published numbers (release wheel v0.1.0 predates the GDN/PLE/QSA optimization commits): git clone @ pin, python -m pip install -e ./python with CUDA 13.2 nvcc first on PATH (CUDA_HOME/CUDA_PATH/CUDACXX exported) + FLASHINFER_CUDA_ARCH_LIST=12.0f; needs Rust storage extension (bundled in the build) for the NVMe PLE backend. Validated env: Ubuntu 22.04 / Linux 6.8 / driver 595.45.04 / CUDA 13.2 nvcc 13.2.51 / Python 3.12. The 2026-10-06 HEAD (4cacd4e5) adds an opt-in byte-budgeted FP8 row cache (#1) on top of the measured 2b56a39e70 baseline - measurement pins below cite the measured commit.

Model

Qwen3.8-Flash-Next-NVFP4 (nvfp4) · Qwen/Qwen3.8-Flash-Next↗
AA intelligence 40 (Artificial Analysis · measured)

Hardware

NVIDIA RTX PRO 6000 Blackwell 96GB ×1

Context

max context: 256K tokens

KV pool

256K tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
8,192	1	14235.1 tok/s	222.6 tok/s	575 ms	reported
32,768	1	13945.9 tok/s	227.8 tok/s	2350 ms	reported
131,072	1	12975.6 tok/s	224.1 tok/s	10101 ms	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Author-measured 2026-09-23 @ commit 2b56a39e7045d6ec7c2e426b8d4b9df57030b61b (README performance table, published-source baseline, one SM120 GPU, C1, 262,144-token ctx, 8,192-token chunks, 512 output tokens, NEXTN/EAGLE 3/1/4, NVMe io_uring PLE w/ 20GiB host LRU, no profiler; 5 fixed prompts per length, median; each length gets a 512-output conditioning request and a 1-output PLE warm before every measured request, KV/radix flushed; all 15 samples cached_tokens=0, exactly 512 out, temp 0, ignore

notes

Author-measured 2026-09-23 @ commit 2b56a39e7045d6ec7c2e426b8d4b9df57030b61b (README performance table, published-source baseline, one SM120 GPU, C1, 262,144-token ctx, 8,192-token chunks, 512 output tokens, NEXTN/EAGLE 3/1/4, NVMe io_uring PLE w/ 20GiB host LRU, no profiler; 5 fixed prompts per length, median; each length gets a 512-output conditioning request and a 1-output PLE warm before every measured request, KV/radix flushed; all 15 samples cached_tokens=0, exactly 512 out, temp 0, ignore

Downloads

Weights https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4
checkpoint used for validation (README does not pin an hf_revision - PLE tables FP8, ~47.68 GiB in the tested checkpoint); NVMe volume with O_DIRECT + io_uring support required
Reference https://github.com/yhfgyyf/sglang-qwen38-flash-next-sm120
4cacd4e586743600c35ae6748aa7b2b4fc5b151a (HEAD 2026-10-06, opt-in FP8 row cache #1); MEASURED baseline @ 2b56a39e7045d6ec7c2e426b8d4b9df57030b61b (optimization commits 7df4aea02b hot-PLE LRU, b2f0b02763 GDN fusion); upstream pins PR #36497 @ 73a255206f, PR

Revision notes

Ingested from GitHub repo yhfgyyf/sglang-qwen38-flash-next-sm120 @ 4cacd4e586743600c35ae6748aa7b2b4fc5b151a (wave 3, kanban t_21a3a083). SM120 = RTX PRO 6000 Blackwell, same hw family as yepapa-nest's recipe but a DIFFERENT fork lineage (this repo's own SM120 QSA sparse-decode extension + breakable-graph dataclass outputs vs yepapa's jzinno-tree overlay). Highest-quality measurement section of wave 3: medians of 5 with cache hygiene documented. Bilingual README (EN + Chinese). The native sparse-GQA prototype is explicitly NOT published/enabled - do not attribute its numbers to this recipe.

Revision history

1131f04bbc recipe: update qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang (r2: /max_context_len, /kv_pool_tokens) 10 d ago diff
a889eb4390 recipe: create qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang 23 d ago

Fork

Export

recipe.yaml (raw)