{"slug":"qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang","title":"Qwen3.8-Flash-Next NVFP4 RTX PRO 6000 Blackwell SM120 SGLang","summary":"Experimental SGLang fork for Qwen3.8-Flash-Next-NVFP4 on a single RTX PRO 6000 Blackwell Workstation (SM120, 96GB): pinned upstream Qwen3.8 (PR #36497 @ 73a255206f), NVMe-backed PLE (PR #36567 @ e14d1c3cb6/405e494199/2179ad9bcb/d4477bd298), SM120 QSA sparse decode (PR #36556, snapshot 93fd20af6f - extends the sparse GQA decode path from SM121 DOWN to SM120 so no incompatible fallback fires) + repo additions: (1) breakable-CUDA-graph dataclass outputs (recursive alloc/copy/slice so a 1-request speculative target-verify graph safely holds 4 rows of verify logits + hidden states; CPU unit coverage), (2) hot-PLE row gathering (all-hit LRU fast path avoiding page plans + duplicate-row reuse for repetitive batches; commit 7df4aea02b), (3) GDN projection + Conv1D decode fusion (commit b2f0b02763, SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV=1 default), (4) QSA indexer query-tile tuning to 8 rows for SM120 w/ 4 index heads dim 128 + text RoPE bound from known CPU seq lens (commit 2b56a39e70). Meas","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"3b38bdadad8817372ec43bc98243ffb2e37523d6","content_hash":"372a4539d8514d0fbc2308f988545bff470cf6ae","created_at":"2026-10-20 15:34:12.156597-06","notes":"Ingested from GitHub repo yhfgyyf/sglang-qwen38-flash-next-sm120 @ 4cacd4e586743600c35ae6748aa7b2b4fc5b151a (wave 3, kanban t_21a3a083). SM120 = RTX PRO 6000 Blackwell, same hw family as yepapa-nest's recipe but a DIFFERENT fork lineage (this repo's own SM120 QSA sparse-decode extension + breakable-graph dataclass outputs vs yepapa's jzinno-tree overlay). Highest-quality measurement section of wave 3: medians of 5 with cache hygiene documented. Bilingual README (EN + Chinese). The native sparse-GQA prototype is explicitly NOT published/enabled - do not attribute its numbers to this recipe."},"engine_source":{"type":"fork","git_url":"https://github.com/yhfgyyf/sglang-qwen38-flash-next-sm120","base_engine":"sglang","pip_package":null,"base_version":"0.5.19.dev series (repo fork)","image_digest":null,"install_note":"Source install to reproduce the published numbers (release wheel v0.1.0 predates the GDN/PLE/QSA optimization commits): git clone @ pin, python -m pip install -e ./python with CUDA 13.2 nvcc first on PATH (CUDA_HOME/CUDA_PATH/CUDACXX exported) + FLASHINFER_CUDA_ARCH_LIST=12.0f; needs Rust storage extension (bundled in the build) for the NVMe PLE backend. Validated env: Ubuntu 22.04 / Linux 6.8 / driver 595.45.04 / CUDA 13.2 nvcc 13.2.51 / Python 3.12. The 2026-10-06 HEAD (4cacd4e5) adds an opt-in byte-budgeted FP8 row cache (#1) on top of the measured 2b56a39e70 baseline - measurement pins below cite the measured commit.","pinned_commit":"4cacd4e586743600c35ae6748aa7b2b4fc5b151a"},"credit":[{"name":"yhfgyyf","url":"https://github.com/yhfgyyf/sglang-qwen38-flash-next-sm120"},{"name":"sgl-project (SGLang upstream + PRs 36497/36567/36556)","url":"https://github.com/sgl-project/sglang"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["python","-m","sglang.launch_server","--model-path","$MODEL_PATH","--quantization","modelopt_fp4","--fp4-gemm-backend","flashinfer_cutlass","--tp-size","1","--dtype","bfloat16","--page-size","64","--mamba-radix-cache-strategy","extra_buffer","--mamba-track-interval","64","--linear-attn-backend","triton","--chunked-prefill-size","8192","--max-prefill-tokens","16384","--max-running-requests","1","--context-length","262144","--max-total-tokens","262144","--mem-fraction-static","0.98","--speculative-algorithm","NEXTN","--speculative-num-steps","3","--speculative-eagle-topk","1","--speculative-num-draft-tokens","4","--cuda-graph-backend-decode","breakable","--cuda-graph-backend-prefill","tc_piecewise","--cuda-graph-max-bs-decode","1","--cuda-graph-bs-decode","1","--cuda-graph-max-bs-prefill","8192","--cuda-graph-bs-prefill","8192","--host","127.0.0.1","--port","30000"],"runtime":{"env":"CUDA_VISIBLE_DEVICES=0 | CUDA_HOME/CUDA_PATH/CUDACXX=<cuda 13.2 root> | FLASHINFER_CUDA_ARCH_LIST=12.0f | PYTORCH_ALLOC_CONF=expandable_segments:True | SGLANG_QWEN4_PLE_NVME_PATH=$MODEL_PATH | SGLANG_QWEN4_PLE_NVME_BACKEND=io_uring | SGLANG_QWEN4_PLE_NVME_QUEUE_DEPTH=512 | SGLANG_QWEN4_PLE_NVME_MAX_BATCH_PAGES=4096 | SGLANG_QWEN4_PLE_NVME_CACHE_PAGES=5242880 (20GiB LRU payload) | SGLANG_QWEN4_PLE_NVME_LOG_INTERVAL=0 | SGLANG_QWEN4_PLE_NVME_NVTX=1 | SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV=1","notes":"Launch = README 'Launch: default 256K context' @ 4cacd4e5 verbatim (env block + argv transcribed token-for-token). With this EAGLE + FP4/MoE combination the correctness safeguard auto-disables target-prefill tc_piecewise replay: actual ordinary prefill was EAGER in all reported A/B arms - do not bypass or interpret the configured prefill graph size as proof replay was active (repo's own callout). Total input+output must not exceed --max-total-tokens; PLE path requires --tp-size 1. Optional 512MiB exact-row cache (SGLANG_QWEN4_PLE_NVME_CACHE_MODE=row + CACHE_BYTES=536870912 + CACHE_PAGES=0): byte budget counts retained payload + Python containers, NOT process RSS; mmap backend does not support bounded caching. Multi-turn checks use --reasoning-parser qwen3 --tool-call-parser qwen3_coder. NVTX ranges (disk_read_pages/read_rows/future_wait/stage_and_copy) are profiling-only; future_wait includes CPU work - do not sum across overlap."},"kv_pool_tokens":262144,"max_context_len":262144},"models":[{"slug":"qwen3-8-flash-next-nvfp4","name":"Qwen3.8-Flash-Next-NVFP4","hf_repo":"Qwen/Qwen3.8-Flash-Next","quant":"nvfp4"}],"hardware":[{"slug":"nvidia-rtx-pro-6000-blackwell","vendor":"nvidia","card":"rtx-pro-6000-blackwell","card_family":"rtx-pro-6000-blackwell","vram_gb":96,"device_count":1}],"generation":"blackwell","tags":["breakable-graphs","gdn-fusion","io-uring","mtp-speculative","nvme-ple","qsa-sparse-decode","sglang","sm120","yhfgyyf"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured 2026-09-23 @ commit 2b56a39e7045d6ec7c2e426b8d4b9df57030b61b (README performance table, published-source baseline, one SM120 GPU, C1, 262,144-token ctx, 8,192-token chunks, 512 output tokens, NEXTN/EAGLE 3/1/4, NVMe io_uring PLE w/ 20GiB host LRU, no profiler; 5 fixed prompts per length, median; each length gets a 512-output conditioning request and a 1-output PLE warm before every measured request, KV/radix flushed; all 15 samples cached_tokens=0, exactly 512 out, temp 0, ignore\", \"tok_s\": 222.599999999999994315658113919198513031005859375, \"points\": [{\"notes\": \"README perf table @ 2b56a39e70 (median of 5): 8K prefill 14,235.125 / decode 222.632 (agg 223.173) / TTFT 575.478ms\", \"ttft_ms\": 575, \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 222.6299999999999954525264911353588104248046875, \"prefill_tok_s\": 14235.129999999999199644662439823150634765625}, {\"notes\": \"README perf table: 32K prefill 13,945.911 / decode 227.779 (agg 227.814) / TTFT 2,349.649ms\", \"ttft_ms\": 2350, \"concurrency\": 1, \"context_len\": 32768, \"decode_tok_s\": 227.780000000000001136868377216160297393798828125, \"prefill_tok_s\": 13945.90999999999985448084771633148193359375}, {\"notes\": \"README perf table: 128K prefill 12,975.567 / decode 224.134 (agg 222.922) / TTFT 10,101.447ms\", \"ttft_ms\": 10101, \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 224.1299999999999954525264911353588104248046875, \"prefill_tok_s\": 12975.5699999999997089616954326629638671875}], \"context_len\": 8192}"}],"perf_points":[{"id":1054,"context_len":8192,"prefill_tok_s":14235.129999999999199644662439823150634765625,"decode_tok_s":222.6299999999999954525264911353588104248046875,"ttft_ms":575,"concurrency":1,"claimed_by":"submitter","notes":"README perf table @ 2b56a39e70 (median of 5): 8K prefill 14,235.125 / decode 222.632 (agg 223.173) / TTFT 575.478ms"},{"id":1055,"context_len":32768,"prefill_tok_s":13945.90999999999985448084771633148193359375,"decode_tok_s":227.780000000000001136868377216160297393798828125,"ttft_ms":2350,"concurrency":1,"claimed_by":"submitter","notes":"README perf table: 32K prefill 13,945.911 / decode 227.779 (agg 227.814) / TTFT 2,349.649ms"},{"id":1056,"context_len":131072,"prefill_tok_s":12975.5699999999997089616954326629638671875,"decode_tok_s":224.1299999999999954525264911353588104248046875,"ttft_ms":10101,"concurrency":1,"claimed_by":"submitter","notes":"README perf table: 128K prefill 12,975.567 / decode 224.134 (agg 222.922) / TTFT 10,101.447ms"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4","pin":"checkpoint used for validation (README does not pin an hf_revision - PLE tables FP8, ~47.68 GiB in the tested checkpoint); NVMe volume with O_DIRECT + io_uring support required"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/yhfgyyf/sglang-qwen38-flash-next-sm120","pin":"4cacd4e586743600c35ae6748aa7b2b4fc5b151a (HEAD 2026-10-06, opt-in FP8 row cache #1); MEASURED baseline @ 2b56a39e7045d6ec7c2e426b8d4b9df57030b61b (optimization commits 7df4aea02b hot-PLE LRU, b2f0b02763 GDN fusion); upstream pins PR #36497 @ 73a255206f, PR"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang.json","alternate":"/r/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:12.156597-06","created_at":"2026-10-07 11:30:13.387252-06","speeds":{"decode_tok_s":224.099999999999994315658113919198513031005859375,"prefill_tok_s":13460.70000000000072759576141834259033203125,"leaderboard":"/api/v1/leaderboard.json"}}