Qwen3.8-Flash-Next NVFP4 RTX PRO 6000 Blackwell SGLang NVMe PLE
00 upvotes — sign in to vote sign in to votecommunity Community submission — verify on your own hardware.
recipe by yepapa-nest↗, jzinno (NVMe PLE streaming PR #36567 source tree)↗, sgl-project (SGLang upstream + PRs 36556/36749/36750)↗ · listed by localai-recipes
revision r2 · commit 783bd7b941 · updated 10 d ago · license CC-BY-4.0
chat code mtp-speculative yepapa-nest sglang nvme-ple io-uring cuda-graphs breakable-graphs sm120
Single-card RTX PRO 6000 Blackwell (SM120, 96GB) recipe for Qwen3.8-Flash-Next on SGLang: the 51B N-gram PLE table is read straight off NVMe with io_uring as tokens need it (measured gather 0.7-2.7 ms, under 2% of decode budget) instead of landing in the 95GB host-RAM PLE worker the official recipes assume - that is what leaves the GPU free for 78.2GB weights plus a real 233,856-token FP8 KV cache. Build: lmsysorg/sglang:qwen38flashnext day-0 image + source overlay of jzinno/sglang @ d4477bd298 (tree head of PR #36567 NVMe PLE streaming stacked on #36497 model support) + cherry-picked PRs 36556 (SM120/SM121 sparse decode - without it the QSA kernel MLIRErrors at startup), 36749 (BCG buffers sized by token rows - the ONLY way breakable CUDA graphs compose with NEXTN speculation; took single-stream decode 57.6 -> 167 tok/s), 36750 (max_thinking_tokens on the OpenAI endpoint) + local patches/0001-qsa-fp8-kv-dequant-on-read.patch (chunked-prompt FP8 KV crash fix; upstream #36644 preferred
Launch command
python -m sglang.launch_server --model-path /model --served-model-name Qwen3.8-Flash-Next --host 0.0.0.0 --port 8000 --quantization modelopt_fp4 --fp4-gemm-backend flashinfer_cudnn --kv-cache-dtype fp8_e4m3 --page-size 64 --mamba-radix-cache-strategy extra_buffer --mamba-track-interval 64 --chunked-prefill-size 8192 --max-running-requests 8 --context-length 262144 --max-total-tokens 393216 --max-mamba-cache-size 25 --mem-fraction-static 0.95 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --cuda-graph-backend-decode breakable --cuda-graph-backend-prefill disabled --reasoning-parser auto --tool-call-parser auto --enable-strict-thinking --enable-metricsruntime
env: 'SGLANG_QWEN4_PLE_NVME_PATH=/model | SGLANG_QWEN4_PLE_NVME_BACKEND=io_uring | SGLANG_QWEN4_PLE_NVME_QUEUE_DEPTH=512 | SGLANG_QWEN4_PLE_NVME_LOG_INTERVAL=10000'
notes: 'Serve profile = serve.sh @ 2ef81d57 verbatim (docker run --gpus device=0 --ipc host --shm-size 32g --security-opt seccomp=seccomp-iouring.json -p 127.0.0.1:8000:8000 -v $MODEL_DIR:/model:ro; engine clamps max-total-tokens 393216 request to ~233,856 with fp8 KV). Docker seccomp: default profile BLOCKS io_uring_setup/enter/register - serve.sh passes --security-opt seccomp=seccomp-iouring.json (default + those 3 syscalls, file in repo). Speculative draft depth capped at 4 by the QSA compression ratio - architectural, higher --speculative-num-draft-tokens does not help. --enable-strict-thinking only needed for per-request max_thinking_tokens (PR #36750; STRICT_ARG dropped when STRICT_THINKING=0). Patch 4 (qsa-fp8-kv-dequant-on-read) assumes direct-cast e4m3 store scale 1.0 - check checkpoint KV scales; upstream #36644 (per-layer KV descale + FA fallback) preferred once merged, or drop --kv-cache-dtype fp8_e4m3 to skip patch 4 at half the KV cache. Concurrency: ~5 streams real limit (MAMBA_SLOTS=25 default; aggregate flat 408 @c4 -> 391 @c16). Prefix cache small: 233K tokens vs ~1.8M dense-27B. Cold start ~3 min (~1.5 warm).'
Use this recipe with your agent
Paste this into any agent chat — it fetches and applies the recipe:
Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.Or fetch it directly:
curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple.jsonDownloads
- Weights https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4revision 7b719225242aacd3dbd3f9407468c2ee9a9d2594 - day-0 uploads get silently amended hours later without announcement: pin the revision (repo's own warning); 78.2GB weights + PLE table read from NVMe
- Engine image https://hub.docker.com/r/lmsysorg/sglangbase lmsysorg/sglang:qwen38flashnext (day-0, pushed 2026-08-26T12:30Z - predates PR #36567: qwen4_exp.py present, qwen4_ple_nvme.py absent inside the image) + source overlay jzinno/sglang @ d4477bd298aef3edae611eb7b2e533d5526e324b + PRs 36556/36749/36750 +
- Reference https://github.com/yepapa-nest/qwen38-flashnext-rtx60002ef81d575c9bb89b9b1a19b749c51f1b6eb02e4d - build.sh (pinned tree + PR cherry-picks), serve.sh (seccomp-iouring.json), patches/0001-qsa-fp8-kv-dequant-on-read.patch, BENCHMARKS.md
Revision notes
Ingested from GitHub repo yepapa-nest/qwen38-flashnext-rtx6000 @ 2ef81d575c9bb89b9b1a19b749c51f1b6eb02e4d (wave 3, kanban t_21a3a083). First single-card cell for the official SGLang Flash-Next recipe matrix (official recipes target H200/B200/B300/GB300/MI350X/MI355X). Four of five changes are open upstream PRs - build shortens as they merge. Patches + scripts Apache-2.0; checkpoint has its own license.
Revision history
- 1c435700e9 recipe: update qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple (r2: /max_context_len, /kv_pool_tokens) 10 d ago diff
- 94eb408f2f recipe: create qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple 23 d ago
Fork
Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.