Qwen3.8-Flash-Next NVFP4 DGX Spark GB10 SGLang NEXTN MTP

community Community submission — verify on your own hardware.

recipe by Michael Bemler (Bemler Labs)↗ · listed by localai-recipes

revision r3 · commit 2c5220aa85 · updated 21 h ago · license CC-BY-4.0

chat code mtp-speculative sglang nvme-ple bemlerlabs dgx-spark gb10 sm121 triton-prefill trtllm-mha

Production single-node DGX Spark (Grace Blackwell GB10, SM121, 128GB unified memory) recipe for Qwen3.8-Flash-Next on SGLang: ModelOpt NVFP4 quantization with zero-copy NVMe mmap of the 47.7 GiB ple_table.bin via --ple-offload-embedding (eliminates 51 GiB of unified-memory footprint), custom Triton FP4 prefill kernels + trtllm_mha decode kernels on SM121, NEXTN MTP (3 verification steps / 4 draft tokens) lifting output to 110-152 tok/s, GDN+QSA hybrid (131,072-token served window, 262k native capacity), 96-slot mamba recurrence buffers, sharp froggeric v22.1 Jinja chat template (-18% response length), persistent Triton/Inductor SM121 JIT caches on NVMe, 8-worker parallel NVMe pre-warm (206 shards in 17s) + 16-core Grace OMP. Author-measured vs same-box 27B dense baseline: 110.4-152.8 tok/s decode w/ MTP (baseline 12.1 base / 21.5 MTP), single-stream CoT 25.1-28.8 tok/s (2.62x speculative gain), TTFT ~347ms radix hit / ~465ms cold, 113 GiB / 121 GiB RAM used (~8 GiB headroom), 103.5 GiB

Launch command

python3 -m sglang.launch_server --model-path RadixArk/Qwen3.8-Flash-Next-NVFP4 --revision 7b719225242aacd3dbd3f9407468c2ee9a9d2594 --tp-size 1 --served-model-name qwen3.8-flash-next --quantization modelopt_fp4 --prefill-attention-backend triton --decode-attention-backend trtllm_mha --ple-offload-embedding --mamba-radix-cache-strategy extra_buffer --max-mamba-cache-size 96 --mem-fraction-static 0.90 --context-length 131072 --chunked-prefill-size 4096 --enable-mixed-chunk --schedule-conservativeness 0.8 --max-running-requests 4 --allow-auto-truncate --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-draft-model-quantization unquant --reasoning-parser qwen3 --tool-call-parser qwen3_coder --chat-template /out/chat-template-flashnext.jinja --api-key local --host 0.0.0.0 --port 8000

runtime

env: 'SGLANG_QWEN4_PLE_MMAP_DIR=/ple (47.7 GiB ple_table.bin, zero-copy mmap) | TRITON_CACHE_DIR=/root/.cache/sglang/triton | TORCHINDUCTOR_CACHE_DIR=/root/.cache/sglang/inductor | OMP_NUM_THREADS=16 | MKL_NUM_THREADS=16'
notes: "Serve profile = launch.sh @ 06fd72a6 verbatim (docker run --rm --name qwen38-flash --gpus all --memory 110g --memory-swap 110g --shm-size 16g --network host --ipc=host, mounts: HF cache, /out config dir w/ chat-template-flashnext.jinja, /ple PLE dir, sglang cache). Model downloaded at pinned revision (hf snapshot under ~/.cache/huggingface); 8-worker parallel pre-warm cats 206 safetensors shards + 4-worker pre-warm of the PLE table into page cache (<25s) before docker exec. Background health monitor polls :8000/health every 10s x270. Rollback note per README: the 0.90 mem-fraction + full-PLE state is the HEAD 'perfectly working state' (commit subject); systemd unit systemd/qwen38-flash.service provided for 24/7 with crash recovery. Sampling: OpenAI-compatible :8000/v1, api_key 'local' default. Chat template: repo file chat-template-flashnext.jinja (sharp froggeric v22.1, mounted READ-ONLY via /out)."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp.json

Engine

sglang · source↗ · 06fd72a6↗ · qwen38-flash:v1.5↗

Install note: docker build from the repo Dockerfile @ 06fd72a6: FROM lmsysorg/sglang:latest (UNPINNED upstream base - rebuilds may drift; env TORCH_CUDA_ARCH_LIST=12.0;12.1, ENTRYPOINT python3 -m sglang.launch_server), tagged qwen38-flash:v1.5 locally. launch.sh runs it with --gpus all --memory 110g --shm-size 16g --network host and the NVMe/JIT-cache mounts. No digest pin on the base image - author ships tag-level reproducibility only (honest gap vs the Felliks/YSLAB pin discipline).

Model

Qwen3.8-Flash-Next-NVFP4 (nvfp4) · Qwen/Qwen3.8-Flash-Next↗
AA intelligence 40 (Artificial Analysis · measured)

Hardware

NVIDIA DGX Spark 128GB ×1 (Blackwell)

Context

max context: 128K tokens

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
131,072	1	—	110.4 tok/s	465 ms	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Author-measured (README telemetry table @ 06fd72a6 + Qwen3.8_FlashNext_DGX_Spark_Benchmark_Report.pdf in-repo, single DGX Spark GB10 @ 273 GB/s): decode 110.4-152.8 tok/s with NEXTN MTP; single-stream CoT 25.1-28.8 tok/s (2.62x speculative gain); TTFT ~347ms radix hit / ~465ms cold; total RAM 113 GiB / 121 GiB (~8 GiB headroom); SGLang VRAM allocation 103.5 GiB static 0.90; PLE table 47.7 GiB zero-copy mmap; swap buffer 1.2 GiB / 99 GiB free; cold boot/restart ~8.5 min (51.8s prefetch, zero swap [t_71325891: the author states the NEXTN MTP decode band 110.4-152.8 tok/s and the single-stream CoT range 25.1-28.8 tok/s at the same served 131,072 context. A point is unique on (context_len, concurrency, claimed_by), so only the 110.4 lower bound is storable at (131072, 1); the upper bound and the CoT range stay documented here rather than being given an invented context. No additional measurement is claimed.]

notes

Author-measured (README telemetry table @ 06fd72a6 + Qwen3.8_FlashNext_DGX_Spark_Benchmark_Report.pdf in-repo, single DGX Spark GB10 @ 273 GB/s): decode 110.4-152.8 tok/s with NEXTN MTP; single-stream CoT 25.1-28.8 tok/s (2.62x speculative gain); TTFT ~347ms radix hit / ~465ms cold; total RAM 113 GiB / 121 GiB (~8 GiB headroom); SGLang VRAM allocation 103.5 GiB static 0.90; PLE table 47.7 GiB zero-copy mmap; swap buffer 1.2 GiB / 99 GiB free; cold boot/restart ~8.5 min (51.8s prefetch, zero swap [t_71325891: the author states the NEXTN MTP decode band 110.4-152.8 tok/s and the single-stream CoT range 25.1-28.8 tok/s at the same served 131,072 context. A point is unique on (context_len, concurrency, claimed_by), so only the 110.4 lower bound is storable at (131072, 1); the upper bound and the CoT range stay documented here rather than being given an invented context. No additional measurement is claimed.]

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4
revision 7b719225242aacd3dbd3f9407468c2ee9a9d2594 (MODEL_REVISION default in launch.sh) - 126GB NVFP4 checkpoint shards + 51GB ple_table.bin mmap'd from NVMe
Engine image https://hub.docker.com/r/lmsysorg/sglang
FROM lmsysorg/sglang:latest built per repo Dockerfile @ 06fd72a6 -> local tag qwen38-flash:v1.5 (base tag UNPINNED upstream - noted in install_note); TORCH_CUDA_ARCH_LIST=12.0;12.1
Reference https://github.com/bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang
06fd72a673284696e55b52b3adb58aa3cc6daf35 - launch.sh (full argv), Dockerfile, chat-template-flashnext.jinja, systemd/qwen38-flash.service, benchmark.py + PDF benchmark report

Revision notes

Ingested from GitHub repo bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang @ 06fd72a673284696e55b52b3adb58aa3cc6daf35 (wave 3, kanban t_21a3a083). Recipe architected by Michael Bemler per README. First DGX Spark + first SGLang-serving recipe on the site (new hardware family dgx-spark seeded via 0051). Apache-2.0.

Revision history

2c5220aa85 recipe: update qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp (r3: /performance) 15 h ago diff
b6a1e1bd26 recipe: update qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp (r2: /max_context_len) 10 d ago diff
4c80531e24 recipe: create qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp 23 d ago diff
8717df2959 wave-3 fix: remove same-run mislinked dgx-spark ingests (hw.card resolved to legacy silent-created profile; re-ingesting with corrected card stem) t_21a3a083 23 d ago diff
5a56a83664 recipe: create qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp 23 d ago

Fork

Export

recipe.yaml (raw)