{"slug":"qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp","title":"Qwen3.8-Flash-Next NVFP4 DGX Spark GB10 SGLang NEXTN MTP","summary":"Production single-node DGX Spark (Grace Blackwell GB10, SM121, 128GB unified memory) recipe for Qwen3.8-Flash-Next on SGLang: ModelOpt NVFP4 quantization with zero-copy NVMe mmap of the 47.7 GiB ple_table.bin via --ple-offload-embedding (eliminates 51 GiB of unified-memory footprint), custom Triton FP4 prefill kernels + trtllm_mha decode kernels on SM121, NEXTN MTP (3 verification steps / 4 draft tokens) lifting output to 110-152 tok/s, GDN+QSA hybrid (131,072-token served window, 262k native capacity), 96-slot mamba recurrence buffers, sharp froggeric v22.1 Jinja chat template (-18% response length), persistent Triton/Inductor SM121 JIT caches on NVMe, 8-worker parallel NVMe pre-warm (206 shards in 17s) + 16-core Grace OMP. Author-measured vs same-box 27B dense baseline: 110.4-152.8 tok/s decode w/ MTP (baseline 12.1 base / 21.5 MTP), single-stream CoT 25.1-28.8 tok/s (2.62x speculative gain), TTFT ~347ms radix hit / ~465ms cold, 113 GiB / 121 GiB RAM used (~8 GiB headroom), 103.5 GiB","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"2c5220aa85b5379d7373abd0aa14bb13b1ec993f","content_hash":"bf0c02f95b99494d6720a515e2f8f438f126bc87","created_at":"2026-10-29 21:32:27.869274-06","notes":"Ingested from GitHub repo bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang @ 06fd72a673284696e55b52b3adb58aa3cc6daf35 (wave 3, kanban t_21a3a083). Recipe architected by Michael Bemler per README. First DGX Spark + first SGLang-serving recipe on the site (new hardware family dgx-spark seeded via 0051). Apache-2.0."},"engine_source":{"type":"docker_image","git_url":"https://github.com/bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang","base_engine":"sglang","pip_package":null,"base_version":"image tag qwen38-flash:v1.5 (FROM lmsysorg/sglang:latest)","image_digest":"qwen38-flash:v1.5","install_note":"docker build from the repo Dockerfile @ 06fd72a6: FROM lmsysorg/sglang:latest (UNPINNED upstream base - rebuilds may drift; env TORCH_CUDA_ARCH_LIST=12.0;12.1, ENTRYPOINT python3 -m sglang.launch_server), tagged qwen38-flash:v1.5 locally. launch.sh runs it with --gpus all --memory 110g --shm-size 16g --network host and the NVMe/JIT-cache mounts. No digest pin on the base image - author ships tag-level reproducibility only (honest gap vs the Felliks/YSLAB pin discipline).","pinned_commit":"06fd72a673284696e55b52b3adb58aa3cc6daf35"},"credit":[{"name":"Michael Bemler (Bemler Labs)","url":"https://github.com/bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["python3","-m","sglang.launch_server","--model-path","RadixArk/Qwen3.8-Flash-Next-NVFP4","--revision","7b719225242aacd3dbd3f9407468c2ee9a9d2594","--tp-size","1","--served-model-name","qwen3.8-flash-next","--quantization","modelopt_fp4","--prefill-attention-backend","triton","--decode-attention-backend","trtllm_mha","--ple-offload-embedding","--mamba-radix-cache-strategy","extra_buffer","--max-mamba-cache-size","96","--mem-fraction-static","0.90","--context-length","131072","--chunked-prefill-size","4096","--enable-mixed-chunk","--schedule-conservativeness","0.8","--max-running-requests","4","--allow-auto-truncate","--speculative-algorithm","NEXTN","--speculative-num-steps","3","--speculative-eagle-topk","1","--speculative-num-draft-tokens","4","--speculative-draft-model-quantization","unquant","--reasoning-parser","qwen3","--tool-call-parser","qwen3_coder","--chat-template","/out/chat-template-flashnext.jinja","--api-key","local","--host","0.0.0.0","--port","8000"],"runtime":{"env":"SGLANG_QWEN4_PLE_MMAP_DIR=/ple (47.7 GiB ple_table.bin, zero-copy mmap) | TRITON_CACHE_DIR=/root/.cache/sglang/triton | TORCHINDUCTOR_CACHE_DIR=/root/.cache/sglang/inductor | OMP_NUM_THREADS=16 | MKL_NUM_THREADS=16","notes":"Serve profile = launch.sh @ 06fd72a6 verbatim (docker run --rm --name qwen38-flash --gpus all --memory 110g --memory-swap 110g --shm-size 16g --network host --ipc=host, mounts: HF cache, /out config dir w/ chat-template-flashnext.jinja, /ple PLE dir, sglang cache). Model downloaded at pinned revision (hf snapshot under ~/.cache/huggingface); 8-worker parallel pre-warm cats 206 safetensors shards + 4-worker pre-warm of the PLE table into page cache (<25s) before docker exec. Background health monitor polls :8000/health every 10s x270. Rollback note per README: the 0.90 mem-fraction + full-PLE state is the HEAD 'perfectly working state' (commit subject); systemd unit systemd/qwen38-flash.service provided for 24/7 with crash recovery. Sampling: OpenAI-compatible :8000/v1, api_key 'local' default. Chat template: repo file chat-template-flashnext.jinja (sharp froggeric v22.1, mounted READ-ONLY via /out)."},"max_context_len":131072},"models":[{"slug":"qwen3-8-flash-next-nvfp4","name":"Qwen3.8-Flash-Next-NVFP4","hf_repo":"Qwen/Qwen3.8-Flash-Next","quant":"nvfp4"}],"hardware":[{"slug":"dgx-spark","vendor":"nvidia","card":"DGX Spark (GB10, unified memory)","card_family":"dgx-spark","vram_gb":128,"device_count":1}],"generation":"blackwell","tags":["bemlerlabs","dgx-spark","gb10","mtp-speculative","nvme-ple","sglang","sm121","triton-prefill","trtllm-mha"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured (README telemetry table @ 06fd72a6 + Qwen3.8_FlashNext_DGX_Spark_Benchmark_Report.pdf in-repo, single DGX Spark GB10 @ 273 GB/s): decode 110.4-152.8 tok/s with NEXTN MTP; single-stream CoT 25.1-28.8 tok/s (2.62x speculative gain); TTFT ~347ms radix hit / ~465ms cold; total RAM 113 GiB / 121 GiB (~8 GiB headroom); SGLang VRAM allocation 103.5 GiB static 0.90; PLE table 47.7 GiB zero-copy mmap; swap buffer 1.2 GiB / 99 GiB free; cold boot/restart ~8.5 min (51.8s prefetch, zero swap [t_71325891: the author states the NEXTN MTP decode band 110.4-152.8 tok/s and the single-stream CoT range 25.1-28.8 tok/s at the same served 131,072 context. A point is unique on (context_len, concurrency, claimed_by), so only the 110.4 lower bound is storable at (131072, 1); the upper bound and the CoT range stay documented here rather than being given an invented context. No additional measurement is claimed.]\", \"tok_s\": 110.400000000000005684341886080801486968994140625, \"context_len\": 131072}"}],"perf_points":[{"id":2130,"context_len":131072,"prefill_tok_s":null,"decode_tok_s":110.400000000000005684341886080801486968994140625,"ttft_ms":465,"concurrency":1,"claimed_by":"submitter","notes":"README telemetry @ 06fd72a6: MTP decode band 110.4-152.8 tok/s; TTFT ~347ms radix hit / ~465ms cold; NEXTN 3 steps / 4 draft"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4","pin":"revision 7b719225242aacd3dbd3f9407468c2ee9a9d2594 (MODEL_REVISION default in launch.sh) - 126GB NVFP4 checkpoint shards + 51GB ple_table.bin mmap'd from NVMe"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/lmsysorg/sglang","pin":"FROM lmsysorg/sglang:latest built per repo Dockerfile @ 06fd72a6 -> local tag qwen38-flash:v1.5 (base tag UNPINNED upstream - noted in install_note); TORCH_CUDA_ARCH_LIST=12.0;12.1"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang","pin":"06fd72a673284696e55b52b3adb58aa3cc6daf35 - launch.sh (full argv), Dockerfile, chat-template-flashnext.jinja, systemd/qwen38-flash.service, benchmark.py + PDF benchmark report"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp.json","alternate":"/r/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:27.869274-06","created_at":"2026-10-07 11:30:09.673775-06","speeds":{"decode_tok_s":110.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}