{"slug":"qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce","title":"Qwen3.8-Flash-Next Uncensored NVFP4 2x DGX Spark SGLang TP2 ROCE","summary":"Recipe by ttlogan. Production 2-node GB10 (ASUS GX10 / DGX Spark) cluster serve of jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 (OrcaRouter uncensored weights repacked to ModelOpt NVFP4 with the PLE n-gram table in FP8, ~126 GiB) on the MiaAI native vLLM stack --tp 2 2-node over ConnectX/ROCE (slug name says SGLang: historical identifier - the repo now runs vLLM at ec4d38624b), MTP speculative decoding, and a REQUIRED qwen4_exp tokenizer registry overlay (BPE tokenizer.json, NOT stock tokenizer behaviour). Podman rootless deployment with boot autostart and a deepseek/glm/qwen swap orchestrator. Measured numbers unchanged (no re-measurement in d5bd003301..ec4d38624b): 40.3 tok/s single, 117.3 tok/s agg at c=4 (the locked optimum), prefill ~2900-3000 tok/s flat; 30-min c=4 soak clean (2432 req / 0 fail / health 200). License notes: PixelML patches MIT; QSA fp8-KV overlay vendored AGPL-3.0-or-later from the single-Spark recipe.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"1217f615379960f758d79b04ceb4ca2a71e8e6cc","content_hash":"db96361cfc2e1999cd1d8ce8f34678e090df5f9b","created_at":"2026-10-28 16:04:52.791091-06","notes":"Ingested from the sanitized production repo @ d5bd0033015d (65 commits, main; created 2026-10-14, found via GitHub search). Submitter/listed_by = localai-recipes; recipe author ttlogan. Corrections to the scout card: (1) the card's 'no measured tok/s table' caveat is STALE - tuning/RESULTS.md at HEAD carries a full measured battery (used above); (2) the SGLang version is stated NOWHERE in the repo (scripts, README, patches) - only the digest-pinned lmsysorg/sglang image, so engine.version is omitted; (3) the launch lane is SGLang via launch_secure.py - recipes/qwen38-nvfp4-262k.yaml in the repo is a stale vLLM-era artifact, not the live lane. --weight-loader-disable-mmap root cause (repo-documented): the mmap loader accumulates cold checkpoint pages on top of the GPU-resident model and the H2D copy_ page-fault stalls under the CUDA driver lock - on rank 0 only. Head-first launch is required (rank 0 binds TCPStore :29522, worker joins ~25 s later; worker-first intermittently deadlocks). Host thermal fix (both nodes, runtime-agnostic): schedutil governor + X925 P-cores capped 2.8 GHz (SoC -9C, ~no single-stream cost); mods/busy-loop-fix is vLLM-only, not used by this lane; nvidia-smi -pl is a NO-OP on GB10. Orca-native NVFP4 (bf16 PLE ~95 GiB) OOMs 2x GB10 at TP2 and its ple_embedding.shard_* names will not bind - the ModelOpt repack (FP8 E4M3 PLE) is required; MAMBA_SSM_CACHE_DTYPE=bfloat16 + MM_ENCODER_TP_MODE=data are required companions. 262K native serving is deliberate: 1M via YaRN x4 clamps concurrency (GDN/mamba state pool) and has a ~294-390K single-request wedge (MiaAI-Lab #11); 512K YaRN x2 is the validated stretch lane. License notes: recipe repo has no LICENSE file (ingested under CC-BY-4.0); SM121 QSA + token-0 patches are PixelML/MiaAI MIT; the QSA fp8-KV overlay is vendored AGPL-3.0-or-later from the single-Spark recipe (README cites files/patch_qsa_fp8_kv.py; not in this sanitized tree). PIN REVISION (kanban t_c7afb7d6, owner go Casey 2026-10-28, parent t_384d283b): pin d5bd0033015d -> ec4d38624b3f (+17 commits, 6 files: 5 code + 1 doc). The qwen38 lane MOVED SGLang->vLLM (MiaAI native stack) at f53802a: scripts/swap-model.sh qwen38 case now CONTAINER vllm-fn (was qwen38fn-sglang), LAUNCHER env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh (was sglang-qwen38.sh; launched setsid nohup detached), MODEL_DIR /home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 (was /models/orca-qwen3.8-modelopt-nvfp4) = HF snapshot drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 (124 GiB, ABLIT=1, snapshot hash a393318fb56d9aedc56d91b6f4962d9af26d2fe7, docs/qwen-vllm-bringup.md). SERVED_MODEL_NAME unchanged qwen3.8-flash-next-uncensored (litellm unchanged). Slug identifier keeps \"sglang\" as a historical name (says sglang, runs vLLM; rename out of scope, orchestrator decision). REQUIRED tokenizer overlay published: mods/tokenizer-registry-overlay/registry.py adds qwen4_exp to _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS -> TokenizersBackend (hub-declared Qwen2Tokenizer sentencepiece is wrong; checkpoint ships BPE tokenizer.json, no spiece.model), bind-mounted over vllm/tokenizers/registry.py via the runbook add_overlay mechanism \u2014 the site must not claim stock tokenizer behaviour. docs/qwen-vllm-bringup.md cold-boot ~880 s = bring-up figure, not a benchmark re-measure. default_model_vllm.sh (thin wrapper exec'ing swap-model.sh), tools/model-tui.py + tools/test_model_tui_esc.py (host TUI tooling) and the GLM-lane rank-order fix are NO-OP for what the site publishes. Measured numbers UNCHANGED: no re-measurement in d5bd003301..ec4d38624b (zero benchmarks/ changes; tuning/RESULTS.md byte-identical at both pins), measured table + perf points kept 1:1 (7 points before and after)."},"engine_source":{"type":"docker_image","git_url":"https://github.com/ttlogan/qwen38-flash-next-recipe","base_engine":"vllm","pip_package":null,"base_version":null,"image_digest":null,"install_note":"Engine lane moved SGLang->vLLM at ec4d38624b (slug name sglang is historical only): the qwen38 lane runs the MiaAI native vLLM stack in podman container vllm-fn (rootless); scripts/swap-model.sh launches it via sudo -u wfoster env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh (setsid nohup detached; the MiaAI runbook start.sh --launch orchestrates head+worker itself). MODEL_DIR /home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 = HF snapshot drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 (124 GiB, ABLIT=1, snapshot hash a393318fb56d9aedc56d91b6f4962d9af26d2fe7 per docs/qwen-vllm-bringup.md). TP=2 2-node, 262K native ctx; SERVED_MODEL_NAME qwen3.8-flash-next-uncensored (litellm unchanged). REQUIRED tokenizer overlay: mods/tokenizer-registry-overlay/registry.py adds qwen4_exp to _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS -> TokenizersBackend (checkpoint ships a BPE tokenizer.json, no spiece.model); bind-mounted over vllm/tokenizers/registry.py via the runbook add_overlay mechanism - NOT stock tokenizer behaviour. First cold boot ~880 s at 2-node TP=2 262K ctx (bring-up figure, not a benchmark re-measure).","pinned_commit":"ec4d38624b3fa818fd21c3715f6c071fe76fc5b5"},"credit":[{"name":"ttlogan","url":"https://github.com/ttlogan/qwen38-flash-next-recipe"},{"name":"PixelML/MiaAI (SM121 QSA fallback + token-0 guard patches, MIT)"},{"name":"jpezzulli (OrcaRouter uncensored ModelOpt NVFP4 + FP8 PLE repack)","url":"https://huggingface.co/jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["# qwen38 lane AT ec4d38624b \u2014 vLLM (MiaAI native stack), NOT SGLang; slug name \"sglang\" is historical only (scripts/swap-model.sh, qwen38 case)","CONTAINER=vllm-fn   # podman rootless, both nodes","LAUNCHER=\"sudo -u wfoster env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh\"","# swap-model.sh launches it detached: setsid nohup ... >/home/wfoster/qwen38_launch.log 2>&1 </dev/null &","# the MiaAI runbook start.sh --launch orchestrates head+worker itself; MODEL_ID / SERVED_MODEL_NAME / ABLIT come from the qwen model.conf + flags .env","MODEL_DIR=/home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47   # HF snapshot drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47, 124 GiB, ABLIT=1","# snapshot dir snapshots/a393318fb56d9aedc56d91b6f4962d9af26d2fe7/ \u2014 real files for config/tokenizer, HARDLINKS for the safetensors shards (docs/qwen-vllm-bringup.md)","SERVED_MODEL_NAME=qwen3.8-flash-next-uncensored   # litellm unchanged","# TP=2, 2-node, 262144 native ctx; port :8000 on the head","# REQUIRED tokenizer overlay (docs/qwen-vllm-bringup.md #1): add_overlay mods/tokenizer-registry-overlay/registry.py -> $VLLM_PKG/tokenizers/registry.py (qwen4_exp -> TokenizersBackend; NOT stock tokenizer behaviour)","# first cold boot ~880 s at 2-node TP=2 262K ctx (bring-up figure, not a benchmark)"],"runtime":{"notes":"Launch stack at ec4d38624b: vLLM MiaAI native \u2014 CONTAINER vllm-fn, LAUNCHER env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh (sudo -u wfoster, setsid nohup detached), MODEL_DIR /home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47, TP=2 2-node, 262K ctx, SERVED_MODEL_NAME qwen3.8-flash-next-uncensored (scripts/swap-model.sh qwen38 case). Slug name \"sglang\" = historical identifier only (orchestrator decision, rename out of scope). TOKENIZER NOT STOCK: mods/tokenizer-registry-overlay/registry.py adds \"qwen4_exp\" to _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS so vLLM bypasses the hub-declared Qwen2Tokenizer (sentencepiece) and uses the generic fast TokenizersBackend \u2014 the checkpoint ships a BPE tokenizer.json (no spiece.model); bind-mounted over vllm/tokenizers/registry.py via the runbook add_overlay mechanism (docs/qwen-vllm-bringup.md #1). Cold boot ~880 s (bring-up figure). Sources: scripts/swap-model.sh, docs/qwen-vllm-bringup.md, mods/tokenizer-registry-overlay/registry.py @ ec4d38624b."},"max_context_len":262144},"models":[{"slug":"qwen3-8-flash-next-uncensored-modelopt-nvfp4-fp8ple","name":"Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4-FP8PLE","hf_repo":"jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4","quant":"nvfp4"}],"hardware":[{"slug":"dgx-spark-x2","vendor":"nvidia","card":"DGX Spark (GB10, unified memory)","card_family":"dgx-spark","vram_gb":128,"device_count":2}],"generation":"blackwell","tags":["dgx-spark","fp8-kv","fp8-ple","gb10","modelopt","mtp","multinode","nextn","nvfp4","podman","qsa","roce","sglang","thermal","uncensored"],"tasks":["chat","code","vision"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured (tuning/RESULTS.md @ d5bd0033, 2x GB10 cluster TP2 over RoCE, 262K native, mem 0.80, max-running-requests 4, NEXTN 3, chunked-prefill 4096, page 64; the promoted POST-fix baseline run 2026-10-14): decode c1 40.3 / c2 67.1 / c4 117.3 (locked optimum) / c6 106.4 (queues, no gain) / c8 120.5 (no crash post-fix); prefill flat ~2900-3000 tok/s (measured at 25,779 and 51,536 actual prompt tokens). 30-min c=4 soak: 2432 req, 0 fail, health 200 continuous. Tuning arms T1 (chunked 8192) and T3 (NEXTN off) both hit a nondeterministic SM121 device-side fault (sglang#37052/#37111 class) - baseline kept; T4 mitigations (disable-flashinfer-autotune + cuda-graph-max-bs 8) REGRESSED decode 10-16% and were rejected. context_len on decode rows = the 262K serve context (short-prompt ladder, prompt len not stated in RESULTS.md).\", \"tok_s\": 117.2999999999999971578290569595992565155029296875, \"points\": [{\"notes\": \"RESULTS.md POST-fix decode ladder, short prompt, max_tokens 320\", \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 40.2999999999999971578290569595992565155029296875}, {\"notes\": \"RESULTS.md POST-fix ladder; 33.5 tok/s per stream\", \"concurrency\": 2, \"context_len\": 262144, \"decode_tok_s\": 67.0499999999999971578290569595992565155029296875}, {\"notes\": \"LOCKED default (max-running-requests 4); 29.3 tok/s per stream; throughput optimum AND stability point\", \"concurrency\": 4, \"context_len\": 262144, \"decode_tok_s\": 117.2999999999999971578290569595992565155029296875}, {\"notes\": \"queues above the max-running-requests 4 cap - no throughput gain, worse latency\", \"concurrency\": 6, \"context_len\": 262144, \"decode_tok_s\": 106.43000000000000682121026329696178436279296875}, {\"notes\": \"queued but stable post-fix (pre-fix c=8 watchdog-crashed); 15.1 tok/s per stream\", \"concurrency\": 8, \"context_len\": 262144, \"decode_tok_s\": 120.5100000000000051159076974727213382720947265625}, {\"notes\": \"RESULTS.md prefill row: requested 50K prompt, actual prompt_tok 25779 in 8.66 s (context_len = actual prompt tokens)\", \"concurrency\": 1, \"context_len\": 25779, \"prefill_tok_s\": 2976.6999999999998181010596454143524169921875}, {\"notes\": \"RESULTS.md prefill row: requested 100K prompt, actual prompt_tok 51536 in 17.55 s; prefill flat ~2900-3000 across context\", \"concurrency\": 1, \"context_len\": 51536, \"prefill_tok_s\": 2936.6999999999998181010596454143524169921875}], \"context_len\": 262144}"}],"perf_points":[{"id":1891,"context_len":25779,"prefill_tok_s":2976.6999999999998181010596454143524169921875,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"RESULTS.md prefill row: requested 50K prompt, actual prompt_tok 25779 in 8.66 s (context_len = actual prompt tokens)"},{"id":1892,"context_len":51536,"prefill_tok_s":2936.6999999999998181010596454143524169921875,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"RESULTS.md prefill row: requested 100K prompt, actual prompt_tok 51536 in 17.55 s; prefill flat ~2900-3000 across context"},{"id":1893,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":40.2999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"RESULTS.md POST-fix decode ladder, short prompt, max_tokens 320"},{"id":1894,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":67.0499999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"RESULTS.md POST-fix ladder; 33.5 tok/s per stream"},{"id":1895,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":117.2999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"LOCKED default (max-running-requests 4); 29.3 tok/s per stream; throughput optimum AND stability point"},{"id":1896,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":106.43000000000000682121026329696178436279296875,"ttft_ms":null,"concurrency":6,"claimed_by":"submitter","notes":"queues above the max-running-requests 4 cap - no throughput gain, worse latency"},{"id":1897,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":120.5100000000000051159076974727213382720947265625,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"queued but stable post-fix (pre-fix c=8 watchdog-crashed); 15.1 tok/s per stream"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce.json","alternate":"/r/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce","describedby":"/openapi.json"},"updated_at":"2026-10-28 16:04:52.791091-06","created_at":"2026-10-20 21:49:18.502531-06","speeds":{"decode_tok_s":40.2999999999999971578290569595992565155029296875,"prefill_tok_s":2936.6999999999998181010596454143524169921875,"leaderboard":"/api/v1/leaderboard.json"}}