localairecipes launch configs for local inference

Qwen3.8-Flash-Next Uncensored NVFP4 2x DGX Spark SGLang TP2 ROCE

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by ttlogan↗, PixelML/MiaAI (SM121 QSA fallback + token-0 guard patches, MIT), jpezzulli (OrcaRouter uncensored ModelOpt NVFP4 + FP8 PLE repack)↗ · listed by localai-recipes

revision r3 · commit 1217f61537 · updated 2 d ago · license CC-BY-4.0

chat code vision fp8-kv uncensored sglang dgx-spark gb10 roce mtp nvfp4 nextn modelopt fp8-ple multinode podman qsa thermal

Recipe by ttlogan. Production 2-node GB10 (ASUS GX10 / DGX Spark) cluster serve of jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 (OrcaRouter uncensored weights repacked to ModelOpt NVFP4 with the PLE n-gram table in FP8, ~126 GiB) on the MiaAI native vLLM stack --tp 2 2-node over ConnectX/ROCE (slug name says SGLang: historical identifier - the repo now runs vLLM at ec4d38624b), MTP speculative decoding, and a REQUIRED qwen4_exp tokenizer registry overlay (BPE tokenizer.json, NOT stock tokenizer behaviour). Podman rootless deployment with boot autostart and a deepseek/glm/qwen swap orchestrator. Measured numbers unchanged (no re-measurement in d5bd003301..ec4d38624b): 40.3 tok/s single, 117.3 tok/s agg at c=4 (the locked optimum), prefill ~2900-3000 tok/s flat; 30-min c=4 soak clean (2432 req / 0 fail / health 200). License notes: PixelML patches MIT; QSA fp8-KV overlay vendored AGPL-3.0-or-later from the single-Spark recipe.

Launch command

'# qwen38 lane AT ec4d38624b — vLLM (MiaAI native stack), NOT SGLang; slug name "sglang" is historical only (scripts/swap-model.sh, qwen38 case)' 'CONTAINER=vllm-fn   # podman rootless, both nodes' 'LAUNCHER="sudo -u wfoster env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh"' '# swap-model.sh launches it detached: setsid nohup ... >/home/wfoster/qwen38_launch.log 2>&1 </dev/null &' '# the MiaAI runbook start.sh --launch orchestrates head+worker itself; MODEL_ID / SERVED_MODEL_NAME / ABLIT come from the qwen model.conf + flags .env' 'MODEL_DIR=/home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47   # HF snapshot drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47, 124 GiB, ABLIT=1' '# snapshot dir snapshots/a393318fb56d9aedc56d91b6f4962d9af26d2fe7/ — real files for config/tokenizer, HARDLINKS for the safetensors shards (docs/qwen-vllm-bringup.md)' 'SERVED_MODEL_NAME=qwen3.8-flash-next-uncensored   # litellm unchanged' '# TP=2, 2-node, 262144 native ctx; port :8000 on the head' '# REQUIRED tokenizer overlay (docs/qwen-vllm-bringup.md #1): add_overlay mods/tokenizer-registry-overlay/registry.py -> $VLLM_PKG/tokenizers/registry.py (qwen4_exp -> TokenizersBackend; NOT stock tokenizer behaviour)' '# first cold boot ~880 s at 2-node TP=2 262K ctx (bring-up figure, not a benchmark)'

runtime

notes: 'Launch stack at ec4d38624b: vLLM MiaAI native — CONTAINER vllm-fn, LAUNCHER env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh (sudo -u wfoster, setsid nohup detached), MODEL_DIR /home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47, TP=2 2-node, 262K ctx, SERVED_MODEL_NAME qwen3.8-flash-next-uncensored (scripts/swap-model.sh qwen38 case). Slug name "sglang" = historical identifier only (orchestrator decision, rename out of scope). TOKENIZER NOT STOCK: mods/tokenizer-registry-overlay/registry.py adds "qwen4_exp" to _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS so vLLM bypasses the hub-declared Qwen2Tokenizer (sentencepiece) and uses the generic fast TokenizersBackend — the checkpoint ships a BPE tokenizer.json (no spiece.model); bind-mounted over vllm/tokenizers/registry.py via the runbook add_overlay mechanism (docs/qwen-vllm-bringup.md #1). Cold boot ~880 s (bring-up figure). Sources: scripts/swap-model.sh, docs/qwen-vllm-bringup.md, mods/tokenizer-registry-overlay/registry.py @ ec4d38624b.'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce.json

Engine

Install note: Engine lane moved SGLang->vLLM at ec4d38624b (slug name sglang is historical only): the qwen38 lane runs the MiaAI native vLLM stack in podman container vllm-fn (rootless); scripts/swap-model.sh launches it via sudo -u wfoster env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh (setsid nohup detached; the MiaAI runbook start.sh --launch orchestrates head+worker itself). MODEL_DIR /home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 = HF snapshot drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 (124 GiB, ABLIT=1, snapshot hash a393318fb56d9aedc56d91b6f4962d9af26d2fe7 per docs/qwen-vllm-bringup.md). TP=2 2-node, 262K native ctx; SERVED_MODEL_NAME qwen3.8-flash-next-uncensored (litellm unchanged). REQUIRED tokenizer overlay: mods/tokenizer-registry-overlay/registry.py adds qwen4_exp to _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS -> TokenizersBackend (checkpoint ships a BPE tokenizer.json, no spiece.model); bind-mounted over vllm/tokenizers/registry.py via the runbook add_overlay mechanism - NOT stock tokenizer behaviour. First cold boot ~880 s at 2-node TP=2 262K ctx (bring-up figure, not a benchmark re-measure).

Model

Hardware

Context

  • max context: 256K tokens

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
25,77912976.7 tok/s——reported
51,53612936.7 tok/s——reported
262,1441—40.3 tok/s—reported
262,1442—67.1 tok/s—reported
262,1444—117.3 tok/s—reported
262,1446—106.4 tok/s—reported
262,1448—120.5 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured (tuning/RESULTS.md @ d5bd0033, 2x GB10 cluster TP2 over RoCE, 262K native, mem 0.80, max-running-requests 4, NEXTN 3, chunked-prefill 4096, page 64; the promoted POST-fix baseline run 2026-10-14): decode c1 40.3 / c2 67.1 / c4 117.3 (locked optimum) / c6 106.4 (queues, no gain) / c8 120.5 (no crash post-fix); prefill flat ~2900-3000 tok/s (measured at 25,779 and 51,536 actual prompt tokens). 30-min c=4 soak: 2432 req, 0 fail, health 200 continuous. Tuning arms T1 (chunked 8192) and T3 (NEXTN off) both hit a nondeterministic SM121 device-side fault (sglang#37052/#37111 class) - baseline kept; T4 mitigations (disable-flashinfer-autotune + cuda-graph-max-bs 8) REGRESSED decode 10-16% and were rejected. context_len on decode rows = the 262K serve context (short-prompt ladder, prompt len not stated in RESULTS.md).

Install requirements

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Revision notes

Ingested from the sanitized production repo @ d5bd0033015d (65 commits, main; created 2026-10-14, found via GitHub search). Submitter/listed_by = localai-recipes; recipe author ttlogan. Corrections to the scout card: (1) the card's 'no measured tok/s table' caveat is STALE - tuning/RESULTS.md at HEAD carries a full measured battery (used above); (2) the SGLang version is stated NOWHERE in the repo (scripts, README, patches) - only the digest-pinned lmsysorg/sglang image, so engine.version is omitted; (3) the launch lane is SGLang via launch_secure.py - recipes/qwen38-nvfp4-262k.yaml in the repo is a stale vLLM-era artifact, not the live lane. --weight-loader-disable-mmap root cause (repo-documented): the mmap loader accumulates cold checkpoint pages on top of the GPU-resident model and the H2D copy_ page-fault stalls under the CUDA driver lock - on rank 0 only. Head-first launch is required (rank 0 binds TCPStore :29522, worker joins ~25 s later; worker-first intermittently deadlocks). Host thermal fix (both nodes, runtime-agnostic): schedutil governor + X925 P-cores capped 2.8 GHz (SoC -9C, ~no single-stream cost); mods/busy-loop-fix is vLLM-only, not used by this lane; nvidia-smi -pl is a NO-OP on GB10. Orca-native NVFP4 (bf16 PLE ~95 GiB) OOMs 2x GB10 at TP2 and its ple_embedding.shard_* names will not bind - the ModelOpt repack (FP8 E4M3 PLE) is required; MAMBA_SSM_CACHE_DTYPE=bfloat16 + MM_ENCODER_TP_MODE=data are required companions. 262K native serving is deliberate: 1M via YaRN x4 clamps concurrency (GDN/mamba state pool) and has a ~294-390K single-request wedge (MiaAI-Lab #11); 512K YaRN x2 is the validated stretch lane. License notes: recipe repo has no LICENSE file (ingested under CC-BY-4.0); SM121 QSA + token-0 patches are PixelML/MiaAI MIT; the QSA fp8-KV overlay is vendored AGPL-3.0-or-later from the single-Spark recipe (README cites files/patch_qsa_fp8_kv.py; not in this sanitized tree). PIN REVISION (kanban t_c7afb7d6, owner go Casey 2026-10-28, parent t_384d283b): pin d5bd0033015d -> ec4d38624b3f (+17 commits, 6 files: 5 code + 1 doc). The qwen38 lane MOVED SGLang->vLLM (MiaAI native stack) at f53802a: scripts/swap-model.sh qwen38 case now CONTAINER vllm-fn (was qwen38fn-sglang), LAUNCHER env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh (was sglang-qwen38.sh; launched setsid nohup detached), MODEL_DIR /home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 (was /models/orca-qwen3.8-modelopt-nvfp4) = HF snapshot drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 (124 GiB, ABLIT=1, snapshot hash a393318fb56d9aedc56d91b6f4962d9af26d2fe7, docs/qwen-vllm-bringup.md). SERVED_MODEL_NAME unchanged qwen3.8-flash-next-uncensored (litellm unchanged). Slug identifier keeps "sglang" as a historical name (says sglang, runs vLLM; rename out of scope, orchestrator decision). REQUIRED tokenizer overlay published: mods/tokenizer-registry-overlay/registry.py adds qwen4_exp to _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS -> TokenizersBackend (hub-declared Qwen2Tokenizer sentencepiece is wrong; checkpoint ships BPE tokenizer.json, no spiece.model), bind-mounted over vllm/tokenizers/registry.py via the runbook add_overlay mechanism — the site must not claim stock tokenizer behaviour. docs/qwen-vllm-bringup.md cold-boot ~880 s = bring-up figure, not a benchmark re-measure. default_model_vllm.sh (thin wrapper exec'ing swap-model.sh), tools/model-tui.py + tools/test_model_tui_esc.py (host TUI tooling) and the GLM-lane rank-order fix are NO-OP for what the site publishes. Measured numbers UNCHANGED: no re-measurement in d5bd003301..ec4d38624b (zero benchmarks/ changes; tuning/RESULTS.md byte-identical at both pins), measured table + perf points kept 1:1 (7 points before and after).

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)