diff --git a/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce/recipe.yaml b/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce/recipe.yaml index bf683d1ae..db96361cf 100644 --- a/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce/recipe.yaml +++ b/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce/recipe.yaml @@ -1,13 +1,13 @@ slug: qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce title: 'Qwen3.8-Flash-Next Uncensored NVFP4 2x DGX Spark SGLang TP2 ROCE' engine: - name: sglang + name: vllm source_type: docker_image git_url: 'https://github.com/ttlogan/qwen38-flash-next-recipe' - pinned_commit: d5bd0033015d5e5a6baa35850c6a9dc4c455c07c - image_digest: 'docker.io/lmsysorg/sglang@sha256:12d3392bdc8be8d35e9a95f191df6aef99c5114bdbefd41bfdc7e760e6d25ec1' - install_note: 'NOT a plain pip install: per-node podman (rootless) run of the digest-pinned lmsysorg/sglang image with the SM121 QSA + token-0 patches staged FROM the image (apply-sm121-podman.sh) and bind-mounted, launch via launch_secure.py (injects --api-key from a mounted secret, --model-loader-extra-config {enable_multithread_load:false}, --weight-loader-disable-mmap). Head-first two-node launch (rank 0 binds TCPStore :29522, worker joins). NCCL over RoCE (NCCL_IB_HCA rocep1s0f0, GID 3, ROCE v2). SM121 patches: PixelML MIT; QSA fp8-KV overlay vendored AGPL-3.0-or-later (single-Spark recipe).' -summary: 'Recipe by ttlogan. Production 2-node GB10 (ASUS GX10 / DGX Spark) cluster serve of jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 (OrcaRouter uncensored weights repacked to ModelOpt NVFP4 with the PLE n-gram table in FP8, ~126 GiB) with SGLang --tp 2 --nnodes 2 over ConnectX/ROCE, NEXTN/MTP speculative decoding, PixelML SM121 QSA + token-0 guard patches, and a --weight-loader-disable-mmap fix for the rank-0 load_weights hang. First networked-cluster (multi-node) lane in the dgx-spark-x2 bucket. Podman rootless deployment with boot autostart and a deepseek/glm/qwen swap orchestrator. Author-measured at the pinned sha: 40.3 tok/s single, 117.3 tok/s agg at c=4 (the locked optimum), prefill ~2900-3000 tok/s flat; 30-min c=4 soak clean (2432 req / 0 fail / health 200). License notes: PixelML patches MIT; QSA fp8-KV overlay vendored AGPL-3.0-or-later from the single-Spark recipe.' + pinned_commit: ec4d38624b3fa818fd21c3715f6c071fe76fc5b5 + image_digest: null + install_note: 'Engine lane moved SGLang->vLLM at ec4d38624b (slug name sglang is historical only): the qwen38 lane runs the MiaAI native vLLM stack in podman container vllm-fn (rootless); scripts/swap-model.sh launches it via sudo -u wfoster env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh (setsid nohup detached; the MiaAI runbook start.sh --launch orchestrates head+worker itself). MODEL_DIR /home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 = HF snapshot drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 (124 GiB, ABLIT=1, snapshot hash a393318fb56d9aedc56d91b6f4962d9af26d2fe7 per docs/qwen-vllm-bringup.md). TP=2 2-node, 262K native ctx; SERVED_MODEL_NAME qwen3.8-flash-next-uncensored (litellm unchanged). REQUIRED tokenizer overlay: mods/tokenizer-registry-overlay/registry.py adds qwen4_exp to _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS -> TokenizersBackend (checkpoint ships a BPE tokenizer.json, no spiece.model); bind-mounted over vllm/tokenizers/registry.py via the runbook add_overlay mechanism - NOT stock tokenizer behaviour. First cold boot ~880 s at 2-node TP=2 262K ctx (bring-up figure, not a benchmark re-measure).' +summary: 'Recipe by ttlogan. Production 2-node GB10 (ASUS GX10 / DGX Spark) cluster serve of jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 (OrcaRouter uncensored weights repacked to ModelOpt NVFP4 with the PLE n-gram table in FP8, ~126 GiB) on the MiaAI native vLLM stack --tp 2 2-node over ConnectX/ROCE (slug name says SGLang: historical identifier - the repo now runs vLLM at ec4d38624b), MTP speculative decoding, and a REQUIRED qwen4_exp tokenizer registry overlay (BPE tokenizer.json, NOT stock tokenizer behaviour). Podman rootless deployment with boot autostart and a deepseek/glm/qwen swap orchestrator. Measured numbers unchanged (no re-measurement in d5bd003301..ec4d38624b): 40.3 tok/s single, 117.3 tok/s agg at c=4 (the locked optimum), prefill ~2900-3000 tok/s flat; 30-min c=4 soak clean (2432 req / 0 fail / health 200). License notes: PixelML patches MIT; QSA fp8-KV overlay vendored AGPL-3.0-or-later from the single-Spark recipe.' license: CC-BY-4.0 model: name: Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4-FP8PLE @@ -46,70 +46,19 @@ tags: - thermal config: launch: - - python3 - - '-u' - - /opt/qwen/launch_secure.py - - '--model-path' - - /models/qwen38-flash-next - - '--served-model-name' - - qwen3.8-flash-next-uncensored - - '--tp' - - '2' - - '--nnodes' - - '2' - - '--node-rank' - - '0' - - '--dist-init-addr' - - ':29522' - - '--quantization' - - modelopt_fp4 - - '--fp4-gemm-backend' - - flashinfer_cutlass - - '--page-size' - - '64' - - '--linear-attn-prefill-backend' - - triton - - '--linear-attn-decode-backend' - - flashinfer - - '--mamba-ssm-dtype' - - bfloat16 - - '--mamba-scheduler-strategy' - - extra_buffer - - '--mamba-track-interval' - - '64' - - '--speculative-algorithm' - - NEXTN - - '--speculative-num-steps' - - '3' - - '--speculative-eagle-topk' - - '1' - - '--speculative-num-draft-tokens' - - '4' - - '--chunked-prefill-size' - - '4096' - - '--max-running-requests' - - '4' - - '--context-length' - - '262144' - - '--mem-fraction-static' - - '0.80' - - '--allow-auto-truncate' - - '--reasoning-parser' - - auto - - '--tool-call-parser' - - auto - - '--sampling-defaults' - - model - - '--watchdog-timeout' - - '1200' - - '--enable-metrics' - - '--host' - - 0.0.0.0 - - '--port' - - '8000' - - '--model-loader-extra-config' - - '{"enable_multithread_load": false}' - - '--weight-loader-disable-mmap' + - '# qwen38 lane AT ec4d38624b — vLLM (MiaAI native stack), NOT SGLang; slug name "sglang" is historical only (scripts/swap-model.sh, qwen38 case)' + - 'CONTAINER=vllm-fn # podman rootless, both nodes' + - 'LAUNCHER="sudo -u wfoster env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh"' + - '# swap-model.sh launches it detached: setsid nohup ... >/home/wfoster/qwen38_launch.log 2>&1 $VLLM_PKG/tokenizers/registry.py (qwen4_exp -> TokenizersBackend; NOT stock tokenizer behaviour)' + - '# first cold boot ~880 s at 2-node TP=2 262K ctx (bring-up figure, not a benchmark)' + runtime: + notes: 'Launch stack at ec4d38624b: vLLM MiaAI native — CONTAINER vllm-fn, LAUNCHER env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh (sudo -u wfoster, setsid nohup detached), MODEL_DIR /home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47, TP=2 2-node, 262K ctx, SERVED_MODEL_NAME qwen3.8-flash-next-uncensored (scripts/swap-model.sh qwen38 case). Slug name "sglang" = historical identifier only (orchestrator decision, rename out of scope). TOKENIZER NOT STOCK: mods/tokenizer-registry-overlay/registry.py adds "qwen4_exp" to _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS so vLLM bypasses the hub-declared Qwen2Tokenizer (sentencepiece) and uses the generic fast TokenizersBackend — the checkpoint ships a BPE tokenizer.json (no spiece.model); bind-mounted over vllm/tokenizers/registry.py via the runbook add_overlay mechanism (docs/qwen-vllm-bringup.md #1). Cold boot ~880 s (bring-up figure). Sources: scripts/swap-model.sh, docs/qwen-vllm-bringup.md, mods/tokenizer-registry-overlay/registry.py @ ec4d38624b.' performance: claimed_by: submitter metrics: @@ -165,5 +114,5 @@ provenance: source_urls: - 'https://github.com/ttlogan/qwen38-flash-next-recipe' - 'https://huggingface.co/jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4' - notes: "Ingested from the sanitized production repo @ d5bd0033015d (65 commits, main; created 2026-10-14, found via GitHub search). Submitter/listed_by = localai-recipes; recipe author ttlogan. Corrections to the scout card: (1) the card's 'no measured tok/s table' caveat is STALE - tuning/RESULTS.md at HEAD carries a full measured battery (used above); (2) the SGLang version is stated NOWHERE in the repo (scripts, README, patches) - only the digest-pinned lmsysorg/sglang image, so engine.version is omitted; (3) the launch lane is SGLang via launch_secure.py - recipes/qwen38-nvfp4-262k.yaml in the repo is a stale vLLM-era artifact, not the live lane. --weight-loader-disable-mmap root cause (repo-documented): the mmap loader accumulates cold checkpoint pages on top of the GPU-resident model and the H2D copy_ page-fault stalls under the CUDA driver lock - on rank 0 only. Head-first launch is required (rank 0 binds TCPStore :29522, worker joins ~25 s later; worker-first intermittently deadlocks). Host thermal fix (both nodes, runtime-agnostic): schedutil governor + X925 P-cores capped 2.8 GHz (SoC -9C, ~no single-stream cost); mods/busy-loop-fix is vLLM-only, not used by this lane; nvidia-smi -pl is a NO-OP on GB10. Orca-native NVFP4 (bf16 PLE ~95 GiB) OOMs 2x GB10 at TP2 and its ple_embedding.shard_* names will not bind - the ModelOpt repack (FP8 E4M3 PLE) is required; MAMBA_SSM_CACHE_DTYPE=bfloat16 + MM_ENCODER_TP_MODE=data are required companions. 262K native serving is deliberate: 1M via YaRN x4 clamps concurrency (GDN/mamba state pool) and has a ~294-390K single-request wedge (MiaAI-Lab #11); 512K YaRN x2 is the validated stretch lane. License notes: recipe repo has no LICENSE file (ingested under CC-BY-4.0); SM121 QSA + token-0 patches are PixelML/MiaAI MIT; the QSA fp8-KV overlay is vendored AGPL-3.0-or-later from the single-Spark recipe (README cites files/patch_qsa_fp8_kv.py; not in this sanitized tree)." + notes: "Ingested from the sanitized production repo @ d5bd0033015d (65 commits, main; created 2026-10-14, found via GitHub search). Submitter/listed_by = localai-recipes; recipe author ttlogan. Corrections to the scout card: (1) the card's 'no measured tok/s table' caveat is STALE - tuning/RESULTS.md at HEAD carries a full measured battery (used above); (2) the SGLang version is stated NOWHERE in the repo (scripts, README, patches) - only the digest-pinned lmsysorg/sglang image, so engine.version is omitted; (3) the launch lane is SGLang via launch_secure.py - recipes/qwen38-nvfp4-262k.yaml in the repo is a stale vLLM-era artifact, not the live lane. --weight-loader-disable-mmap root cause (repo-documented): the mmap loader accumulates cold checkpoint pages on top of the GPU-resident model and the H2D copy_ page-fault stalls under the CUDA driver lock - on rank 0 only. Head-first launch is required (rank 0 binds TCPStore :29522, worker joins ~25 s later; worker-first intermittently deadlocks). Host thermal fix (both nodes, runtime-agnostic): schedutil governor + X925 P-cores capped 2.8 GHz (SoC -9C, ~no single-stream cost); mods/busy-loop-fix is vLLM-only, not used by this lane; nvidia-smi -pl is a NO-OP on GB10. Orca-native NVFP4 (bf16 PLE ~95 GiB) OOMs 2x GB10 at TP2 and its ple_embedding.shard_* names will not bind - the ModelOpt repack (FP8 E4M3 PLE) is required; MAMBA_SSM_CACHE_DTYPE=bfloat16 + MM_ENCODER_TP_MODE=data are required companions. 262K native serving is deliberate: 1M via YaRN x4 clamps concurrency (GDN/mamba state pool) and has a ~294-390K single-request wedge (MiaAI-Lab #11); 512K YaRN x2 is the validated stretch lane. License notes: recipe repo has no LICENSE file (ingested under CC-BY-4.0); SM121 QSA + token-0 patches are PixelML/MiaAI MIT; the QSA fp8-KV overlay is vendored AGPL-3.0-or-later from the single-Spark recipe (README cites files/patch_qsa_fp8_kv.py; not in this sanitized tree). PIN REVISION (kanban t_c7afb7d6, owner go Casey 2026-10-28, parent t_384d283b): pin d5bd0033015d -> ec4d38624b3f (+17 commits, 6 files: 5 code + 1 doc). The qwen38 lane MOVED SGLang->vLLM (MiaAI native stack) at f53802a: scripts/swap-model.sh qwen38 case now CONTAINER vllm-fn (was qwen38fn-sglang), LAUNCHER env SPARK_VLLM_MODEL_CONF=/home/wfoster/.config/spark-vllm/qwen-model.conf bash /home/wfoster/.local/bin/miaai_launch.sh (was sglang-qwen38.sh; launched setsid nohup detached), MODEL_DIR /home/wfoster/.cache/huggingface/hub/models--drowzeys--keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 (was /models/orca-qwen3.8-modelopt-nvfp4) = HF snapshot drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47 (124 GiB, ABLIT=1, snapshot hash a393318fb56d9aedc56d91b6f4962d9af26d2fe7, docs/qwen-vllm-bringup.md). SERVED_MODEL_NAME unchanged qwen3.8-flash-next-uncensored (litellm unchanged). Slug identifier keeps \"sglang\" as a historical name (says sglang, runs vLLM; rename out of scope, orchestrator decision). REQUIRED tokenizer overlay published: mods/tokenizer-registry-overlay/registry.py adds qwen4_exp to _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS -> TokenizersBackend (hub-declared Qwen2Tokenizer sentencepiece is wrong; checkpoint ships BPE tokenizer.json, no spiece.model), bind-mounted over vllm/tokenizers/registry.py via the runbook add_overlay mechanism — the site must not claim stock tokenizer behaviour. docs/qwen-vllm-bringup.md cold-boot ~880 s = bring-up figure, not a benchmark re-measure. default_model_vllm.sh (thin wrapper exec'ing swap-model.sh), tools/model-tui.py + tools/test_model_tui_esc.py (host TUI tooling) and the GLM-lane rank-order fix are NO-OP for what the site publishes. Measured numbers UNCHANGED: no re-measurement in d5bd003301..ec4d38624b (zero benchmarks/ changes; tuning/RESULTS.md byte-identical at both pins), measured table + perf points kept 1:1 (7 points before and after)." max_context_len: 262144