diff --git a/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce/recipe.yaml b/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce/recipe.yaml index 07b7f2eff..bf683d1ae 100644 --- a/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce/recipe.yaml +++ b/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce/recipe.yaml @@ -166,3 +166,4 @@ provenance: - 'https://github.com/ttlogan/qwen38-flash-next-recipe' - 'https://huggingface.co/jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4' notes: "Ingested from the sanitized production repo @ d5bd0033015d (65 commits, main; created 2026-10-14, found via GitHub search). Submitter/listed_by = localai-recipes; recipe author ttlogan. Corrections to the scout card: (1) the card's 'no measured tok/s table' caveat is STALE - tuning/RESULTS.md at HEAD carries a full measured battery (used above); (2) the SGLang version is stated NOWHERE in the repo (scripts, README, patches) - only the digest-pinned lmsysorg/sglang image, so engine.version is omitted; (3) the launch lane is SGLang via launch_secure.py - recipes/qwen38-nvfp4-262k.yaml in the repo is a stale vLLM-era artifact, not the live lane. --weight-loader-disable-mmap root cause (repo-documented): the mmap loader accumulates cold checkpoint pages on top of the GPU-resident model and the H2D copy_ page-fault stalls under the CUDA driver lock - on rank 0 only. Head-first launch is required (rank 0 binds TCPStore :29522, worker joins ~25 s later; worker-first intermittently deadlocks). Host thermal fix (both nodes, runtime-agnostic): schedutil governor + X925 P-cores capped 2.8 GHz (SoC -9C, ~no single-stream cost); mods/busy-loop-fix is vLLM-only, not used by this lane; nvidia-smi -pl is a NO-OP on GB10. Orca-native NVFP4 (bf16 PLE ~95 GiB) OOMs 2x GB10 at TP2 and its ple_embedding.shard_* names will not bind - the ModelOpt repack (FP8 E4M3 PLE) is required; MAMBA_SSM_CACHE_DTYPE=bfloat16 + MM_ENCODER_TP_MODE=data are required companions. 262K native serving is deliberate: 1M via YaRN x4 clamps concurrency (GDN/mamba state pool) and has a ~294-390K single-request wedge (MiaAI-Lab #11); 512K YaRN x2 is the validated stretch lane. License notes: recipe repo has no LICENSE file (ingested under CC-BY-4.0); SM121 QSA + token-0 patches are PixelML/MiaAI MIT; the QSA fp8-KV overlay is vendored AGPL-3.0-or-later from the single-Spark recipe (README cites files/patch_qsa_fp8_kv.py; not in this sanitized tree)." +max_context_len: 262144