diff --git a/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3/recipe.yaml b/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3/recipe.yaml index c83d78276..948eebc6e 100644 --- a/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3/recipe.yaml +++ b/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3/recipe.yaml @@ -6,7 +6,7 @@ engine: source_type: docker_image image_digest: 'localinferencelab/vllm:jovian-judgement-community-20260910-r34' git_url: 'https://github.com/local-inference-lab/rtx6kpro' -summary: 'Single-card Blackwell recipe for Qwen3.8-Flash-Next (hybrid GDN/QSA MoE, 48 layers, 512 experts) on one 96 GB RTX PRO 6000: the Community image (localinferencelab/vllm, v0.26.1rc0+glm53.r34), TP1 with MTP3 speculative decoding (b12x MoE backend), FP8 KV cache, PLE n-gram table offloaded to host RAM (~27 GiB) so the 96 GB card fits 262,144-token context, modelopt NVFP4 checkpoint bind-mounted locally. Vision on (mm cache disabled), 64 seqs, 8192 batched tokens, util 0.98, FULL_AND_PIECEWISE cudagraphs, GLP-47 abliteration overlay alpha 1.0 with the projective steering hotfix. B12x PCIe allreduce stack pinned via env; measured 262K KV pool 619,559 tokens (2.36x admission), single-stream decode p50 ~117 tok/s (MTP acceptance 2.6/3), prompt chunks to ~2,900 tok/s p90. Follow-up runs on the r35 community image (same stack, newer build) measured 157.6 tok/s decode at C1 and 12,104 tok/s prefill (nominal 32K) - both runs kept in the perf history below.' +summary: 'Single-card Blackwell recipe for Qwen3.8-Flash-Next (hybrid GDN/QSA MoE, 48 layers, 512 experts) on one 96 GB RTX PRO 6000: the Community image community image R34 (localinferencelab/vllm, v0.26.1rc0+glm53.r34), TP1 with MTP3 speculative decoding (b12x MoE backend), FP8 KV cache, PLE n-gram table offloaded to host RAM (~27 GiB) so the 96 GB card fits 262,144-token context, modelopt NVFP4 checkpoint bind-mounted locally. Vision on (mm cache disabled), 64 seqs, 8192 batched tokens, util 0.98, FULL_AND_PIECEWISE cudagraphs, GLP-47 abliteration overlay alpha 1.0 with the projective steering hotfix. B12x PCIe allreduce stack pinned via env; measured 262K KV pool 619,559 tokens (2.36x admission), single-stream decode p50 ~117 tok/s (MTP acceptance 2.6/3), prompt chunks to ~2,900 tok/s p90. Weights: JMNI-Labs/Qwen3.8-Flash-Next-NVFP4-QAD5500-Hybrid — step-5500 QAD trained tensors (qad-step5500-ple1000 60215d26) recombined with main 7c4f1bc1 MXFP8 text-attention + W4A16 NVFP4 MTP experts; same-stack measured +5% MTP-normalized decode / +25% with MTP / slightly better prefill vs the main checkpoint (evals within noise, MMLU-Pro 66.9 vs 67.2).' model: name: Qwen3.8-Flash-Next-NVFP4 family: qwen @@ -141,5 +141,5 @@ provenance: notes: 'First-party, from the live launcher on the production host (container up 2026-09-27). Engine-log-derived perf percentiles are claims pending a proper run file — import one via POST /api/v1/benchmarks to upgrade this recipe to T1.' credit: - - name: 'local-inference-lab' + name: local-inference-lab url: 'https://github.com/local-inference-lab/rtx6kpro'