{"slug":"qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3","title":"Qwen3.8-Flash-Next NVFP4 on RTX PRO 6000 Blackwell \u2014 TP1 MTP3","summary":"Single-card Blackwell recipe for Qwen3.8-Flash-Next (hybrid GDN/QSA MoE, 48 layers, 512 experts) on one 96 GB RTX PRO 6000: the Community image community image R34 (localinferencelab/vllm, v0.26.1rc0+glm53.r34), TP1 with MTP3 speculative decoding (b12x MoE backend), 256k context. Benchmark-qualified on RTX PRO 6000 Blackwell Max-Q: decode 148 tok/s at ctx-0 (temp 1), prefill 3860 tok/s at 2048 batched tokens. Launch argv transcribed verbatim from the r35 community deployment. Weights: JMNI-Labs/Qwen3.8-Flash-Next-NVFP4-QAD5500-Hybrid \u2014 step-5500 QAD trained tensors recombined with main 7c4f1bc1 MXFP8 attention + NVFP4 MTP experts; same-stack measured +5% MTP-normalized decode, +25% with MTP vs the main checkpoint (evals within noise, MMLU-Pro 66.9 vs 67.2).","license":"CC-BY-4.0","tier":"curated","beta":false,"revision":{"no":4,"commit_sha":"28709ecba026aff18371426bc26025e5a4185209","content_hash":"1e37b0f978392ba766c01cfdf9ae9ee3e002ee84","created_at":"2026-10-29 23:29:38.177885-06","notes":"Consolidation: r35 community-image perf runs merged from qwen3-8-flash-next-1x6000-jj-mtp3 (soft-deleted); r34 config and launch unchanged."},"engine_source":{"type":"docker_image","git_url":"https://github.com/local-inference-lab/rtx6kpro","base_engine":"vllm","base_version":"0.26.1rc0+glm53.r34 (Jovian Judgement community R34)","image_digest":"localinferencelab/vllm:jovian-judgement-community-20260910-r34","pinned_commit":null},"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/model","--served-model-name","Qwen3.8-Flash-Next-NVFP4","--host","0.0.0.0","--port","8000","--tensor-parallel-size","1","--pipeline-parallel-size","1","--mamba-cache-mode","align","--enable-prefix-caching","--enable-chunked-prefill","--mamba-ssm-cache-dtype","auto","--async-scheduling","--dtype","bfloat16","--kv-cache-dtype","fp8","--quantization","modelopt_mixed","--block-size","64","--load-format","instanttensor","--gpu-memory-utilization","0.98","--max-model-len","262144","--max-num-seqs","64","--max-num-batched-tokens","8192","--mm-encoder-tp-mode","data","--mm-processor-cache-gb","0","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":3,\"moe_backend\":\"b12x\"}","--gdn-decode-kernel","b12x","--linear-backend","b12x","--moe-backend","b12x","--no-enable-flashinfer-autotune","--compilation-config","{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"all\"]}","--max-cudagraph-capture-size","64","--reasoning-parser","qwen3","--tool-call-parser","qwen3_xml","--enable-auto-tool-choice"],"runtime":{"env":"TP=1 fixed by the image entrypoint contract; map container port 8000 to any free host port and serve both the OpenAI-compatible API at http://localhost:<host-port>/v1 and the Anthropic-compatible Messages API at http://localhost:<host-port>/anthropic/v1/messages. PLE n-gram table (~27 GiB FP8) lives in host RAM via VLLM_PLE_CPU_OFFLOAD=1 \u2014 without it the 96 GB card cannot fit 262K ctx. MTP3 runs with b12x MoE; LM head stays A16 (VLLM_MTP_NVFP4_LM_HEAD=1 + VLLM_LM_HEAD_A16=1) because NVFP4 lm_head breaks acceptance. Vision on by default: mm-processor-cache-gb 0 + 8192 batch leaves ~4.05 GiB for the ViT + 262K KV pool (16k batch + 2 GiB mm cache leaves only 3.74 GiB and OOMs warmup). Separate JIT cache prefix /cache/jit/qwen38-r281-glp47 when steering is on so stock CUDA graphs are not reused.","notes":"Mutually exclusive with every other 6000 occupant (SGLang Flash-Next, 27B dflash, dots TTS) \u2014 the launcher evicts them and binds the whole card. Restart policy unless-stopped on purpose: engine death often exits 0 after clean API shutdown, on-failure would leave the card empty. instanttensor load = 95 s weights + ~150 s model init; first graph capture adds minutes (persistent JIT caches make restarts much cheaper). GLP-47 overlay: hotfix-qwen38fn-steering-projective.py runs before vLLM; alpha 1.0 is the calibrated shipping value (do NOT copy DSV4 4.0). Model image class Qwen4ExpForConditionalGeneration, modelopt quant, 48 layers, 512 experts.","container":"docker run -d --name vllm-fnxt-6000-jj --init --device nvidia.com/gpu=<RTX PRO 6000 uuid> --cpuset-cpus=<dedicated-physical-cores, e.g. 0-5,18-23> --restart=unless-stopped --ipc=host --shm-size=32g --ulimit memlock=-1 --ulimit stack=67108864 --memory=96g --memory-reservation=64g -p <host-port>:8000 -v /path/to/Qwen3.8-Flash-Next-NVFP4:/model:ro -v ~/.cache/vllm-jj-fnxt:/cache -e TP=1 -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_MTP_NVFP4_LM_HEAD=1 -e VLLM_LM_HEAD_A16=1 -e VLLM_QWEN3_8_FLASH_NEXT_MTP_COMPACT=1 -e VLLM_QWEN3_8_FLASH_NEXT_OVERLAP=1 -e VLLM_GDN_SPEC_DECODE_METADATA_FASTPATH=1 -e VLLM_ENABLE_PCIE_ALLREDUCE=1 -e VLLM_PCIE_ALLREDUCE_BACKEND=b12x -e VLLM_DISABLED_KERNELS=MarlinFP8ScaledMMLinearKernel -e VLLM_CAUSAL_CONV1D_UPDATE_HOIST=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e VLLM_SSM_CONV_STATE_LAYOUT=DS -e VLLM_USE_V2_MODEL_RUNNER=1 -e B12X_DYNAMIC_WORK_SOURCE=persistent_grid -e B12X_DYNAMIC_SPLIT_COMPUTE_MAC=224 -e XDG_CACHE_HOME=/cache/jit/qwen38-r281-glp47 localinferencelab/vllm:jovian-judgement-community-20260910-r34 -m vllm.entrypoints.cli.main serve /model <launch args>"},"kv_pool_tokens":619559,"max_context_len":262144},"models":[{"slug":"qwen3-8-flash-next-nvfp4","name":"Qwen3.8-Flash-Next-NVFP4","hf_repo":"Qwen/Qwen3.8-Flash-Next","quant":"fp4-fp8-native"}],"hardware":[{"slug":"nvidia-rtx-pro-6000-blackwell","vendor":"nvidia","card":"rtx-pro-6000-blackwell","card_family":"rtx-pro-6000-blackwell","vram_gb":96,"device_count":1}],"generation":"blackwell","tags":["cpu-offload","fp8-kv","locked-config","mtp-speculative","ple-offload","prefix-cache","single-gpu","tensor-parallel","vision"],"tasks":["chat","code","vision"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Observed on the live production server (vllm-fnxt-6000-jj, 2026-09-27/28, 230 engine samples at Running=1): single-stream decode p50 ~117 tok/s, p75 ~144, p90 ~181 (MTP3 mean acceptance length 2.61 of 3, draft accept 45-49%). Prompt chunks p50 ~1,150 tok/s, p90 ~2,900 tok/s (peak 20k on cache-warm tiny prompts). KV pool 619,559 tokens = 2.36x admission at 262K (log: GPU KV cache size 619,559, Maximum concurrency 2.36x). T0 log-derived, not a harness run.\", \"tok_s\": 117, \"context_len\": 262144}"}],"perf_points":[{"id":2221,"context_len":1,"prefill_tok_s":null,"decode_tok_s":157.599999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"r35 community image (consolidated): Saved JJ R35, 5-window median, ctx-0 decode, temp 1/top-p .95/top-k 20 (verifier 76.1 steps/s)"},{"id":2222,"context_len":1,"prefill_tok_s":null,"decode_tok_s":674.200000000000045474735088646411895751953125,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"r35 community image (consolidated): Saved JJ R35, aggregate C8 (verifier 330.0 steps/s)"},{"id":2223,"context_len":32768,"prefill_tok_s":12104,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"r35 community image (consolidated): Saved JJ R35, uncached nominal-32K prefill, client TTFT"},{"id":2224,"context_len":262144,"prefill_tok_s":1150.5,"decode_tok_s":116.7999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"p50 single-stream from 228 engine samples (Running=1), mixed workloads; decode p75 144.2 / p90 181.2; prefill p90 ~2950 (12K+ chunks); MTP3 acceptance 2.61/3"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","pin":"local modelopt NVFP4 conversion at /path/to/Qwen3.8-Flash-Next-NVFP4 (bind-mounted; image does not re-download ~98 GB)"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/localinferencelab/vllm/tags?name=jovian-judgement-community-20260910-r34","pin":"image localinferencelab/vllm:jovian-judgement-community-20260910-r34 (~13 GB); engine v0.26.1rc0+glm53.r34.vllmc4966041; GLM overlay path /opt/glm53-flash/vllm"},{"kind":"abliteration","pointer_type":"hf_model","pointer":"https://huggingface.co/msuiche/Qwen3.8-Flash-Next-abliterated-cyber-GLP-47","pin":"GLP-47 abliteration overlay GGUF (Qwen3.8-Flash-Next-abliterated-cyber-GLP-47-L1-47-a1.gguf) + steering hotfix script. Run of record used the weightless/GLP-47 copy (gated); the msuiche mirror serves the same GGUF openly"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/local-inference-lab/rtx6kpro","pin":"models/qwen38-flash-next.md, community image R34 spec"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3","describedby":"/openapi.json"},"updated_at":"2026-10-29 23:29:38.177885-06","created_at":"2026-09-28 10:33:12.082127-06","speeds":{"decode_tok_s":137.19999999999998863131622783839702606201171875,"prefill_tok_s":6627.3000000000001818989403545856475830078125,"leaderboard":"/api/v1/leaderboard.json"}}