Qwen3.8-Flash-Next NVFP4 on RTX PRO 6000 Blackwell — TP1 MTP3

vllm serve /model --served-model-name Qwen3.8-Flash-Next-NVFP4 --host 0.0.0.0 --port 8000 --tensor-parallel-size 1 --pipeline-parallel-size 1 --mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill --mamba-ssm-cache-dtype auto --async-scheduling --dtype bfloat16 --kv-cache-dtype fp8 --quantization modelopt_mixed --block-size 64 --load-format instanttensor --gpu-memory-utilization 0.98 --max-model-len 262144 --max-num-seqs 64 --max-num-batched-tokens 8192 --mm-encoder-tp-mode data --mm-processor-cache-gb 0 --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"b12x"}' --gdn-decode-kernel b12x --linear-backend b12x --moe-backend b12x --no-enable-flashinfer-autotune --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' --max-cudagraph-capture-size 64 --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

env: 'TP=1 fixed by the image entrypoint contract; map container port 8000 to any free host port and serve both the OpenAI-compatible API at http://localhost:<host-port>/v1 and the Anthropic-compatible Messages API at http://localhost:<host-port>/anthropic/v1/messages. PLE n-gram table (~27 GiB FP8) lives in host RAM via VLLM_PLE_CPU_OFFLOAD=1 — without it the 96 GB card cannot fit 262K ctx. MTP3 runs with b12x MoE; LM head stays A16 (VLLM_MTP_NVFP4_LM_HEAD=1 + VLLM_LM_HEAD_A16=1) because NVFP4 lm_head breaks acceptance. Vision on by default: mm-processor-cache-gb 0 + 8192 batch leaves ~4.05 GiB for the ViT + 262K KV pool (16k batch + 2 GiB mm cache leaves only 3.74 GiB and OOMs warmup). Separate JIT cache prefix /cache/jit/qwen38-r281-glp47 when steering is on so stock CUDA graphs are not reused.'
notes: 'Mutually exclusive with every other 6000 occupant (SGLang Flash-Next, 27B dflash, dots TTS) — the launcher evicts them and binds the whole card. Restart policy unless-stopped on purpose: engine death often exits 0 after clean API shutdown, on-failure would leave the card empty. instanttensor load = 95 s weights + ~150 s model init; first graph capture adds minutes (persistent JIT caches make restarts much cheaper). GLP-47 overlay: hotfix-qwen38fn-steering-projective.py runs before vLLM; alpha 1.0 is the calibrated shipping value (do NOT copy DSV4 4.0). Model image class Qwen4ExpForConditionalGeneration, modelopt quant, 48 layers, 512 experts.'
container: 'docker run -d --name vllm-fnxt-6000-jj --init --device nvidia.com/gpu=<RTX PRO 6000 uuid> --cpuset-cpus=<dedicated-physical-cores, e.g. 0-5,18-23> --restart=unless-stopped --ipc=host --shm-size=32g --ulimit memlock=-1 --ulimit stack=67108864 --memory=96g --memory-reservation=64g -p <host-port>:8000 -v /path/to/Qwen3.8-Flash-Next-NVFP4:/model:ro -v ~/.cache/vllm-jj-fnxt:/cache -e TP=1 -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_MTP_NVFP4_LM_HEAD=1 -e VLLM_LM_HEAD_A16=1 -e VLLM_QWEN3_8_FLASH_NEXT_MTP_COMPACT=1 -e VLLM_QWEN3_8_FLASH_NEXT_OVERLAP=1 -e VLLM_GDN_SPEC_DECODE_METADATA_FASTPATH=1 -e VLLM_ENABLE_PCIE_ALLREDUCE=1 -e VLLM_PCIE_ALLREDUCE_BACKEND=b12x -e VLLM_DISABLED_KERNELS=MarlinFP8ScaledMMLinearKernel -e VLLM_CAUSAL_CONV1D_UPDATE_HOIST=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e VLLM_SSM_CONV_STATE_LAYOUT=DS -e VLLM_USE_V2_MODEL_RUNNER=1 -e B12X_DYNAMIC_WORK_SOURCE=persistent_grid -e B12X_DYNAMIC_SPLIT_COMPUTE_MAC=224 -e XDG_CACHE_HOME=/cache/jit/qwen38-r281-glp47 localinferencelab/vllm:jovian-judgement-community-20260910-r34 -m vllm.entrypoints.cli.main serve /model <launch args>'

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json