DeepSeek V4.1 Flash 1017 FP8 TP1xPP2 DSpark 2x DGX Station GB300 RoCE vLLM 0.31 overlay

vllm serve /model --served-model-name deepseek-v4.1-flash --trust-remote-code --tokenizer-mode deepseek_v41 --tensor-parallel-size 1 --pipeline-parallel-size 2 --distributed-executor-backend mp --nnodes 2 --node-rank 0 --master-addr '<rank0-rail0-ip>' --master-port 29501 --disable-custom-all-reduce --max-model-len 1048576 --max-num-seqs 128 --max-num-batched-tokens 16384 --gpu-memory-utilization 0.9 --engram-config '{"cpu_offload": true, "use_thp": true}' --reasoning-parser deepseek_v41 --tool-call-parser deepseek_v41 --enable-auto-tool-choice --enable-prefix-caching --no-enable-flashinfer-autotune --kv-cache-dtype fp8 --attention-config '{"backend": "FLASHINFER_MLA_SPARSE_DSV41", "indexer_kv_dtype": "mxfp4", "indexer_sparse_logits": true}' --default-chat-template-kwargs '{"preserve_thinking": true}' --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}' --mm-encoder-tp-mode data --host 0.0.0.0 --port 8001

env: 'Per-rank container env (serve_vllm_node.sh): VLLM_HOST_IP=<node rail0 ip>; NCCL_HCAS=mlx5_0,mlx5_1 dual-rail RoCE with NCCL_ALGO=RING NCCL_PROTO=SIMPLE NCCL_MIN/MAX_NCHANNELS=8 NCCL_IB_QPS_PER_CONNECTION=4 NCCL_DMABUF_ENABLE=1 NCCL_NET_GDR_LEVEL=SYS (tuned striping, measured best on this fabric; LL/LL128 auto-selection crashes on RoCE+QPS); GLOO/NCCL_SOCKET_IFNAME=cx8r0; VLLM_ALLREDUCE_USE_SYMM_MEM=0 VLLM_ALLREDUCE_USE_FLASHINFER=0 VLLM_USE_NCCL_SYMM_MEM=0; VLLM_DEEP_GEMM_WARMUP=relax; PYTHONPATH=/opt/pp2-flag (overlay). Host: HOST_RDMA=1 bind-mounts host MOFED libibverbs/libmlx5 for the ConnectX-8 Data Direct DMA path. VLLM_PATCH_PP2_FLAG=1 gates the pp+DSpark launch; MODE=low-latency selects DSpark n=5. ENGRAM_HOST_TABLE=1 ENGRAM_LAYOUT=private ENGRAM_PIN=1 ENGRAM_THP=1 (Engram shard cudaHostRegister, 2 MiB THP lookups). VLLM_FLASHINFER_AUTOTUNE=0 -> --no-enable-flashinfer-autotune (distributed autotune deadlocks cross-node). MODEL_DIR_RANK0=derived KeysAblit checkpoint, MODEL_DIR_RANK1=official 1017 (identical pinned revision, local NVMe per node).'
notes: "Two-node launcher: stage the recipe dir at the same path on both nodes, fill placeholder IPs in config.env, then 'bash -n recipe/config.env && bash recipe/launch_cluster.sh' (health-waits, streams both ranks; NODE_RANK via launch_cluster per node, rank 0 = API host :8001, rank 1 headless). VLLM_PARALLEL=pp -> TP1xPP2 (40 layers 20/20; the flag-only fix refuses partitions whose last stage starts above layer 37 - drafts need layers 37-39 there). Persistent JIT caches seeded from the image's /root/.cache on first use. preflight.sh checks RDMA devices, retained HBM, jumbo ping on rail 1, MemAvailable > 300 GiB/node. Verify lane: python3 verify/texts_prod.py (determinism md5 x3) && python3 verify/pp2_soak.py (C1/C2/C4/C8 aggregate + host memory gates); receipts under logs/. MEM_FRACTION_STATIC=0.80 in config.env is the sglang lane - unused by the vLLM path."

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-fp8-tp1xpp2-dspark-2x-dgx-station-gb300.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-fp8-tp1xpp2-dspark-2x-dgx-station-gb300.json