{"slug":"deepseek-v4-1-flash-1017-fp8-tp1xpp2-dspark-2x-dgx-station-gb300","title":"DeepSeek V4.1 Flash 1017 FP8 TP1xPP2 DSpark 2x DGX Station GB300 RoCE vLLM 0.31 overlay","summary":"Pipeline-parallel TP1xPP2 + DSpark serving of DeepSeek V4.1 Flash (1017 official FP8 weights) across two DGX Station GB300 nodes over 400G CX8 RoCE. PP2 splits the 40 layers 20/20 and removes all 84 per-step all-reduces, replacing them with two point-to-point hops: single-stream temp-0 391 tok/s vs TP2 stock 290.5 (+34%), step 12.3 ms -> ~9.4 ms; aggregate soak 323/406/595/974 tok/s at C1/C2/C4/C8 (3.01x C1); GSM8K 97/100, 0 errors; 1M-token context, FP8 KV, Engram host table pinned in Grace RAM on rank 0. Ships the five-file vLLM v0.31.0 source-fix overlay (bind-mounted over stock vllm/vllm-openai:v0.31.0) that makes DSpark work under PP, plus preflight, launch scripts and soak/receipt verifiers. ALL perf numbers author-measured on their own pair (2026-10-25) - not reproduced locally; site has no GB300 hardware. Recipe by ebfio (port by Iris, a GLM-5.3 model, with Eduardo & Kestrel).","license":"NOASSERTION (repo); overlaid vLLM sources Apache-2.0","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"2acdf45851ff045dc48344004a9ec5fde3af3de4","content_hash":"739e619707a224aa4cd8782dc5ed8d43f68dce86","created_at":"2026-10-27 16:38:42.712204-06","notes":"Scout card t_ec419289; owner ADMIT ruling 2026-10-27 admitted the NEW hardware profile dgx-station-gb300-x2 (seeded via migration 0074; site has no GB300 hardware). All performance numbers are author-measured on the submitter's own pair (2026-10-25) and NOT locally reproduced; the repo ships verify/ instruments (texts_prod.py determinism receipt, pp2_soak.py C1-C8 aggregate + host-memory gates, gsm8k_gate.py, pp2_receipts.py, prefix_probe_agentscale.py). Corrections to scout card: overlay is FIVE files for four fixes (README 'The four fixes (five files)'), card prose said 4 overlays; config.env MEM_FRACTION_STATIC=0.80 belongs to the sglang lane - the vLLM lane uses --gpu-memory-utilization 0.9 (VLLM_GPU_MEMORY_UTILIZATION). Repo ships derived local checkpoints (MODEL_DIR_RANK0=derived KeysAblit, RANK1=official); official pinned revision 1bae0e10 used for provenance. Fix-3 note: upstream vLLM main already merged the input_ids gate - a future tag may retire 2 of the 5 overlays; re-check before bumping the image tag."},"engine_source":{"type":"docker_image","git_url":"https://github.com/ebfio/deepseek-v41-flash-pp2-pair","base_engine":"vllm","pip_package":null,"base_version":"vLLM v0.31.0 + 5-file PP2+DSpark source overlay (bind-mount)","image_digest":"docker.io/vllm/vllm-openai:v0.31.0","install_note":"Runs on the STOCK image vllm/vllm-openai:v0.31.0 - no custom registry image. serve_vllm_node.sh bind-mounts patches/vllm-pp2-flag/ (sitecustomize.py via PYTHONPATH=/opt/pp2-flag, dspark.py, v1/worker/gpu/model_runner.py, v1/worker/gpu/cudagraph_utils.py, v1/core/kv_cache_utils.py) over the image's vLLM sources; the whole-file overlays are v0.31.0 copies with the edits - valid ONLY on that tag; on any other tag use the shipped .diff files. VLLM_PATCH_PP2_FLAG=1 gates pp+DSpark (stock 0.31.0 refuses DSpark under PP). Two nodes: stage the recipe dir at the same path on both, fill placeholder IPs in config.env, then 'bash recipe/launch_cluster.sh' (rank 0 API :8001, rank 1 headless). Needs host RDMA staging (MOFED libibverbs/libmlx5 bind-mounts), >300 GiB MemAvailable per node (Engram host table ~189 GiB on rank 0), checkpoint on local NVMe per node. Verify: verify/texts_prod.py && verify/pp2_soak.py. vLLM main merged the fix-3 gate - a future tag may retire 2 of the 5 overlays.","pinned_commit":"71699b4a93ec68ea9340d07cfa3691be891516f9"},"credit":[{"name":"ebfio - DeepSeek V4.1 Flash PP2+DSpark GB300 pair recipe (port by Iris on 4x RTX Pro 6000, with Eduardo & Kestrel)","url":"https://github.com/ebfio/deepseek-v41-flash-pp2-pair"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/model","--served-model-name","deepseek-v4.1-flash","--trust-remote-code","--tokenizer-mode","deepseek_v41","--tensor-parallel-size","1","--pipeline-parallel-size","2","--distributed-executor-backend","mp","--nnodes","2","--node-rank","0","--master-addr","<rank0-rail0-ip>","--master-port","29501","--disable-custom-all-reduce","--max-model-len","1048576","--max-num-seqs","128","--max-num-batched-tokens","16384","--gpu-memory-utilization","0.9","--engram-config","{\"cpu_offload\": true, \"use_thp\": true}","--reasoning-parser","deepseek_v41","--tool-call-parser","deepseek_v41","--enable-auto-tool-choice","--enable-prefix-caching","--no-enable-flashinfer-autotune","--kv-cache-dtype","fp8","--attention-config","{\"backend\": \"FLASHINFER_MLA_SPARSE_DSV41\", \"indexer_kv_dtype\": \"mxfp4\", \"indexer_sparse_logits\": true}","--default-chat-template-kwargs","{\"preserve_thinking\": true}","--speculative-config","{\"method\":\"dspark\",\"num_speculative_tokens\":5,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":false}","--mm-encoder-tp-mode","data","--host","0.0.0.0","--port","8001"],"layouts":["pp"],"runtime":{"env":"Per-rank container env (serve_vllm_node.sh): VLLM_HOST_IP=<node rail0 ip>; NCCL_HCAS=mlx5_0,mlx5_1 dual-rail RoCE with NCCL_ALGO=RING NCCL_PROTO=SIMPLE NCCL_MIN/MAX_NCHANNELS=8 NCCL_IB_QPS_PER_CONNECTION=4 NCCL_DMABUF_ENABLE=1 NCCL_NET_GDR_LEVEL=SYS (tuned striping, measured best on this fabric; LL/LL128 auto-selection crashes on RoCE+QPS); GLOO/NCCL_SOCKET_IFNAME=cx8r0; VLLM_ALLREDUCE_USE_SYMM_MEM=0 VLLM_ALLREDUCE_USE_FLASHINFER=0 VLLM_USE_NCCL_SYMM_MEM=0; VLLM_DEEP_GEMM_WARMUP=relax; PYTHONPATH=/opt/pp2-flag (overlay). Host: HOST_RDMA=1 bind-mounts host MOFED libibverbs/libmlx5 for the ConnectX-8 Data Direct DMA path. VLLM_PATCH_PP2_FLAG=1 gates the pp+DSpark launch; MODE=low-latency selects DSpark n=5. ENGRAM_HOST_TABLE=1 ENGRAM_LAYOUT=private ENGRAM_PIN=1 ENGRAM_THP=1 (Engram shard cudaHostRegister, 2 MiB THP lookups). VLLM_FLASHINFER_AUTOTUNE=0 -> --no-enable-flashinfer-autotune (distributed autotune deadlocks cross-node). MODEL_DIR_RANK0=derived KeysAblit checkpoint, MODEL_DIR_RANK1=official 1017 (identical pinned revision, local NVMe per node).","notes":"Two-node launcher: stage the recipe dir at the same path on both nodes, fill placeholder IPs in config.env, then 'bash -n recipe/config.env && bash recipe/launch_cluster.sh' (health-waits, streams both ranks; NODE_RANK via launch_cluster per node, rank 0 = API host :8001, rank 1 headless). VLLM_PARALLEL=pp -> TP1xPP2 (40 layers 20/20; the flag-only fix refuses partitions whose last stage starts above layer 37 - drafts need layers 37-39 there). Persistent JIT caches seeded from the image's /root/.cache on first use. preflight.sh checks RDMA devices, retained HBM, jumbo ping on rail 1, MemAvailable > 300 GiB/node. Verify lane: python3 verify/texts_prod.py (determinism md5 x3) && python3 verify/pp2_soak.py (C1/C2/C4/C8 aggregate + host memory gates); receipts under logs/. MEM_FRACTION_STATIC=0.80 in config.env is the sglang lane - unused by the vLLM path."},"max_context_len":1048576,"storage_requirements":{"note":"Engram host table ~189 GiB pinned on rank 0 (preflight gates MemAvailable > 300 GiB per node); checkpoint on local NVMe per node","ram_gb":300}},"models":[{"slug":"deepseek-v4-1-flash-1017","name":"DeepSeek-V4.1-Flash-1017","hf_repo":"deepseek-ai/DeepSeek-V4.1-Flash-1017","quant":null}],"hardware":[{"slug":"dgx-station-gb300-x2","vendor":"nvidia","card":"DGX Station (GB300)","card_family":"dgx-station-gb300","vram_gb":288,"device_count":2}],"generation":"blackwell","tags":["dspark","engram","fp8-kv","gb300-pair","long-context","multi-node","pp2","rdma","roce","speculative"],"tasks":["chat","code","vision"],"aa":{"aa_index":45,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash-1017","aa_index_version":"Index v4.3.2 (2026-10-18)","aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-19 12:00:00-06","aa_family_slug":"deepseek-v4-1-flash-1017","aa_resolved_slug":"deepseek-v4-1-flash-1017"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"ALL numbers submitter-claimed: measured by the recipe author on their own DGX Station GB300 pair (receipts 2026-10-25); the site has no GB300 hardware and has not reproduced them. GSM8K 100-question slice temp-0: 97/100, 0 errors (verify/gsm8k_gate.py). Free KV cache: rank0 72.7 GiB / rank1 57.1 GiB. Host memory flat through the C8 soak, zero swap. TP2-vs-PP2 at C4-C16 not yet measured on this build; author recommends capping concurrency 4-6 for interactive per-stream latency. Aggregate rows are the round-robin soak corpus (verify/pp2_soak.py), NOT the single-stream prompt. Overlay valid on vllm/vllm-openai:v0.31.0 only.\", \"tok_s\": 391, \"points\": [{\"notes\": \"single-stream temp-0 identical prompt, PP2 + DSpark n=5 probabilistic; vs TP2 stock 290.5 (+34%); step 12.3 ms -> ~9.4 ms, 0 all-reduces; author receipt 2026-10-25\", \"concurrency\": 1, \"context_len\": 1048576, \"decode_tok_s\": 391}, {\"notes\": \"aggregate soak C1 (round-robin corpus, verify/pp2_soak.py); context_len is the tuple marker 1048576+10240 (disambiguates from the single-stream C1 row)\", \"concurrency\": 1, \"context_len\": 1059816, \"decode_tok_s\": 323}, {\"notes\": \"aggregate soak C2; context_len is the tuple marker 1048576+20480\", \"concurrency\": 2, \"context_len\": 1069056, \"decode_tok_s\": 406}, {\"notes\": \"aggregate soak C4; context_len is the tuple marker 1048576+30720; cap 4-6 interactive per author\", \"concurrency\": 4, \"context_len\": 1079296, \"decode_tok_s\": 595}, {\"notes\": \"aggregate soak C8 = 3.01x C1; context_len is the tuple marker 1048576+40960; host memory flat, zero swap\", \"concurrency\": 8, \"context_len\": 1089536, \"decode_tok_s\": 974}], \"context_len\": 1048576}"}],"perf_points":[{"id":1852,"context_len":1048576,"prefill_tok_s":null,"decode_tok_s":391,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream temp-0 identical prompt, PP2 + DSpark n=5 probabilistic; vs TP2 stock 290.5 (+34%); step 12.3 ms -> ~9.4 ms, 0 all-reduces; author receipt 2026-10-25"},{"id":1853,"context_len":1059816,"prefill_tok_s":null,"decode_tok_s":323,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"aggregate soak C1 (round-robin corpus, verify/pp2_soak.py); context_len is the tuple marker 1048576+10240 (disambiguates from the single-stream C1 row)"},{"id":1854,"context_len":1069056,"prefill_tok_s":null,"decode_tok_s":406,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"aggregate soak C2; context_len is the tuple marker 1048576+20480"},{"id":1855,"context_len":1079296,"prefill_tok_s":null,"decode_tok_s":595,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"aggregate soak C4; context_len is the tuple marker 1048576+30720; cap 4-6 interactive per author"},{"id":1856,"context_len":1089536,"prefill_tok_s":null,"decode_tok_s":974,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate soak C8 = 3.01x C1; context_len is the tuple marker 1048576+40960; host memory flat, zero swap"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-fp8-tp1xpp2-dspark-2x-dgx-station-gb300.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-fp8-tp1xpp2-dspark-2x-dgx-station-gb300.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017","pin":"1bae0e1088993b9b82de08f59de5be5a34e42170"},{"kind":"engine_image","pointer_type":"url","pointer":"https://hub.docker.com/r/vllm/vllm-openai/tags?name=v0.31.0","pin":"vllm/vllm-openai:v0.31.0 (tag, not digest)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/ebfio/deepseek-v41-flash-pp2-pair","pin":"71699b4a93ec68ea9340d07cfa3691be891516f9"}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-fp8-tp1xpp2-dspark-2x-dgx-station-gb300.json","alternate":"/r/deepseek-v4-1-flash-1017-fp8-tp1xpp2-dspark-2x-dgx-station-gb300","describedby":"/openapi.json"},"updated_at":"2026-10-27 16:38:42.712204-06","created_at":"2026-10-27 16:38:42.712204-06","speeds":{"decode_tok_s":357,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}