{"slug":"deepseek-v4-1-flash-1017-flare-tp3-3x-rtx-pro-6000-blackwell","title":"DeepSeek-V4.1-Flash-1017 Flare TP3 3x RTX PRO 6000 Blackwell","summary":"NVIDIA Flare TP3 recipe for DeepSeek-V4.1-Flash-1017 on 3x 96 GB RTX PRO 6000 Blackwell, full 1M-token context: personal Flare fork (main @ 78b18b1c + source.patch) adds a merged VRAM/host hot-cold expert bank (300 hot experts per parked layer 20-33), MXFP4 Engram tables in host RAM, vocabulary-parallel sampling, and world-3 PCIe all-reduce/planner fixes; FlashInfer @ ddbe93fc b12x plus four tp3-patches. Package exported locally with export.sh (~40 min); serve.sh starts in ~60 s on port 15015. Measured 2026-11-05: KV pool 2,991,488 tokens (2.99 x 1M); decode 32K c1/c4/c8 ~254/595/805 tok/s; prefill 32K/128K 12,548/12,160 tok/s; coding peak median 379 tok/s; eval 43/43, tool calls 60/60, lavd + hotel-lights 30/30 each at c4, LiveBench coding 82.9. First Flare-engine recipe on the site. Launch argv verbatim from serve.sh at the pinned commit.","license":"MIT","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"bfadd44fa78c32e00ba23603004cc8dd7177592b","content_hash":"a91234a6aee31d51a6cba9fbdebd94dafebc3cf1","created_at":"2026-11-05 18:55:44.135181-07","notes":"Community ingest of BoxTops4Coochie's recipe (listed-by localai-recipes; recipe, Flare TP3 fork and all measurements by BoxTops4Coochie). Re-verified at pinned HEAD a5269231 2026-11-06. Corrections to the scout card: the card's folder path 'DeepSeek-V4.1-Flash/Flare-tp3-20261105' is actually a single top-level folder 'DeepSeek-V4.1-Flash Flare-tp3-20261105' (space, not a nested path); card family deepseek-v4-1-flash corrected to deepseek-v4-1-flash-1017 - the site convention for all 1017-variant recipes (all 11 existing 1017 recipes use family deepseek-v4-1-flash-1017). README numbers confirmed verbatim at the pin: KV 2,991,488; decode 32K 253.8/595.0/804.5 (card rounded ~254/595/805); prefill 12,548/12,160; startup ~60 s; eval 43/43; tools 60/60; LiveBench coding 82.9 (LCB generation 71.8, coding completion 94.0). Not a dupe of the vLLM Kraken TP3 recipe on this box (different engine, package/export layer vs docker image). Not upstreamable as written per Flare contribution rules (model-specific offload policy + AI-authored GPU-execution changes) - personal fork. Upstream flare-local-inference pyproject currently floats a different FlashInfer sha (9c9e94ef on main); this recipe pins ddbe93fc per the folder's own lock/uv sync."},"engine_source":{"type":"fork","git_url":"https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes","base_engine":"flare","pip_package":null,"base_version":"flare main 78b18b1c + ds41-tp3-v2 fork + FlashInfer ddbe93fc","image_digest":null,"install_note":"No prebuilt package or image: build the fork per the folder EXPORT.md - clone NVIDIA/flare-local-inference at 78b18b1c, git apply source.patch (32 files), uv sync --locked --extra prepare --extra test (Python 3.12, pulls FlashInfer @ ddbe93fc from local-inference-lab/flashinfer), re-apply tp3-patches/*.patch into the installed FlashInfer after every uv sync, cargo build --release --manifest-path flare/Cargo.toml. Download both public checkpoints (~412 GB), assemble the checkpoint view with make_checkpoint_view.sh + this folder's corrected config.json/model.safetensors.index.json, then CKPT=... export.sh (~40 min on 3 GPUs, ~900 GB free disk, memlock unlimited, CUDA 13.1, Rust 1.97). Package is hardware-specific: 3x 96 GB SM120 only, max concurrency 8.","pinned_commit":"a5269231a570286898ba041690d524fdf1769c7e"},"credit":[{"name":"BoxTops4Coochie (recipe, Flare TP3 fork, measurements)","url":"https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes"},{"name":"NVIDIA - Flare runtime + MXFP4 Engram tables","url":"https://github.com/NVIDIA/flare-local-inference"},{"name":"Local Inference Lab - FlashInfer b12x kernels","url":"https://github.com/local-inference-lab/flashinfer"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["FLARE=<fork>/flare/target/release/flare PACKAGE=ds41-1017-engram-mxfp4 ./serve.sh","# serve.sh default (ENGRAM_RAM=1) execs:","/usr/bin/env","FLARE_ENGRAM_RAM_TABLES=0","<fork>/flare/target/release/flare","run","ds41-1017-engram-mxfp4","--gpus 0,1,2","--port 15015","--host 127.0.0.1","--served-model-name DeepSeek-V4.1-Flash","--prefix-cache-memory 16G","--mtp-draft-tokens 5","--mtp-adaptive-verification","--mtp-max-verification-token-budget 128","--mtp-adaptive-verification-alpha 0.6","--scheduler-policy balanced"],"layouts":["tp"],"runtime":{"env":"serve.sh: FLARE=<fork>/flare/target/release/flare (default flare/target/release/flare) | PACKAGE=ds41-1017-engram-mxfp4 (Flare model-cache default version) | ENGRAM_RAM=1 default sets FLARE_ENGRAM_RAM_TABLES=0 (Engram tables in host RAM); ENGRAM_RAM=0 unsets it -> tables read from drive via page cache ~3-5% slower at c1 | GPUS=0,1,2 | PORT=15015 | HOST=127.0.0.1 | PREFIX_CACHE=16G (verbatim from serve.sh @ a5269231). Export-time (export.sh): FLARE_DS41_HOST_EXPERT_LAYERS=20-33 | FLARE_DS41_HOT_EXPERTS=300 (HOT) | FLARE_DS41_MERGED_BANK=1 | FLARE_DS41_EXPERT_COUNTS_DIR=expert-order/ | FLARE_DS41_TP3_VOCAB_PARALLEL=1 | FLARE_DS41_TP3_GRAPH_GIB=1 | HF_HUB_OFFLINE=1; ignition export --max-concurrency 8 --prefill-chunk-size 4096 --mtp-draft-tokens 5 --draft-head-quantization nvfp4 --markov-head-quantization fp8 --vocabulary-parallel-sampling.","notes":"Flare serves an EXPORTED package (weights laid out per GPU, kernels autotuned, CUDA graphs captured) - the package is tied to the hardware/clocks it was exported with (this one: 350 W, +6000 MHz memory offset, PCIe Gen5 x16). Alternative exports (same checkpoint, export.sh only changes HOT/PREFILL): HOT=267 PREFILL=4096 -> 6.05M KV, decode 32K 237/510/688 c1/c4/c8, for six simultaneous 1M sessions; HOT=313 PREFILL=2048 -> 3.47M KV. top_p is fixed at 1.0 (vocabulary-parallel sampling has no top-p filter; requests with lower top_p are clamped) and temperature fixed at 1.0 by Flare's DeepSeek recipe. Spec decode: DSpark (checkpoint's 3 mtp.* layers), 5 drafts, NVFP4 draft head, FP8 Markov head, adaptive verification alpha 0.6 budget 128. TP3 required 4 FlashInfer patches in tp3-patches/ (swiglu_limit in the tiny_decode W4A8 kernel, world-3 DCP a2a + tuning tables, MXFP4 engram lookup) and the fork's planner/bootstrap fixes in source.patch."},"kv_pool_tokens":2991488,"max_context_len":1048576,"storage_requirements":{"note":"~407 GB exported package on NVMe (~60 s startup read). Engram ~102 GiB + parked experts ~13 GiB in host RAM (~150 GB used). ENGRAM_RAM=0: tables via page cache, ~3-5% slower c1.","ram_gb":256,"nvme_gb":410}},"models":[{"slug":"deepseek-v4-1-flash-1017","name":"DeepSeek-V4.1-Flash-1017","hf_repo":"deepseek-ai/DeepSeek-V4.1-Flash-1017","quant":"native-weights-mxfp4-engram-tables"}],"hardware":[{"slug":"rtx-pro-6000-blackwell-x3","vendor":"nvidia","card":"rtx-pro-6000-blackwell","card_family":"rtx-pro-6000-blackwell","vram_gb":96,"device_count":3}],"generation":"blackwell","tags":["1m-ctx","community-recipe","dspark","engram","expert-offload","flare","long-context","mxfp4","speculative","tensor-parallel"],"tasks":["chat","code"],"aa":{"aa_index":45,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash-1017","aa_index_version":"Index v4.3.2 (2026-10-18)","aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-19 12:00:00-06","aa_family_slug":"deepseek-v4-1-flash-1017","aa_resolved_slug":"deepseek-v4-1-flash-1017"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"claimed_by=submitter T0, author measurements 2026-11-05 with llm-inference-bench v0.9.1 on this box (350 W, +6000 MHz memory offset, 256 GB DDR5-6000), release export HOT=300 PREFILL=4096, Engram in RAM. Decode 32K: 253.8 / 595.0 / 804.5 tok/s at c1/c4/c8 (93.8 / 62.0 / 48.1 engine steps/s); a separate c1 run measured 304.6 tok/s at 93.2 steps/s (3.27 tokens/step) - tok/s depends on draft acceptance. c1 steps/s at 8K/32K/128K: 96.3 / 95.2 / 92.6. Coding peak (--coding-peak, 5 runs, median) 378.8 tok/s (range 350-417). Prefill: 10,977 / 12,548 / 12,160 tok/s at 8K / 32K / 128K (TTFT 0.75 / 2.57 / 10.57 s). Prefill trails the vLLM Kraken .11 build on the same box by ~12% at 32K, ~4% at 128K; decode leads (~254/595/805 vs 231-260/470-490/~790). Startup ~60 s (Engram tables into RAM) vs ~3.5 min for Kraken.\", \"tok_s\": 253.80000000000001136868377216160297393798828125, \"points\": [{\"notes\": \"llm-inference-bench v0.9.1, release export HOT=300 PREFILL=4096, 93.8 engine steps/s; separate run 304.6 tok/s at 93.2 steps/s (3.27 tokens/step) - tok/s depends on draft acceptance.\", \"concurrency\": 1, \"context_len\": 32768, \"decode_tok_s\": 253.80000000000001136868377216160297393798828125}, {\"notes\": \"62.0 engine steps/s. c4 is where Flare leads most vs the vLLM Kraken build (~595 vs 470-490).\", \"concurrency\": 4, \"context_len\": 32768, \"decode_tok_s\": 595}, {\"notes\": \"48.1 engine steps/s; max concurrency of this package is 8.\", \"concurrency\": 8, \"context_len\": 32768, \"decode_tok_s\": 804.5}, {\"notes\": \"Prefill 8K.\", \"ttft_ms\": 750, \"context_len\": 8192, \"prefill_tok_s\": 10977}, {\"notes\": \"Prefill 32K; ~12% behind vLLM Kraken .11 (14,195) on the same box.\", \"ttft_ms\": 2570, \"context_len\": 32768, \"prefill_tok_s\": 12548}, {\"notes\": \"Prefill 128K; ~4% behind Kraken (12,716).\", \"ttft_ms\": 10570, \"context_len\": 131072, \"prefill_tok_s\": 12160}, {\"notes\": \"coding peak (--coding-peak, 5 runs median, range 350-417). context_len is the tuple marker 32768+10240 to avoid colliding with the plain decode c1 row; the bench cell is 32K.\", \"concurrency\": 1, \"context_len\": 43008, \"decode_tok_s\": 378.80000000000001136868377216160297393798828125}], \"context_len\": 32768}"}],"perf_points":[{"id":2450,"context_len":8192,"prefill_tok_s":10977,"decode_tok_s":null,"ttft_ms":750,"concurrency":null,"claimed_by":"submitter","notes":"Prefill 8K."},{"id":2451,"context_len":32768,"prefill_tok_s":12548,"decode_tok_s":null,"ttft_ms":2570,"concurrency":null,"claimed_by":"submitter","notes":"Prefill 32K; ~12% behind vLLM Kraken .11 (14,195) on the same box."},{"id":2447,"context_len":32768,"prefill_tok_s":null,"decode_tok_s":253.80000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"llm-inference-bench v0.9.1, release export HOT=300 PREFILL=4096, 93.8 engine steps/s; separate run 304.6 tok/s at 93.2 steps/s (3.27 tokens/step) - tok/s depends on draft acceptance."},{"id":2448,"context_len":32768,"prefill_tok_s":null,"decode_tok_s":595,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"62.0 engine steps/s. c4 is where Flare leads most vs the vLLM Kraken build (~595 vs 470-490)."},{"id":2449,"context_len":32768,"prefill_tok_s":null,"decode_tok_s":804.5,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"48.1 engine steps/s; max concurrency of this package is 8."},{"id":2453,"context_len":43008,"prefill_tok_s":null,"decode_tok_s":378.80000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"coding peak (--coding-peak, 5 runs median, range 350-417). context_len is the tuple marker 32768+10240 to avoid colliding with the plain decode c1 row; the bench cell is 32K."},{"id":2452,"context_len":131072,"prefill_tok_s":12160,"decode_tok_s":null,"ttft_ms":10570,"concurrency":null,"claimed_by":"submitter","notes":"Prefill 128K; ~4% behind Kraken (12,716)."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-flare-tp3-3x-rtx-pro-6000-blackwell.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-flare-tp3-3x-rtx-pro-6000-blackwell.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017","pin":"dense + routed-expert weights @ 1bae0e1088993b9b82de08f59de5be5a34e42170 (MIT), shards 1-46"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/nvidia/DeepSeek-V4.1-Flash-1017-NVFP4","pin":"both Engram n-gram tables in MXFP4 (97 GiB, MIT), shards 47-48"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes","pin":"recipe folder 'DeepSeek-V4.1-Flash Flare-tp3-20261105' @ a5269231a570286898ba041690d524fdf1769c7e (watched)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/NVIDIA/flare-local-inference","pin":"flare main @ 78b18b1c + this folder's source.patch (32 files, fork branch ds41-tp3-v2)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/local-inference-lab/flashinfer","pin":"FlashInfer b12x @ ddbe93fc (gb202 pin) + four patches in tp3-patches/"}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-flare-tp3-3x-rtx-pro-6000-blackwell.json","alternate":"/r/deepseek-v4-1-flash-1017-flare-tp3-3x-rtx-pro-6000-blackwell","describedby":"/openapi.json"},"updated_at":"2026-11-05 18:55:44.135181-07","created_at":"2026-11-05 18:55:44.135181-07","speeds":{"decode_tok_s":316.30000000000001136868377216160297393798828125,"prefill_tok_s":12354,"leaderboard":"/api/v1/leaderboard.json"}}