DeepSeek-V4.1-Flash-1017 Flare TP3 3x RTX PRO 6000 Blackwell

community Community submission — verify on your own hardware.

recipe by BoxTops4Coochie (recipe, Flare TP3 fork, measurements)↗, NVIDIA - Flare runtime + MXFP4 Engram tables↗, Local Inference Lab - FlashInfer b12x kernels↗, DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)↗ · listed by localai-recipes

Initial version · updated 5 h ago · license MIT

chat code 1m-ctx tensor-parallel dspark long-context engram speculative expert-offload mxfp4 community-recipe flare

NVIDIA Flare TP3 recipe for DeepSeek-V4.1-Flash-1017 on 3x 96 GB RTX PRO 6000 Blackwell, full 1M-token context: personal Flare fork (main @ 78b18b1c + source.patch) adds a merged VRAM/host hot-cold expert bank (300 hot experts per parked layer 20-33), MXFP4 Engram tables in host RAM, vocabulary-parallel sampling, and world-3 PCIe all-reduce/planner fixes; FlashInfer @ ddbe93fc b12x plus four tp3-patches. Package exported locally with export.sh (~40 min); serve.sh starts in ~60 s on port 15015. Measured 2026-11-05: KV pool 2,991,488 tokens (2.99 x 1M); decode 32K c1/c4/c8 ~254/595/805 tok/s; prefill 32K/128K 12,548/12,160 tok/s; coding peak median 379 tok/s; eval 43/43, tool calls 60/60, lavd + hotel-lights 30/30 each at c4, LiveBench coding 82.9. First Flare-engine recipe on the site. Launch argv verbatim from serve.sh at the pinned commit.

Launch command

'FLARE=<fork>/flare/target/release/flare PACKAGE=ds41-1017-engram-mxfp4 ./serve.sh' '# serve.sh default (ENGRAM_RAM=1) execs:' /usr/bin/env FLARE_ENGRAM_RAM_TABLES=0 '<fork>/flare/target/release/flare' run ds41-1017-engram-mxfp4 '--gpus 0,1,2' '--port 15015' '--host 127.0.0.1' '--served-model-name DeepSeek-V4.1-Flash' '--prefix-cache-memory 16G' '--mtp-draft-tokens 5' --mtp-adaptive-verification '--mtp-max-verification-token-budget 128' '--mtp-adaptive-verification-alpha 0.6' '--scheduler-policy balanced'

runtime

env: 'serve.sh: FLARE=<fork>/flare/target/release/flare (default flare/target/release/flare) | PACKAGE=ds41-1017-engram-mxfp4 (Flare model-cache default version) | ENGRAM_RAM=1 default sets FLARE_ENGRAM_RAM_TABLES=0 (Engram tables in host RAM); ENGRAM_RAM=0 unsets it -> tables read from drive via page cache ~3-5% slower at c1 | GPUS=0,1,2 | PORT=15015 | HOST=127.0.0.1 | PREFIX_CACHE=16G (verbatim from serve.sh @ a5269231). Export-time (export.sh): FLARE_DS41_HOST_EXPERT_LAYERS=20-33 | FLARE_DS41_HOT_EXPERTS=300 (HOT) | FLARE_DS41_MERGED_BANK=1 | FLARE_DS41_EXPERT_COUNTS_DIR=expert-order/ | FLARE_DS41_TP3_VOCAB_PARALLEL=1 | FLARE_DS41_TP3_GRAPH_GIB=1 | HF_HUB_OFFLINE=1; ignition export --max-concurrency 8 --prefill-chunk-size 4096 --mtp-draft-tokens 5 --draft-head-quantization nvfp4 --markov-head-quantization fp8 --vocabulary-parallel-sampling.'
notes: "Flare serves an EXPORTED package (weights laid out per GPU, kernels autotuned, CUDA graphs captured) - the package is tied to the hardware/clocks it was exported with (this one: 350 W, +6000 MHz memory offset, PCIe Gen5 x16). Alternative exports (same checkpoint, export.sh only changes HOT/PREFILL): HOT=267 PREFILL=4096 -> 6.05M KV, decode 32K 237/510/688 c1/c4/c8, for six simultaneous 1M sessions; HOT=313 PREFILL=2048 -> 3.47M KV. top_p is fixed at 1.0 (vocabulary-parallel sampling has no top-p filter; requests with lower top_p are clamped) and temperature fixed at 1.0 by Flare's DeepSeek recipe. Spec decode: DSpark (checkpoint's 3 mtp.* layers), 5 drafts, NVFP4 draft head, FP8 Markov head, adaptive verification alpha 0.6 budget 128. TP3 required 4 FlashInfer patches in tp3-patches/ (swiglu_limit in the tiny_decode W4A8 kernel, world-3 DCP a2a + tuning tables, MXFP4 engram lookup) and the fork's planner/bootstrap fixes in source.patch."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-flare-tp3-3x-rtx-pro-6000-blackwell.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-flare-tp3-3x-rtx-pro-6000-blackwell.json

Engine

flare · source↗ · a5269231↗

Install note: No prebuilt package or image: build the fork per the folder EXPORT.md - clone NVIDIA/flare-local-inference at 78b18b1c, git apply source.patch (32 files), uv sync --locked --extra prepare --extra test (Python 3.12, pulls FlashInfer @ ddbe93fc from local-inference-lab/flashinfer), re-apply tp3-patches/*.patch into the installed FlashInfer after every uv sync, cargo build --release --manifest-path flare/Cargo.toml. Download both public checkpoints (~412 GB), assemble the checkpoint view with make_checkpoint_view.sh + this folder's corrected config.json/model.safetensors.index.json, then CKPT=... export.sh (~40 min on 3 GPUs, ~900 GB free disk, memlock unlimited, CUDA 13.1, Rust 1.97). Package is hardware-specific: 3x 96 GB SM120 only, max concurrency 8.

Model

DeepSeek-V4.1-Flash-1017 (native-weights-mxfp4-engram-tables) · deepseek-ai/DeepSeek-V4.1-Flash-1017↗
AA intelligence 45 (Artificial Analysis · measured)

Hardware

NVIDIA RTX PRO 6000 Blackwell 96GB ×3

Context

max context: 1M tokens

KV pool

2,991,488 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
8,192	—	10977 tok/s	—	750 ms	reported
32,768	—	12548 tok/s	—	2570 ms	reported
32,768	1	—	253.8 tok/s	—	reported
32,768	4	—	595 tok/s	—	reported
32,768	8	—	804.5 tok/s	—	reported
43,008	1	—	378.8 tok/s	—	reported
131,072	—	12160 tok/s	—	10570 ms	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	claimed_by=submitter T0, author measurements 2026-11-05 with llm-inference-bench v0.9.1 on this box (350 W, +6000 MHz memory offset, 256 GB DDR5-6000), release export HOT=300 PREFILL=4096, Engram in RAM. Decode 32K: 253.8 / 595.0 / 804.5 tok/s at c1/c4/c8 (93.8 / 62.0 / 48.1 engine steps/s); a separate c1 run measured 304.6 tok/s at 93.2 steps/s (3.27 tokens/step) - tok/s depends on draft acceptance. c1 steps/s at 8K/32K/128K: 96.3 / 95.2 / 92.6. Coding peak (--coding-peak, 5 runs, median) 378.8 tok/s (range 350-417). Prefill: 10,977 / 12,548 / 12,160 tok/s at 8K / 32K / 128K (TTFT 0.75 / 2.57 / 10.57 s). Prefill trails the vLLM Kraken .11 build on the same box by ~12% at 32K, ~4% at 128K; decode leads (~254/595/805 vs 231-260/470-490/~790). Startup ~60 s (Engram tables into RAM) vs ~3.5 min for Kraken.

notes

claimed_by=submitter T0, author measurements 2026-11-05 with llm-inference-bench v0.9.1 on this box (350 W, +6000 MHz memory offset, 256 GB DDR5-6000), release export HOT=300 PREFILL=4096, Engram in RAM. Decode 32K: 253.8 / 595.0 / 804.5 tok/s at c1/c4/c8 (93.8 / 62.0 / 48.1 engine steps/s); a separate c1 run measured 304.6 tok/s at 93.2 steps/s (3.27 tokens/step) - tok/s depends on draft acceptance. c1 steps/s at 8K/32K/128K: 96.3 / 95.2 / 92.6. Coding peak (--coding-peak, 5 runs, median) 378.8 tok/s (range 350-417). Prefill: 10,977 / 12,548 / 12,160 tok/s at 8K / 32K / 128K (TTFT 0.75 / 2.57 / 10.57 s). Prefill trails the vLLM Kraken .11 build on the same box by ~12% at 32K, ~4% at 128K; decode leads (~254/595/805 vs 231-260/470-490/~790). Startup ~60 s (Engram tables into RAM) vs ~3.5 min for Kraken.

Downloads

Weights https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
dense + routed-expert weights @ 1bae0e1088993b9b82de08f59de5be5a34e42170 (MIT), shards 1-46
Weights https://huggingface.co/nvidia/DeepSeek-V4.1-Flash-1017-NVFP4
both Engram n-gram tables in MXFP4 (97 GiB, MIT), shards 47-48
Reference https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes
recipe folder 'DeepSeek-V4.1-Flash Flare-tp3-20261105' @ a5269231a570286898ba041690d524fdf1769c7e (watched)
Reference https://github.com/NVIDIA/flare-local-inference
flare main @ 78b18b1c + this folder's source.patch (32 files, fork branch ds41-tp3-v2)
Reference https://github.com/local-inference-lab/flashinfer
FlashInfer b12x @ ddbe93fc (gb202 pin) + four patches in tp3-patches/

Revision notes

Initial version.

Revision history

Initial version.

Fork

Export

recipe.yaml (raw)