{"slug":"deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell","title":"DeepSeek-V4.1-Flash-1017 Kraken TP3 3x RTX PRO 6000 Blackwell","summary":"DeepSeek-V4.1-Flash-1017 at TP3 on three 96 GB RTX PRO 6000 Blackwell cards with the full 1M-token context, serving DeepSeek's original weights (FP8 dense + MXFP4 routed experts, nothing requantized). Expert layers 20-33 park in host RAM against a 1.5 GiB promotion reserve; once healthy their hottest experts promote back to GPU (hot-first order from 1017 route counts, lossless FlashInfer CSF scale compression). Spreading that fixed budget across 14 layers instead of 6 shallows each layer's cold tail - the author reports 68.0 -> 72.3 engine steps/s at 32K decode (+6.3%), no quality, KV or prefill cost. Engram layer 1 pinned in host RAM, layer 14 streamed from NVMe. DSpark speculative decoding, 5 drafts verified, NVFP4 drafter head. 1,048,576-token window over a 2,185,581-token KV pool, up to 4 sequences. Measured on image 20261019.3: prefill 13,486/12,245 tok/s (32K/128K), 7,193 at 983K; decode 186.7/340.7 c1/c4 at 32K, 179.2/307.1 at 128K, 474 aggregate on 4 shorts, 271.8 coding peak.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"fe33370ee6a18643174519c8b9e956f05e807711","content_hash":"252307b64eed922ec9291880d6efb1938f670372","created_at":"2026-10-29 23:29:37.139691-06","notes":"Community ingest of BoxTops4Coochie's recipe (repo folder 'DeepSeek-V4.1-Flash Kraken-tp3-20261019' @ bcf72fe6, self-contained: compose.yaml, ds41-launch.yaml, ds41-serve, Dockerfile, src/ TP3 port, bench/ + bench-20261019.3/, qualification/). Listed-by localai-recipes; recipe + measurements by BoxTops4Coochie. TP3 expert-parking design credited to infernix (ds41f-tp3 on karmic-kraken); runtime/kernels Local Inference Lab (vLLM fork, FlashInfer b12x); weights DeepSeek original 1017 @ 1bae0e10 (MIT). Discord corroboration (local-inference-lab #general, submitter-T0, not re-verified): Eric 2026-10-23 05:28Z \"been running tp3 with Box's recipe that runs on kraken. it stores unused experts in RAM. very nice so far\"; pete8359 04:39Z \"TP3 dsv4.1f works lossless with Box's recipe\"; andomort corroborated. Quality at top-p 1.0: lavd + hotel-lights 30/30 exact each @ 4 concurrent, 128K/900K retrieval pass, long generation 4/4 clean, vision 117/120 hard set. NOT a dupe of the site DS4.1 lanes (4x CMP 170HX Engram-NVMe TP4/PP4, 6x A100 SM80, EXL3 4x 170HX): first 3x RTX PRO 6000 Blackwell TP3 DS4.1-Flash recipe with full 1M context on original weights. Corrections to the scout card: image-inspect.json in the repo records the 20261019.2 image (sha256:06578acf...), not .3 - the engine_source digest is the 20261019.3 sha from README/Docker Hub. Facts re-verified at pinned HEAD 2026-10-24. UPDATE 2026-10-28 (repo @ 4fc4f0c9825d): config-only change on the same image 20261019.3 - DS41_OFFLOAD_TO 25 -> 33 (layers 20-33, 14 layers instead of 6), reserve 1 GiB -> 1.5 GiB, restart policy no -> unless-stopped, and the vLLM/Inductor/Triton compile caches persisted to /cache. The author publishes this as llm-bench decode 68.0 -> 72.3 ENGINE STEPS/S at 32K, a two-run mean with +/-1-2 steps run-to-run noise; that is a step-rate figure and is NOT converted to tok/s here, because tok/s = steps/s x draft acceptance and acceptance was not re-reported for this config. The tok/s points below remain the author's measured 20261019.3 fresh-restart figures and were not re-measured at the new offload range. Author's stated limits: do not widen past ~34 (20-34/20-36 OOM at load because DS41_VMM_COLD_FRACTION=1 host-maps every wrapped layer up front); keep dspark draft layers 37-39 resident; at the 1 GiB reserve 20-33 runs ~5 MiB free and throws repeated expandable_segments memory-mapping OOM retries under concurrent long-context, while 1.5 GiB holds ~480 MiB free under a 4x600K-token prefill-during-decode stress. Supersedes the earlier 'parking 20-29 instead of 20-25 - no further gain' note, which the author retracts because that was a tok/s A/B that cannot resolve ~6% through spec-decode acceptance noise."},"engine_source":{"type":"docker_image","git_url":"https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes","base_engine":"vllm","pip_package":null,"base_version":"karmic-kraken-beta-20261016.2 + FlashInfer main 5b065abe","image_digest":"docker.io/azallaza/ds41-kraken-tp3@sha256:c4c2de529da6d3762951e7f5c5b696866f20d740a5f16d9cdda0f13786352646","install_note":"Pull azallaza/ds41-kraken-tp3:20261019.3 and run the repo compose.yaml: HF_CACHE_DIR=/path/to/huggingface docker compose up -d (port 15015, served model name DeepSeek-V4.1-Flash). Needs 3x 96 GB GPUs (measured at 350 W cap, +6000 MiB memory offset), ~256 GB host RAM and the deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint in the HF cache on an NVMe drive. Image = LIL vLLM karmic-kraken-beta-20261016.2 (vLLM 6d66eed35) + FlashInfer main @ 5b065abe b12x (Python only, compiled at startup - a fresh cache volume adds ~10 min) + the 16-file TP3 port in src/. Expert promotion self-arms after /health: serve traffic only after the log line \"ds41-serve: expert promotion armed and warmed\".","pinned_commit":"4fc4f0c9825da7d01f1985c1d3571d214c48085e"},"credit":[{"name":"BoxTops4Coochie (recipe, TP3 port, measurements)","url":"https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes"},{"name":"infernix - ds41f-tp3 expert-parking design on karmic-kraken","url":"https://github.com/infernix"},{"name":"Local Inference Lab - vLLM karmic-kraken fork + FlashInfer b12x kernels","url":"https://github.com/local-inference-lab"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["docker pull azallaza/ds41-kraken-tp3:20261019.3","docker run -d --name ds41-kraken-tp3 --restart unless-stopped --stop-timeout 90 --ipc host","--security-opt seccomp=unconfined --ulimit memlock=-1 --ulimit nofile=1048576 --ulimit stack=67108864","--gpus '\"device=0,1,2\"' -p 15015:15015","-v /path/to/huggingface:/root/.cache/huggingface:ro -v ds41-runtime-cache:/cache -e VLLM_CACHE_ROOT=/cache/vllm -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor -e TRITON_CACHE_DIR=/cache/triton","-e MODEL=deepseek-ai/DeepSeek-V4.1-Flash-1017 -e KV_CACHE_MEMORY_BYTES=2415919104","-e DS41_OFFLOAD_FROM_ORDINAL=20 -e DS41_OFFLOAD_TO_ORDINAL=33","-e B12X_DEFERRED_BANK_RESERVE_BYTES=1610612736","-e DS41_ENGRAM_RAM_LAYERS=1 -e DS41_ENGRAM_ROW_CACHE_ROWS=8388608","-e VLLM_DS41_DRAFT_NVFP4_HEAD=1 -e DS41_LAUNCH_SETTINGS=/etc/ds41/launch.yaml -e HF_HUB_OFFLINE=1","azallaza/ds41-kraken-tp3:20261019.3","# ENTRYPOINT ds41-serve runs the lil launcher (repo ds41-launch.yaml baked as /etc/ds41/launch.yaml):","/usr/local/bin/lil-entrypoint --profile=ds41-flash --hardware=rtx-pro-6000-pcie --settings=/etc/ds41/launch.yaml --port=15015 --scheduling-policy=priority","-- --kv-cache-memory-bytes=2415919104 --cpu-offload-gb=8 --cpu-offload-params experts vision aligner","lil settings: served-model-name=DeepSeek-V4.1-Flash tensor-parallel-size=3 cache-mode=vram","lil settings: max-model-len=1048576 max-num-seqs=4 max-num-batched-tokens=4096 max-cudagraph-capture-size=24","lil settings: gpu-memory-utilization=0.93 max-parallel-prefills=auto prefill-policy=round-robin","lil settings: mode=dspark draft-tokens=5 adaptive-verification=false","lil settings: override-generation-config={temperature:1.0,top_p:1.0}","lil settings: engram-table-memory=disk engram-disk-resident-scales=true jit-monitor-mode=warn"],"layouts":["tp"],"kv_pool_tokens":2185581,"max_context_len":1048576,"storage_requirements":{"note":"Checkpoint (~510 GB, 48 FP8 shards incl. both 94 GiB Engram tables) on NVMe - layer 14 is read from it during serving; ~256 GB host RAM, ~150 GiB used while serving.","ram_gb":256,"nvme_gb":510}},"models":[{"slug":"deepseek-v4-1-flash-1017","name":"DeepSeek-V4.1-Flash-1017","hf_repo":"deepseek-ai/DeepSeek-V4.1-Flash-1017","quant":"native-weights-mxfp4-experts-fp8-dense-deepseek-ai-release"}],"hardware":[{"slug":"rtx-pro-6000-blackwell-x3","vendor":"nvidia","card":"rtx-pro-6000-blackwell","card_family":"rtx-pro-6000-blackwell","vram_gb":96,"device_count":3}],"generation":"blackwell","tags":["1m-ctx","b12x","community-recipe","dspark","engram","expert-offload","long-context","speculative","tensor-parallel"],"tasks":["chat","code","vision"],"aa":{"aa_index":45,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash-1017","aa_index_version":"Index v4.3.2 (2026-10-18)","aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-19 12:00:00-06","aa_family_slug":"deepseek-v4-1-flash-1017","aa_resolved_slug":"deepseek-v4-1-flash-1017"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"claimed_by=submitter T0 (author bench-20261019.3/, fresh restart 2026-10-20, image 20261019.3: NVFP4 drafter head + top-p 1.0 server default). llm_decode_bench 0.7.8 + bench.py quick + prefill_real.py + bench_long.py needle on threadripper-aabduh (Threadripper PRO 7965WX, 256 GB, driver 615.71.09), 350 W/GPU, +6000 MiB offset. DSpark 5 drafts verified: tok/s = steps/s x accept_len (c1 accept 2.71-2.77 tok/step). context_len markers: 1024 = ~59-token short prompt, 4096 = coding prompt, 28553 = 28K needle prompt length, 143360 = 131072+12288 real-code prefill marker, 983040 = 983K needle nominal. UPDATE 2026-10-28 (repo @ 4fc4f0c9825d): config-only change on the same image 20261019.3 - DS41_OFFLOAD_TO 25 -> 33 (layers 20-33, 14 layers instead of 6), reserve 1 GiB -> 1.5 GiB, restart policy no -> unless-stopped, and the vLLM/Inductor/Triton compile caches persisted to /cache. The author publishes this as llm-bench decode 68.0 -> 72.3 ENGINE STEPS/S at 32K, a two-run mean with +/-1-2 steps run-to-run noise; that is a step-rate figure and is NOT converted to tok/s here, because tok/s = steps/s x draft acceptance and acceptance was not re-reported for this config. The tok/s points below remain the author's measured 20261019.3 fresh-restart figures and were not re-measured at the new offload range. Author's stated limits: do not widen past ~34 (20-34/20-36 OOM at load because DS41_VMM_COLD_FRACTION=1 host-maps every wrapped layer up front); keep dspark draft layers 37-39 resident; at the 1 GiB reserve 20-33 runs ~5 MiB free and throws repeated expandable_segments memory-mapping OOM retries under concurrent long-context, while 1.5 GiB holds ~480 MiB free under a 4x600K-token prefill-during-decode stress. Supersedes the earlier 'parking 20-29 instead of 20-25 - no further gain' note, which the author retracts because that was a tok/s A/B that cannot resolve ~6% through spec-decode acceptance noise.\", \"tok_s\": 186.69999999999998863131622783839702606201171875, \"context_len\": 32768}"}],"perf_points":[{"id":2175,"context_len":1024,"prefill_tok_s":null,"decode_tok_s":228.5,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Short-prompt decode c1, 2 runs 228.5 / 223.3 tok/s (79.1 / 79.4 steps/s x 2.89 / 2.81 tok/step). context_len 1024 is the marker for the ~59-token prompt row."},{"id":2176,"context_len":1024,"prefill_tok_s":null,"decode_tok_s":474.30000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"4 concurrent ~59-token requests: aggregate 474.3 tok/s (per-request 123.0-130.4). context_len 1024 marker."},{"id":2177,"context_len":4096,"prefill_tok_s":null,"decode_tok_s":271.80000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Coding peak (cc1 Sieve of Eratosthenes, 3 runs: mean 271.8 / median 267.1 / max 282.8 tok/s). context_len 4096 is the marker for the short coding prompt."},{"id":2178,"context_len":28553,"prefill_tok_s":null,"decode_tok_s":253.099999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"28,553-token needle decode, pass (bench-20261019.3 quick.json). context_len is the actual needle prompt length."},{"id":2179,"context_len":32768,"prefill_tok_s":13486,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"llm_decode_bench prefill 32K (32,251 tokens, TTFT 2.39 s). Concurrency omitted (NULL)."},{"id":2171,"context_len":32768,"prefill_tok_s":null,"decode_tok_s":186.69999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"llm_decode_bench 32K C1 (32,251-token context): aggregate 186.7 tok/s, TTFT 0.25 s, 67.5 engine steps/s, accept 2.77 tok/step."},{"id":2173,"context_len":32768,"prefill_tok_s":null,"decode_tok_s":340.69999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"llm_decode_bench 32K C4 aggregate: 340.7 tok/s (85.2 per request), accept 2.44."},{"id":2180,"context_len":131072,"prefill_tok_s":12245,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"llm_decode_bench prefill 128K (128,511 tokens, TTFT 10.5 s)."},{"id":2172,"context_len":131072,"prefill_tok_s":null,"decode_tok_s":179.19999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"llm_decode_bench 128K C1 (128,510-token context): 179.2 tok/s, 66.0 steps/s, accept 2.71."},{"id":2174,"context_len":131072,"prefill_tok_s":null,"decode_tok_s":307.1000000000000227373675443232059478759765625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"llm_decode_bench 128K C4 aggregate: 307.1 tok/s - the noisiest cell, 307-349 tok/s across runs of the same image; accept 2.26."},{"id":2181,"context_len":143360,"prefill_tok_s":10646,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"Real FlashInfer Python source prefill, 3 files (105K / 259K / 102K tokens): 10,646 / 10,228 / 11,354 tok/s (median shown). context_len 143360 is the tuple marker 131072+12288."},{"id":2182,"context_len":983040,"prefill_tok_s":7193,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"983K-token needle prefill (994,842 prompt tokens, TTFT 138.3 s, pass, peak 97,319 MiB). context_len 983040 nominal."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017","pin":"original weights @ 1bae0e1088993b9b82de08f59de5be5a34e42170 (FP8 dense + MXFP4 routed experts, nothing requantized)"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/azallaza/ds41-kraken-tp3","pin":"tag 20261019.3 (NVFP4 drafter head + top-p 1.0 default); .2 added max-parallel-prefills auto"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes","pin":"recipe folder 'DeepSeek-V4.1-Flash Kraken-tp3-20261019' @ bcf72fe69b949dce447b2974e1315e1a757e73bf (watched)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/local-inference-lab/flashinfer","pin":"b12x kernels @ 5b065abe0ce787364d26f01d753c7b73d2c0d57e (FlashInfer main, fetched by the Dockerfile)"}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell.json","alternate":"/r/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell","describedby":"/openapi.json"},"updated_at":"2026-10-29 23:29:37.139691-06","created_at":"2026-10-24 10:35:18.978794-06","speeds":{"decode_tok_s":228.5,"prefill_tok_s":11445.5,"leaderboard":"/api/v1/leaderboard.json"}}