localairecipes launch configs for local inference

DeepSeek-V4.1-Flash-1017 Kraken TP3 3x RTX PRO 6000 Blackwell

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by BoxTops4Coochie (recipe, TP3 port, measurements)↗, infernix - ds41f-tp3 expert-parking design on karmic-kraken↗, Local Inference Lab - vLLM karmic-kraken fork + FlashInfer b12x kernels↗, DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)↗ · listed by localai-recipes

revision r3 · commit fe33370ee6 · updated 19 h ago · license CC-BY-4.0

chat code vision 1m-ctx tensor-parallel dspark long-context engram speculative expert-offload b12x community-recipe

DeepSeek-V4.1-Flash-1017 at TP3 on three 96 GB RTX PRO 6000 Blackwell cards with the full 1M-token context, serving DeepSeek's original weights (FP8 dense + MXFP4 routed experts, nothing requantized). Expert layers 20-33 park in host RAM against a 1.5 GiB promotion reserve; once healthy their hottest experts promote back to GPU (hot-first order from 1017 route counts, lossless FlashInfer CSF scale compression). Spreading that fixed budget across 14 layers instead of 6 shallows each layer's cold tail - the author reports 68.0 -> 72.3 engine steps/s at 32K decode (+6.3%), no quality, KV or prefill cost. Engram layer 1 pinned in host RAM, layer 14 streamed from NVMe. DSpark speculative decoding, 5 drafts verified, NVFP4 drafter head. 1,048,576-token window over a 2,185,581-token KV pool, up to 4 sequences. Measured on image 20261019.3: prefill 13,486/12,245 tok/s (32K/128K), 7,193 at 983K; decode 186.7/340.7 c1/c4 at 32K, 179.2/307.1 at 128K, 474 aggregate on 4 shorts, 271.8 coding peak.

Launch command

'docker pull azallaza/ds41-kraken-tp3:20261019.3' 'docker run -d --name ds41-kraken-tp3 --restart unless-stopped --stop-timeout 90 --ipc host' '--security-opt seccomp=unconfined --ulimit memlock=-1 --ulimit nofile=1048576 --ulimit stack=67108864' '--gpus '\''"device=0,1,2"'\'' -p 15015:15015' '-v /path/to/huggingface:/root/.cache/huggingface:ro -v ds41-runtime-cache:/cache -e VLLM_CACHE_ROOT=/cache/vllm -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor -e TRITON_CACHE_DIR=/cache/triton' '-e MODEL=deepseek-ai/DeepSeek-V4.1-Flash-1017 -e KV_CACHE_MEMORY_BYTES=2415919104' '-e DS41_OFFLOAD_FROM_ORDINAL=20 -e DS41_OFFLOAD_TO_ORDINAL=33' '-e B12X_DEFERRED_BANK_RESERVE_BYTES=1610612736' '-e DS41_ENGRAM_RAM_LAYERS=1 -e DS41_ENGRAM_ROW_CACHE_ROWS=8388608' '-e VLLM_DS41_DRAFT_NVFP4_HEAD=1 -e DS41_LAUNCH_SETTINGS=/etc/ds41/launch.yaml -e HF_HUB_OFFLINE=1' azallaza/ds41-kraken-tp3:20261019.3 '# ENTRYPOINT ds41-serve runs the lil launcher (repo ds41-launch.yaml baked as /etc/ds41/launch.yaml):' '/usr/local/bin/lil-entrypoint --profile=ds41-flash --hardware=rtx-pro-6000-pcie --settings=/etc/ds41/launch.yaml --port=15015 --scheduling-policy=priority' '-- --kv-cache-memory-bytes=2415919104 --cpu-offload-gb=8 --cpu-offload-params experts vision aligner' 'lil settings: served-model-name=DeepSeek-V4.1-Flash tensor-parallel-size=3 cache-mode=vram' 'lil settings: max-model-len=1048576 max-num-seqs=4 max-num-batched-tokens=4096 max-cudagraph-capture-size=24' 'lil settings: gpu-memory-utilization=0.93 max-parallel-prefills=auto prefill-policy=round-robin' 'lil settings: mode=dspark draft-tokens=5 adaptive-verification=false' 'lil settings: override-generation-config={temperature:1.0,top_p:1.0}' 'lil settings: engram-table-memory=disk engram-disk-resident-scales=true jit-monitor-mode=warn'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell.json

Engine

Install note: Pull azallaza/ds41-kraken-tp3:20261019.3 and run the repo compose.yaml: HF_CACHE_DIR=/path/to/huggingface docker compose up -d (port 15015, served model name DeepSeek-V4.1-Flash). Needs 3x 96 GB GPUs (measured at 350 W cap, +6000 MiB memory offset), ~256 GB host RAM and the deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint in the HF cache on an NVMe drive. Image = LIL vLLM karmic-kraken-beta-20261016.2 (vLLM 6d66eed35) + FlashInfer main @ 5b065abe b12x (Python only, compiled at startup - a fresh cache volume adds ~10 min) + the 16-file TP3 port in src/. Expert promotion self-arms after /health: serve traffic only after the log line "ds41-serve: expert promotion armed and warmed".

Model

Hardware

Context

  • max context: 1M tokens

KV pool

  • 2,185,581 tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
1,0241—228.5 tok/s—reported
1,0244—474.3 tok/s—reported
4,0961—271.8 tok/s—reported
28,5531—253.1 tok/s—reported
32,768—13486 tok/s——reported
32,7681—186.7 tok/s—reported
32,7684—340.7 tok/s—reported
131,072—12245 tok/s——reported
131,0721—179.2 tok/s—reported
131,0724—307.1 tok/s—reported
143,360—10646 tok/s——reported
983,040—7193 tok/s——reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesclaimed_by=submitter T0 (author bench-20261019.3/, fresh restart 2026-10-20, image 20261019.3: NVFP4 drafter head + top-p 1.0 server default). llm_decode_bench 0.7.8 + bench.py quick + prefill_real.py + bench_long.py needle on threadripper-aabduh (Threadripper PRO 7965WX, 256 GB, driver 615.71.09), 350 W/GPU, +6000 MiB offset. DSpark 5 drafts verified: tok/s = steps/s x accept_len (c1 accept 2.71-2.77 tok/step). context_len markers: 1024 = ~59-token short prompt, 4096 = coding prompt, 28553 = 28K needle prompt length, 143360 = 131072+12288 real-code prefill marker, 983040 = 983K needle nominal. UPDATE 2026-10-28 (repo @ 4fc4f0c9825d): config-only change on the same image 20261019.3 - DS41_OFFLOAD_TO 25 -> 33 (layers 20-33, 14 layers instead of 6), reserve 1 GiB -> 1.5 GiB, restart policy no -> unless-stopped, and the vLLM/Inductor/Triton compile caches persisted to /cache. The author publishes this as llm-bench decode 68.0 -> 72.3 ENGINE STEPS/S at 32K, a two-run mean with +/-1-2 steps run-to-run noise; that is a step-rate figure and is NOT converted to tok/s here, because tok/s = steps/s x draft acceptance and acceptance was not re-reported for this config. The tok/s points below remain the author's measured 20261019.3 fresh-restart figures and were not re-measured at the new offload range. Author's stated limits: do not widen past ~34 (20-34/20-36 OOM at load because DS41_VMM_COLD_FRACTION=1 host-maps every wrapped layer up front); keep dspark draft layers 37-39 resident; at the 1 GiB reserve 20-33 runs ~5 MiB free and throws repeated expandable_segments memory-mapping OOM retries under concurrent long-context, while 1.5 GiB holds ~480 MiB free under a 4x600K-token prefill-during-decode stress. Supersedes the earlier 'parking 20-29 instead of 20-25 - no further gain' note, which the author retracts because that was a tok/s A/B that cannot resolve ~6% through spec-decode acceptance noise.

Install requirements

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Downloads

Revision notes

Community ingest of BoxTops4Coochie's recipe (repo folder 'DeepSeek-V4.1-Flash Kraken-tp3-20261019' @ bcf72fe6, self-contained: compose.yaml, ds41-launch.yaml, ds41-serve, Dockerfile, src/ TP3 port, bench/ + bench-20261019.3/, qualification/). Listed-by localai-recipes; recipe + measurements by BoxTops4Coochie. TP3 expert-parking design credited to infernix (ds41f-tp3 on karmic-kraken); runtime/kernels Local Inference Lab (vLLM fork, FlashInfer b12x); weights DeepSeek original 1017 @ 1bae0e10 (MIT). Discord corroboration (local-inference-lab #general, submitter-T0, not re-verified): Eric 2026-10-23 05:28Z "been running tp3 with Box's recipe that runs on kraken. it stores unused experts in RAM. very nice so far"; pete8359 04:39Z "TP3 dsv4.1f works lossless with Box's recipe"; andomort corroborated. Quality at top-p 1.0: lavd + hotel-lights 30/30 exact each @ 4 concurrent, 128K/900K retrieval pass, long generation 4/4 clean, vision 117/120 hard set. NOT a dupe of the site DS4.1 lanes (4x CMP 170HX Engram-NVMe TP4/PP4, 6x A100 SM80, EXL3 4x 170HX): first 3x RTX PRO 6000 Blackwell TP3 DS4.1-Flash recipe with full 1M context on original weights. Corrections to the scout card: image-inspect.json in the repo records the 20261019.2 image (sha256:06578acf...), not .3 - the engine_source digest is the 20261019.3 sha from README/Docker Hub. Facts re-verified at pinned HEAD 2026-10-24. UPDATE 2026-10-28 (repo @ 4fc4f0c9825d): config-only change on the same image 20261019.3 - DS41_OFFLOAD_TO 25 -> 33 (layers 20-33, 14 layers instead of 6), reserve 1 GiB -> 1.5 GiB, restart policy no -> unless-stopped, and the vLLM/Inductor/Triton compile caches persisted to /cache. The author publishes this as llm-bench decode 68.0 -> 72.3 ENGINE STEPS/S at 32K, a two-run mean with +/-1-2 steps run-to-run noise; that is a step-rate figure and is NOT converted to tok/s here, because tok/s = steps/s x draft acceptance and acceptance was not re-reported for this config. The tok/s points below remain the author's measured 20261019.3 fresh-restart figures and were not re-measured at the new offload range. Author's stated limits: do not widen past ~34 (20-34/20-36 OOM at load because DS41_VMM_COLD_FRACTION=1 host-maps every wrapped layer up front); keep dspark draft layers 37-39 resident; at the 1 GiB reserve 20-33 runs ~5 MiB free and throws repeated expandable_segments memory-mapping OOM retries under concurrent long-context, while 1.5 GiB holds ~480 MiB free under a 4x600K-token prefill-during-decode stress. Supersedes the earlier 'parking 20-29 instead of 20-25 - no further gain' note, which the author retracts because that was a tok/s A/B that cannot resolve ~6% through spec-decode acceptance noise.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)