diff --git a/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell/recipe.yaml b/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell/recipe.yaml index 4301cce27..252307b64 100644 --- a/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell/recipe.yaml +++ b/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell/recipe.yaml @@ -5,10 +5,10 @@ engine: version: 'karmic-kraken-beta-20261016.2 + FlashInfer main 5b065abe' source_type: docker_image git_url: 'https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes' - pinned_commit: bcf72fe69b949dce447b2974e1315e1a757e73bf + pinned_commit: 4fc4f0c9825da7d01f1985c1d3571d214c48085e image_digest: 'docker.io/azallaza/ds41-kraken-tp3@sha256:c4c2de529da6d3762951e7f5c5b696866f20d740a5f16d9cdda0f13786352646' install_note: 'Pull azallaza/ds41-kraken-tp3:20261019.3 and run the repo compose.yaml: HF_CACHE_DIR=/path/to/huggingface docker compose up -d (port 15015, served model name DeepSeek-V4.1-Flash). Needs 3x 96 GB GPUs (measured at 350 W cap, +6000 MiB memory offset), ~256 GB host RAM and the deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint in the HF cache on an NVMe drive. Image = LIL vLLM karmic-kraken-beta-20261016.2 (vLLM 6d66eed35) + FlashInfer main @ 5b065abe b12x (Python only, compiled at startup - a fresh cache volume adds ~10 min) + the 16-file TP3 port in src/. Expert promotion self-arms after /health: serve traffic only after the log line "ds41-serve: expert promotion armed and warmed".' -summary: "DeepSeek-V4.1-Flash-1017 at TP3 on three 96 GB RTX PRO 6000 Blackwell cards with the full 1M-token context, serving DeepSeek's original weights (FP8 dense + MXFP4 routed experts, nothing requantized). Expert layers 20-25 park in host RAM at startup; once healthy their hottest experts promote back to GPU (hot-first order from 1017 route counts, lossless FlashInfer CSF scale compression). Engram layer 1 pinned in host RAM, layer 14 streamed from the NVMe checkpoint. DSpark speculative decoding, 5 drafts, every draft verified, NVFP4 drafter head. 1,048,576-token window over a 2,185,581-token KV pool, up to 4 sequences. Measured on image 20261019.3, fresh restart: prefill 13,486 / 12,245 tok/s (32K/128K), 10,228-11,354 on real code, 7,193 at 983K (needle pass); decode 186.7 / 340.7 tok/s at 32K and 179.2 / 307.1 at 128K (c1/c4), 223-229 short, 474 aggregate on 4 concurrent shorts, 271.8 coding peak; 5/5 known answers, lavd + hotel-lights 30/30 exact, vision 117/120 hard." +summary: "DeepSeek-V4.1-Flash-1017 at TP3 on three 96 GB RTX PRO 6000 Blackwell cards with the full 1M-token context, serving DeepSeek's original weights (FP8 dense + MXFP4 routed experts, nothing requantized). Expert layers 20-33 park in host RAM against a 1.5 GiB promotion reserve; once healthy their hottest experts promote back to GPU (hot-first order from 1017 route counts, lossless FlashInfer CSF scale compression). Spreading that fixed budget across 14 layers instead of 6 shallows each layer's cold tail - the author reports 68.0 -> 72.3 engine steps/s at 32K decode (+6.3%), no quality, KV or prefill cost. Engram layer 1 pinned in host RAM, layer 14 streamed from NVMe. DSpark speculative decoding, 5 drafts verified, NVFP4 drafter head. 1,048,576-token window over a 2,185,581-token KV pool, up to 4 sequences. Measured on image 20261019.3: prefill 13,486/12,245 tok/s (32K/128K), 7,193 at 983K; decode 186.7/340.7 c1/c4 at 32K, 179.2/307.1 at 128K, 474 aggregate on 4 shorts, 271.8 coding peak." model: name: DeepSeek-V4.1-Flash-1017 family: deepseek-v4-1-flash-1017 @@ -49,13 +49,13 @@ kv_pool_tokens: 2185581 config: launch: - 'docker pull azallaza/ds41-kraken-tp3:20261019.3' - - 'docker run -d --name ds41-kraken-tp3 --restart no --stop-timeout 90 --ipc host' + - 'docker run -d --name ds41-kraken-tp3 --restart unless-stopped --stop-timeout 90 --ipc host' - '--security-opt seccomp=unconfined --ulimit memlock=-1 --ulimit nofile=1048576 --ulimit stack=67108864' - '--gpus ''"device=0,1,2"'' -p 15015:15015' - - '-v /path/to/huggingface:/root/.cache/huggingface:ro -v ds41-runtime-cache:/cache' + - '-v /path/to/huggingface:/root/.cache/huggingface:ro -v ds41-runtime-cache:/cache -e VLLM_CACHE_ROOT=/cache/vllm -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor -e TRITON_CACHE_DIR=/cache/triton' - '-e MODEL=deepseek-ai/DeepSeek-V4.1-Flash-1017 -e KV_CACHE_MEMORY_BYTES=2415919104' - - '-e DS41_OFFLOAD_FROM_ORDINAL=20 -e DS41_OFFLOAD_TO_ORDINAL=25' - - '-e B12X_DEFERRED_BANK_RESERVE_BYTES=1073741824' + - '-e DS41_OFFLOAD_FROM_ORDINAL=20 -e DS41_OFFLOAD_TO_ORDINAL=33' + - '-e B12X_DEFERRED_BANK_RESERVE_BYTES=1610612736' - '-e DS41_ENGRAM_RAM_LAYERS=1 -e DS41_ENGRAM_ROW_CACHE_ROWS=8388608' - '-e VLLM_DS41_DRAFT_NVFP4_HEAD=1 -e DS41_LAUNCH_SETTINGS=/etc/ds41/launch.yaml -e HF_HUB_OFFLINE=1' - 'azallaza/ds41-kraken-tp3:20261019.3' @@ -73,7 +73,7 @@ performance: metrics: tok_s: 186.7 context_len: 32768 - notes: 'claimed_by=submitter T0 (author bench-20261019.3/, fresh restart 2026-10-20, image 20261019.3: NVFP4 drafter head + top-p 1.0 server default). llm_decode_bench 0.7.8 + bench.py quick + prefill_real.py + bench_long.py needle on threadripper-aabduh (Threadripper PRO 7965WX, 256 GB, driver 615.71.09), 350 W/GPU, +6000 MiB offset. DSpark 5 drafts verified: tok/s = steps/s x accept_len (c1 accept 2.71-2.77 tok/step). context_len markers: 1024 = ~59-token short prompt, 4096 = coding prompt, 28553 = 28K needle prompt length, 143360 = 131072+12288 real-code prefill marker, 983040 = 983K needle nominal.' + notes: "claimed_by=submitter T0 (author bench-20261019.3/, fresh restart 2026-10-20, image 20261019.3: NVFP4 drafter head + top-p 1.0 server default). llm_decode_bench 0.7.8 + bench.py quick + prefill_real.py + bench_long.py needle on threadripper-aabduh (Threadripper PRO 7965WX, 256 GB, driver 615.71.09), 350 W/GPU, +6000 MiB offset. DSpark 5 drafts verified: tok/s = steps/s x accept_len (c1 accept 2.71-2.77 tok/step). context_len markers: 1024 = ~59-token short prompt, 4096 = coding prompt, 28553 = 28K needle prompt length, 143360 = 131072+12288 real-code prefill marker, 983040 = 983K needle nominal. UPDATE 2026-10-28 (repo @ 4fc4f0c9825d): config-only change on the same image 20261019.3 - DS41_OFFLOAD_TO 25 -> 33 (layers 20-33, 14 layers instead of 6), reserve 1 GiB -> 1.5 GiB, restart policy no -> unless-stopped, and the vLLM/Inductor/Triton compile caches persisted to /cache. The author publishes this as llm-bench decode 68.0 -> 72.3 ENGINE STEPS/S at 32K, a two-run mean with +/-1-2 steps run-to-run noise; that is a step-rate figure and is NOT converted to tok/s here, because tok/s = steps/s x draft acceptance and acceptance was not re-reported for this config. The tok/s points below remain the author's measured 20261019.3 fresh-restart figures and were not re-measured at the new offload range. Author's stated limits: do not widen past ~34 (20-34/20-36 OOM at load because DS41_VMM_COLD_FRACTION=1 host-maps every wrapped layer up front); keep dspark draft layers 37-39 resident; at the 1 GiB reserve 20-33 runs ~5 MiB free and throws repeated expandable_segments memory-mapping OOM retries under concurrent long-context, while 1.5 GiB holds ~480 MiB free under a 4x600K-token prefill-during-decode stress. Supersedes the earlier 'parking 20-29 instead of 20-25 - no further gain' note, which the author retracts because that was a tok/s A/B that cannot resolve ~6% through spec-decode acceptance noise." points: - context_len: 32768 @@ -170,4 +170,4 @@ provenance: - 'https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes' - 'https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017' - 'https://hub.docker.com/r/azallaza/ds41-kraken-tp3' - notes: "Community ingest of BoxTops4Coochie's recipe (repo folder 'DeepSeek-V4.1-Flash Kraken-tp3-20261019' @ bcf72fe6, self-contained: compose.yaml, ds41-launch.yaml, ds41-serve, Dockerfile, src/ TP3 port, bench/ + bench-20261019.3/, qualification/). Listed-by localai-recipes; recipe + measurements by BoxTops4Coochie. TP3 expert-parking design credited to infernix (ds41f-tp3 on karmic-kraken); runtime/kernels Local Inference Lab (vLLM fork, FlashInfer b12x); weights DeepSeek original 1017 @ 1bae0e10 (MIT). Discord corroboration (local-inference-lab #general, submitter-T0, not re-verified): Eric 2026-10-23 05:28Z \"been running tp3 with Box's recipe that runs on kraken. it stores unused experts in RAM. very nice so far\"; pete8359 04:39Z \"TP3 dsv4.1f works lossless with Box's recipe\"; andomort corroborated. Quality at top-p 1.0: lavd + hotel-lights 30/30 exact each @ 4 concurrent, 128K/900K retrieval pass, long generation 4/4 clean, vision 117/120 hard set. NOT a dupe of the site DS4.1 lanes (4x CMP 170HX Engram-NVMe TP4/PP4, 6x A100 SM80, EXL3 4x 170HX): first 3x RTX PRO 6000 Blackwell TP3 DS4.1-Flash recipe with full 1M context on original weights. Corrections to the scout card: image-inspect.json in the repo records the 20261019.2 image (sha256:06578acf...), not .3 - the engine_source digest is the 20261019.3 sha from README/Docker Hub. Facts re-verified at pinned HEAD 2026-10-24." + notes: "Community ingest of BoxTops4Coochie's recipe (repo folder 'DeepSeek-V4.1-Flash Kraken-tp3-20261019' @ bcf72fe6, self-contained: compose.yaml, ds41-launch.yaml, ds41-serve, Dockerfile, src/ TP3 port, bench/ + bench-20261019.3/, qualification/). Listed-by localai-recipes; recipe + measurements by BoxTops4Coochie. TP3 expert-parking design credited to infernix (ds41f-tp3 on karmic-kraken); runtime/kernels Local Inference Lab (vLLM fork, FlashInfer b12x); weights DeepSeek original 1017 @ 1bae0e10 (MIT). Discord corroboration (local-inference-lab #general, submitter-T0, not re-verified): Eric 2026-10-23 05:28Z \"been running tp3 with Box's recipe that runs on kraken. it stores unused experts in RAM. very nice so far\"; pete8359 04:39Z \"TP3 dsv4.1f works lossless with Box's recipe\"; andomort corroborated. Quality at top-p 1.0: lavd + hotel-lights 30/30 exact each @ 4 concurrent, 128K/900K retrieval pass, long generation 4/4 clean, vision 117/120 hard set. NOT a dupe of the site DS4.1 lanes (4x CMP 170HX Engram-NVMe TP4/PP4, 6x A100 SM80, EXL3 4x 170HX): first 3x RTX PRO 6000 Blackwell TP3 DS4.1-Flash recipe with full 1M context on original weights. Corrections to the scout card: image-inspect.json in the repo records the 20261019.2 image (sha256:06578acf...), not .3 - the engine_source digest is the 20261019.3 sha from README/Docker Hub. Facts re-verified at pinned HEAD 2026-10-24. UPDATE 2026-10-28 (repo @ 4fc4f0c9825d): config-only change on the same image 20261019.3 - DS41_OFFLOAD_TO 25 -> 33 (layers 20-33, 14 layers instead of 6), reserve 1 GiB -> 1.5 GiB, restart policy no -> unless-stopped, and the vLLM/Inductor/Triton compile caches persisted to /cache. The author publishes this as llm-bench decode 68.0 -> 72.3 ENGINE STEPS/S at 32K, a two-run mean with +/-1-2 steps run-to-run noise; that is a step-rate figure and is NOT converted to tok/s here, because tok/s = steps/s x draft acceptance and acceptance was not re-reported for this config. The tok/s points below remain the author's measured 20261019.3 fresh-restart figures and were not re-measured at the new offload range. Author's stated limits: do not widen past ~34 (20-34/20-36 OOM at load because DS41_VMM_COLD_FRACTION=1 host-maps every wrapped layer up front); keep dspark draft layers 37-39 resident; at the 1 GiB reserve 20-33 runs ~5 MiB free and throws repeated expandable_segments memory-mapping OOM retries under concurrent long-context, while 1.5 GiB holds ~480 MiB free under a 4x600K-token prefill-during-decode stress. Supersedes the earlier 'parking 20-29 instead of 20-25 - no further gain' note, which the author retracts because that was a tok/s A/B that cannot resolve ~6% through spec-decode acceptance noise."