{"slug":"deepseek-v4-1-flash-1017-exl3-3bpw-2x-rtx-pro-6000-engram-on-ssd","title":"DeepSeek-V4.1-Flash-1017 EXL3 3bpw 2x RTX PRO 6000 Engram on SSD","summary":"DeepSeek-V4.1-Flash-1017 (diffbot's EXL3 3bpw pack) on 2x RTX PRO 6000 96 GB with only 128 GB of host RAM instead of the ~256 GB diffbot's own recipe pins. No fork, no rebuild: five vLLM DeepSeek-V4.1 files (engram, engram_ssd, engram_gather.cpp, model, model_state) are bind-mounted read-only over diffbot's image so the two int4 Engram n-gram tables (~111 GB) stay in the pack's NVMe shards and are read on demand through a native row-gather thread pool, with prefetch during prefill, per-layer deferred reads, resident scales and a 736-slot host-expert cache per GPU; rows are bit-identical to the pinned-RAM path in parity mode. 1M context, vision, tool calls and DSpark k=5 preserved; host RAM drops from ~230 GiB pinned to ~82 GiB plus page cache. Author-measured: 162.7-163.3 tok/s 2K decode, 298.7-315.9 aggregate at 4x46K streams, prefill 7,263-7,457 never-seen text, GSM8K-200 98.5%, 1.04M needle in 111.0 s, 97.0/97.0 GB VRAM. recipe by yoloyash; pack/kernels/serve.sh by diffbot.","license":"MIT","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"68888a175cbffbc4afcbe12badc1bbe668e6c931","content_hash":"d6cc0b03481cfbc1c8ed12b595a348ba231ed88d","created_at":"2026-11-04 22:24:27.932509-07","notes":"Filed from the lar-scout 2026-11-05 sweep; surfaced in local-inference-lab Discord general (2026-11-04 03:05, author 'King'), verified live. Repo @ ca6491e08314f2fc926290a2bd3b05ee3e27d52a (HEAD, pushed 2026-11-02T00:57:34Z): MIT (LICENSE 'Copyright (c) 2026 Yash Khurana'); the three modified vLLM files in overlay/ stay Apache-2.0 (NOTICE). Verified at the pin: serve/serve-ssd.sh bind-mounts FIVE overlay files (card said four: 4 Python + 1 C++ engram_gather.cpp) read-only over the image's vllm/models/deepseek_v41 and execs diffbot's $PACK/recipe/serve/serve.sh unchanged; plans/plan-1017-agent30-c512-p375.json present; every README number traces to results/raw-logs.txt (162.7/163.3, 165.9/168.7, 298.7/315.9, 7263/7457, 9394, 98.5%, 96.3-97.6%, needle 111.0 s found=True). diffbot pack @ 451f7dcb (lastModified 2026-10-25, 2372 downloads / 35 likes) verified; deepseek-ai upstream @ 1bae0e10 verified as main HEAD release commit. Corrections to card: card's hardware vram_gb=192 corrected to 96 per card (2x96; site profile nvidia-rtx-pro-6000-blackwell-max-qx2 carries vram_gb 96, the profile the site's other 2x-6000 recipes use); README states 600 W Workstation cards, Max-Q is the site's seed wording; card's artifacts pointer_type 'other' submitted as 'hf_model'. Dedupe: corpus grep for yoloyash/dsv41-flash-1017-2x/ssd-engram returned 0 recipes; the site's 2x-6000 rows are other models (ds4-flash-2x6000-jj-dspark-k5, ds4-vision-2x6000-jj-dspark-k3); DS41 rows are 4x/3x/1x other boards. First time in pipeline."},"engine_source":{"type":"docker_image","git_url":"https://github.com/yoloyash/dsv41-flash-1017-2x-rtx-pro-6000-ssd-engram","base_engine":"vllm","pip_package":null,"base_version":null,"image_digest":null,"install_note":"Needs 2x RTX PRO 6000 (96 GB, sm_120) with nothing else on them (or set HOST_CACHE=512), NVIDIA driver with CUDA >= 13, Docker with the NVIDIA runtime, ~128 GB RAM and ~330 GB on fast NVMe. Steps: hf download the diffbot pack to /models/dsv41-1017 (PACK env), build diffbot's image and kernels from $PACK/recipe (./docker/build.sh && ./kernels/build.sh, 30-60 min), clone this repo, then serve/serve-ssd.sh. RECIPE defaults to $PACK/recipe so the vendored diffbot serve.sh is used unchanged; overlay/ files are bind-mounted over /usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v41 (five files: engram.py, engram_ssd.py, engram_gather.cpp, model.py, model_state.py). Parity mode (ENGRAM_SSD_CHECK=1) re-reads sampled rows with pread and compares bit for bit every step. Image is built locally by diffbot's docker/build.sh (IMAGE=dsv41-flash-1017-exl3-3bpw-sm120), not pulled.","pinned_commit":"ca6491e08314f2fc926290a2bd3b05ee3e27d52a"},"credit":[{"name":"yoloyash (recipe author: SSD Engram overlay, native row gather, SSD plans, measurements, MIT)","url":"https://github.com/yoloyash"},{"name":"diffbot (pack, image, kernels, serve.sh, model-card numbers: diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000)"},{"name":"deepseek-ai/DeepSeek-V4.1-Flash-1017 - upstream weights"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"env":["PACK=/models/dsv41-1017","ENGRAM_SSD=1","ENGRAM_SSD_PREFETCH=16384","ENGRAM_SSD_RESIDENT_SCALES=1","ENGRAM_SSD_EVICT=1","ENGRAM_SSD_DEFER=1","ENGRAM_SSD_NATIVE=1","ENGRAM_SSD_THREADS=128","ENGRAM_SSD_MIN_CHUNK=8","ENGRAM_SSD_INLINE_ROWS=0","ENGRAM_SSD_LOG_EVERY=500","ENGRAM_SSD_CHECK=0","CONTAINER_MEM=120g","HOST_CACHE=736","SSD_PLAN=plan-1017-agent30-c512-p375.json","CACHE_METRICS=1"],"flags":"serve/serve-ssd.sh runs diffbot's $PACK/recipe/serve/serve.sh UNCHANGED (every serve.sh knob still works: API_KEY, PORT, SPEC, MAX_NUM_SEQS, EAGER ...) with EXTRA_ARGS='--enable-prompt-tokens-details' by default and these bind mounts / env added to the container: -v overlay/<f>:$VP/<f>:ro for common/engram.py, common/engram_ssd.py, common/engram_gather.cpp, nvidia/model.py, nvidia/model_state.py (VP=/usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v41); -v $ROOT/plans:/opt/ssd-plans:ro; -e EXL3_HOST_EXPERTS=/opt/ssd-plans/<SSD_PLAN>; -e DSV41_ENGRAM_SSD* (the knobs above). DSpark k=5 comes from diffbot's serve.sh default; SSD_PLAN= (empty) reverts to diffbot's own host-expert plan.","notes":"ENGRAM_SSD=0 reverts to diffbot's pinned-RAM path, which needs ~230 GiB RAM. CONTAINER_MEM caps the container at 120g instead of diffbot's 260g - page cache counts against the cap, so it also caps the Engram cache; raise it only with RAM to match. HOST_CACHE=736 is sized for cards nothing else uses; with a display or another process on them use HOST_CACHE=512 (~1.4 GiB/GPU less). Right after a boot decode is slower until the page cache warms (80.8 / 114.9 / 218.8 tok/s on an earlier version). DSpark off falls to 84.3 / 87.3 / 205.2 tok/s - leave it on. Batched greedy decoding is not bit-deterministic (HumanEval flips 1-2 problems between runs).","launch":["PACK=/models/dsv41-1017","API_KEY=change-me","bash","serve/serve-ssd.sh"],"kv_pool_tokens":1071558,"max_context_len":1048576,"storage_requirements":{"note":"~128 GB host RAM (container capped at 120g, page cache holds Engram rows) and ~330 GB on fast NVMe for the pack incl. the ~111 GB int4 Engram shards.","ram_gb":128,"nvme_gb":330}},"models":[{"slug":"deepseek-v4-1-flash-1017-exl3-3bpw","name":"DeepSeek-V4.1-Flash-1017-EXL3-3bpw","hf_repo":"diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000","quant":"exl3-3bpw"}],"hardware":[{"slug":"nvidia-rtx-pro-6000-blackwell-max-qx2","vendor":"nvidia","card":"RTX PRO 6000 Blackwell Max-Q","card_family":"rtx-pro-6000-blackwell","vram_gb":96,"device_count":2}],"generation":"blackwell","tags":["1m-ctx","community-recipe","dspark","engram-nvme","exl3","expert-offload","long-context","speculative","tp2"],"tasks":["chat","code","vision"],"aa":{"aa_index":45,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash-1017","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-19 12:00:00-06","aa_family_slug":"deepseek-v4-1-flash-1017","aa_resolved_slug":"deepseek-v4-1-flash-1017"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured, warm page cache, Threadripper PRO 5955WX host (Zen 3, no AVX-512, diffbot's GPU-only path), 8x16 GB DDR4-2933, 2x RTX PRO 6000 Workstation 600 W on PCIe 4.0 x16, Engram shards on a WD_BLACK SN7100 4 TB (~195K random 4 KiB reads/s across the tables' 110 GB, fio), Linux 7.0. Decode DSpark k=5 via diffbot's bench-fixed.py, two runs with a restart per run: 1 stream 2K 162.7-163.3 (diffbot 156.9), 1 stream 46K 165.9-168.7 (172.8), 4 streams 46K aggregate 298.7-315.9 (362.9). Prefill ~40K-token prompt, cold KV: never-seen text 7,263-7,457 (diffbot 11,157-11,467); 9,394 when Engram rows are already cached. Quality: GSM8K-200 thinking-off 98.5% (equal to diffbot), HumanEval pass@1 96.3-97.6% (97.0%). ~1.04M-token needle found in 111.0 s at 97.0/97.0 GB peak VRAM. DSpark off falls to 84.3/87.3/205.2 tok/s - leave it on. Point values are run midpoints; raw output for every number: results/raw-logs.txt in the repo at the pinned commit.\", \"tok_s\": 163, \"points\": [{\"notes\": \"DSpark k=5, diffbot bench-fixed.py, warm page cache; author-measured 162.7-163.3 across two runs (restart per run); diffbot pinned-RAM path 156.9.\", \"concurrency\": 1, \"context_len\": 2048, \"decode_tok_s\": 163}, {\"notes\": \"single 46K-prompt stream; author-measured 165.9-168.7 across two runs; diffbot pinned-RAM 172.8.\", \"concurrency\": 1, \"context_len\": 46080, \"decode_tok_s\": 167.30000000000001136868377216160297393798828125}, {\"notes\": \"4 streams x 46K prompts, aggregate tok/s; author-measured 298.7-315.9 across two runs; diffbot pinned-RAM 362.9 (their host is PCIe 5.0 / DDR5).\", \"concurrency\": 4, \"context_len\": 46080, \"decode_tok_s\": 307.30000000000001136868377216160297393798828125}, {\"notes\": \"~40K-token prompt, nothing in KV cache, median of 8 never-seen Python-source slices (bench/prefill_ab.py), two runs: 7,263-7,457 (diffbot 11,157-11,467); 9,394 when Engram rows are already cached (value in notes, not a separate point).\", \"context_len\": 40960, \"prefill_tok_s\": 7360}], \"context_len\": 2048}"}],"perf_points":[{"id":2443,"context_len":2048,"prefill_tok_s":null,"decode_tok_s":163,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"DSpark k=5, diffbot bench-fixed.py, warm page cache; author-measured 162.7-163.3 across two runs (restart per run); diffbot pinned-RAM path 156.9."},{"id":2446,"context_len":40960,"prefill_tok_s":7360,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"~40K-token prompt, nothing in KV cache, median of 8 never-seen Python-source slices (bench/prefill_ab.py), two runs: 7,263-7,457 (diffbot 11,157-11,467); 9,394 when Engram rows are already cached (value in notes, not a separate point)."},{"id":2444,"context_len":46080,"prefill_tok_s":null,"decode_tok_s":167.30000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single 46K-prompt stream; author-measured 165.9-168.7 across two runs; diffbot pinned-RAM 172.8."},{"id":2445,"context_len":46080,"prefill_tok_s":null,"decode_tok_s":307.30000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"4 streams x 46K prompts, aggregate tok/s; author-measured 298.7-315.9 across two runs; diffbot pinned-RAM 362.9 (their host is PCIe 5.0 / DDR5)."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-3bpw-2x-rtx-pro-6000-engram-on-ssd.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-3bpw-2x-rtx-pro-6000-engram-on-ssd.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/yoloyash/dsv41-flash-1017-2x-rtx-pro-6000-ssd-engram","pin":"ca6491e08314f2fc926290a2bd3b05ee3e27d52a"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000","pin":"451f7dcb8ecb0502a963f3314e16b34c5a8ce5ae"},{"kind":"reference","pointer_type":"hf_model","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017","pin":"1bae0e1088993b9b82de08f59de5be5a34e42170"}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-3bpw-2x-rtx-pro-6000-engram-on-ssd.json","alternate":"/r/deepseek-v4-1-flash-1017-exl3-3bpw-2x-rtx-pro-6000-engram-on-ssd","describedby":"/openapi.json"},"updated_at":"2026-11-04 22:24:27.932509-07","created_at":"2026-11-04 22:24:27.932509-07","speeds":{"decode_tok_s":165.19999999999998863131622783839702606201171875,"prefill_tok_s":7360,"leaderboard":"/api/v1/leaderboard.json"}}