localairecipes launch configs for local inference

DeepSeek-V4.1-Flash-1017 EXL3 3bpw 2x RTX PRO 6000 Engram on SSD

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by yoloyash (recipe author: SSD Engram overlay, native row gather, SSD plans, measurements, MIT)↗, diffbot (pack, image, kernels, serve.sh, model-card numbers: diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000), deepseek-ai/DeepSeek-V4.1-Flash-1017 - upstream weights · listed by localai-recipes

Initial version · updated 16 h ago · license MIT

chat code vision 1m-ctx dspark long-context exl3 speculative expert-offload community-recipe tp2 engram-nvme

DeepSeek-V4.1-Flash-1017 (diffbot's EXL3 3bpw pack) on 2x RTX PRO 6000 96 GB with only 128 GB of host RAM instead of the ~256 GB diffbot's own recipe pins. No fork, no rebuild: five vLLM DeepSeek-V4.1 files (engram, engram_ssd, engram_gather.cpp, model, model_state) are bind-mounted read-only over diffbot's image so the two int4 Engram n-gram tables (~111 GB) stay in the pack's NVMe shards and are read on demand through a native row-gather thread pool, with prefetch during prefill, per-layer deferred reads, resident scales and a 736-slot host-expert cache per GPU; rows are bit-identical to the pinned-RAM path in parity mode. 1M context, vision, tool calls and DSpark k=5 preserved; host RAM drops from ~230 GiB pinned to ~82 GiB plus page cache. Author-measured: 162.7-163.3 tok/s 2K decode, 298.7-315.9 aggregate at 4x46K streams, prefill 7,263-7,457 never-seen text, GSM8K-200 98.5%, 1.04M needle in 111.0 s, 97.0/97.0 GB VRAM. recipe by yoloyash; pack/kernels/serve.sh by diffbot.

Launch command

PACK=/models/dsv41-1017 API_KEY=change-me bash serve/serve-ssd.sh

env

- PACK=/models/dsv41-1017
- ENGRAM_SSD=1
- ENGRAM_SSD_PREFETCH=16384
- ENGRAM_SSD_RESIDENT_SCALES=1
- ENGRAM_SSD_EVICT=1
- ENGRAM_SSD_DEFER=1
- ENGRAM_SSD_NATIVE=1
- ENGRAM_SSD_THREADS=128
- ENGRAM_SSD_MIN_CHUNK=8
- ENGRAM_SSD_INLINE_ROWS=0
- ENGRAM_SSD_LOG_EVERY=500
- ENGRAM_SSD_CHECK=0
- CONTAINER_MEM=120g
- HOST_CACHE=736
- SSD_PLAN=plan-1017-agent30-c512-p375.json
- CACHE_METRICS=1

flags

"serve/serve-ssd.sh runs diffbot's $PACK/recipe/serve/serve.sh UNCHANGED (every serve.sh knob still works: API_KEY, PORT, SPEC, MAX_NUM_SEQS, EAGER ...) with EXTRA_ARGS='--enable-prompt-tokens-details' by default and these bind mounts / env added to the container: -v overlay/<f>:$VP/<f>:ro for common/engram.py, common/engram_ssd.py, common/engram_gather.cpp, nvidia/model.py, nvidia/model_state.py (VP=/usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v41); -v $ROOT/plans:/opt/ssd-plans:ro; -e EXL3_HOST_EXPERTS=/opt/ssd-plans/<SSD_PLAN>; -e DSV41_ENGRAM_SSD* (the knobs above). DSpark k=5 comes from diffbot's serve.sh default; SSD_PLAN= (empty) reverts to diffbot's own host-expert plan."

notes

"ENGRAM_SSD=0 reverts to diffbot's pinned-RAM path, which needs ~230 GiB RAM. CONTAINER_MEM caps the container at 120g instead of diffbot's 260g - page cache counts against the cap, so it also caps the Engram cache; raise it only with RAM to match. HOST_CACHE=736 is sized for cards nothing else uses; with a display or another process on them use HOST_CACHE=512 (~1.4 GiB/GPU less). Right after a boot decode is slower until the page cache warms (80.8 / 114.9 / 218.8 tok/s on an earlier version). DSpark off falls to 84.3 / 87.3 / 205.2 tok/s - leave it on. Batched greedy decoding is not bit-deterministic (HumanEval flips 1-2 problems between runs)."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-3bpw-2x-rtx-pro-6000-engram-on-ssd.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-3bpw-2x-rtx-pro-6000-engram-on-ssd.json

Engine

Install note: Needs 2x RTX PRO 6000 (96 GB, sm_120) with nothing else on them (or set HOST_CACHE=512), NVIDIA driver with CUDA >= 13, Docker with the NVIDIA runtime, ~128 GB RAM and ~330 GB on fast NVMe. Steps: hf download the diffbot pack to /models/dsv41-1017 (PACK env), build diffbot's image and kernels from $PACK/recipe (./docker/build.sh && ./kernels/build.sh, 30-60 min), clone this repo, then serve/serve-ssd.sh. RECIPE defaults to $PACK/recipe so the vendored diffbot serve.sh is used unchanged; overlay/ files are bind-mounted over /usr/local/lib/python3.12/dist-packages/vllm/models/deepseek_v41 (five files: engram.py, engram_ssd.py, engram_gather.cpp, model.py, model_state.py). Parity mode (ENGRAM_SSD_CHECK=1) re-reads sampled rows with pread and compares bit for bit every step. Image is built locally by diffbot's docker/build.sh (IMAGE=dsv41-flash-1017-exl3-3bpw-sm120), not pulled.

Model

Hardware

Context

  • max context: 1M tokens

KV pool

  • 1,071,558 tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
2,0481—163 tok/s—reported
40,960—7360 tok/s——reported
46,0801—167.3 tok/s—reported
46,0804—307.3 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured, warm page cache, Threadripper PRO 5955WX host (Zen 3, no AVX-512, diffbot's GPU-only path), 8x16 GB DDR4-2933, 2x RTX PRO 6000 Workstation 600 W on PCIe 4.0 x16, Engram shards on a WD_BLACK SN7100 4 TB (~195K random 4 KiB reads/s across the tables' 110 GB, fio), Linux 7.0. Decode DSpark k=5 via diffbot's bench-fixed.py, two runs with a restart per run: 1 stream 2K 162.7-163.3 (diffbot 156.9), 1 stream 46K 165.9-168.7 (172.8), 4 streams 46K aggregate 298.7-315.9 (362.9). Prefill ~40K-token prompt, cold KV: never-seen text 7,263-7,457 (diffbot 11,157-11,467); 9,394 when Engram rows are already cached. Quality: GSM8K-200 thinking-off 98.5% (equal to diffbot), HumanEval pass@1 96.3-97.6% (97.0%). ~1.04M-token needle found in 111.0 s at 97.0/97.0 GB peak VRAM. DSpark off falls to 84.3/87.3/205.2 tok/s - leave it on. Point values are run midpoints; raw output for every number: results/raw-logs.txt in the repo at the pinned commit.

Install requirements

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Downloads

Revision notes

Initial version.

Revision history

Initial version.

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)