DeepSeek-V4.1-Flash-1017 Engram-NVMe TP4/PP4 4x CMP 170HX

community Community submission — verify on your own hardware.

recipe by GNmac-lab / greencandle (recipe, patches, measurements)↗, CurtisAccelerate - TP4 hybrid-checkpoint recipe + tensor-core EXL3 MoE port↗, zebgop-ops - dsv41reap-pp sm_80 sparse attention, PP shadow KV, DSpark V1, Engram row store↗, vcruz305 - vllm-exl3 (AGPL-3.0-only)↗, LibertAI - EXL3 2.9-bpw quantization (MIT)↗, DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)↗, kaka86mm - dsv41-flash-pp4-170hx (watched sibling)↗, Morrowmake - CMP 170HX recipe format, P2P work, tensor-core MoE kernels↗, Mia-AiLab - decode benchmark protocol + prompts (used as data)↗, FlashInfer - PCIe IPC all-reduce↗ · listed by localai-recipes

revision r2 · commit 1b9a60d70c · updated 43 h ago · license MIT; patches/vllm-exl3 AGPL-3.0-only; vendored tc_moe kernels Apache-2.0 (built image AGPL)

chat code vision dspark exl3 speculative sm80 tp4 community-recipe engram-nvme pp4

DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards power-capped at 180 W, with the model's 189 GiB Engram n-gram tables streamed from NVMe on demand. Recipe 1.2.0 (author measurements, claimed_by=submitter T0): 355.7/258.4/183.6 tok/s one user (structured/code/prose), 867.6/544.0/440.5 across eight, 2,416 tok/s cold prefill (TP4; PP4 2,311), 1,395,731-token KV pool at 262,144 context (1,270,789 at 1M), HumanEval 154/164 (6 length-capped; McNemar p>=0.5 vs 156/164), GSM8K 1,275/1,319 (TP4). Hybrid checkpoint assembled without copying tensors: LibertAI EXL3 2.9-bpw routed experts (layers 0-39) + official FP8 attention/shared experts/head/DSpark drafter/Engram shards + BF16 embeddings/LM head/vision, fp8_ds_mla KV. Engine: vLLM on the community sm_80 stack (zebgop-ops sm80 sparse attention + DSpark V1 + Engram row store, FlashInfer PCIe IPC all-reduce) plus this recipe's measured keep-or-drop patches. TP4 (default, needs x16 + P2P) and PP4 (x1/x4, no P2P) from one .env switch.

Launch command

'docker run -d --name ds41-flash --gpus all --ipc=host --cap-add SYS_NICE --ulimit memlock=-1 --ulimit stack=67108864' '-v <MODELS_DIR>:/models:ro -v ./cache/root-cache:/root/.cache -v ./cache/triton:/root/.triton -v ./cache/tilelang:/root/.tilelang' '-p 127.0.0.1:8000:8000' '-e VLLM_PLUGINS=vllm_exl3 -e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e VLLM_EXL3_MODEL_DIR=/models/DeepSeek-V4.1-Flash-1017-hybrid' '-e VLLM_USE_V2_MODEL_RUNNER=0 -e DSV41_DSPARK_V1=1 -e VLLM_EXL3_COOP=1' '-e VLLM_EXL3_TC_MOE_V2=1 -e VLLM_EXL3_TC_MOE_PREFILL=1 -e VLLM_EXL3_TC_PREFILL_ROWS=2048' '-e VLLM_EXL3_TC_MOE_SO=/opt/dsv41-tc-moe/_dsv41_tc_moe_C.abi3.so -e VLLM_EXL3_TC_HOOK=/opt/dsv41-tc-moe/exl3_tc_hook.py' '-e VLLM_EXL3_MOE_TP_ALIGN=128 -e VLLM_EXL3_MOE_TP_ROTATE=1' '-e NCCL_CUMEM_ENABLE=0 -e NCCL_CUMEM_HOST_ENABLE=0 -e NCCL_P2P_LEVEL=SYS' '-e DSV41_ENGRAM_STORAGE=ssd -e DSV41_ENGRAM_DIR=/models/DeepSeek-V4.1-Flash-1017-engram' '-e DSV41_EXPERTS_ONLY_FILES_RE='\''^model-[0-9]+-of-000(43|39)\.safetensors$'\'' -e TORCH_CUDA_ARCH_LIST=8.0 -e TORCHINDUCTOR_CACHE_DIR=/root/.cache/inductor' '-e VLLM_DSV41_DECODE_SPLITK=1 -e DSV41_SHARED_HEAVY_BLOCKS=1 -e DSV41_DRAFT_HEAD_FP8=1 -e DSV41_PREFILL_CHUNK_REQS=1 -e DSV41_INDEXER_BUF_MULT=4 -e DSV41_DYNK=accept:3,5' '-e VLLM_ALLREDUCE_USE_FLASHINFER_PCIE_IPC=1' '--entrypoint vllm ds41-cmp170hx:latest' 'serve /models/DeepSeek-V4.1-Flash-1017-hybrid' '--quantization exl3 --load-format safetensors --safetensors-load-strategy lazy' '--tensor-parallel-size 4 --pipeline-parallel-size 1' '--served-model-name deepseek-v4.1-flash' '--host 0.0.0.0 --port 8000' '--max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.95' '--kv-cache-dtype fp8_ds_mla --enable-prefix-caching' '--compilation-config {"cudagraph_mode": "FULL_AND_PIECEWISE", "cudagraph_capture_sizes": [1, 2, 3, 4, 5, 6, 8, 12, 16, 18, 20, 24, 28, 30, 32, 36, 42, 48]}' --async-scheduling '--speculative-config {"method": "dspark", "num_speculative_tokens": 5}' '--tokenizer-mode deepseek_v41' '--enable-auto-tool-choice --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41'

env_defaults

dsv41_dynk: 'DSV41_DYNK=accept:3,5 is the tp4-only default set in serve.sh:56 (per-step DSpark verify width 3 or 5 from recent acceptance; empty = always 5 drafts). pp4 sets no default (off: k = num_speculative_tokens)'
dsv41_gate_tau: 'DSV41_GATE_TAU is never set by serve.sh; code default 0.10 (patches/dsv41_variants/variants/dsv41_gate.py:65 DEFAULT_TAU; 0 = off); forwarded via the OPT_ENV loop serve.sh:68-74 only when set'
dsv41_variants: 'tp4: unset in serve.sh - falls through to the module default of TWELVE variants (patches/dsv41_variants/dsv41_variants.py:14 DEFAULT: topk_meta,h2d_lean,ragged_lean,gate,ce_ring,split_reduce,moe_try,engram_prefetch,mhc_pcfg,moe_v2try,prefill_pack,attn_slices). pp4: serve.sh:64 pins the FOUR of 1.1.0 (dsv41_topk_meta,dsv41_h2d_lean,dsv41_ragged_lean,dsv41_gate) - pipeline parallel keeps the 1.1.0 list'
tp4_cudagraph_sizes: 'tp4 18 sizes [1,2,3,4,5,6,8,12,16,18,20,24,28,30,32,36,42,48] (serve.sh:54; 16/20/28/32 added for decode steps that verify 3 drafts per request via DSV41_DYNK); pp4 keeps the 1.1.0 14 sizes with cudagraph_mode PIECEWISE'
dsv41_draft_head_fp8: 'DSV41_DRAFT_HEAD_FP8=1 in serve.sh COMMON_ENV (serve.sh:46); applies to tp4 AND pp4 (layout-independent); code default 0 (opt-in), consumed by patches/patch_v38.py:76'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx.json

Engine

vllm · source↗ · 2ad5741a↗

Install note: One command from a fresh clone: git clone https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx && cd deepseek-v41-flash-1017-cmp170hx && cp .env.example .env && tools/download.sh && ./start.sh - builds the image on first use (~30-45 min), starts, waits until ready; ./start.sh smoke = one chat + one tool call + one image. Model parts + hybrid assembly are resumable. Host: 4x CMP 170HX unlocked to 64 GiB / 74 SMs; TP4 needs PCIe x16 + P2P (tools/p2p_check.sh verifies; LAYOUT=pp4 for x1/x4); NVIDIA driver with CUDA 13 (measured 610.43.02); Docker + NVIDIA Container Toolkit; ~400 GiB fast NVMe for checkpoints + ~40 GiB image; 64 GB+ host RAM (spare RAM becomes page cache for Engram rows). Optional 180 W power cap per card used for every published number. 1.2.0 @ 2ad5741ab1: Dockerfile patch steps 21->34 (dsv41_fp8w/woa ext, dsv41_variants, tc_moe_try builds); tc_moe kernels vendored Apache-2.0 (Morrowmake a0c57b5). Env knobs: config.env_defaults.

Model

DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw-Hybrid (exl3-2-9bpw-routed-experts-fp8-core-hybrid) · LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw↗
AA intelligence 45 (Artificial Analysis · measured)

Hardware

NVIDIA CMP 170HX 64GB ×4 (Ampere)

Context

max context: 256K tokens

KV pool

1,395,731 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
10,240	—	2416 tok/s	—	—	reported
20,480	—	2311 tok/s	—	—	reported
262,144	1	—	355.7 tok/s	—	reported
262,144	8	—	867.6 tok/s	—	reported
272,384	1	—	258.4 tok/s	—	reported
272,384	8	—	544 tok/s	—	reported
282,624	1	—	183.6 tok/s	—	reported
282,624	8	—	440.5 tok/s	—	reported
303,104	1	—	117.5 tok/s	—	reported
303,104	8	—	340.5 tok/s	—	reported
313,344	1	—	90.5 tok/s	—	reported
313,344	8	—	213.9 tok/s	—	reported
323,584	1	—	63.2 tok/s	—	reported
323,584	8	—	163.1 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	claimed_by=submitter T0 (author's bench; recipe 1.2.0, MiaAI-Lab protocol via bench/mia.py unmodified, measured 2026-10-26 on 4x CMP 170HX: 74 SMs/card, PCIe 2.0 x16, P2P on, 180 W/card, Engram on NVMe, 262,144-token context, 8 sequences, .env defaults — docs/results.md:3-5 @ 2ad5741). c1 = median of 5 streaming runs; c2/c4/c8 = median of 3 aggregate runs (results.md:11-15). 1.2.0 TP4: c1 355.7/258.4/183.6, c2 433.1/243.8/238.8, c4 626.4/387.8/348.7, c8 867.6/544.0/440.5 (results.md:19-22). 1.2.0 PP4: c1 117.5/90.5/63.2, c2 118.4/83.4/65.2, c4 205.9/142.9/105.1, c8 340.5/213.9/163.1 (results.md:19-22). Previously published 1.0.0 comparators: TP4 296/212/150 c1 and 796/477/359 c8; PP4 111/81/52 c1 and 320/188/144 c8. Cold prefill TP4 2,416 (1.1.0 1,764; +37.4 % vs the published 1,759); PP4 2,311 (1.1.0 2,457; -2.7 % vs the published 2,375, inside the +/-3-7 % boot spread; results.md:41-43, 51-52). KV pool at 262,144: TP4 1,395,731 tokens = 2.95 GiB/card, 5.3 full-length requests (results.md:44; docs/kv-and-context.md:13-14); at 1,048,576: 1,270,789 tokens = 2.27 GiB, 1.21 (results.md:45; kv-and-context.md:13). DSpark drafts 5 tokens/step; TP4 verify width DSV41_DYNK accept:3,5 (prose A/B: 2 users 232->248, 4 users 306/316->349, 8 users 420/419->445; results.md:31-34); confidence gate default tau 0.10 (prose c1 174.7 ungated vs 179.9 gated; results.md:35-37). 1.2.0 DynK/variant gains are described in results.md:31-37 prose; docs/tuning-log.md Kept table last row remains the 1.1.0 confidence gate (tuning-log.md:32); 1.2.0 tried-and-dropped incl. the 4,096-token prefill chunk (~+1 %, cold bench OOM; tuning-log.md:91) and moe_prefill_v2 (slower; tuning-log.md:95) are in the 'Tried and dropped for 1.2.0' section (tuning-log.md:86-99). Quality: HumanEval 154/164 with 6 length-capped (regression vs the published 156/164; author McNemar p >= 0.5 — run-to-run spread; results.md:60-67), GSM8K 1,275/1,319 (results.md:63); paired NLL +0.57 % (1.7 sigma) vs 1.1.0 (results.md:69-71); needle 6/6 at 19K/72K, 3/3 at 388K 1M option (results.md:80-81). Soak 20 min x 8 streams: 1,018 requests, 403,570 tokens, 0 errors, 334.8 tok/s mean (results.md:85-86). Nobody has booted 1.2.0 in this fleet — every number is the author's own measurement. Comparators: sibling EXL3/TC-MoE recipe measured 180.4/137.1/96.6 c1 and 621.1/376.1/282.5 c8 at 140 W — NOT power-matched (180 W caps + Engram-NVMe tier).

notes

claimed_by=submitter T0 (author's bench; recipe 1.2.0, MiaAI-Lab protocol via bench/mia.py unmodified, measured 2026-10-26 on 4x CMP 170HX: 74 SMs/card, PCIe 2.0 x16, P2P on, 180 W/card, Engram on NVMe, 262,144-token context, 8 sequences, .env defaults — docs/results.md:3-5 @ 2ad5741). c1 = median of 5 streaming runs; c2/c4/c8 = median of 3 aggregate runs (results.md:11-15). 1.2.0 TP4: c1 355.7/258.4/183.6, c2 433.1/243.8/238.8, c4 626.4/387.8/348.7, c8 867.6/544.0/440.5 (results.md:19-22). 1.2.0 PP4: c1 117.5/90.5/63.2, c2 118.4/83.4/65.2, c4 205.9/142.9/105.1, c8 340.5/213.9/163.1 (results.md:19-22). Previously published 1.0.0 comparators: TP4 296/212/150 c1 and 796/477/359 c8; PP4 111/81/52 c1 and 320/188/144 c8. Cold prefill TP4 2,416 (1.1.0 1,764; +37.4 % vs the published 1,759); PP4 2,311 (1.1.0 2,457; -2.7 % vs the published 2,375, inside the +/-3-7 % boot spread; results.md:41-43, 51-52). KV pool at 262,144: TP4 1,395,731 tokens = 2.95 GiB/card, 5.3 full-length requests (results.md:44; docs/kv-and-context.md:13-14); at 1,048,576: 1,270,789 tokens = 2.27 GiB, 1.21 (results.md:45; kv-and-context.md:13). DSpark drafts 5 tokens/step; TP4 verify width DSV41_DYNK accept:3,5 (prose A/B: 2 users 232->248, 4 users 306/316->349, 8 users 420/419->445; results.md:31-34); confidence gate default tau 0.10 (prose c1 174.7 ungated vs 179.9 gated; results.md:35-37). 1.2.0 DynK/variant gains are described in results.md:31-37 prose; docs/tuning-log.md Kept table last row remains the 1.1.0 confidence gate (tuning-log.md:32); 1.2.0 tried-and-dropped incl. the 4,096-token prefill chunk (~+1 %, cold bench OOM; tuning-log.md:91) and moe_prefill_v2 (slower; tuning-log.md:95) are in the 'Tried and dropped for 1.2.0' section (tuning-log.md:86-99). Quality: HumanEval 154/164 with 6 length-capped (regression vs the published 156/164; author McNemar p >= 0.5 — run-to-run spread; results.md:60-67), GSM8K 1,275/1,319 (results.md:63); paired NLL +0.57 % (1.7 sigma) vs 1.1.0 (results.md:69-71); needle 6/6 at 19K/72K, 3/3 at 388K 1M option (results.md:80-81). Soak 20 min x 8 streams: 1,018 requests, 403,570 tokens, 0 errors, 334.8 tok/s mean (results.md:85-86). Nobody has booted 1.2.0 in this fleet — every number is the author's own measurement. Comparators: sibling EXL3/TC-MoE recipe measured 180.4/137.1/96.6 c1 and 621.1/376.1/282.5 c8 at 140 W — NOT power-matched (180 W caps + Engram-NVMe tier).

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
official FP8 attention/shared experts/head/DSpark drafter (incl. FP4 drafter experts)/Engram shards 47-48; bf16 embeddings + LM head + vision tower
Weights https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw
routed experts layers 0-39 ONLY (LibertAI 2.9 bpw avg; layers 18-22 at 2 bpw); repo watched @ 0be63c3337efbb2a0061d5b8bb4006fa186d433f
Engine image https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx
repo HEAD 2ad5741ab1 (author recipe 1.2.0) at sweep 2026-10-28 (was bbaf367291 at ingest sweep 2026-10-22); image built locally on first use (~30-45 min) from pinned upstream sources via patches/ (34 patch steps)
Reference https://github.com/zebgop-ops/dsv41reap-pp
sm_80 sparse attention + PP shadow KV + DSpark V1 + Engram row store drop-ins (@ 138e7488, watched)
Reference https://github.com/vcruz305/vllm-exl3
EXL3 plugin (AGPL-3.0-only) this recipe's patches modify
Reference https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE
credited for the TP4 hybrid-checkpoint recipe + tensor-core EXL3 MoE port lineage (watched @ e559217ac4)

Revision notes

Community ingest of GNmac-lab's recipe (MIT; patches/vllm-exl3 AGPL-3.0-only as it modifies vcruz305/vllm-exl3; built image contains AGPL-3.0 code, see NOTICE.md). All perf numbers are the author's own measurements (claimed_by=submitter T0; docs/results.md + docs/tuning-log.md include what did NOT help). Hardware note: 180 W/card power caps, 74-SM unlock, PCIe 2.0 x16 with P2P; PP4 lane documented for x1/x4 non-P2P rigs. Announced cmpunlocker #clanker-talk 2026-10-21 13:59Z by greencandle, cross-posted #development-general 14:06Z; Discord numbers match README. NOT a dupe of the sibling deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4: same hybrid checkpoint approach but a different engine stack (zebgop-ops dsv41reap-pp drop-ins + FlashInfer PCIe IPC all-reduce vs Morrowmake host all-reduce + arena cache), its own measured tables and quality scores, and the site's only 1017 PP4 row. Perf-point context_len markers: TP4 rows base 262144 (+10240 code, +20480 prose; PP4 base +40960); prefill rows use markers 10240 (TP4) / 20480 (PP4) to avoid (ctx,conc,claimed_by) dedup collisions - cold prefill was measured on ~24-38K-token prompts, median of 9. Also credits turboderp (exllamav3 + EXL3 codec) and HalfVulpes (Deepinfer-SM80, credited upstream). Engine watch on ingest: GNmac-lab/deepseek-v41-flash-1017-cmp170hx @ bbaf367291 (pin at ingest; repo created 2026-10-21, expect movement); zebgop-ops/dsv41reap-pp already watched @ 138e7488; LibertAIDAI EXL3 pack already watched @ 0be63c33; deepseek-ai/DeepSeek-V4.1-Flash-1017 already watched @ 1bae0e10. Facts re-verified at pinned HEAD 2026-10-22 - no corrections to the scout card. 1.2.0 PUBLISH (kanban t_d9bb5ffd, owner go 2026-10-28): author release 1.2.0 @ 2ad5741ab1 (both commits since the pin: 45ced96 2026-10-23 = 1.1.0, 2ad5741 2026-10-26 = 1.2.0; the 1.0.0 pin bbaf367291 is superseded). Engine pins added since the 1.0.0 pin: at 1.1.0 - dsv41_fp8_gemv.py, dsv41_fp8w/, dsv41_markov.py, dsv41_woa/, 14 patch_*.py appliers incl. patch_v38 (DSV41_DRAFT_HEAD_FP8), patches/vllm-exl3/, patches/deepinfer_sm80_*; at 1.2.0 - dsv41_dynk.py + patch_dynk.py, eight new variant modules (attn_slices, ce_ring, engram_prefetch, mhc_pcfg, moe_try, moe_v2try, prefill_pack, split_reduce), tc_moe_try/ + tc_moe_v2try/ vendored Apache-2.0 tensor-core kernels. No upstream dependency SHA moved; only the Dockerfile patch ordering grew (21 -> 34 lines). Quality note (docs/results.md:62 @ 2ad5741ab1): HumanEval 154/164 (6 length-capped) is a DOWNGRADE vs the published 156/164; GSM8K unchanged 1,275/1,319. All numbers remain author-measured T0 - no fleet re-bench.

Revision history

1b9a60d70c recipe: update deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx (r2: prod sync of QC-approved preprod r5 a2cd9530 / content_hash dadcb54c — engine pin 1.0.0 bbaf367291 -> 1.2.0 2ad5741ab1, 37 h ago diff
3d76177099 recipe: create deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx 9 d ago

Fork

Export

recipe.yaml (raw)