DeepSeek-V4.1-Flash (764B) on 4x CMP 170HX — PP4, shadow KV, EXL3 2bpw, DSpark K5, 512k ctx, native vision (kaka86mm)

community Community submission — verify on your own hardware.

recipe by kaka86mm↗, zebgop-ops (dsv41reap-pp shadow KV overlay)↗, vcruz305 (vllm-exl3 plugin)↗, turboderp-org (exllamav3)↗, wtdcode (vllm-backport)↗ · listed by localai-recipes

revision r2 · commit 363a35de53 · updated 10 d ago · license CC-BY-4.0

chat code vision pipeline-parallel dspark long-context engram exl3 shadow-kv speculative

Four-card Ampere recipe for full DeepSeek-V4.1-Flash (764B, native vision) on 4x CMP 170HX (sm80, Gen2 x4, no P2P — 256 GB VRAM can't hold 510 GB, so: EXL3 2bpw + fp8 KV): PP4 x TP1 partition 10,10,11,9 (rank3 = 9 layers + drafter), shadow KV via dsv41reap-pp overlay (54 files), EXL3 2bpw pack — engram tables kept FP8 at 203 GB, pinned 189 GiB host RAM (quantizing destroys retrieval), DSpark n=5 (conf=0; truncation is -30% here; ngram is a 7.6 tok/s disaster), 524k ctx, fp8_ds_mla KV, 5.9M-token pool, native ViT, GLOO_SOCKET_IFNAME=lo — the 09-28 one-line fix for PP gloo via docker bridge (535 ms/send) that collapsed long-context decode. Measured by the author (09-28 v37): decode 60.7 @32K / 58.3 @128K / 54.7 @524K tok/s (slope -10% vs -37% pre-fix), cold prefill 2,771 @32K to 4,061 tok/s @524K (flat), decode-only C8 200.6 / C24 252.1, quality 7/7 + vision 2/2. Patches: img_ids PP relay, engram sm80 fp8, cand-key fix, exl3 shim.

Launch command

docker run -d --name dsv41-pp4 --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v '$HOME/models:/models:ro' -v '$GRAFT/dsv41reap/overlay:/overlay:ro' -v '$GRAFT/v41_sitecustom.py:/usr/lib/python3.12/sitecustomize.py:ro' -v '$HOME/tilelang-cache:/root/.tilelang' -p 8095:8095 -e VLLM_PLUGINS=vllm_exl3 -e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e NCCL_CUMEM_ENABLE=0 -e GLOO_SOCKET_IFNAME=lo -e VLLM_USE_V2_MODEL_RUNNER=0 -e VLLM_PP_LAYER_PARTITION=10,10,11,9 -e PYTHONPATH=/overlay -e TORCH_CUDA_ARCH_LIST=8.0 --entrypoint vllm pp4-exl3:v14 serve /models/v41-pollard-calibrated --pipeline-parallel-size 4 --tensor-parallel-size 1 --served-model-name deepseek-v4.1-flash --host 0.0.0.0 --port 8095 --max-model-len 524288 --max-num-seqs 32 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.93 --kv-cache-dtype fp8_ds_mla --engram-config '{"cpu_offload": true}' --enable-prefix-caching --compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,3,4,5,6,8,10,12,16,20,24,32]}' --speculative-config '{"method":"dspark","num_speculative_tokens":5}' --tokenizer-mode deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41

runtime

env: 'VLLM_PLUGINS=vllm_exl3 | VLLM_ENGINE_READY_TIMEOUT_S=3600 | NCCL_CUMEM_ENABLE=0 | GLOO_SOCKET_IFNAME=lo | VLLM_USE_V2_MODEL_RUNNER=0 | VLLM_PP_LAYER_PARTITION=10,10,11,9 | PYTHONPATH=/overlay | TORCH_CUDA_ARCH_LIST=8.0 (verbatim from serve/launch-pp4.sh @ 950f4c0, v37 with the 09-28 GLOO fix).'
notes: "serve/launch-pp4.sh pre-flight guards: GPUs must be free (nvidia-smi sum < 2000 MiB) and host RAM available ≥200 GB (engram pin 189 GiB). VLLM_USE_V2_MODEL_RUNNER=0 is REQUIRED — the dspark port + engram piecewise static-meta live on the V1 runner; V2 raises 'dspark with PP not supported'. use_local_argmax_reduction is not supported by the overlay's DSparkProposer. Partition 10,10,11,9: rank3 = 9 layers + drafter — KV pool 2.92M → 4.55M (+56%); shadow allows arbitrary cuts. k=6 rejected (must divide n_predict=5); k=10 measured -8-12%; k=5 optimal. DSpark acceptance: thinking ~85-100%, prose ~45-60%, mixed C32 ~62%; without speculation 12-16 tok/s across all contexts (DSpark is a 3.5x lifeline here, not a tax). Prefix cache matches by block content hash — header nonces do NOT defeat it; honest cold prefill needs per-point fully unique filler text. Decode rate is content-sensitive (35-56 tok/s at 524K with acceptance swings). Rejected configs (measured): ngram speculation 7.6 tok/s; DSV41_DSPARK_CONF=0.7 -30% (their CPU-expert setup benefits, this all-GPU bandwidth-bound arch does not); FULL_DECODE_ONLY graph = PIECEWISE; seqs 64 < C32 219 platform. Ops: zombie requests (client-disconnect leaves Running requests holding ~250K KV tokens → periodic 0 tok/s stalls; restart engine); docker rm -f may hang on shutdown. exllamav3 v1.5.3: speed-neutral ±2%, fixes memory leak (v1.5.1+ two-stage coop kernels already in v1.5.0). Cross-machine deployments should NOT use GLOO_SOCKET_IFNAME=lo — point it at the real internal NIC."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw.json

Engine

vllm · source↗ · 138e7488↗ · docker.io/vllm/vllm-openai:deepseekv41-flash-0909↗

Install note: Image pp4-exl3:v14 is LOCALLY BUILT: docker pull vllm/vllm-openai:deepseekv41-flash-0909 (official, ships nvcc/Triton/lmcache), clone zebgop-ops/dsv41reap-pp @ 138e748, copy overlay/vllm/ (54 .py) into the image's dist-packages/vllm/ paths and overlay/hybrid/ into site-packages (PYTHONPATH does not reach worker subprocesses — it must land in the package tree), copy the ampere/ SM8x sparse-attention dir from wtdcode/vllm-backport's tree into vllm/models/deepseek_v4_1/ampere/, apply the repo's 4 idempotent patch scripts in order (pp_relay_img_ids.py, engram_sm80_fp8.py, moe_cand_key_fix.py, plugin_v150_shim.py only for pre-94c29ba vllm-exl3), install the vllm-exl3 plugin (AGPL-3.0) + build exllamav3 v1.5.0 ext in-container with TORCH_CUDA_ARCH_LIST=8.0, convert weights to EXL3 2bpw (exllamav3 convert.py) and docker commit. Host: ≥373 GB RAM (engram pin 189 GiB + ≥24 GiB margin is a hard line), ~700 GB disk, driver 610.57.04. Startup ~7.5 min with the persistent TileLang JIT cache.

Model

DeepSeek-V4.1-Flash (exl3-2bpw) · deepseek-ai/DeepSeek-V4.1-Flash↗
AA intelligence 39 (Artificial Analysis · measured)

Hardware

NVIDIA CMP 170HX 64GB ×4 (Ampere)

Context

max context: 512K tokens

KV pool

5,895,670 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
1	8	—	200.6 tok/s	—	reported
1	24	—	252.1 tok/s	—	reported
32,768	1	—	60.7 tok/s	—	reported
131,072	1	—	58.3 tok/s	—	reported
307,200	1	—	56.7 tok/s	—	reported
524,288	1	—	54.7 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Author-measured on the 4x170HX box, v37 (post GLOO_SOCKET_IFNAME=lo fix, exllamav3 v1.5.3; README 09-28 table + FINDINGS.md §7 @ repo 950f4c0, clean engine, temp 0, streaming with TTFT separated, 2-round warmup): single-stream decode 60.7 @32K / 58.3 @128K / 56.7 @300K / 54.7 @524K tok/s (pre-fix 524K was 35.8, slope -37% → -10%); cold prefill (all-unique text, nonce at head) 32K 2,771 / 128K 3,756 / 300K 3,736 / 524K 4,061 tok/s (TTFT 129 s @524K), flat across depth; concurrent decode-only C8 200.6 / C16 199.4 / C24 252.1 tok/s (broke the old ~220 platform — half of it was gloo tax); KV pool 5,895,670 tokens (11.25 concurrent 524k windows); quality 7/7 + vision 2/2 unchanged. claimed_by=submitter T0 — no run file ingested; import via POST /api/v1/benchmarks to upgrade.

notes

Author-measured on the 4x170HX box, v37 (post GLOO_SOCKET_IFNAME=lo fix, exllamav3 v1.5.3; README 09-28 table + FINDINGS.md §7 @ repo 950f4c0, clean engine, temp 0, streaming with TTFT separated, 2-round warmup): single-stream decode 60.7 @32K / 58.3 @128K / 56.7 @300K / 54.7 @524K tok/s (pre-fix 524K was 35.8, slope -37% → -10%); cold prefill (all-unique text, nonce at head) 32K 2,771 / 128K 3,756 / 300K 3,736 / 524K 4,061 tok/s (TTFT 129 s @524K), flat across depth; concurrent decode-only C8 200.6 / C16 199.4 / C24 252.1 tok/s (broke the old ~220 platform — half of it was gloo tax); KV pool 5,895,670 tokens (11.25 concurrent 524k windows); quality 7/7 + vision 2/2 unchanged. claimed_by=submitter T0 — no run file ingested; import via POST /api/v1/benchmarks to upgrade.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash
2cba9e42aa026125f3ed06c6d98c1db82f7ca027 — original fp8 release (510 GB); serving pack = local EXL3 2bpw conversion (exllamav3 convert.py, Pollard-calibrated, 334 GiB: engram 197B tables FP8 = 203 GB kept unquantized, pinned 189 GiB UVA)
Engine image https://hub.docker.com/r/vllm/vllm-openai
vllm/vllm-openai:deepseekv41-flash-0909 + dsv41reap overlay 54 files (zebgop-ops/dsv41reap-pp @ 138e74881130a1510b8bcf22cc01456bd6e46219, Apache-2.0) + recipe patches + vllm-exl3 plugin (vcruz305, AGPL-3.0) + exllamav3 v1.5.0 ext → local image pp4-exl3:v14
Reference https://github.com/kaka86mm/dsv41-flash-pp4-170hx
950f4c003c074be9434735a37ec67466559909ec — source recipe repo @ HEAD (serve/launch-pp4.sh is the verbatim argv/env source; patches/, bench/accept.py, bench/reports/2026-09-14-v13-512k.md, FINDINGS.md all at this commit)

Revision notes

Community ingest of kaka86mm's recipe repo @ 950f4c0 (repo owner-named by Casey Oct 6 as part of the 170HX focus wave; Apache-2.0). Launch argv + runtime env transcribed verbatim from serve/launch-pp4.sh @ 950f4c0 (v37, includes the 09-28 GLOO_SOCKET_IFNAME=lo fix). Measured numbers are the author's README 09-28 v37 table + bench/reports/2026-09-14-v13-512k.md + FINDINGS.md §7, logged claimed_by=submitter T0 (no run file) — the repo's own measurement-discipline rules (bench/README, nonce-at-head, unique-text cold prefill, warmup 2 rounds) are noted because four of its earlier numbers were retracted in FINDINGS §4. Model: 764B native vision (image-text-to-text pipeline tag on HF); V4.1-Flash is its own model generation — separate row + family deepseek-v4-1-flash, NOT a variant of V4-Flash-0731 (site already carries both families since the jj Blackwell recipe). Built-on credits per the repo's own acknowledgements: zebgop-ops/dsv41reap-pp (shadow KV overlay, Apache-2.0), vcruz305/vllm-exl3 (EXL3 vLLM plugin, AGPL-3.0), turboderp-org/exllamav3 (quantization + kernels), wtdcode/vllm-backport (sm80 sparse-MLA lineage; ampere/ sparse-attention dir copied from its tree).

Revision history

ccec0230fe recipe: update deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw (r2: /max_context_len, /kv_pool_tokens, /storage_requirements) 10 d ago diff
cef3d9939e recipe: create deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw 24 d ago

Fork

Export

recipe.yaml (raw)