{"slug":"deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw","title":"DeepSeek-V4.1-Flash (764B) on 4x CMP 170HX \u2014 PP4, shadow KV, EXL3 2bpw, DSpark K5, 512k ctx, native vision (kaka86mm)","summary":"Four-card Ampere recipe for full DeepSeek-V4.1-Flash (764B, native vision) on 4x CMP 170HX (sm80, Gen2 x4, no P2P \u2014 256 GB VRAM can't hold 510 GB, so: EXL3 2bpw + fp8 KV): PP4 x TP1 partition 10,10,11,9 (rank3 = 9 layers + drafter), shadow KV via dsv41reap-pp overlay (54 files), EXL3 2bpw pack \u2014 engram tables kept FP8 at 203 GB, pinned 189 GiB host RAM (quantizing destroys retrieval), DSpark n=5 (conf=0; truncation is -30% here; ngram is a 7.6 tok/s disaster), 524k ctx, fp8_ds_mla KV, 5.9M-token pool, native ViT, GLOO_SOCKET_IFNAME=lo \u2014 the 09-28 one-line fix for PP gloo via docker bridge (535 ms/send) that collapsed long-context decode. Measured by the author (09-28 v37): decode 60.7 @32K / 58.3 @128K / 54.7 @524K tok/s (slope -10% vs -37% pre-fix), cold prefill 2,771 @32K to 4,061 tok/s @524K (flat), decode-only C8 200.6 / C24 252.1, quality 7/7 + vision 2/2. Patches: img_ids PP relay, engram sm80 fp8, cand-key fix, exl3 shim.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"363a35de531734dabec247825b6103dc00425670","content_hash":"49f43cc59d2bfe7c33b889b5fdc5129673136cc3","created_at":"2026-10-20 15:34:08.130558-06","notes":"Community ingest of kaka86mm's recipe repo @ 950f4c0 (repo owner-named by Casey Oct 6 as part of the 170HX focus wave; Apache-2.0). Launch argv + runtime env transcribed verbatim from serve/launch-pp4.sh @ 950f4c0 (v37, includes the 09-28 GLOO_SOCKET_IFNAME=lo fix). Measured numbers are the author's README 09-28 v37 table + bench/reports/2026-09-14-v13-512k.md + FINDINGS.md \u00a77, logged claimed_by=submitter T0 (no run file) \u2014 the repo's own measurement-discipline rules (bench/README, nonce-at-head, unique-text cold prefill, warmup 2 rounds) are noted because four of its earlier numbers were retracted in FINDINGS \u00a74. Model: 764B native vision (image-text-to-text pipeline tag on HF); V4.1-Flash is its own model generation \u2014 separate row + family deepseek-v4-1-flash, NOT a variant of V4-Flash-0731 (site already carries both families since the jj Blackwell recipe). Built-on credits per the repo's own acknowledgements: zebgop-ops/dsv41reap-pp (shadow KV overlay, Apache-2.0), vcruz305/vllm-exl3 (EXL3 vLLM plugin, AGPL-3.0), turboderp-org/exllamav3 (quantization + kernels), wtdcode/vllm-backport (sm80 sparse-MLA lineage; ampere/ sparse-attention dir copied from its tree)."},"engine_source":{"type":"docker_image","git_url":"https://github.com/zebgop-ops/dsv41reap-pp","base_engine":"vllm","pip_package":null,"base_version":"official vllm-openai:deepseekv41-flash-0909 + dsv41reap overlay","image_digest":"docker.io/vllm/vllm-openai:deepseekv41-flash-0909","install_note":"Image pp4-exl3:v14 is LOCALLY BUILT: docker pull vllm/vllm-openai:deepseekv41-flash-0909 (official, ships nvcc/Triton/lmcache), clone zebgop-ops/dsv41reap-pp @ 138e748, copy overlay/vllm/ (54 .py) into the image's dist-packages/vllm/ paths and overlay/hybrid/ into site-packages (PYTHONPATH does not reach worker subprocesses \u2014 it must land in the package tree), copy the ampere/ SM8x sparse-attention dir from wtdcode/vllm-backport's tree into vllm/models/deepseek_v4_1/ampere/, apply the repo's 4 idempotent patch scripts in order (pp_relay_img_ids.py, engram_sm80_fp8.py, moe_cand_key_fix.py, plugin_v150_shim.py only for pre-94c29ba vllm-exl3), install the vllm-exl3 plugin (AGPL-3.0) + build exllamav3 v1.5.0 ext in-container with TORCH_CUDA_ARCH_LIST=8.0, convert weights to EXL3 2bpw (exllamav3 convert.py) and docker commit. Host: \u2265373 GB RAM (engram pin 189 GiB + \u226524 GiB margin is a hard line), ~700 GB disk, driver 610.57.04. Startup ~7.5 min with the persistent TileLang JIT cache.","pinned_commit":"138e74881130a1510b8bcf22cc01456bd6e46219"},"credit":[{"name":"kaka86mm","url":"https://github.com/kaka86mm/dsv41-flash-pp4-170hx"},{"name":"zebgop-ops (dsv41reap-pp shadow KV overlay)","url":"https://github.com/zebgop-ops/dsv41reap-pp"},{"name":"vcruz305 (vllm-exl3 plugin)","url":"https://github.com/vcruz305/vllm-exl3"},{"name":"turboderp-org (exllamav3)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"wtdcode (vllm-backport)","url":"https://github.com/wtdcode/vllm-backport"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["docker","run","-d","--name","dsv41-pp4","--gpus","all","--ipc=host","--ulimit","memlock=-1","--ulimit","stack=67108864","-v","$HOME/models:/models:ro","-v","$GRAFT/dsv41reap/overlay:/overlay:ro","-v","$GRAFT/v41_sitecustom.py:/usr/lib/python3.12/sitecustomize.py:ro","-v","$HOME/tilelang-cache:/root/.tilelang","-p","8095:8095","-e","VLLM_PLUGINS=vllm_exl3","-e","VLLM_ENGINE_READY_TIMEOUT_S=3600","-e","NCCL_CUMEM_ENABLE=0","-e","GLOO_SOCKET_IFNAME=lo","-e","VLLM_USE_V2_MODEL_RUNNER=0","-e","VLLM_PP_LAYER_PARTITION=10,10,11,9","-e","PYTHONPATH=/overlay","-e","TORCH_CUDA_ARCH_LIST=8.0","--entrypoint","vllm","pp4-exl3:v14","serve","/models/v41-pollard-calibrated","--pipeline-parallel-size","4","--tensor-parallel-size","1","--served-model-name","deepseek-v4.1-flash","--host","0.0.0.0","--port","8095","--max-model-len","524288","--max-num-seqs","32","--max-num-batched-tokens","8192","--gpu-memory-utilization","0.93","--kv-cache-dtype","fp8_ds_mla","--engram-config","{\"cpu_offload\": true}","--enable-prefix-caching","--compilation-config","{\"cudagraph_mode\":\"PIECEWISE\",\"cudagraph_capture_sizes\":[1,2,3,4,5,6,8,10,12,16,20,24,32]}","--speculative-config","{\"method\":\"dspark\",\"num_speculative_tokens\":5}","--tokenizer-mode","deepseek_v41","--enable-auto-tool-choice","--tool-call-parser","deepseek_v41","--reasoning-parser","deepseek_v41"],"runtime":{"env":"VLLM_PLUGINS=vllm_exl3 | VLLM_ENGINE_READY_TIMEOUT_S=3600 | NCCL_CUMEM_ENABLE=0 | GLOO_SOCKET_IFNAME=lo | VLLM_USE_V2_MODEL_RUNNER=0 | VLLM_PP_LAYER_PARTITION=10,10,11,9 | PYTHONPATH=/overlay | TORCH_CUDA_ARCH_LIST=8.0 (verbatim from serve/launch-pp4.sh @ 950f4c0, v37 with the 09-28 GLOO fix).","notes":"serve/launch-pp4.sh pre-flight guards: GPUs must be free (nvidia-smi sum < 2000 MiB) and host RAM available \u2265200 GB (engram pin 189 GiB). VLLM_USE_V2_MODEL_RUNNER=0 is REQUIRED \u2014 the dspark port + engram piecewise static-meta live on the V1 runner; V2 raises 'dspark with PP not supported'. use_local_argmax_reduction is not supported by the overlay's DSparkProposer. Partition 10,10,11,9: rank3 = 9 layers + drafter \u2014 KV pool 2.92M \u2192 4.55M (+56%); shadow allows arbitrary cuts. k=6 rejected (must divide n_predict=5); k=10 measured -8-12%; k=5 optimal. DSpark acceptance: thinking ~85-100%, prose ~45-60%, mixed C32 ~62%; without speculation 12-16 tok/s across all contexts (DSpark is a 3.5x lifeline here, not a tax). Prefix cache matches by block content hash \u2014 header nonces do NOT defeat it; honest cold prefill needs per-point fully unique filler text. Decode rate is content-sensitive (35-56 tok/s at 524K with acceptance swings). Rejected configs (measured): ngram speculation 7.6 tok/s; DSV41_DSPARK_CONF=0.7 -30% (their CPU-expert setup benefits, this all-GPU bandwidth-bound arch does not); FULL_DECODE_ONLY graph = PIECEWISE; seqs 64 < C32 219 platform. Ops: zombie requests (client-disconnect leaves Running requests holding ~250K KV tokens \u2192 periodic 0 tok/s stalls; restart engine); docker rm -f may hang on shutdown. exllamav3 v1.5.3: speed-neutral \u00b12%, fixes memory leak (v1.5.1+ two-stage coop kernels already in v1.5.0). Cross-machine deployments should NOT use GLOO_SOCKET_IFNAME=lo \u2014 point it at the real internal NIC."},"kv_pool_tokens":5895670,"max_context_len":524288,"storage_requirements":{"ram_gb":373}},"models":[{"slug":"deepseek-v4-1-flash","name":"DeepSeek-V4.1-Flash","hf_repo":"deepseek-ai/DeepSeek-V4.1-Flash","quant":"exl3-2bpw"}],"hardware":[{"slug":"nvidia-cmp-170hxx4","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":4}],"generation":"ampere","tags":["dspark","engram","exl3","long-context","pipeline-parallel","shadow-kv","speculative"],"tasks":["chat","code","vision"],"aa":{"aa_index":39,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash","aa_index_version":"Index v4.3.2","aa_methodology_version":"Index v4.3.2 (2026-10-18)","aa_published_fetched_at":"2026-10-20 21:49:11.309778-06","aa_family_slug":"deepseek-v4-1-flash","aa_resolved_slug":"deepseek-v4-1-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured on the 4x170HX box, v37 (post GLOO_SOCKET_IFNAME=lo fix, exllamav3 v1.5.3; README 09-28 table + FINDINGS.md \u00a77 @ repo 950f4c0, clean engine, temp 0, streaming with TTFT separated, 2-round warmup): single-stream decode 60.7 @32K / 58.3 @128K / 56.7 @300K / 54.7 @524K tok/s (pre-fix 524K was 35.8, slope -37% \u2192 -10%); cold prefill (all-unique text, nonce at head) 32K 2,771 / 128K 3,756 / 300K 3,736 / 524K 4,061 tok/s (TTFT 129 s @524K), flat across depth; concurrent decode-only C8 200.6 / C16 199.4 / C24 252.1 tok/s (broke the old ~220 platform \u2014 half of it was gloo tax); KV pool 5,895,670 tokens (11.25 concurrent 524k windows); quality 7/7 + vision 2/2 unchanged. claimed_by=submitter T0 \u2014 no run file ingested; import via POST /api/v1/benchmarks to upgrade.\", \"tok_s\": 54.7000000000000028421709430404007434844970703125, \"points\": [{\"notes\": \"single-stream decode @32K, v37 post-gloo-fix (pre-fix 57.0)\", \"concurrency\": 1, \"context_len\": 32768, \"decode_tok_s\": 60.7000000000000028421709430404007434844970703125}, {\"notes\": \"single-stream decode @128K (pre-fix 46.7)\", \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 58.2999999999999971578290569595992565155029296875}, {\"notes\": \"single-stream decode @300K (pre-fix ~40)\", \"concurrency\": 1, \"context_len\": 307200, \"decode_tok_s\": 56.7000000000000028421709430404007434844970703125}, {\"notes\": \"single-stream decode @524K (pre-fix 35.8); content-sensitive 35-56 with acceptance swings\", \"concurrency\": 1, \"context_len\": 524288, \"decode_tok_s\": 54.7000000000000028421709430404007434844970703125}, {\"notes\": \"cold prefill @524K, fully unique text (block-content-hash prefix cache \u2014 header nonces do not defeat it), TTFT 129 s; flat 3.7-4K tok/s across depth\", \"ttft_ms\": 129000, \"concurrency\": 1, \"context_len\": 524288, \"prefill_tok_s\": 4061}, {\"notes\": \"cold prefill @32K, v37\", \"concurrency\": 1, \"context_len\": 32768, \"prefill_tok_s\": 2771}, {\"notes\": \"aggregate decode-only C8, v37 (mixed scenarios)\", \"concurrency\": 8, \"context_len\": 1, \"decode_tok_s\": 200.599999999999994315658113919198513031005859375}, {\"notes\": \"aggregate decode-only C24 \u2014 broke the pre-fix ~220 pipeline-saturation platform\", \"concurrency\": 24, \"context_len\": 1, \"decode_tok_s\": 252.099999999999994315658113919198513031005859375}], \"context_len\": 524288}"}],"perf_points":[{"id":1030,"context_len":1,"prefill_tok_s":null,"decode_tok_s":200.599999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate decode-only C8, v37 (mixed scenarios)"},{"id":1031,"context_len":1,"prefill_tok_s":null,"decode_tok_s":252.099999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":24,"claimed_by":"submitter","notes":"aggregate decode-only C24 \u2014 broke the pre-fix ~220 pipeline-saturation platform"},{"id":1026,"context_len":32768,"prefill_tok_s":null,"decode_tok_s":60.7000000000000028421709430404007434844970703125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode @32K, v37 post-gloo-fix (pre-fix 57.0)"},{"id":1027,"context_len":131072,"prefill_tok_s":null,"decode_tok_s":58.2999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode @128K (pre-fix 46.7)"},{"id":1028,"context_len":307200,"prefill_tok_s":null,"decode_tok_s":56.7000000000000028421709430404007434844970703125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode @300K (pre-fix ~40)"},{"id":1029,"context_len":524288,"prefill_tok_s":null,"decode_tok_s":54.7000000000000028421709430404007434844970703125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode @524K (pre-fix 35.8); content-sensitive 35-56 with acceptance swings"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash","pin":"2cba9e42aa026125f3ed06c6d98c1db82f7ca027 \u2014 original fp8 release (510 GB); serving pack = local EXL3 2bpw conversion (exllamav3 convert.py, Pollard-calibrated, 334 GiB: engram 197B tables FP8 = 203 GB kept unquantized, pinned 189 GiB UVA)"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/vllm/vllm-openai","pin":"vllm/vllm-openai:deepseekv41-flash-0909 + dsv41reap overlay 54 files (zebgop-ops/dsv41reap-pp @ 138e74881130a1510b8bcf22cc01456bd6e46219, Apache-2.0) + recipe patches + vllm-exl3 plugin (vcruz305, AGPL-3.0) + exllamav3 v1.5.0 ext \u2192 local image pp4-exl3:v14"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/kaka86mm/dsv41-flash-pp4-170hx","pin":"950f4c003c074be9434735a37ec67466559909ec \u2014 source recipe repo @ HEAD (serve/launch-pp4.sh is the verbatim argv/env source; patches/, bench/accept.py, bench/reports/2026-09-14-v13-512k.md, FINDINGS.md all at this commit)"}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw.json","alternate":"/r/deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:08.130558-06","created_at":"2026-10-06 17:02:47.638712-06","speeds":{"decode_tok_s":57.5,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}