DeepSeek V4 Flash-0731 abliterated on 3x CMP 170HX — vLLM patched PP3 DSpark K5

recipe by allover326↗ · listed by localai-recipes

revision r5 · commit 24a9ac9015 · updated 21 h ago · license CC-BY-4.0

code chat fp8-kv pipeline-parallel no-swap dspark

The proven 3-card pipeline-parallel recipe on CMP 170HX (sm_80, 64GB unlocked): PP=3 with manual layer partition 15,16,12 at util 0.98, FP8 KV, block-size 256, DSpark block drafting at exactly 5 speculative tokens, async scheduling, CPU pinning (dedicated physical cores, placeholder in launch args), hard memory cap 64g with swap disallowed. Structured decode 181 tok/s at 99% acceptance. Upstream reference repo: allover326/deepseek-v4-cmp170hx @ 3dd2d88 (Aug 13 2026) — documents 15,15,13 3-card split, retracted 4-card requirement, INT4 524288 ctx w/ DSV4_LOGITS_ROW_CHUNK=64, +7% decode via c3046d1 rebase patches (full source build required). First-party 15,16,12 numbers on this page are ours.

Launch command

vllm serve /model --served-model-name DeepSeek-V4-Flash-0731-FP8-abliterated --pipeline-parallel-size 3 --kv-cache-dtype fp8 --block-size 256 --max-model-len 262144 --max-num-batched-tokens 2048 --trust-remote-code --gpu-memory-utilization 0.98 --max-num-seqs 8 --no-enable-flashinfer-autotune --tokenizer-mode deepseek_v4 --enable-auto-tool-choice --tool-call-parser deepseek_v4 --async-scheduling --speculative-config '{"method":"dspark","num_speculative_tokens":5}'

runtime

env: 'CUDA_DEVICE_ORDER=PCI_BUS_ID, HF_HUB_OFFLINE=1, VLLM_WORKER_MULTIPROC_METHOD=spawn, DSV4_LOGITS_ROW_CHUNK=64, VLLM_SPARSE_DENSE_QUERY_BLOCK=0, VLLM_MARLIN_FP8_DEQUANT_BF16=1'
notes: 'DSpark block size MUST be exactly 5 for this checkpoint (block-5 Markov head); 7 measured acceptance 0% and token salad. max_num_batched_tokens 2048 is pinned by VRAM (~40 MiB free at util 0.98); 4096 OOMs rank1 and was slower before dying. Swap must be disallowed (cgroup v2: equal --memory/--memory-swap); 7.6 GB of paged-out PP worker stalls every stage behind it. CPU pinning matters: unpinned PP workers competing with other host jobs measurably reduce throughput.'
container: '--runtime=nvidia --cpuset-cpus <dedicated-physical-cores, e.g. 0-5,18-23> --memory 64g --memory-swap 64g --shm-size=16g'
layer_partition: '15,16,12 (VLLM_PP_LAYER_PARTITION; last rank also carries lm_head + DSpark drafter)'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/ds4-flash-3x170hx-pp3-dspark-k5.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/ds4-flash-3x170hx-pp3-dspark-k5.json

Engine

vllm · source↗ · c3046d1↗

Install note: Rebuild required - engine is a source build from haosdent/vllm branch dsv4-flash-a100 (commit c3046d1, unreachable by git; tarball + tree-SHA reconstruction + ~115 min full sm_80 source build). No engine image exists yet - not installable as written until one is published.

Model

DeepSeek-V4-Flash-0731-abliterated (fp8-original)
AA intelligence 34 (Artificial Analysis · measured)

Hardware

NVIDIA CMP 170HX 64GB ×3 (Ampere)

Context

max context: 256K tokens

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
131,072	1	—	78.8 tok/s	—	reported
262,144	1	2350 tok/s	181 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	T1 self-measured: 181 tok/s structured at 99% accept (33 ms/cycle), 76 tok/s prose at 30%, ceiling 183. Prefill 2,350 tok/s at 2048 batched tokens.
prefill_tok_s	2350

Downloads

Reference https://github.com/allover326/deepseek-v4-cmp170hx
HEAD 3dd2d8817e7deae00d998edde0d227e7254ea71e (Aug 13 2026) — upstream 3-card reference documents 15,15,13; our 15,16,12 is first-party tuning (both splits valid). c3046d1 base unreachable (force-pushed); repo docs carry tree-SHA tarball reconstruction.
Weights https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash
0731 checkpoint (46-shard original, not the 46-shard DeepSeek-V4-Flash wrong variant)

Revision notes

First-party 3-card launch procedure per the first-party RESULTS.md 3-card documentation. Layer partition 15,16,12 is first-party tuning; the upstream reference (allover326) documents 15,15,13 for this 3-card setup. Reported numbers are from 15,16,12. Layer partition and DSpark token count verified first-party on a 3x CMP 170HX rig.

capacity backfill (kanban t_3b48e9df, 2026-10-20): max_context_len 262144 — source: launch --max-model-len 262144. Same revision repairs the provenance block broken by r3 26510552f (an artifacts-style list item appended inside the provenance mapping after credit — invalid YAML, unparseable, which 500'd every PATCH/FORK on this recipe). The stray list item moves to its correct top-level artifacts block, byte-identical content otherwise.

Revision history

24a9ac9015 recipe: update ds4-flash-3x170hx-pp3-dspark-k5 (r5: /performance) 15 h ago diff
d43dfde98c recipe: update ds4-flash-3x170hx-pp3-dspark-k5 (r5: /kv_pool_tokens) 10 d ago diff
4ed4070f8e recipe: update ds4-flash-3x170hx-pp3-dspark-k5 (r4: /max_context_len, /provenance yaml-repair) 10 d ago diff
26510552ff recipe: add upstream reference artifact allover326/deepseek-v4-cmp170hx @ 3dd2d88 (r3: /artifacts) 24 d ago diff
7f658bb910 recipe: update ds4-flash-3x170hx-pp3-dspark-k5 (r2: /summary) 24 d ago diff
bfdf9daa3e history: re-squash corpus to a single root commit after the slug reconciliation (prior tip kept at refs/lar-archive/pre-single-slug-resquash) 25 d ago

Fork

Export

recipe.yaml (raw)