{"slug":"ds4-flash-3x170hx-pp3-dspark-k5","title":"DeepSeek V4 Flash-0731 abliterated on 3x CMP 170HX \u2014 vLLM patched PP3 DSpark K5","summary":"The proven 3-card pipeline-parallel recipe on CMP 170HX (sm_80, 64GB unlocked): PP=3 with manual layer partition 15,16,12 at util 0.98, FP8 KV, block-size 256, DSpark block drafting at exactly 5 speculative tokens, async scheduling, CPU pinning (dedicated physical cores, placeholder in launch args), hard memory cap 64g with swap disallowed. Structured decode 181 tok/s at 99% acceptance. Upstream reference repo: allover326/deepseek-v4-cmp170hx @ 3dd2d88 (Aug 13 2026) \u2014 documents 15,15,13 3-card split, retracted 4-card requirement, INT4 524288 ctx w/ DSV4_LOGITS_ROW_CHUNK=64, +7% decode via c3046d1 rebase patches (full source build required). First-party 15,16,12 numbers on this page are ours.","license":"CC-BY-4.0","tier":"curated","beta":false,"revision":{"no":5,"commit_sha":"24a9ac90156459e4efb49c951590edbea775df38","content_hash":"02d56ece2249f2b3e2b1ee2ffce158e841e20d26","created_at":"2026-10-29 21:32:23.860633-06","notes":"First-party 3-card launch procedure per the first-party RESULTS.md 3-card documentation. Layer partition 15,16,12 is first-party tuning; the upstream reference (allover326) documents 15,15,13 for this 3-card setup. Reported numbers are from 15,16,12. Layer partition and DSpark token count verified first-party on a 3x CMP 170HX rig.\n\ncapacity backfill (kanban t_3b48e9df, 2026-10-20): max_context_len 262144 \u2014 source: launch --max-model-len 262144. Same revision repairs the provenance block broken by r3 26510552f (an artifacts-style list item appended inside the provenance mapping after credit \u2014 invalid YAML, unparseable, which 500'd every PATCH/FORK on this recipe). The stray list item moves to its correct top-level artifacts block, byte-identical content otherwise."},"engine_source":{"type":"fork","git_url":"https://github.com/haosdent/vllm","base_engine":"vllm","base_version":"patched (vllm-patched tree + dspark)","image_digest":null,"install_note":"Rebuild required - engine is a source build from haosdent/vllm branch dsv4-flash-a100 (commit c3046d1, unreachable by git; tarball + tree-SHA reconstruction + ~115 min full sm_80 source build). No engine image exists yet - not installable as written until one is published.","pinned_commit":"c3046d1"},"credit":[{"name":"allover326","url":"https://github.com/allover326/deepseek-v4-cmp170hx"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/model","--served-model-name","DeepSeek-V4-Flash-0731-FP8-abliterated","--pipeline-parallel-size","3","--kv-cache-dtype","fp8","--block-size","256","--max-model-len","262144","--max-num-batched-tokens","2048","--trust-remote-code","--gpu-memory-utilization","0.98","--max-num-seqs","8","--no-enable-flashinfer-autotune","--tokenizer-mode","deepseek_v4","--enable-auto-tool-choice","--tool-call-parser","deepseek_v4","--async-scheduling","--speculative-config","{\"method\":\"dspark\",\"num_speculative_tokens\":5}"],"runtime":{"env":"CUDA_DEVICE_ORDER=PCI_BUS_ID, HF_HUB_OFFLINE=1, VLLM_WORKER_MULTIPROC_METHOD=spawn, DSV4_LOGITS_ROW_CHUNK=64, VLLM_SPARSE_DENSE_QUERY_BLOCK=0, VLLM_MARLIN_FP8_DEQUANT_BF16=1","notes":"DSpark block size MUST be exactly 5 for this checkpoint (block-5 Markov head); 7 measured acceptance 0% and token salad. max_num_batched_tokens 2048 is pinned by VRAM (~40 MiB free at util 0.98); 4096 OOMs rank1 and was slower before dying. Swap must be disallowed (cgroup v2: equal --memory/--memory-swap); 7.6 GB of paged-out PP worker stalls every stage behind it. CPU pinning matters: unpinned PP workers competing with other host jobs measurably reduce throughput.","container":"--runtime=nvidia --cpuset-cpus <dedicated-physical-cores, e.g. 0-5,18-23> --memory 64g --memory-swap 64g --shm-size=16g","layer_partition":"15,16,12 (VLLM_PP_LAYER_PARTITION; last rank also carries lm_head + DSpark drafter)"},"max_context_len":262144},"models":[{"slug":"deepseek-v4-flash-0731-abliterated","name":"DeepSeek-V4-Flash-0731-abliterated","hf_repo":null,"quant":"fp8-original"}],"hardware":[{"slug":"nvidia-cmp-170hxx3","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":3}],"generation":"ampere","tags":["dspark","fp8-kv","no-swap","pipeline-parallel"],"tasks":["chat","code"],"aa":{"aa_index":34,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-07 18:00:00-06","aa_family_slug":"deepseek-v4-flash-0731","aa_resolved_slug":"deepseek-v4-flash-0731"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"T1 self-measured: 181 tok/s structured at 99% accept (33 ms/cycle), 76 tok/s prose at 30%, ceiling 183. Prefill 2,350 tok/s at 2048 batched tokens.\", \"tok_s\": 181, \"context_len\": 262144, \"prefill_tok_s\": 2350}"}],"perf_points":[{"id":2041,"context_len":131072,"prefill_tok_s":null,"decode_tok_s":78.7999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"allover326/deepseek-v4-cmp170hx RESULTS.md @ 3dd2d88, section \"3 cards + the INT4 repack: 512k verified\" -> \"The 3-card run, in full\": 3x CMP 170HX, PP3, DSpark num_speculative_tokens 5, --max-model-len 131072, decode ~78.8 tok/s median. The author run used the INT4 repack, layer partition 15,15,13, DSV4_LOGITS_ROW_CHUNK 256, util 0.95; this recipe is the FP8 abliterated checkpoint at 262144 ctx, partition 15,16,12, row chunk 64, util 0.98."},{"id":2040,"context_len":262144,"prefill_tok_s":2350,"decode_tok_s":181,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"T1 self-measured: 181 tok/s structured decode at 99% accept (33 ms/cycle), 76 tok/s prose at 30%, ceiling 183; prefill 2,350 tok/s at 2048 batched tokens"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/ds4-flash-3x170hx-pp3-dspark-k5.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/ds4-flash-3x170hx-pp3-dspark-k5.json"},"requirements":null,"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/allover326/deepseek-v4-cmp170hx","pin":"HEAD 3dd2d8817e7deae00d998edde0d227e7254ea71e (Aug 13 2026) \u2014 upstream 3-card reference documents 15,15,13; our 15,16,12 is first-party tuning (both splits valid). c3046d1 base unreachable (force-pushed); repo docs carry tree-SHA tarball reconstruction."},{"kind":"model_weights","pointer_type":"url","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","pin":"0731 checkpoint (46-shard original, not the 46-shard DeepSeek-V4-Flash wrong variant)"}],"links":{"self":"/api/v1/recipes/ds4-flash-3x170hx-pp3-dspark-k5.json","alternate":"/r/ds4-flash-3x170hx-pp3-dspark-k5","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:23.860633-06","created_at":"2026-09-27 21:34:43.912323-06","speeds":{"decode_tok_s":129.900000000000005684341886080801486968994140625,"prefill_tok_s":2350,"leaderboard":"/api/v1/leaderboard.json"}}