{"slug":"deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5","title":"DeepSeek-V4.1-Flash SM80 DeepInfer kernel port 6x CMP 170HX DSpark K5","summary":"DeepSeek-V4.1-Flash served on six 64 GB SM80 GPUs (GA100 die, owner-classified 170HX) with HalfVulpes' Deepinfer-SM80 patch set: a 30-file vLLM V1 overlay porting DeepSeek's DeepInfer kernel architecture (SM100 tcgen05/TMEM) to Ampere mma.sync tensor cores - unified 512-dim SM80 sparse-MLA decode/prefill kernels with bitshift UE8M0 scale decoding, a bit-exact single-CTA Triton TopK+SWA combiner (22.17x), direct-store KV-cache gather (7.64x @48K), SM80-aligned TileLang mHC routing (2.89x), and YOCO L20-L39 ragged indexer-metadata reuse (19 redundant kernel chains eliminated per chunk). TP2 x PP3 (14/14/12) + DSpark K=5 + 1M context, 4.65M-token GPU-resident KV. Author-measured: 64K cold prefill 4,507 tok/s (+24.2% vs baseline vLLM SM80), 64K TTFT 10.7 s (-18.9%), c1 DSpark decode 160.8 tok/s code / 188.8 tok/s 16K+ long-gen, c4 170.5 aggregate. Microbenchmarks bit-exact (0 diff) or within BF16 noise. Topology guide: PP6 = 2.1x slower single-stream; TP2+TP4 asymmetric hurts Marlin tiles.","license":"Apache-2.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"ced28259b90a43ec103ecd450b68455a56d2808d","content_hash":"e00265eca8b84734c2c7de2b67d558b0d702d25e","created_at":"2026-10-28 15:48:17.404025-06","notes":"Community ingest of HalfVulpes' port (Apache-2.0). All numbers claimed_by=submitter T0 (microbench + E2E harness + raw JSON in-repo) - import via POST /api/v1/benchmarks to upgrade. HARDWARE CAVEAT: validated on GA100/A100 64 GB SM80; CMP 170HX is the same GA100 silicon at 140 W (74 SMs) but the author did not measure CMP-class power/link configs, and the 6x rig uses PCIe x16 Gen4/Gen5 - do NOT present the numbers as CMP-measured. Engine is a vLLM patch, NOT a new engine enum. Upstream deepseek-ai/DeepInfer itself stays SM100/Ascend-only and is NOT separately ingestable. Discord corroboration: cmpunlocker #testing-general / #clanker-talk 2026-10-20 17:12-17:22Z ('160.8-188.8 tok/s... even pp6 65.5-68.5... double of GLM 5.3... multi-model') and reala/segmond linking the repo (not re-verified). docs/05's KV math section names the -1017 revision while README/summary serve the base V4.1-Flash checkpoint. A 170hx-x4 CMP variant is the natural follow-up once someone measures it. HARDWARE RECLASSIFICATION (owner ruling 2026-10-21): the author labeled the rig 6x A100; 64 GB SM80 cards are CMP 170HX (GA100 die) - real A100 is 40/80 GB and never 64 - so the recipe is owner-classified as 6x CMP 170HX; the -6x-a100- slug keeps its historical token. PIN REVISION (kanban t_ef4644ee, owner go Casey 2026-10-28, parent t_ab1b8c82): pin 2a828fc2122b -> 8895510cae8f (+3 commits: 8895510 fix(pp) pp_share.py + TP2xPP3/TP4xPP2 + DSpark + 1M indexer modules; a11809d 8-GPU TP4xPP2 benchmarks; e375de2 base vLLM PR #56201 (79a7108d9a) + haosdent/vllm SM80 fork link + 00-vllm-pr56201-sm80-base.patch). Launch args re-transcribed from the repo README (Base vLLM Version, Fork Lineage & Quick Start Option A/B) + docs/05_TOPOLOGY_1M_CONTEXT_ANALYSIS_SM80.md section 1: the repo ships no serve script; previously-published args applied only patches/vllm-sm80-deepinfer.patch with no SM80 base, stale at 8895510cae (patch no longer applies to upstream main: PR #56741 rename + SM80 backend missing). Scaling rows: omitted - recipe publishes no scaling table (8-GPU TP4xPP2 data: 13,306,380 KV / 12.69x1M, C=1,2,4,12 decode+prefill, left for a future 8-GPU variant card). UI chart charts/05_8gpu_tp4pp2_scaling_1_2_4_12.png deliberately not attached (t_ab1b8c82). Newer-commits-deferred: NONE - live GitHub 2026-10-28 main HEAD == 8895510cae8f (4 commits total); haosdent/vllm dsv4-flash-a100 branch head moved to 12810046c799 (informational; recipe pins the bundled patch file, not the branch). Perf fields untouched (measurement provenance separate lane)."},"engine_source":{"type":"fork","git_url":"https://github.com/HalfVulpes/Deepinfer-SM80","base_engine":"vllm","pip_package":null,"base_version":"V1 @ vllm PR #56201 base 79a7108d9a + haosdent/vllm dsv4-flash-a100 SM80 base + Deepinfer-SM80 patch @ 8895510cae8f","image_digest":null,"install_note":"Engine pin chain as stated by the repo at 8895510cae: base = vllm-project/vllm PR #56201 base commit 79a7108d9aea27ddab99ce1779290d300b17fc23 (v0.28.1rc0); SM80 base enablement = haosdent/vllm dsv4-flash-a100 (deepseek-v4-a100-support, commits 62195e9..c3046d1) rebased onto 79a7108d9a, shipped as patches/00-vllm-pr56201-sm80-base.patch (NEW at e375de2, +15,359 lines); DeepInfer-SM80 kernel + YOCO/PP optimizations via patches/vllm-sm80-deepinfer.patch (MODIFIED at 8895510: 30 files, 4,289 lines). Option A (reproducible full build from upstream): git clone vllm; git fetch origin 79a7108d9aea27ddab99ce1779290d300b17fc23; git checkout it; git apply patches/00-vllm-pr56201-sm80-base.patch; then git apply patches/vllm-sm80-deepinfer.patch. Option B (existing SM80 base tree, 11cf4a67f9 on top of 79a7108d9a): apply only patches/vllm-sm80-deepinfer.patch, or copy/bind-mount all 30 drop-in files: cp -r vllm_sm80_deepinfer/vllm/* site-packages/vllm/ (docker bind-mount read-only overlays work - the CMP-recipe pattern). Why vllm-sm80-deepinfer.patch does not apply to latest upstream vllm/main: upstream PR #56741 (cf1584f373) renamed vllm/models/deepseek_v4_1/ to deepseek_v41/, and upstream main ships only SM100 Blackwell + AMD ROCm backends for DeepSeek-V4.1-Flash. Verify with the three in-repo microbenchmarks (microbench_sm80_kernels.py / _part2 / _part3) which print bit-exactness + latency tables; charts via generate_charts.py; full dataset in benchmarks/results/sm80_benchmark_summary.json. Requirements: SM80 (A100 40/64/80 GB, A800, GA100) or SM86 (RTX 3090/3090 Ti, RTX A6000, A40/A10/A30 - all kernels <=48 KB SRAM/CTA); vLLM V1 engine; Triton + TileLang JIT (startup cache-warm included). Base checkpoint: official deepseek-ai/DeepSeek-V4.1-Flash FP8.","pinned_commit":"8895510cae8f8f1733401686e67a4d1e11e7e94b"},"credit":[{"name":"HalfVulpes (Deepinfer-SM80 port + measurements)","url":"https://github.com/HalfVulpes/Deepinfer-SM80"},{"name":"DeepSeek-AI (DeepInfer engine architecture + DeepSeek-V4.1-Flash checkpoint)","url":"https://github.com/deepseek-ai/DeepInfer"},{"name":"vLLM project (V1 engine + TPxPP runtime)","url":"https://github.com/vllm-project/vllm"},{"name":"TileLang + OpenAI Triton (SM80 kernel synthesis)","url":"https://github.com/tile-ai/tilelang"},{"name":"FlashInfer + IST-DASLab Marlin (sparse-MLA + MXFP4/W4A16 Ampere GEMM primitives)","url":"https://github.com/flashinfer-ai/flashinfer"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["git clone https://github.com/HalfVulpes/Deepinfer-SM80 && cd Deepinfer-SM80","git clone https://github.com/vllm-project/vllm.git && cd vllm && git fetch origin 79a7108d9aea27ddab99ce1779290d300b17fc23 && git checkout 79a7108d9aea27ddab99ce1779290d300b17fc23","# Option A: apply BOTH patches in order (00- base first, then DeepInfer-SM80):","git apply /path/to/Deepinfer-SM80/patches/00-vllm-pr56201-sm80-base.patch","git apply /path/to/Deepinfer-SM80/patches/vllm-sm80-deepinfer.patch","# Option B: on an existing SM80 base tree (11cf4a67f9 on 79a7108d9a) apply only patches/vllm-sm80-deepinfer.patch, or bind-mount/copy all 30 drop-in files: cp -r vllm_sm80_deepinfer/vllm/* /path/to/site-packages/vllm/","# verify: python3 benchmarks/microbench_sm80_kernels.py && python3 benchmarks/microbench_sm80_part2.py && python3 benchmarks/microbench_sm80_part3.py","# serve DeepSeek-V4.1-Flash 6x SM80 64GB: TP2 x PP3 VLLM_PP_LAYER_PARTITION=14,14,12 + DSpark K=5 + max context 1,048,576 (README Quick Start + docs/05 section 1; repo ships no serve script)"],"layouts":["tp","pp"],"runtime":{"env":"Author topology on 6x 64 GB: TP2 x PP3 partition 14/14/12 (VLLM_PP_LAYER_PARTITION=14,14,12 per benchmarks/results/sm80_benchmark_summary.json), DSpark K=5 speculative, max context 1,048,576, 4.65M GPU-resident KV. Alternative topologies measured in docs/05: PP6 (TP1xPP6) = 65.5-68.5 tok/s c1 (2.15x slower), TP2+TP4 asymmetric = ~55.5 tok/s unspeculated (moie_intermediate 2304/4=576 hurts Marlin 4.5x128 tiling). Deployment env/args: the patch is engine-internal; no special env vars documented in the README (DSpark handled by vLLM's speculative-config; the mHC/Triton kernels self-configure per SM count).","notes":"Drop-in modules (30 files at 8895510cae) touch: vllm/models/deepseek_v4_1/pp_share.py (PPSharePlan cross-PP YOCO shared-KV/indexer-K/candidate replication - missing in 2a828fc212, the PP patch set was BROKEN without it), nvidia/{dspark,vl_model}.py (TP-sharded DSpark on PP; VL PP input_ids propagation), common/engram.py (exact-size cudaHostRegister + pthread pread Engram host shards), ampere/ampere_sparse.py (SM80 SWA metadata builder, no FlashMLA dependency), attention.py (per-head Triton indexer warmup); vllm/v1/attention/backends/mla/{indexer,streamed_workspace}.py (bounded 1M-context streamed indexer workspace ~135 MB vs 5.28 GB), v1/attention/backends/utils.py, v1/attention/ops/mqa_logits_candidates_triton.py (candidate-only indexer scoring for L24/28/32/36), v1/core/kv_cache_utils.py + v1/core/single_type_kv_cache_manager.py + v1/core/sched/{scheduler,output,async_scheduler}.py, v1/structured_output/utils.py, v1/worker/gpu/model_runner.py + v1/worker/gpu/cudagraph_utils.py, v1/worker/gpu/sample/batch_shard.py, v1/worker/gpu/spec_decode/rejection_sampler.py + v1/worker/gpu/spec_decode/dspark/utils.py; plus the initial-release files v1/attention/ops/rocm_aiter_mla_sparse.py (unified 512-dim SM80 sparse MLA), models/deepseek_v4_1/amd/rocm.py (single-CTA TopK+SWA combiner), common/ops/cache_utils.py (bitshift UE8M0 KV gather), model_executor/kernels/mhc/{tilelang,warmup}.py (SM80 mHC + JIT warm), models/deepseek_v4_1/nvidia/model.py (YOCO L20-L39 metadata reuse), model_executor/layers/sparse_attn_indexer.py + model_executor/model_loader/weight_utils.py + model_executor/models/qwen3_dspark.py. Numerics: 100% bit-exact on 4 of the 5 kernel families (0.00e+00 diff); mHC fused kernels <1.9e-3 (BF16 MMA order) and sparse-attn prefill <4.8e-3 - documented per-kernel, no accuracy regression claimed. Author rig also tiers KV to pinned /dev/shm RAM and 512 GiB NVMe (docs/05) - rig infra, not a patch requirement."},"kv_pool_tokens":4648480,"max_context_len":1048576},"models":[{"slug":"deepseek-v4-1-flash","name":"DeepSeek-V4.1-Flash","hf_repo":"deepseek-ai/DeepSeek-V4.1-Flash","quant":"fp8-official-checkpoint-fp8-latent-ue8m0-scales"}],"hardware":[{"slug":"cmp-170hx-x6","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":6}],"generation":"ampere","tags":["ampere","deepinfer","dspark","kernel-port","mhc","mla","sm80","sparse-attention","speculative","yoco"],"tasks":["chat","code"],"aa":{"aa_index":39,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash","aa_index_version":"Index v4.3.2","aa_methodology_version":"Index v4.3.2 (2026-10-18)","aa_published_fetched_at":"2026-10-20 21:49:11.309778-06","aa_family_slug":"deepseek-v4-1-flash","aa_resolved_slug":"deepseek-v4-1-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"claimed_by=submitter T0 (author's production bench on 6x A100/GA100 64 GB, TP2xPP3 + DSpark K5, 1M ctx): 16K cold prefill 4,037 tok/s (peak 4,059) vs 3,349 baseline (+20.5%); 64K cold prefill 4,506.8 vs 3,627.3 (+24.2%); 64K TTFT 10.7 s vs 13.2 s (-18.9%); c1 DSpark decode 160.8 tok/s code / 188.8 tok/s 16K+ long-gen / 65.9 Chinese prose vs 151.8 baseline (+5.9%); c4 170.5 aggregate (41.8/seq, peak 45.3) vs 163.8 (+4.1%); PLP NLL within noise (0.2836/0.5492/0.9123 vs 0.2787/0.5533/0.8827); KV 4,648,480 tokens both sides (zero VRAM overhead). Microbench (baseline->SM80, us): topk+SWA combiner T=1536 1,083.72->48.88 (22.17x, bit-exact); KV gather 48K 263.43->34.47 (7.64x) / 16K 80.10->19.93 (4.02x), bit-exact; mHC T=12 357.70->123.77 (2.89x) / T=6 163.64->125.59 (1.30x); sparse-attn decode B=6 149.64->69.92 (2.14x) / B=24 349.12->167.51 (2.08x), bit-exact; sparse-attn prefill 3,742.80->3,202.11 (1.17x); mHC post_sqrsum 275.27->232.62 (1.18x).\", \"tok_s\": 160.80000000000001136868377216160297393798828125, \"points\": [{\"notes\": \"c1 DSpark K5, Python code median; 16K+ long-gen 188.8 tok/s; Chinese prose median 65.9 (baseline 151.8 code)\", \"concurrency\": 1, \"context_len\": 1048576, \"decode_tok_s\": 160.80000000000001136868377216160297393798828125}, {\"notes\": \"c4 aggregate prose 41.8/seq (peak 45.3); baseline 163.8 (+4.1%)\", \"concurrency\": 4, \"context_len\": 1048576, \"decode_tok_s\": 170.5}, {\"notes\": \"64K cold prefill +24.2% vs baseline 3,627.3 (peak 4,512); TTFT baseline 13,200 ms (-18.9%)\", \"ttft_ms\": 10700, \"concurrency\": 1, \"context_len\": 65536, \"prefill_tok_s\": 4506.8000000000001818989403545856475830078125}, {\"notes\": \"16K cold prefill +20.5% vs baseline 3,349.4 (peak 4,059)\", \"concurrency\": 1, \"context_len\": 16384, \"prefill_tok_s\": 4037}], \"context_len\": 1048576}"}],"perf_points":[{"id":1881,"context_len":16384,"prefill_tok_s":4037,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"16K cold prefill +20.5% vs baseline 3,349.4 (peak 4,059)"},{"id":1882,"context_len":65536,"prefill_tok_s":4506.8000000000001818989403545856475830078125,"decode_tok_s":null,"ttft_ms":10700,"concurrency":1,"claimed_by":"submitter","notes":"64K cold prefill +24.2% vs baseline 3,627.3 (peak 4,512); TTFT baseline 13,200 ms (-18.9%)"},{"id":1883,"context_len":1048576,"prefill_tok_s":null,"decode_tok_s":160.80000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"c1 DSpark K5, Python code median; 16K+ long-gen 188.8 tok/s; Chinese prose median 65.9 (baseline 151.8 code)"},{"id":1884,"context_len":1048576,"prefill_tok_s":null,"decode_tok_s":170.5,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"c4 aggregate prose 41.8/seq (peak 45.3); baseline 163.8 (+4.1%)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5.json"},"requirements":null,"artifacts":[{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/HalfVulpes/Deepinfer-SM80","pin":"HEAD 8895510cae8f8f1733401686e67a4d1e11e7e94b at revision 2026-10-28 (verified: 30 drop-in files; patches/00-vllm-pr56201-sm80-base.patch new +15,359 lines + patches/vllm-sm80-deepinfer.patch modified 30 files/4,289 lines; README+summary JSON match)"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash","pin":"official FP8 checkpoint (no revision pinned upstream)"}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5.json","alternate":"/r/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5","describedby":"/openapi.json"},"updated_at":"2026-10-28 15:48:17.404025-06","created_at":"2026-10-20 21:36:52.669706-06","speeds":{"decode_tok_s":160.80000000000001136868377216160297393798828125,"prefill_tok_s":4506.8000000000001818989403545856475830078125,"leaderboard":"/api/v1/leaderboard.json"}}