localairecipes launch configs for local inference

DeepSeek-V4.1-Flash SM80 DeepInfer kernel port 6x CMP 170HX DSpark K5

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by HalfVulpes (Deepinfer-SM80 port + measurements)↗, DeepSeek-AI (DeepInfer engine architecture + DeepSeek-V4.1-Flash checkpoint)↗, vLLM project (V1 engine + TPxPP runtime)↗, TileLang + OpenAI Triton (SM80 kernel synthesis)↗, FlashInfer + IST-DASLab Marlin (sparse-MLA + MXFP4/W4A16 Ampere GEMM primitives)↗ · listed by localai-recipes

revision r3 · commit ced28259b9 · updated 2 d ago · license Apache-2.0

chat code dspark speculative sm80 ampere deepinfer kernel-port mhc mla sparse-attention yoco

DeepSeek-V4.1-Flash served on six 64 GB SM80 GPUs (GA100 die, owner-classified 170HX) with HalfVulpes' Deepinfer-SM80 patch set: a 30-file vLLM V1 overlay porting DeepSeek's DeepInfer kernel architecture (SM100 tcgen05/TMEM) to Ampere mma.sync tensor cores - unified 512-dim SM80 sparse-MLA decode/prefill kernels with bitshift UE8M0 scale decoding, a bit-exact single-CTA Triton TopK+SWA combiner (22.17x), direct-store KV-cache gather (7.64x @48K), SM80-aligned TileLang mHC routing (2.89x), and YOCO L20-L39 ragged indexer-metadata reuse (19 redundant kernel chains eliminated per chunk). TP2 x PP3 (14/14/12) + DSpark K=5 + 1M context, 4.65M-token GPU-resident KV. Author-measured: 64K cold prefill 4,507 tok/s (+24.2% vs baseline vLLM SM80), 64K TTFT 10.7 s (-18.9%), c1 DSpark decode 160.8 tok/s code / 188.8 tok/s 16K+ long-gen, c4 170.5 aggregate. Microbenchmarks bit-exact (0 diff) or within BF16 noise. Topology guide: PP6 = 2.1x slower single-stream; TP2+TP4 asymmetric hurts Marlin tiles.

Launch command

'git clone https://github.com/HalfVulpes/Deepinfer-SM80 && cd Deepinfer-SM80' 'git clone https://github.com/vllm-project/vllm.git && cd vllm && git fetch origin 79a7108d9aea27ddab99ce1779290d300b17fc23 && git checkout 79a7108d9aea27ddab99ce1779290d300b17fc23' '# Option A: apply BOTH patches in order (00- base first, then DeepInfer-SM80):' 'git apply /path/to/Deepinfer-SM80/patches/00-vllm-pr56201-sm80-base.patch' 'git apply /path/to/Deepinfer-SM80/patches/vllm-sm80-deepinfer.patch' '# Option B: on an existing SM80 base tree (11cf4a67f9 on 79a7108d9a) apply only patches/vllm-sm80-deepinfer.patch, or bind-mount/copy all 30 drop-in files: cp -r vllm_sm80_deepinfer/vllm/* /path/to/site-packages/vllm/' '# verify: python3 benchmarks/microbench_sm80_kernels.py && python3 benchmarks/microbench_sm80_part2.py && python3 benchmarks/microbench_sm80_part3.py' '# serve DeepSeek-V4.1-Flash 6x SM80 64GB: TP2 x PP3 VLLM_PP_LAYER_PARTITION=14,14,12 + DSpark K=5 + max context 1,048,576 (README Quick Start + docs/05 section 1; repo ships no serve script)'

runtime

env: "Author topology on 6x 64 GB: TP2 x PP3 partition 14/14/12 (VLLM_PP_LAYER_PARTITION=14,14,12 per benchmarks/results/sm80_benchmark_summary.json), DSpark K=5 speculative, max context 1,048,576, 4.65M GPU-resident KV. Alternative topologies measured in docs/05: PP6 (TP1xPP6) = 65.5-68.5 tok/s c1 (2.15x slower), TP2+TP4 asymmetric = ~55.5 tok/s unspeculated (moie_intermediate 2304/4=576 hurts Marlin 4.5x128 tiling). Deployment env/args: the patch is engine-internal; no special env vars documented in the README (DSpark handled by vLLM's speculative-config; the mHC/Triton kernels self-configure per SM count)."
notes: 'Drop-in modules (30 files at 8895510cae) touch: vllm/models/deepseek_v4_1/pp_share.py (PPSharePlan cross-PP YOCO shared-KV/indexer-K/candidate replication - missing in 2a828fc212, the PP patch set was BROKEN without it), nvidia/{dspark,vl_model}.py (TP-sharded DSpark on PP; VL PP input_ids propagation), common/engram.py (exact-size cudaHostRegister + pthread pread Engram host shards), ampere/ampere_sparse.py (SM80 SWA metadata builder, no FlashMLA dependency), attention.py (per-head Triton indexer warmup); vllm/v1/attention/backends/mla/{indexer,streamed_workspace}.py (bounded 1M-context streamed indexer workspace ~135 MB vs 5.28 GB), v1/attention/backends/utils.py, v1/attention/ops/mqa_logits_candidates_triton.py (candidate-only indexer scoring for L24/28/32/36), v1/core/kv_cache_utils.py + v1/core/single_type_kv_cache_manager.py + v1/core/sched/{scheduler,output,async_scheduler}.py, v1/structured_output/utils.py, v1/worker/gpu/model_runner.py + v1/worker/gpu/cudagraph_utils.py, v1/worker/gpu/sample/batch_shard.py, v1/worker/gpu/spec_decode/rejection_sampler.py + v1/worker/gpu/spec_decode/dspark/utils.py; plus the initial-release files v1/attention/ops/rocm_aiter_mla_sparse.py (unified 512-dim SM80 sparse MLA), models/deepseek_v4_1/amd/rocm.py (single-CTA TopK+SWA combiner), common/ops/cache_utils.py (bitshift UE8M0 KV gather), model_executor/kernels/mhc/{tilelang,warmup}.py (SM80 mHC + JIT warm), models/deepseek_v4_1/nvidia/model.py (YOCO L20-L39 metadata reuse), model_executor/layers/sparse_attn_indexer.py + model_executor/model_loader/weight_utils.py + model_executor/models/qwen3_dspark.py. Numerics: 100% bit-exact on 4 of the 5 kernel families (0.00e+00 diff); mHC fused kernels <1.9e-3 (BF16 MMA order) and sparse-attn prefill <4.8e-3 - documented per-kernel, no accuracy regression claimed. Author rig also tiers KV to pinned /dev/shm RAM and 512 GiB NVMe (docs/05) - rig infra, not a patch requirement.'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5.json

Engine

Install note: Engine pin chain as stated by the repo at 8895510cae: base = vllm-project/vllm PR #56201 base commit 79a7108d9aea27ddab99ce1779290d300b17fc23 (v0.28.1rc0); SM80 base enablement = haosdent/vllm dsv4-flash-a100 (deepseek-v4-a100-support, commits 62195e9..c3046d1) rebased onto 79a7108d9a, shipped as patches/00-vllm-pr56201-sm80-base.patch (NEW at e375de2, +15,359 lines); DeepInfer-SM80 kernel + YOCO/PP optimizations via patches/vllm-sm80-deepinfer.patch (MODIFIED at 8895510: 30 files, 4,289 lines). Option A (reproducible full build from upstream): git clone vllm; git fetch origin 79a7108d9aea27ddab99ce1779290d300b17fc23; git checkout it; git apply patches/00-vllm-pr56201-sm80-base.patch; then git apply patches/vllm-sm80-deepinfer.patch. Option B (existing SM80 base tree, 11cf4a67f9 on top of 79a7108d9a): apply only patches/vllm-sm80-deepinfer.patch, or copy/bind-mount all 30 drop-in files: cp -r vllm_sm80_deepinfer/vllm/* site-packages/vllm/ (docker bind-mount read-only overlays work - the CMP-recipe pattern). Why vllm-sm80-deepinfer.patch does not apply to latest upstream vllm/main: upstream PR #56741 (cf1584f373) renamed vllm/models/deepseek_v4_1/ to deepseek_v41/, and upstream main ships only SM100 Blackwell + AMD ROCm backends for DeepSeek-V4.1-Flash. Verify with the three in-repo microbenchmarks (microbench_sm80_kernels.py / _part2 / _part3) which print bit-exactness + latency tables; charts via generate_charts.py; full dataset in benchmarks/results/sm80_benchmark_summary.json. Requirements: SM80 (A100 40/64/80 GB, A800, GA100) or SM86 (RTX 3090/3090 Ti, RTX A6000, A40/A10/A30 - all kernels <=48 KB SRAM/CTA); vLLM V1 engine; Triton + TileLang JIT (startup cache-warm included). Base checkpoint: official deepseek-ai/DeepSeek-V4.1-Flash FP8.

Model

Hardware

Context

  • max context: 1M tokens

KV pool

  • 4,648,480 tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
16,38414037 tok/s——reported
65,53614506.8 tok/s—10700 msreported
1,048,5761—160.8 tok/s—reported
1,048,5764—170.5 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesclaimed_by=submitter T0 (author's production bench on 6x A100/GA100 64 GB, TP2xPP3 + DSpark K5, 1M ctx): 16K cold prefill 4,037 tok/s (peak 4,059) vs 3,349 baseline (+20.5%); 64K cold prefill 4,506.8 vs 3,627.3 (+24.2%); 64K TTFT 10.7 s vs 13.2 s (-18.9%); c1 DSpark decode 160.8 tok/s code / 188.8 tok/s 16K+ long-gen / 65.9 Chinese prose vs 151.8 baseline (+5.9%); c4 170.5 aggregate (41.8/seq, peak 45.3) vs 163.8 (+4.1%); PLP NLL within noise (0.2836/0.5492/0.9123 vs 0.2787/0.5533/0.8827); KV 4,648,480 tokens both sides (zero VRAM overhead). Microbench (baseline->SM80, us): topk+SWA combiner T=1536 1,083.72->48.88 (22.17x, bit-exact); KV gather 48K 263.43->34.47 (7.64x) / 16K 80.10->19.93 (4.02x), bit-exact; mHC T=12 357.70->123.77 (2.89x) / T=6 163.64->125.59 (1.30x); sparse-attn decode B=6 149.64->69.92 (2.14x) / B=24 349.12->167.51 (2.08x), bit-exact; sparse-attn prefill 3,742.80->3,202.11 (1.17x); mHC post_sqrsum 275.27->232.62 (1.18x).

Downloads

Revision notes

Community ingest of HalfVulpes' port (Apache-2.0). All numbers claimed_by=submitter T0 (microbench + E2E harness + raw JSON in-repo) - import via POST /api/v1/benchmarks to upgrade. HARDWARE CAVEAT: validated on GA100/A100 64 GB SM80; CMP 170HX is the same GA100 silicon at 140 W (74 SMs) but the author did not measure CMP-class power/link configs, and the 6x rig uses PCIe x16 Gen4/Gen5 - do NOT present the numbers as CMP-measured. Engine is a vLLM patch, NOT a new engine enum. Upstream deepseek-ai/DeepInfer itself stays SM100/Ascend-only and is NOT separately ingestable. Discord corroboration: cmpunlocker #testing-general / #clanker-talk 2026-10-20 17:12-17:22Z ('160.8-188.8 tok/s... even pp6 65.5-68.5... double of GLM 5.3... multi-model') and reala/segmond linking the repo (not re-verified). docs/05's KV math section names the -1017 revision while README/summary serve the base V4.1-Flash checkpoint. A 170hx-x4 CMP variant is the natural follow-up once someone measures it. HARDWARE RECLASSIFICATION (owner ruling 2026-10-21): the author labeled the rig 6x A100; 64 GB SM80 cards are CMP 170HX (GA100 die) - real A100 is 40/80 GB and never 64 - so the recipe is owner-classified as 6x CMP 170HX; the -6x-a100- slug keeps its historical token. PIN REVISION (kanban t_ef4644ee, owner go Casey 2026-10-28, parent t_ab1b8c82): pin 2a828fc2122b -> 8895510cae8f (+3 commits: 8895510 fix(pp) pp_share.py + TP2xPP3/TP4xPP2 + DSpark + 1M indexer modules; a11809d 8-GPU TP4xPP2 benchmarks; e375de2 base vLLM PR #56201 (79a7108d9a) + haosdent/vllm SM80 fork link + 00-vllm-pr56201-sm80-base.patch). Launch args re-transcribed from the repo README (Base vLLM Version, Fork Lineage & Quick Start Option A/B) + docs/05_TOPOLOGY_1M_CONTEXT_ANALYSIS_SM80.md section 1: the repo ships no serve script; previously-published args applied only patches/vllm-sm80-deepinfer.patch with no SM80 base, stale at 8895510cae (patch no longer applies to upstream main: PR #56741 rename + SM80 backend missing). Scaling rows: omitted - recipe publishes no scaling table (8-GPU TP4xPP2 data: 13,306,380 KV / 12.69x1M, C=1,2,4,12 decode+prefill, left for a future 8-GPU variant card). UI chart charts/05_8gpu_tp4pp2_scaling_1_2_4_12.png deliberately not attached (t_ab1b8c82). Newer-commits-deferred: NONE - live GitHub 2026-10-28 main HEAD == 8895510cae8f (4 commits total); haosdent/vllm dsv4-flash-a100 branch head moved to 12810046c799 (informational; recipe pins the bundled patch file, not the branch). Perf fields untouched (measurement provenance separate lane).

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)