localairecipes launch configs for local inference

DeepSeek-V4.1-Flash-1017 on 1x CMP 170HX XTier Engram NVMe 16K

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by mamutou (recipe author, MIT)↗, XTier v0.1.1 @4b1e7666 - routed-expert streaming engine↗, SGLang main @6162b071 - attention/scheduler lane↗, deepseek-ai/DeepSeek-V4.1-Flash-1017 - upstream weights↗ · listed by localai-recipes

revision r2 · commit b65375e7cd · updated 2 h ago · license MIT

chat cpu-offload single-gpu engram nvme-mmap sm80 moe xtier expert-streaming unquantized

DeepSeek-V4.1-Flash-1017 (491 GB, no requantization) served on a single 64 GB CMP 170HX. XTier (v0.1.1, unmodified upstream) streams the 40x384 routed experts from CPU/NVMe, the 189 GB Engram table is mmapped from NVMe, and SGLang (main @ 6162b071) runs MLA attention, KV pool and scheduling. 1M-token KV pool allocates; measured usable single-request prompt is ~16K (14,666 tokens succeeds, 32K reproducibly dies) - this is a ~16K-usable recipe, not a 1M-context recipe. ~10-11 tok/s warm single-stream decode, 47-64 tok/s prefill. recipe by mamutou.

Launch command

python -m sglang.launch_server --model-path '$HOME/models/ds41-flash-1017' --served-model-name DeepSeek-V4.1-Flash --host 0.0.0.0 --port 30010 --tp-size 1 --trust-remote-code --skip-server-warmup --max-running-requests 2 --chunked-prefill-size 2048 --max-prefill-tokens 2048 --context-length 1048576 --max-total-tokens 1048576 --mem-fraction-static 0.95 --disable-shared-experts-fusion --cuda-graph-backend-prefill disabled --fp8-gemm-backend auto --reasoning-parser deepseek-v41 --cuda-graph-backend-decode full --watchdog-timeout 1200 --enable-metrics --weight-loader-drop-cache-after-load

env

- XTIER=1
- XTIER_THREADS=16
- XTIER_MAIN_CPUS=auto
- XTIER_PROBE_DISPATCH_CPU=1
- XTIER_SPIN_COUNT=1048576
- XTIER_DYNAMIC=2
- XTIER_DOWN_BF16=1
- XTIER_DOWN_GRAIN=24
- XTIER_THP=0
- XTIER_ENGRAM_ROWS=nvme
- XTIER_ENGRAM_MODEL=$M
- XTIER_ENGRAM_CACHE_GIB=2
- XTIER_CACHE_SLOTS=64
- XTIER_CACHE_EVERY=4
- XTIER_CACHE_SWAPS=16
- XTIER_CACHE_PROMPT=1
- XTIER_CACHE_PROMPT_KEEP=0.85
- XTIER_CACHE_PROMPT_ROUNDS=8
- XTIER_CACHE_PROMPT_SWAPS=512
- XTIER_CACHE_LOG=500
- XTIER_COLD=1
- XTIER_COLD_PATHS=$HOME/models/ds41-cold
- XTIER_COLD_FRACTION=0.85
- XTIER_FAST_LOAD=1
- XTIER_PREFILL_FP4=1
- XTIER_PREFILL_UNPACK=1
- XTIER_PREFILL_PREFETCH=1
- XTIER_PREFILL_OVERLAP=1
- XTIER_PREFILL_PREFETCH_CHUNKS=6
- XTIER_PREFILL_EXPERT_CHUNK=8
- XTIER_PREFILL_ROUTE_ACCUM=1
- XTIER_CPU_PREFILL_BATCH=2048
- XTIER_GPU_PREFILL_MIN=1024
- XTIER_HYBRID_PREFILL_MAX=1024
- XTIER_SPLIT_PREFILL_MAX=6144
- XTIER_DOWN_PRECISION=tf32
- XTIER_PREFILL_TILE_M=64
- XTIER_GATE_BN=128
- 'XTIER_GATE_LAUNCH=8,3'
- XTIER_DOWN_BN=128
- XTIER_DOWN_BK=64
- 'XTIER_DOWN_LAUNCH=8,3'
- FLASHINFER_CUDA_ARCH_LIST=8.0
- TVM_FFI_CUDA_ARCH_LIST=8.0
- TVM_FFI_GPU_BACKEND=cuda
- HF_HUB_OFFLINE=1
- 'PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True'

notes

'Launch + env matrix transcribed verbatim from deploy/ds41.sh at recipe repo HEAD f9a8334e9f (systemd user unit deploy/ds41.service wraps it). Model dir $M=ds41-flash-1017 (491 GB bf16, 57 shards, 40 layers x 384 experts top-6, H=5120 I=2304); cold expert pool $HOME/models/ds41-cold on NVMe. 189 GB Engram table is NVMe-mmap (no VRAM/RAM residency; 2 GiB row cache). VRAM 60443/65536 MiB (92%). Cold start 150-330 s (matrix.jsonl boot_s 150; BENCH.md 330 s incl XTier pool init). Host: i9-12900 16C/24T AVX2+VNNI no AVX-512/AMX, 63 GB DDR5 dual-channel, single 2 TB 990 PRO, 63 GB swap. THREADS=16 is a hard top (17+ rejected by XTIER_NODE_CORES topology check). SLOTS=64 here means XTIER_CACHE_SLOTS (resident expert slots), NOT SGLang request slots; --max-running-requests is 2. chunked-prefill 2048 is the sweet spot: 1024 prefill-doubles the fixed overhead (26-30 tok/s), 4096 OOMs ~8500-token prompts. Spec decode is net-negative on this box (DSPARK 4.13-7.37 vs baseline 10.1; MTP/EAGLE unloadable - checkpoint is DSPARK layout, missing e_proj/h_proj/shared_head/enorm).'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-on-1x-cmp-170hx-xtier-engram-nvme-16k.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-on-1x-cmp-170hx-xtier-engram-nvme-16k.json

Engine

Install note: SGLang is only the attention/scheduling lane: pip install -e '.[srt]' from sgl-project/sglang @ 6162b071 (Python 3.12 venv). Expert streaming is delegated to the SEPARATE external engine XTier (a775828b-dot/xtier @ 4b1e7666, v0.1.1, AVX2 CPU backend, used unmodified; pip install -e . builds libxtier_engine_avx2.so). Not installable as a single pip package - both checkouts must be pinned; CUDA 13.4.92 via the venv nvidia cu13 tree. Required env: XTIER=1 plus the full XTier env matrix (see config.env), FLASHINFER_CUDA_ARCH_LIST=8.0 and TVM_FFI_CUDA_ARCH_LIST=8.0 (GA100 sm80; upstream default 12.0 fails at first token), HF_HUB_OFFLINE=1, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.

Model

Hardware

Context

  • max context: 14,666 tokens

KV pool

  • 1M tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
121—10.1 tok/s—reported
122—11.7 tok/s—reported
124—15.5 tok/s—reported
128—18.4 tok/s—reported
471—17.3 tok/s—27170 msreported
923—22 tok/s—42030 msreported
1,850—47.3 tok/s—39120 msreported
1,857—59.5 tok/s—31190 msreported
3,641—61.4 tok/s—59300 msreported
3,684—56 tok/s—65760 msreported
7,340—64.4 tok/s—113940 msreported
7,369—57.5 tok/s—128220 msreported
14,666—59.7 tok/s—245480 msreported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured on the reference box 2026-11-03 (results/raw.jsonl, bench/bench_ds41.py, server-side runs, greedy ignore_eos). '1M' always means the KV pool allocated (--context-length/--max-total-tokens 1048576), NOT a usable 1M prompt: 14,666-token prompt succeeds, 32,768 reproducibly dies (RemoteDisconnected twice, once with 4489 MiB free - a hard limit, not VRAM). Decode rows are warm steady-state (cold run1 7.35, run2 10.01, run3 10.13 tok/s). Prefill cold-page-cache rows 471-14666 tokens; warm rows 1857/3641/7340 listed too (59.5/61.4/64.4). Prefill fits t ~= 9.8 s + 16.1 ms/token above 1850 tokens (platform ceiling ~62 tok/s). Concurrency rows are AGGREGATE tok/s (per-stream: 5.84 / 3.86 / 2.30). All 12 (ctx,conc) tuples unique - see point notes.

Downloads

Revision notes

Recipe repo pinned at HEAD f9a8334e9f (MIT, Copyright (c) 2026 mamutou). Launch argv + env matrix transcribed verbatim from deploy/ds41.sh at that sha; all numbers from results/raw.jsonl + docs/BENCH.md (measured 2026-11-03). corrections_to_card (vs scout card t_d09620cb): card said served_model_name DeepSeek-V4.1-Flash-1017 - ds41.sh default is 'DeepSeek-V4.1-Flash' (SERVED_NAME env override); card's config 'slots: 64' is XTIER_CACHE_SLOTS, not an SGLang slot flag - SGLang --max-running-requests is 2; card's engine_source sidecars/config fields adapted to the live submission schema (engine.install_note + config.env + artifacts); card listed model family deepseek-v4-1-flash - submitted as deepseek-v4-1-flash-1017 (both exist; all sibling -1017 recipes use the -1017 family, correct facet grouping for this checkpoint). Do not promise long context: BENCH.md documents ~16K usable prompt despite the 1M KV pool; max_context_len set to the measured 14666, kv_pool_tokens to the 1M pool. XTier and SGLang both used UNMODIFIED (docs/UPSTREAM.md) - no fork, no patch overlay. Docs are Chinese-language; read the scripts, not the prose. Standing git-watch: mamutou/deepseek-v41-flash-170hx @ f9a8334e9f + HF deepseek-ai/DeepSeek-V4.1-Flash-1017 @ 1bae0e10 + a775828b-dot/xtier @ 4b1e7666.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)