{"slug":"deepseek-v4-1-flash-1017-on-1x-cmp-170hx-xtier-engram-nvme-16k","title":"DeepSeek-V4.1-Flash-1017 on 1x CMP 170HX XTier Engram NVMe 16K","summary":"DeepSeek-V4.1-Flash-1017 (491 GB, no requantization) served on a single 64 GB CMP 170HX. XTier (v0.1.1, unmodified upstream) streams the 40x384 routed experts from CPU/NVMe, the 189 GB Engram table is mmapped from NVMe, and SGLang (main @ 6162b071) runs MLA attention, KV pool and scheduling. 1M-token KV pool allocates; measured usable single-request prompt is ~16K (14,666 tokens succeeds, 32K reproducibly dies) - this is a ~16K-usable recipe, not a 1M-context recipe. ~10-11 tok/s warm single-stream decode, 47-64 tok/s prefill. recipe by mamutou.","license":"MIT","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"b65375e7cd73c7e516c510c7ae0f91ac5449cab6","content_hash":"317b1bdb19cb239baec544d473b9a1b6eaa1758c","created_at":"2026-11-05 21:52:31.750822-07","notes":"Recipe repo pinned at HEAD f9a8334e9f (MIT, Copyright (c) 2026 mamutou). Launch argv + env matrix transcribed verbatim from deploy/ds41.sh at that sha; all numbers from results/raw.jsonl + docs/BENCH.md (measured 2026-11-03). corrections_to_card (vs scout card t_d09620cb): card said served_model_name DeepSeek-V4.1-Flash-1017 - ds41.sh default is 'DeepSeek-V4.1-Flash' (SERVED_NAME env override); card's config 'slots: 64' is XTIER_CACHE_SLOTS, not an SGLang slot flag - SGLang --max-running-requests is 2; card's engine_source sidecars/config fields adapted to the live submission schema (engine.install_note + config.env + artifacts); card listed model family deepseek-v4-1-flash - submitted as deepseek-v4-1-flash-1017 (both exist; all sibling -1017 recipes use the -1017 family, correct facet grouping for this checkpoint). Do not promise long context: BENCH.md documents ~16K usable prompt despite the 1M KV pool; max_context_len set to the measured 14666, kv_pool_tokens to the 1M pool. XTier and SGLang both used UNMODIFIED (docs/UPSTREAM.md) - no fork, no patch overlay. Docs are Chinese-language; read the scripts, not the prose. Standing git-watch: mamutou/deepseek-v41-flash-170hx @ f9a8334e9f + HF deepseek-ai/DeepSeek-V4.1-Flash-1017 @ 1bae0e10 + a775828b-dot/xtier @ 4b1e7666."},"engine_source":{"type":"release","git_url":"https://github.com/sgl-project/sglang","base_engine":"sglang","pip_package":null,"base_version":"sglang main @ 6162b071","image_digest":null,"install_note":"SGLang is only the attention/scheduling lane: pip install -e '.[srt]' from sgl-project/sglang @ 6162b071 (Python 3.12 venv). Expert streaming is delegated to the SEPARATE external engine XTier (a775828b-dot/xtier @ 4b1e7666, v0.1.1, AVX2 CPU backend, used unmodified; pip install -e . builds libxtier_engine_avx2.so). Not installable as a single pip package - both checkouts must be pinned; CUDA 13.4.92 via the venv nvidia cu13 tree. Required env: XTIER=1 plus the full XTier env matrix (see config.env), FLASHINFER_CUDA_ARCH_LIST=8.0 and TVM_FFI_CUDA_ARCH_LIST=8.0 (GA100 sm80; upstream default 12.0 fails at first token), HF_HUB_OFFLINE=1, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.","pinned_commit":"6162b071994fcf67090ebe3d45f7aebcdf03f4d7"},"credit":[{"name":"mamutou (recipe author, MIT)","url":"https://github.com/mamutou/deepseek-v41-flash-170hx"},{"name":"XTier v0.1.1 @4b1e7666 - routed-expert streaming engine","url":"https://github.com/a775828b-dot/xtier"},{"name":"SGLang main @6162b071 - attention/scheduler lane","url":"https://github.com/sgl-project/sglang"},{"name":"deepseek-ai/DeepSeek-V4.1-Flash-1017 - upstream weights","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"env":["XTIER=1","XTIER_THREADS=16","XTIER_MAIN_CPUS=auto","XTIER_PROBE_DISPATCH_CPU=1","XTIER_SPIN_COUNT=1048576","XTIER_DYNAMIC=2","XTIER_DOWN_BF16=1","XTIER_DOWN_GRAIN=24","XTIER_THP=0","XTIER_ENGRAM_ROWS=nvme","XTIER_ENGRAM_MODEL=$M","XTIER_ENGRAM_CACHE_GIB=2","XTIER_CACHE_SLOTS=64","XTIER_CACHE_EVERY=4","XTIER_CACHE_SWAPS=16","XTIER_CACHE_PROMPT=1","XTIER_CACHE_PROMPT_KEEP=0.85","XTIER_CACHE_PROMPT_ROUNDS=8","XTIER_CACHE_PROMPT_SWAPS=512","XTIER_CACHE_LOG=500","XTIER_COLD=1","XTIER_COLD_PATHS=$HOME/models/ds41-cold","XTIER_COLD_FRACTION=0.85","XTIER_FAST_LOAD=1","XTIER_PREFILL_FP4=1","XTIER_PREFILL_UNPACK=1","XTIER_PREFILL_PREFETCH=1","XTIER_PREFILL_OVERLAP=1","XTIER_PREFILL_PREFETCH_CHUNKS=6","XTIER_PREFILL_EXPERT_CHUNK=8","XTIER_PREFILL_ROUTE_ACCUM=1","XTIER_CPU_PREFILL_BATCH=2048","XTIER_GPU_PREFILL_MIN=1024","XTIER_HYBRID_PREFILL_MAX=1024","XTIER_SPLIT_PREFILL_MAX=6144","XTIER_DOWN_PRECISION=tf32","XTIER_PREFILL_TILE_M=64","XTIER_GATE_BN=128","XTIER_GATE_LAUNCH=8,3","XTIER_DOWN_BN=128","XTIER_DOWN_BK=64","XTIER_DOWN_LAUNCH=8,3","FLASHINFER_CUDA_ARCH_LIST=8.0","TVM_FFI_CUDA_ARCH_LIST=8.0","TVM_FFI_GPU_BACKEND=cuda","HF_HUB_OFFLINE=1","PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True"],"notes":"Launch + env matrix transcribed verbatim from deploy/ds41.sh at recipe repo HEAD f9a8334e9f (systemd user unit deploy/ds41.service wraps it). Model dir $M=ds41-flash-1017 (491 GB bf16, 57 shards, 40 layers x 384 experts top-6, H=5120 I=2304); cold expert pool $HOME/models/ds41-cold on NVMe. 189 GB Engram table is NVMe-mmap (no VRAM/RAM residency; 2 GiB row cache). VRAM 60443/65536 MiB (92%). Cold start 150-330 s (matrix.jsonl boot_s 150; BENCH.md 330 s incl XTier pool init). Host: i9-12900 16C/24T AVX2+VNNI no AVX-512/AMX, 63 GB DDR5 dual-channel, single 2 TB 990 PRO, 63 GB swap. THREADS=16 is a hard top (17+ rejected by XTIER_NODE_CORES topology check). SLOTS=64 here means XTIER_CACHE_SLOTS (resident expert slots), NOT SGLang request slots; --max-running-requests is 2. chunked-prefill 2048 is the sweet spot: 1024 prefill-doubles the fixed overhead (26-30 tok/s), 4096 OOMs ~8500-token prompts. Spec decode is net-negative on this box (DSPARK 4.13-7.37 vs baseline 10.1; MTP/EAGLE unloadable - checkpoint is DSPARK layout, missing e_proj/h_proj/shared_head/enorm).","launch":["python","-m","sglang.launch_server","--model-path","$HOME/models/ds41-flash-1017","--served-model-name","DeepSeek-V4.1-Flash","--host","0.0.0.0","--port","30010","--tp-size","1","--trust-remote-code","--skip-server-warmup","--max-running-requests","2","--chunked-prefill-size","2048","--max-prefill-tokens","2048","--context-length","1048576","--max-total-tokens","1048576","--mem-fraction-static","0.95","--disable-shared-experts-fusion","--cuda-graph-backend-prefill","disabled","--fp8-gemm-backend","auto","--reasoning-parser","deepseek-v41","--cuda-graph-backend-decode","full","--watchdog-timeout","1200","--enable-metrics","--weight-loader-drop-cache-after-load"],"kv_pool_tokens":1048576,"max_context_len":14666,"storage_requirements":{"note":"491 GB bf16 weights (57 shards) + 189 GB Engram NVMe-mmap + XTier cold expert pool, all on the reference box's single 2 TB 990 PRO; 63 GB DDR5 host RAM, 63 GB swap; Engram row cache 2 GiB.","ram_gb":63,"nvme_gb":2048}},"models":[{"slug":"deepseek-v4-1-flash-1017","name":"DeepSeek-V4.1-Flash-1017","hf_repo":"deepseek-ai/DeepSeek-V4.1-Flash-1017","quant":"bf16-unrequantized-57-shards"}],"hardware":[{"slug":"nvidia-cmp-170hx","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":1}],"generation":"ampere","tags":["cpu-offload","engram","expert-streaming","moe","nvme-mmap","single-gpu","sm80","unquantized","xtier"],"tasks":["chat"],"aa":{"aa_index":45,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash-1017","aa_index_version":"Index v4.3.2 (2026-10-18)","aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-19 12:00:00-06","aa_family_slug":"deepseek-v4-1-flash-1017","aa_resolved_slug":"deepseek-v4-1-flash-1017"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured on the reference box 2026-11-03 (results/raw.jsonl, bench/bench_ds41.py, server-side runs, greedy ignore_eos). '1M' always means the KV pool allocated (--context-length/--max-total-tokens 1048576), NOT a usable 1M prompt: 14,666-token prompt succeeds, 32,768 reproducibly dies (RemoteDisconnected twice, once with 4489 MiB free - a hard limit, not VRAM). Decode rows are warm steady-state (cold run1 7.35, run2 10.01, run3 10.13 tok/s). Prefill cold-page-cache rows 471-14666 tokens; warm rows 1857/3641/7340 listed too (59.5/61.4/64.4). Prefill fits t ~= 9.8 s + 16.1 ms/token above 1850 tokens (platform ceiling ~62 tok/s). Concurrency rows are AGGREGATE tok/s (per-stream: 5.84 / 3.86 / 2.30). All 12 (ctx,conc) tuples unique - see point notes.\", \"tok_s\": 10.1300000000000007815970093361102044582366943359375, \"points\": [{\"notes\": \"single-stream decode run 3 (warm); raw.jsonl runs: 7.35 cold / 10.01 / 10.13 tok/s, 200 output tokens, 12-token prompt\", \"concurrency\": 1, \"context_len\": 12, \"decode_tok_s\": 10.1300000000000007815970093361102044582366943359375}, {\"notes\": \"aggregate 2 streams x 128 tokens; per-stream 5.84, wall 21.9 s (raw.jsonl)\", \"concurrency\": 2, \"context_len\": 12, \"decode_tok_s\": 11.6899999999999995026200849679298698902130126953125}, {\"notes\": \"aggregate 4 streams; per-stream 3.86, wall 33.12 s (raw.jsonl)\", \"concurrency\": 4, \"context_len\": 12, \"decode_tok_s\": 15.46000000000000085265128291212022304534912109375}, {\"notes\": \"aggregate 8 streams; per-stream 2.30, wall 55.63 s (raw.jsonl); matrix base config 17.72\", \"concurrency\": 8, \"context_len\": 12, \"decode_tok_s\": 18.410000000000000142108547152020037174224853515625}, {\"notes\": \"cold TTFT, fresh random text, no prefix cache\", \"ttft_ms\": 27170, \"context_len\": 471, \"prefill_tok_s\": 17.300000000000000710542735760100185871124267578125}, {\"notes\": \"cold TTFT\", \"ttft_ms\": 42030, \"context_len\": 923, \"prefill_tok_s\": 22}, {\"notes\": \"cold TTFT\", \"ttft_ms\": 39120, \"context_len\": 1850, \"prefill_tok_s\": 47.2999999999999971578290569595992565155029296875}, {\"notes\": \"cold TTFT\", \"ttft_ms\": 65760, \"context_len\": 3684, \"prefill_tok_s\": 56}, {\"notes\": \"cold TTFT\", \"ttft_ms\": 128220, \"context_len\": 7369, \"prefill_tok_s\": 57.5}, {\"notes\": \"cold TTFT; largest prompt that succeeds - 32,768 reproducibly dies (RemoteDisconnected)\", \"ttft_ms\": 245480, \"context_len\": 14666, \"prefill_tok_s\": 59.7000000000000028421709430404007434844970703125}, {\"notes\": \"warm (weights+page-cache hot) counterpart of the 1850 row\", \"ttft_ms\": 31190, \"context_len\": 1857, \"prefill_tok_s\": 59.5}, {\"notes\": \"warm counterpart of the 3684 row\", \"ttft_ms\": 59300, \"context_len\": 3641, \"prefill_tok_s\": 61.39999999999999857891452847979962825775146484375}, {\"notes\": \"warm counterpart of the 7369 row\", \"ttft_ms\": 113940, \"context_len\": 7340, \"prefill_tok_s\": 64.400000000000005684341886080801486968994140625}]}"}],"perf_points":[{"id":2495,"context_len":12,"prefill_tok_s":null,"decode_tok_s":10.1300000000000007815970093361102044582366943359375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode run 3 (warm); raw.jsonl runs: 7.35 cold / 10.01 / 10.13 tok/s, 200 output tokens, 12-token prompt"},{"id":2496,"context_len":12,"prefill_tok_s":null,"decode_tok_s":11.6899999999999995026200849679298698902130126953125,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"aggregate 2 streams x 128 tokens; per-stream 5.84, wall 21.9 s (raw.jsonl)"},{"id":2497,"context_len":12,"prefill_tok_s":null,"decode_tok_s":15.46000000000000085265128291212022304534912109375,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"aggregate 4 streams; per-stream 3.86, wall 33.12 s (raw.jsonl)"},{"id":2498,"context_len":12,"prefill_tok_s":null,"decode_tok_s":18.410000000000000142108547152020037174224853515625,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate 8 streams; per-stream 2.30, wall 55.63 s (raw.jsonl); matrix base config 17.72"},{"id":2499,"context_len":471,"prefill_tok_s":17.300000000000000710542735760100185871124267578125,"decode_tok_s":null,"ttft_ms":27170,"concurrency":null,"claimed_by":"submitter","notes":"cold TTFT, fresh random text, no prefix cache"},{"id":2500,"context_len":923,"prefill_tok_s":22,"decode_tok_s":null,"ttft_ms":42030,"concurrency":null,"claimed_by":"submitter","notes":"cold TTFT"},{"id":2501,"context_len":1850,"prefill_tok_s":47.2999999999999971578290569595992565155029296875,"decode_tok_s":null,"ttft_ms":39120,"concurrency":null,"claimed_by":"submitter","notes":"cold TTFT"},{"id":2505,"context_len":1857,"prefill_tok_s":59.5,"decode_tok_s":null,"ttft_ms":31190,"concurrency":null,"claimed_by":"submitter","notes":"warm (weights+page-cache hot) counterpart of the 1850 row"},{"id":2506,"context_len":3641,"prefill_tok_s":61.39999999999999857891452847979962825775146484375,"decode_tok_s":null,"ttft_ms":59300,"concurrency":null,"claimed_by":"submitter","notes":"warm counterpart of the 3684 row"},{"id":2502,"context_len":3684,"prefill_tok_s":56,"decode_tok_s":null,"ttft_ms":65760,"concurrency":null,"claimed_by":"submitter","notes":"cold TTFT"},{"id":2507,"context_len":7340,"prefill_tok_s":64.400000000000005684341886080801486968994140625,"decode_tok_s":null,"ttft_ms":113940,"concurrency":null,"claimed_by":"submitter","notes":"warm counterpart of the 7369 row"},{"id":2503,"context_len":7369,"prefill_tok_s":57.5,"decode_tok_s":null,"ttft_ms":128220,"concurrency":null,"claimed_by":"submitter","notes":"cold TTFT"},{"id":2504,"context_len":14666,"prefill_tok_s":59.7000000000000028421709430404007434844970703125,"decode_tok_s":null,"ttft_ms":245480,"concurrency":null,"claimed_by":"submitter","notes":"cold TTFT; largest prompt that succeeds - 32,768 reproducibly dies (RemoteDisconnected)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-on-1x-cmp-170hx-xtier-engram-nvme-16k.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-on-1x-cmp-170hx-xtier-engram-nvme-16k.json"},"requirements":null,"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/mamutou/deepseek-v41-flash-170hx","pin":"f9a8334e9f43f6a193605e16c77c4e95e375e1ae"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017","pin":"1bae0e1088993b9b82de08f59de5be5a34e42170"},{"kind":"other","pointer_type":"git_commit","pointer":"https://github.com/a775828b-dot/xtier","pin":"4b1e76665a9ce7585b7327fc929f8236a5b3b56c"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/sgl-project/sglang","pin":"6162b071994fcf67090ebe3d45f7aebcdf03f4d7"}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-on-1x-cmp-170hx-xtier-engram-nvme-16k.json","alternate":"/r/deepseek-v4-1-flash-1017-on-1x-cmp-170hx-xtier-engram-nvme-16k","describedby":"/openapi.json"},"updated_at":"2026-11-05 21:52:31.750822-07","created_at":"2026-11-05 21:44:32.432414-07","speeds":{"decode_tok_s":10.0999999999999996447286321199499070644378662109375,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}