{"slug":"deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81","title":"DeepSeek-V4.1-Flash-1017 Fidelity-First 4x CMP 170HX alanoo81","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four changes: routed experts of layers 18-22 grafted 2-bit to 3-bit (LibertAIDAI EXL3 3.5bpw shards 00020-00024-of-00044 into the 2.9bpw pack, all 40 layers 3-bit); no DSpark drafter (SPEC_TOKENS=0, frees ~2.8 GiB/card for KV); Engram tables genuinely loaded (ENGRAM_STORAGE=ssd from a 200 GiB RAM tmpfs); gpu_memory_utilization 0.98, KV pool 2,717,420 tokens at 262,144 context. Author-measured: 300/300 exact identifier copies, byte-identical repeats; perplexity -1.80% overall / -4.45% French vs the 2.9bpw pack (95% CIs exclude zero); ~105 tok/s decode c1, 449-560 tok/s aggregate c8; cold prefill 2,514 tok/s median. Documents a silent failure: leaving DSV41_ENGRAM_STORAGE unset (host mode) leaves the Engram tables unfilled - layers 1 and 14 act as identity functions, 15/20 copies, temp-0 output drift, no error.","license":"MIT (recipe); engine image contains AGPL-3.0-only vllm-exl3","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"5af1460f4a2dce4f590f6fdfcd108119285e3e7c","content_hash":"67eb1810e7eea071e85e8edab747fd98247dd872","created_at":"2026-10-29 21:57:43.459028-06","notes":"Community ingest by localai-recipes from alanoo81's repo @ da967740 (single commit, verified live 2026-10-30; HEAD at sweep == HEAD at ingest). Recipe by alanoo81 - a fidelity-first variant of GNmac-lab 1.2.0 @ 2ad5741ab1 with four changes (3-bit graft layers 18-22, no DSpark, Engram genuinely loaded, 0.98). KEY FINDING (silent-failure disclosure, the most valuable part of this recipe): DSV41_ENGRAM_STORAGE unset / host mode fails SILENTLY. The stack reserves 2 x 23.6 GiB pinned per rank but never fills it; the hybrid index contains only the small Engram tensors (q, k, wkv), not engram.embed.*; vLLM skips its missing-weights check when a quantization method is set. Symptoms: 15/20 exact copies, identifiers with dropped _ separators, output changing between runs at temperature 0 - no error, and decode speed is unchanged, so benchmarks do not catch it. The upstream author never tested host mode; the documented path is ssd. Startup log data/logs/01-first-start-engram-NOT-loaded.txt; those benchmarks kept as data/bench/INVALID-no-engram*. Fix: ENGRAM_STORAGE=ssd reading a 200 GiB tmpfs (2 x ~101.5 GB shards, ~189 GiB); after the fix copy test 20/20 then 100/100 x 3 passes, byte-identical repeats. NOT a dupe: distinct launch from all three existing DS41 4x-170HX recipes (GNmac EXL3 2.9bpw TC-MoE, Morrowmake hybrid, EXL3 3bpw TC-MoE) - English/French documented, no drafter, RAM-tmpfs Engram, 3-bit graft at 0.98. License: repo MIT (LICENSE 1,065 B); the built engine image contains AGPL-3.0-only vllm-exl3 code per NOTICE.md - do not claim the image is MIT. All perf/fidelity numbers author-reported (T0). corrections_to_card: none - README 17,317 B + README.fr.md, NOTICE.md 5,759 B, MIT LICENSE, HEAD, and every quoted figure match the repo at the pin; the c8 speed rows published per-suite (559.7/482.4/449.2) and the card's 505 aggregate mid-range are both carried (505 as the aggregate headline with the published range 449-560 in notes)."},"engine_source":{"type":"fork","git_url":"https://github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx","base_engine":"vllm","pip_package":null,"base_version":"GNmac-lab recipe 1.2.0 @ 2ad5741ab1, patched","image_digest":null,"install_note":"Local image ds41-cmp170hx:1.2.0-fidelity built via the upstream recipe's docker/build.sh at 2ad5741ab1 (build ~20 min, image 17.8 GB; the patches do not touch the Dockerfile - the image is the upstream 1.2.0 image under another tag). Clone GNmac-lab/deepseek-v41-flash-1017-cmp170hx @ 2ad5741ab1, apply this repo's recipe/*.patch with patch -p1, copy recipe/tools/*, recipe/engram-ram.sh and recipe/env.example (as .env), run tools/download-pinned.sh (~428 GiB, pinned revisions), tools/make_hybrid_graft.py <MODELS_DIR> DeepSeek-V4.1-Flash-1017-hybrid k3, then ./start.sh (engram-ram.sh then serve.sh; first start ~10 min kernel compile). Run in Docker with GPUs passed via CDI (--device nvidia.com/gpu=all); NVIDIA open driver 610.43.02 from a cmpunlocker fork with a P2P patch; bc, curl, python3, git, patch on the host. WARNING: the built image contains AGPL-3.0-only vllm-exl3 code (upstream patches/vllm-exl3/); serving it over a network triggers AGPL-3.0 section 13 - see the repo NOTICE.md.","pinned_commit":"da96774006515269bcf9eebfdc993874b59df1ba"},"credit":[{"name":"alanoo81 (recipe, graft, measurements)","url":"https://github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx"},{"name":"GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"Curtis White (hybrid layout + quantization_config.json)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)","url":"https://github.com/vcruz305/vllm-exl3"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"env":{"P2P":1,"BIND":"127.0.0.1","PORT":8011,"IMAGE":"ds41-cmp170hx:1.2.0-fidelity","LAYOUT":"tp4","EXTRA_ARGS":"--enable-prompt-tokens-details","SPEC_TOKENS":0,"MAX_NUM_SEQS":8,"MAX_MODEL_LEN":262144,"ENGRAM_STORAGE":"ssd","ENGRAM_HOST_DIR":"/opt/engram-ram/DeepSeek-V4.1-Flash-1017-engram","GPU_MEMORY_UTILIZATION":0.979999999999999982236431605997495353221893310546875,"VLLM_EXL3_FAT_THRESHOLD":2048},"launch":["docker run -d --name ds41-flash --device nvidia.com/gpu=all --ipc=host --cap-add SYS_NICE","--ulimit memlock=-1 --ulimit stack=67108864","-v <MODELS_DIR>:/models:ro -v /opt/engram-ram/DeepSeek-V4.1-Flash-1017-engram:/engram:ro","-v <CACHE_DIR>/root-cache:/root/.cache -v <CACHE_DIR>/triton:/root/.triton -v <CACHE_DIR>/tilelang:/root/.tilelang","-p 127.0.0.1:8011:8011","-e VLLM_PLUGINS=vllm_exl3 -e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e VLLM_EXL3_MODEL_DIR=/models/DeepSeek-V4.1-Flash-1017-hybrid","-e VLLM_USE_V2_MODEL_RUNNER=0 -e DSV41_DSPARK_V1=1 -e VLLM_EXL3_COOP=1","-e VLLM_EXL3_TC_MOE_V2=1 -e VLLM_EXL3_TC_MOE_PREFILL=1 -e VLLM_EXL3_TC_PREFILL_ROWS=2048","-e VLLM_EXL3_TC_MOE_SO=/opt/dsv41-tc-moe/_dsv41_tc_moe_C.abi3.so -e VLLM_EXL3_TC_HOOK=/opt/dsv41-tc-moe/exl3_tc_hook.py","-e VLLM_EXL3_MOE_TP_ALIGN=128 -e VLLM_EXL3_MOE_TP_ROTATE=1","-e NCCL_CUMEM_ENABLE=0 -e NCCL_CUMEM_HOST_ENABLE=0 -e NCCL_P2P_LEVEL=SYS -e VLLM_ALLREDUCE_USE_FLASHINFER_PCIE_IPC=1","-e DSV41_ENGRAM_DIR=/engram -e DSV41_ENGRAM_STORAGE=ssd","-e DSV41_EXPERTS_ONLY_FILES_RE=^model-[0-9]+-of-000(43|44|39)\\.safetensors$","-e TORCH_CUDA_ARCH_LIST=8.0 -e TORCHINDUCTOR_CACHE_DIR=/root/.cache/inductor","-e VLLM_DSV41_DECODE_SPLITK=1 -e DSV41_SHARED_HEAVY_BLOCKS=1 -e DSV41_DRAFT_HEAD_FP8=1","-e DSV41_PREFILL_CHUNK_REQS=1 -e DSV41_INDEXER_BUF_MULT=4 -e VLLM_EXL3_FAT_THRESHOLD=2048","--entrypoint vllm ds41-cmp170hx:1.2.0-fidelity serve /models/DeepSeek-V4.1-Flash-1017-hybrid","--quantization exl3 --load-format safetensors --safetensors-load-strategy lazy","--tensor-parallel-size 4 --pipeline-parallel-size 1","--served-model-name deepseek-v4.1-flash --host 0.0.0.0 --port 8011","--max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.98","--kv-cache-dtype fp8_ds_mla --enable-prefix-caching","--compilation-config {\"cudagraph_mode\": \"FULL_AND_PIECEWISE\", \"cudagraph_capture_sizes\": [1,2,3,4,5,6,8,12,16,18,20,24,28,30,32,36,42,48]}","--async-scheduling","--tokenizer-mode deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41","--enable-prompt-tokens-details"],"layouts":["tp"],"runtime":{"notes":"Patched serve.sh: SPEC_TOKENS=0 drops --speculative-config entirely (no DSpark drafter, no DSV41_DYNK); ENGRAM_HOST_DIR mounted read-only at /engram with DSV41_ENGRAM_STORAGE=ssd; start.sh runs engram-ram.sh first when ENGRAM_HOST_DIR is under /opt/engram-ram/ (tmpfs ENGRAM_TMPFS_SIZE=200G, copy ~94 s from NVMe). API has no auth by default - bind stays 127.0.0.1 until EXTRA_ARGS carries --api-key. Startup log must show 'Built engram token map (129280 -> 99092 ids) for layers (1, 14)' per rank and NO 'offloaded to pinned' line; 'GPU KV cache size: 2,717,420 tokens' at 0.98 (2,417,392 at 0.97, 1,817,233 at 0.95). Graft build: tools/make_hybrid_graft.py k3 = 188,241 tensors, 23,040 grafted at 3 bits, 88 links."},"kv_pool_tokens":2717420,"max_context_len":262144,"storage_requirements":{"note":"Author ran a 460 GiB container memory limit (189 GiB Engram tmpfs + engine host memory; 240 GiB plausible but unmeasured floor); ~428 GiB fast local disk for pinned downloads; image 17.8 GB + caches.","ram_gb":460,"nvme_gb":430}},"models":[{"slug":"deepseek-v4-1-flash-1017","name":"DeepSeek-V4.1-Flash-1017","hf_repo":"deepseek-ai/DeepSeek-V4.1-Flash-1017","quant":"exl3-hybrid-3bpw-all-40-layers"}],"hardware":[{"slug":"nvidia-cmp-170hxx4","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":4}],"generation":"ampere","tags":["community-recipe","engram-ram","exl3","fidelity-first","fp8-kv","hybrid-checkpoint","long-context","prefix-caching","sm80","tp4"],"tasks":["chat","code","vision"],"aa":{"aa_index":45,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash-1017","aa_index_version":"Index v4.3.2 (2026-10-18)","aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-19 12:00:00-06","aa_family_slug":"deepseek-v4-1-flash-1017","aa_resolved_slug":"deepseek-v4-1-flash-1017"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"claimed_by=submitter T0 - the author's own harness (docs/results.md @ da967740; MiaAI-Lab decode protocol: temp 0, thinking off, 400-token cap; cold prefill = median of 9 runs of texts ~24k/34k/38k tokens, no cache hits). The speed table was measured at gpu_memory_utilization 0.95 (changes the KV pool size, not the kernels); production runs at 0.98. Cards 200 W, 170tune +200 MHz. Fidelity (paired NLL, 64 x 1,024-token windows per set): perplexity vs the published 2.9bpw pack -1.80% all [-2.00,-1.60], -4.45% French [-4.97,-3.93] (64/64 better), -0.70% English, -0.18% code; copy test 300/300 exact (100 lists x 40 ids x 3 passes), repeats byte-identical; A-B-A restart check bit-identical prompt logprobs on 192 long windows. Stress: 2 waves of 8 unique 100k prompts, 16/16; peak 63,580/64,911 MiB usable per card at 0.98 (~1.3 GiB left); prompt_logprobs on 1,024-token windows peaks 64,676 MiB (~0.23 GiB) - use 0.97 (2,417,392 tokens) or <=512-token windows for that workload.\", \"tok_s\": 105, \"points\": [{\"notes\": \"Structured decode c1 headline (streaming, MiaAI protocol, measured at gpu_mem_util 0.95; code 105.0 / prose 104.8 at markers +10240/+20480).\", \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 105.400000000000005684341886080801486968994140625}, {\"notes\": \"Code decode c1. context_len is the tuple marker 262144+10240 (code row).\", \"concurrency\": 1, \"context_len\": 272384, \"decode_tok_s\": 105}, {\"notes\": \"Prose decode c1. context_len is the tuple marker 262144+20480 (prose row).\", \"concurrency\": 1, \"context_len\": 282624, \"decode_tok_s\": 104.7999999999999971578290569595992565155029296875}, {\"notes\": \"Aggregate c8 headline: published RANGE 449-560 tok/s across structured/code/prose (per-suite 559.7/482.4/449.2 at markers; c2 174.3/148.3/167.6, c4 325.2/282.0/289.2).\", \"concurrency\": 8, \"context_len\": 262144, \"decode_tok_s\": 505}, {\"notes\": \"Aggregate code decode c8 total. context_len is the tuple marker 262144+10240.\", \"concurrency\": 8, \"context_len\": 272384, \"decode_tok_s\": 482.3999999999999772626324556767940521240234375}, {\"notes\": \"Aggregate prose decode c8 total (low end of the published 449-560 range). context_len is the tuple marker 262144+20480.\", \"concurrency\": 8, \"context_len\": 282624, \"decode_tok_s\": 449.19999999999998863131622783839702606201171875}, {\"notes\": \"Aggregate structured decode c2 total. Code 148.3 / prose 167.6 at markers +10240/+20480 for c2 - see c8 marker rows for the per-suite shape.\", \"concurrency\": 2, \"context_len\": 262144, \"decode_tok_s\": 174.30000000000001136868377216160297393798828125}, {\"notes\": \"Aggregate structured decode c4 total. Code 282.0 / prose 289.2 at markers +10240/+20480 for c4.\", \"concurrency\": 4, \"context_len\": 262144, \"decode_tok_s\": 325.19999999999998863131622783839702606201171875}, {\"notes\": \"Cold prefill 2,514 tok/s = MEDIAN of 9 runs of three public-domain texts (~24k, 34k, 38k tokens), no cache hits (measured at gpu_mem_util 0.95). context_len 32768 is the tuple marker for the prefill row; concurrency omitted (NULL).\", \"context_len\": 32768, \"prefill_tok_s\": 2514}], \"context_len\": 262144}"}],"perf_points":[{"id":2157,"context_len":32768,"prefill_tok_s":2514,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"Cold prefill 2,514 tok/s = MEDIAN of 9 runs of three public-domain texts (~24k, 34k, 38k tokens), no cache hits (measured at gpu_mem_util 0.95). context_len 32768 is the tuple marker for the prefill row; concurrency omitted (NULL)."},{"id":2149,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":105.400000000000005684341886080801486968994140625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Structured decode c1 headline (streaming, MiaAI protocol, measured at gpu_mem_util 0.95; code 105.0 / prose 104.8 at markers +10240/+20480)."},{"id":2155,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":174.30000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"Aggregate structured decode c2 total. Code 148.3 / prose 167.6 at markers +10240/+20480 for c2 - see c8 marker rows for the per-suite shape."},{"id":2156,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":325.19999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"Aggregate structured decode c4 total. Code 282.0 / prose 289.2 at markers +10240/+20480 for c4."},{"id":2152,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":505,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Aggregate c8 headline: published RANGE 449-560 tok/s across structured/code/prose (per-suite 559.7/482.4/449.2 at markers; c2 174.3/148.3/167.6, c4 325.2/282.0/289.2)."},{"id":2150,"context_len":272384,"prefill_tok_s":null,"decode_tok_s":105,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Code decode c1. context_len is the tuple marker 262144+10240 (code row)."},{"id":2153,"context_len":272384,"prefill_tok_s":null,"decode_tok_s":482.3999999999999772626324556767940521240234375,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Aggregate code decode c8 total. context_len is the tuple marker 262144+10240."},{"id":2151,"context_len":282624,"prefill_tok_s":null,"decode_tok_s":104.7999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Prose decode c1. context_len is the tuple marker 262144+20480 (prose row)."},{"id":2154,"context_len":282624,"prefill_tok_s":null,"decode_tok_s":449.19999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Aggregate prose decode c8 total (low end of the published 449-560 range). context_len is the tuple marker 262144+20480."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017","pin":"official FP8 1017 @ 1bae0e1088993b9b82de08f59de5be5a34e42170 - non-expert tensors + Engram shards 00047/00048-of-00048"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw","pin":"EXL3 2.9bpw routed experts @ 0be63c3337efbb2a0061d5b8bb4006fa186d433f"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-3.5bpw","pin":"EXL3 3.5bpw @ 04a492fc3b4d25ddbab8f2d2f07f235ec77c9010 - shards 00020-00024-of-00044 grafted into layers 18-22"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx","pin":"da96774006515269bcf9eebfdc993874b59df1ba (patches, scripts, tests, raw measurement data; no weights, no engine code)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx","pin":"2ad5741ab1a278d64f33b842706a41717dfbf181 (base recipe release 1.2.0, image + kernels + benchmarks)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX","pin":"b8b9e99 - hybrid layout + quantization_config.json (Curtis White)"}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81.json","alternate":"/r/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:57:43.459028-06","created_at":"2026-10-29 21:57:43.459028-06","speeds":{"decode_tok_s":105,"prefill_tok_s":2514,"leaderboard":"/api/v1/leaderboard.json"}}