{"slug":"deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw TC-MoE on 4x CMP 170HX TP4","summary":"DeepSeek-V4.1-Flash-1017 (the 2026-10-17 re-release checkpoint, NOT the original DeepSeek-V4.1-Flash) as a hybrid quant on the TC-MoE port: official deepseek-ai/DeepSeek-V4.1-Flash-1017 FP8 tensors for attention, shared experts, head, DSpark draft and Engram (shards 47/48), with LibertAIDAI/...1017-EXL3-2.9bpw routed experts (layers 0-39, mostly 3-bit, layers 18-22 2-bit) TP4 on four CMP 170HX cards (sm_80, 140 W, ~62.5 GB/card). Same Morrowmake kernels + image as the EXL3 3bpw TC-MoE recipe; VLLM_EXL3_COOP=1; EXL3 arena cache (~1.1 GiB/layer, load shortcut only). 1017 quantization_config.json is the switch (dsv41_main_routed_experts_only, mtp_experts=source, non-routed stays FP8, compressor projection BF16). Author-measured: 180.4/137.1/96.6 tok/s structured/code/prose c1; 8-user aggregate 621.1/376.1/282.5 vs 564.7 previous Mia 3.0bpw. Cold 131k prompt 1,330 prefill / 163.9 decode. Needles 9/9. KV pool 1,534,065 tokens; vision on, drafter text-only.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"306225cea85cf158c3036c971b031a9c1740075b","content_hash":"59b7b30fa1a9d7fe2197055fbfc51c994b611ccc","created_at":"2026-10-29 21:32:22.449013-06","notes":"Scouted + ingested by localai-recipes from CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX (created 2026-10-18, pinned @ b8b9e993); 1017 weight selection, hybrid recipe and measurements by Curtis White. This recipe serves the 1017 checkpoint (deepseek-ai/DeepSeek-V4.1-Flash-1017 @ 1bae0e10, official FP8 attention/shared-experts/head/draft/Engram shards 47-48 + LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw @ 0be63c33 routed experts layers 0-39, MIT licensed; the quant is LibertAI's) \u2014 NOT the original deepseek-ai/DeepSeek-V4.1-Flash, and not the Mia-AiLab EXL3 3.0bpw checkpoint of the sibling EXL3 3bpw TC-MoE recipe. Same Morrowmake kernel base vllm-cmp170hx @ 18305948 (Apache-2.0) and EXL3 codec (turboderp exllamav3, MIT) as that sibling. AA Intelligence 45 is AA's published index for DeepSeek V4.1 Flash 1017 (Max), read from the AA model page 2026-10-19 (Index v4.3.2; original V4.1 Flash is a separate 39 entry). All performance numbers author-reported (T0). 1017 measured table refresh (kanban t_1bc89b18, deferred lane of t_22817e0f, preprod only per owner ruling 2026-10-28): performance rebuilt cell-exact from benchmarks/2026-10-25/results.json (sha256 3f8e70fa7e5538b30dfe5db887a574eeb8872781203ade02d76bc898069cb992) at the re-pinned 1017 commit 8b4e52906e; pin honesty \u2014 the cited file exists only from 8b4e52906e (b8b9e993e0 -> 8b4e52906e is 14 commits, GitHub compare 'ahead'); fdb629e/a06e30d9 NOT pinned (Oct-28 gate-kernel commits post-date the audited evidence; next sweep). The comparison measures Morrowmake's repo run on this Broadwell. Those figures are our measurements, not numbers published by Morrowmake. Stale October-18 headline points and the long-context curve retired; perf points 10 -> 10."},"engine_source":{"type":"fork","git_url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE","base_engine":"vllm","pip_package":null,"base_version":"upstream vllm + Morrowmake sm80 host-allreduce 18305948","image_digest":null,"install_note":"Same serving image + EXL3 plugin + TC-MoE .so as the EXL3 3bpw TC-MoE recipe (this checkpoint changes weights, not the stack): build out/_dsv41_tc_moe_C.abi3.so from DeepSeekv41-Flash-EXL3-TC-MoE build_standalone.py against the image's PyTorch. Hybrid checkpoint = official 1017 FP8 tensors (attention/shared/head/draft/Engram shards 47-48) + LibertAI routed experts layers 0-39 + quantization_config.json from the 1017 repo. No published container image; weights are not included.","pinned_commit":"e559217ac4f90dd8c2e4820f97b259e584efcf1f"},"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Measured serve used local image sha256:55d2bd9a6664 (no published image)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"PixelML/sm80vllm branch sm80 (DeepseekV4AmpereMLAAttention) - SM80 attention for this model","url":"https://github.com/PixelML/sm80vllm"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["docker run -d --name dsv41-1017 --cap-add SYS_NICE --gpus all --network host --ipc host --ulimit memlock=-1 --ulimit stack=67108864","-v <hybrid_model_dir>:/model-view:ro -v <arena_cache_dir>:<arena_cache_dir>","-v <exl3_py>:/usr/local/lib/python3.12/dist-packages/vllm_exl3/exl3.py:ro -v <sm80_dir>:/usr/local/lib/python3.12/dist-packages/vllm/sm80:ro","-v <envs_py>:/usr/local/lib/python3.12/dist-packages/vllm/envs.py:ro -v <cuda_py>:/usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py:ro","-v <tc_hook>:/opt/dsv41-tc-moe/exl3_tc_hook.py:ro -v <tc_so>:/opt/dsv41-tc-moe/_dsv41_tc_moe_C.abi3.so:ro","-e VLLM_EXL3_MODEL_DIR=/model-view -e VLLM_EXL3_ARENA_CACHE=1 -e VLLM_EXL3_ARENA_CACHE_DIR=<arena_cache_dir>","-e VLLM_PLUGINS=vllm_exl3 -e VLLM_USE_V2_MODEL_RUNNER=0 -e DSV41_DSPARK_V1=1","-e VLLM_EXL3_COOP=1 -e VLLM_HOST_ALLREDUCE=1 -e NCCL_P2P_DISABLE=1 -e NCCL_IGNORE_DISABLED_P2P=1 -e NCCL_CUMEM_ENABLE=0","-e VLLM_EXL3_TC_MOE_V2=1 -e VLLM_EXL3_TC_MOE_PREFILL=1 -e VLLM_EXL3_TC_PREFILL_ROWS=2048","-e VLLM_EXL3_TC_MOE_SO=/opt/dsv41-tc-moe/_dsv41_tc_moe_C.abi3.so -e VLLM_EXL3_MOE_TP_ALIGN=128 -e VLLM_EXL3_MOE_TP_ROTATE=1 -e TORCH_CUDA_ARCH_LIST=8.0","--entrypoint vllm <dsv41_image>","serve /model-view","--quantization exl3 --load-format safetensors --safetensors-load-strategy lazy","--pipeline-parallel-size 1 --tensor-parallel-size 4","--served-model-name deepseek-v4.1-flash-1017 --host 0.0.0.0 --port 8095","--max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.97","--kv-cache-dtype fp8_ds_mla --engram-config {\"cpu_offload\": true} --enable-prefix-caching","--compilation-config {\"cudagraph_mode\": \"FULL_AND_PIECEWISE\", \"cudagraph_capture_sizes\": [1, 2, 3, 4, 6, 8, 12, 18, 24, 30, 36, 42, 48]}","--async-scheduling","--speculative-config {\"method\": \"dspark\", \"num_speculative_tokens\": 5}","--tokenizer-mode deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41"],"layouts":["tp"],"kv_pool_tokens":1534065,"max_context_len":262144,"storage_requirements":{"note":"Host has 378 GB RAM visible to the OS - the machine, not a minimum. Serving uses ~288 GB (~190 GB Engram pin in system RAM; a 256 GB machine does not have room). ~62.5 GB of each 64 GB card filled.","ram_gb":378,"nvme_gb":674},"serving_selection_2026_10_26":{"band":"batches of 2-8 users; batch 7 pads onto the 48-row batch-8 graph; draft length stays 5 (6 rows per request). Draft length 7 and the Mia weights are NOT part of this selection.","lane":"2-8 user decode lane = experiments/ds1017-r2-b2-b8 (separate source, DEFAULT OFF). The published C1 source stays the one-user bundle in experiments/ds1017-native-mixed-and-split (native mixed experts, split attention, captured K5 drafter).","image":"UNAVAILABLE (never published): the author states 'There is still no pullable image' (SERVING-2026-10-26.md). The only image identifier in the stack, sha256:55d2bd9a6664aea2540e3f8e373d784b88420d35c3cedefc9f03923cd687b6f5, is a LOCAL docker image id recorded for the measured serve (1017 repo README.md:93), not a registry reference; the site's image_digest is null. Cloning the repo does not turn the lane on: no image boots the 2-8 user graphs or the sampled drafter yet.","hardware":"4x CMP 170HX (sm_80) host as measured; the October 26 run used the 1017 weights on this host class.","measurements":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX/blob/main/benchmarks/2026-10-26/README.md","lane_defaults":"DSV41_R2_BATCHES=2-8; DSV41_R2_COMPONENTS=attention,experts,collective,sampler,sampled; DSV41_R2_PAD7=1; DSV41_R2_SPLITS=per batch nearest the one-user 96-CTA shape (empty = default); DSV41_R2_EXPERTS=coop, or tc when tensor-core v2 covers the row count; DSV41_R2_COLLECTIVE=pair at 48 rows, otherwise the host all-reduce size rule; DSV41_R2_CAP_MIB=768 (retained-graph cap checked at startup).","measured_plan":"With DSV41_R2=1, DSV41_R2_PAD7=1 and VLLM_EXL3_TC_MOE_V2=1 and no per-batch overrides the defaults are the measured plan: cooperative experts for batches 2-6, tensor-core experts + pair collective at batch 8, batch 7 served by the batch-8 graph. New attention for every batch size; a temperature-0 request stays greedy at every batch size.","published_from":"CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE @ e559217ac4f90dd8c2e4820f97b259e584efcf1f (experiments/ds1017-native-mixed-and-split/SERVING-2026-10-26.md + experiments/ds1017-r2-b2-b8/README.md; commits 'Publish the 2-8 user lane and its default-off flags' and 'Note the October 26 serving selection and the missing image')","default_off_flags":[{"flag":"DSV41_R2","effect":"Master switch for the 2-8 lane; 1 is the only armed value, malformed flag raises at boot. Being present in the repository does NOT turn the lane on.","default":"0 (off)"},{"flag":"VLLM_EXL3_TC_MOE_V2","effect":"Required for the tensor-core expert path on the 48-row graph; the launch command of this recipe arms it explicitly (-e VLLM_EXL3_TC_MOE_V2=1).","default":"0 (off)"},{"flag":"dsv41_draft_tq","effect":"Per-request sampled-drafter flag; NOT turned on by DSV41_R2. The October 26 selection keeps the drafter greedy at batches 2-8 and uses it only for a single temperature-1 request (outside this lane).","default":"off"},{"flag":"DSV41_ARENA_CACHE_DIR","effect":"Optional gate in scripts/run-tp4-tcmoe.sh: unset/empty = zero argv change; when set it adds -v DIR:DIR, -e VLLM_EXL3_ARENA_CACHE=1 and -e VLLM_EXL3_ARENA_CACHE_DIR=DIR.","default":"unset (off)"}],"launch_consistency":"The launch command above already carries the opt-in arena-cache group (-e VLLM_EXL3_ARENA_CACHE=1 -e VLLM_EXL3_ARENA_CACHE_DIR=<arena_cache_dir>) and arms VLLM_EXL3_TC_MOE_V2=1; DSV41_R2 is NOT set in the launch command, so the published serve keeps the 2-8 lane default-off until armed."}},"models":[{"slug":"deepseek-v4-1-flash-1017-exl3-2-9bpw-hybrid","name":"DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw-Hybrid","hf_repo":"LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw","quant":"exl3-2-9bpw-routed-experts-fp8-core-hybrid"}],"hardware":[{"slug":"nvidia-cmp-170hxx4","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":4}],"generation":"ampere","tags":["arena-cache","dspark","exl3","fp8-kv","host-allreduce","long-context","mtp-speculative","prefix-caching","tensor-core-moe","tp4"],"tasks":["chat","code"],"aa":{"aa_index":45,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash-1017","aa_index_version":"Index v4.3.2 (2026-10-18)","aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-19 12:00:00-06","aa_family_slug":"deepseek-v4-1-flash-1017","aa_resolved_slug":"deepseek-v4-1-flash-1017"},"performance":[{"claimed_by":"submitter","metrics":"{\"metrics\": {\"notes\": \"Measured 2026-10-25 cumulative single-user decode (benchmarks/2026-10-25/results.json @ CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX 8b4e52906e, sha256 3f8e70fa7e5538b30dfe5db887a574eeb8872781203ade02d76bc898069cb992; hardware 4x CMP 170HX; metric \\\"Single-user streamed decode tokens/second\\\"; baseline \\\"Published October22 results, same published configuration across each suite\\\"). Table (Task | Published baseline | Part 1: native mixed | Part 1 change | Part 2: combined | Part 2 change | Final total; tok/s): Short code 144.23 | 146.28 | +1.42% | 155.54 | +6.33% | +7.84%; Short prose 101.16 | 102.82 | +1.64% | 116.05 | +12.87% | +14.72%; Spark-matched code 158.36 | 169.44 | +7.00% | 175.73 | +3.71% | +10.97%; Spark-matched prose 83.64 | 88.46 | +5.76% | 94.65 | +7.00% | +13.16%. Spark-matched rows are the Spark PROMPT SUITE run on THIS host (the same 4x CMP 170HX machine) \u2014 not a DGX Spark result, not a hardware comparison, not a second system. Separate statistic (not a column, not added to the historical percentages, per limitations): median paired Part 2 gains in the same-load three-pair comparison \u2014 short code +6.33%, short prose +12.87%, Spark-matched code +3.71%, Spark-matched prose +6.25%. Controlled counting (same host): 187.723 -> 195.423 tok/s, +4.10% \u2014 24 requests in 6 balanced blocks; matched output/prompt/cache/speculative work; counting only; not a +4.10% claim for every code/prose request. Spark-suite C8 aggregate (this host): 275.11 -> 276.55 tok/s (+0.52%, ratio of medians). Cold prefill 93,335 prompt tokens at 1,389.0 -> 1,391.1 prompt tok/s; prefill = prompt ingestion in the same passes, not decode; no consistent C4/C8 or cold-prefill gain. Matched task-quality screen: code 5/5 vs 4/5, prose strict 0/5 vs 1/5, prose checklist+format 5/5 vs 5/5; quality_equivalence_established false; experimental split attention is not bit-exact, broader verification needed. Historical changes describe configurations across runs, not isolated causal component effects. No row derives from benchmarks/2026-10-24/broadwell-vs-morrowmake.\"}}"}],"perf_points":[{"id":1981,"context_len":2048,"prefill_tok_s":null,"decode_tok_s":155.539999999999992041921359486877918243408203125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Measured 2026-10-25 combined decode, Short code (benchmarks/2026-10-25/results.json @ 8b4e52906e): baseline 144.23, native mixed 146.28 (+1.42%), combined 155.54 (Part 2 +6.33%; paired median +6.33%), total +7.84%. context_len is the tuple marker for the short-code row, not a prompt length."},{"id":1982,"context_len":2080,"prefill_tok_s":null,"decode_tok_s":116.0499999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Measured 2026-10-25 combined decode, Short prose (same source): baseline 101.16, native mixed 102.82 (+1.64%), combined 116.05 (Part 2 +12.87%; paired median +12.87%), total +14.72%. context_len is the tuple marker for the short-prose row, not a prompt length."},{"id":1983,"context_len":2112,"prefill_tok_s":null,"decode_tok_s":175.729999999999989768184605054557323455810546875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Spark-matched code: the Spark PROMPT SUITE run on THIS host (4x CMP 170HX) \u2014 a prompt suite, not a DGX Spark hardware comparison (benchmarks/2026-10-25/results.json @ 8b4e52906e): baseline 158.36, native mixed 169.44 (+7.00%), combined 175.73 (Part 2 +3.71%; paired median +3.71%), total +10.97%. context_len is the tuple marker."},{"id":1984,"context_len":2144,"prefill_tok_s":null,"decode_tok_s":94.650000000000005684341886080801486968994140625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Spark-matched prose: the Spark PROMPT SUITE run on THIS host (4x CMP 170HX) \u2014 a prompt suite, not a DGX Spark hardware comparison (same source): baseline 83.64, native mixed 88.46 (+5.76%), combined 94.65 (Part 2 +7.00%; paired median +6.25% is the separate same-load statistic), total +13.16%. context_len is the tuple marker."},{"id":1985,"context_len":3072,"prefill_tok_s":null,"decode_tok_s":187.72300000000001318767317570745944976806640625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Controlled native-mixed counting BEFORE (same host): 187.723 tok/s \u2014 24 requests in 6 balanced blocks; matched output/prompt/cache/speculative work; counting only. context_len is the tuple marker."},{"id":1986,"context_len":3104,"prefill_tok_s":null,"decode_tok_s":195.423000000000001818989403545856475830078125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Controlled native-mixed counting AFTER (same host): 195.423 tok/s, +4.10% vs 187.723; counting only \u2014 not a claim for every code/prose request. context_len is the tuple marker."},{"id":1987,"context_len":4096,"prefill_tok_s":null,"decode_tok_s":275.1100000000000136424205265939235687255859375,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Spark-suite C8 aggregate BEFORE (prompt suite on THIS host): 275.11 tok/s, 8 concurrent streams."},{"id":1988,"context_len":4128,"prefill_tok_s":null,"decode_tok_s":276.55000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Spark-suite C8 aggregate AFTER (prompt suite on THIS host): 276.55 tok/s, +0.52% ratio of medians; no consistent C4/C8 gain."},{"id":1989,"context_len":93335,"prefill_tok_s":1389,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"Cold prefill of 93,335 prompt tokens BEFORE: 1,389.0 prompt tok/s; prefill = prompt ingestion, not decode. context_len IS the prompt length; concurrency NULL (before-arm of the paired rows)."},{"id":1990,"context_len":93335,"prefill_tok_s":1391.09999999999990905052982270717620849609375,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Cold prefill of 93,335 prompt tokens AFTER: 1,391.1 prompt tok/s; prefill = prompt ingestion, not decode. context_len IS the prompt length; concurrency is the tuple marker for the after-arm row."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/tree/1bae0e1088993b9b82de08f59de5be5a34e42170","pin":"1bae0e1088993b9b82de08f59de5be5a34e42170"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw/tree/0be63c3337efbb2a0061d5b8bb4006fa186d433f","pin":"0be63c3337efbb2a0061d5b8bb4006fa186d433f"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX","pin":"8b4e52906ec255b74bfec546d23d2a8741a86946"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE","pin":"e559217ac4f90dd8c2e4820f97b259e584efcf1f"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/Morrowmake/vllm-cmp170hx","pin":"18305948acba3f4c4d912cdd56f2202d2ebd94e8"}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:22.449013-06","created_at":"2026-10-20 21:49:11.309778-06","speeds":{"decode_tok_s":165.599999999999994315658113919198513031005859375,"prefill_tok_s":1390.09999999999990905052982270717620849609375,"leaderboard":"/api/v1/leaderboard.json"}}