{"slug":"deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash EXL3 3bpw TC-MoE on 4x CMP 170HX TP4 (original checkpoint)","summary":"DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw \u2014 NOT the 2026-10-17 '1017' re-release, which has its own sibling recipe) served EXL3 3bpw tensor-parallel 4 on four CMP 170HX cards (sm_80, 140 W, HBM locked 1728 MHz) in a 2x Xeon E5-2689 v4 Broadwell host, PCIe Gen2 x16, no NVLink, using Morrowmake's tensor-core MoE kernels + host-staged all-reduce ported onto this checkpoint: K3 on every served layer with a split K2 down-projection, row-rule hybrid (coop rows 1-42, moe_out_v2 43-64, moe_out_prefill 65-2048), one shared 540.4 MiB MoE workspace per GPU, zero-register-spill served kernels. Measured by the author: 178.3 tok/s structured decode c1, 587 tok/s across 8 users, median TTFT 8.37 s from 2K through 255K context (10.9 s at 255,424) against the same host's optimized graphs+coop+NCCL stack at 172.7 / 456 / 10.15 s; needles 9/9 through 256K; KV pool 616,916 tokens.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"dc2dbe7cce66fb0f4d3dfad6b6765ed3c4ea8f74","content_hash":"5640bd33a3ca7d6a71e5a4804e0ec780e0c34fb1","created_at":"2026-10-29 21:32:23.373585-06","notes":"Scouted + ingested by localai-recipes from the CurtisAccelerate repo (created 2026-10-17); recipe, port and measurements by Curtis White. Tensor-core EXL3 MoE kernels + host-staged all-reduce ported from Morrowmake/vllm-cmp170hx @ 18305948 (Apache-2.0; built for Qwen3.8-Flash-Next on 2x CMP 170HX). EXL3 codec adapted from turboderp exllamav3 @ d3739fd3 (MIT). Checkpoint Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-3.0bpw @ c5534b90 (MIT; base deepseek-ai/DeepSeek-V4.1-Flash; 40 layers, hidden 5120, 384 experts top-6, MTP). Morrowmake's published CMP 170HX tables are for hardware-class comparability on a different model, not a head-to-head. All performance numbers author-reported (T0)."},"engine_source":{"type":"fork","git_url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE","base_engine":"vllm","pip_package":null,"base_version":"upstream vllm + Morrowmake sm80 host-allreduce 18305948","image_digest":null,"install_note":"NOT a plain pip install: build the tensor-core MoE extension against the same PyTorch the server imports (TORCH_LIBRARY is not a stable ABI): python3 vllm_exl3/tc_moe/build_standalone.py --out ./out --build-dir ./build -> out/_dsv41_tc_moe_C.abi3.so (measured nvcc 13.0.88 + torch 2.13.0+cu130; build needs no GPU, serve does). Mount the .so + exl3_tc_hook.py into the image, cherry-pick Morrowmake/vllm-cmp170hx @ 18305948 (vllm/sm80, envs.py, platforms/cuda.py, host-allreduce) and wire exl3_tc_hook try_build/try_apply into the EXL3 fused-expert path. No published container image; weights are not included. Startup line 'EXL3 TC MoE layer=... v2=on prefill=on ws=540.4MiB' proves the hook is live. Arena cache is opt-in and default-off: run-tp4-tcmoe.sh adds -v <arena_cache_dir>:<arena_cache_dir>, -e VLLM_EXL3_ARENA_CACHE=1 and -e VLLM_EXL3_ARENA_CACHE_DIR=<arena_cache_dir> only when DSV41_ARENA_CACHE_DIR is set; unset (default) the launch carries none of these flags.","pinned_commit":"e559217ac4f90dd8c2e4820f97b259e584efcf1f"},"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["docker run -d --name dsv41-tp4 --cap-add SYS_NICE --gpus all --network host --ipc host --ulimit memlock=-1 --ulimit stack=67108864","-v <model_dir>:/model-view:ro -v <exl3_py>:/usr/local/lib/python3.12/dist-packages/vllm_exl3/exl3.py:ro -v <sm80_dir>:/usr/local/lib/python3.12/dist-packages/vllm/sm80:ro","-v <envs_py>:/usr/local/lib/python3.12/dist-packages/vllm/envs.py:ro -v <cuda_py>:/usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py:ro","-v <tc_hook>:/opt/dsv41-tc-moe/exl3_tc_hook.py:ro -v <tc_so>:/opt/dsv41-tc-moe/_dsv41_tc_moe_C.abi3.so:ro","-e VLLM_EXL3_MODEL_DIR=/model-view -e VLLM_PLUGINS=vllm_exl3 -e VLLM_USE_V2_MODEL_RUNNER=0 -e DSV41_DSPARK_V1=1","-e VLLM_EXL3_COOP=1 -e VLLM_HOST_ALLREDUCE=1 -e NCCL_P2P_DISABLE=1 -e NCCL_IGNORE_DISABLED_P2P=1 -e NCCL_CUMEM_ENABLE=0","-e VLLM_EXL3_TC_MOE_V2=1 -e VLLM_EXL3_TC_MOE_PREFILL=1 -e VLLM_EXL3_TC_PREFILL_ROWS=2048","-e VLLM_EXL3_TC_MOE_SO=/opt/dsv41-tc-moe/_dsv41_tc_moe_C.abi3.so -e VLLM_EXL3_MOE_TP_ALIGN=128 -e VLLM_EXL3_MOE_TP_ROTATE=1 -e TORCH_CUDA_ARCH_LIST=8.0","--entrypoint vllm <dsv41_image>","serve /model-view","--quantization exl3 --load-format safetensors --safetensors-load-strategy lazy","--pipeline-parallel-size 1 --tensor-parallel-size 4","--served-model-name deepseek-v4.1-flash --host 0.0.0.0 --port 8095","--max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.97","--kv-cache-dtype fp8_ds_mla --engram-config {\"cpu_offload\": true} --enable-prefix-caching","--compilation-config {\"cudagraph_mode\": \"FULL_AND_PIECEWISE\", \"cudagraph_capture_sizes\": [1, 2, 3, 4, 6, 8, 12, 18, 24, 30, 36, 42, 48]}","--async-scheduling","--speculative-config {\"method\": \"dspark\", \"num_speculative_tokens\": 5}","--tokenizer-mode deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41"],"layouts":["tp"],"kv_pool_tokens":616916,"max_context_len":262144,"storage_requirements":{"note":"Host RAM: no fixed 384 GB minimum (author machine has 378 GB visible; while serving it uses ~288 GB, ~190 GB of it the pinned Engram tables in system RAM \u2014 a 256 GB machine does not have room for that pin). 674 GB checkpoint streams from disk (lazy). ~62.5 GB of each 64 GB card is filled (48 GB cards do not fit)."},"serving_selection_2026_10_26":{"band":"batches of 2-8 users; batch 7 pads onto the 48-row batch-8 graph; draft length stays 5 (6 rows per request). Draft length 7 and the Mia weights are NOT part of this selection.","lane":"2-8 user decode lane = experiments/ds1017-r2-b2-b8 (separate source, DEFAULT OFF). The published C1 source stays the one-user bundle in experiments/ds1017-native-mixed-and-split (native mixed experts, split attention, captured K5 drafter).","image":"UNAVAILABLE (never published): the author states 'There is still no pullable image' (SERVING-2026-10-26.md). The only image identifier in the stack, sha256:55d2bd9a6664aea2540e3f8e373d784b88420d35c3cedefc9f03923cd687b6f5, is a LOCAL docker image id recorded for the measured serve (1017 repo README.md:93), not a registry reference; the site's image_digest is null. Cloning the repo does not turn the lane on: no image boots the 2-8 user graphs or the sampled drafter yet.","hardware":"4x CMP 170HX (sm_80) host as measured; the October 26 run used the 1017 weights on this host class.","measurements":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX/blob/main/benchmarks/2026-10-26/README.md","lane_defaults":"DSV41_R2_BATCHES=2-8; DSV41_R2_COMPONENTS=attention,experts,collective,sampler,sampled; DSV41_R2_PAD7=1; DSV41_R2_SPLITS=per batch nearest the one-user 96-CTA shape (empty = default); DSV41_R2_EXPERTS=coop, or tc when tensor-core v2 covers the row count; DSV41_R2_COLLECTIVE=pair at 48 rows, otherwise the host all-reduce size rule; DSV41_R2_CAP_MIB=768 (retained-graph cap checked at startup).","measured_plan":"With DSV41_R2=1, DSV41_R2_PAD7=1 and VLLM_EXL3_TC_MOE_V2=1 and no per-batch overrides the defaults are the measured plan: cooperative experts for batches 2-6, tensor-core experts + pair collective at batch 8, batch 7 served by the batch-8 graph. New attention for every batch size; a temperature-0 request stays greedy at every batch size.","published_from":"CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE @ e559217ac4f90dd8c2e4820f97b259e584efcf1f (experiments/ds1017-native-mixed-and-split/SERVING-2026-10-26.md + experiments/ds1017-r2-b2-b8/README.md; commits 'Publish the 2-8 user lane and its default-off flags' and 'Note the October 26 serving selection and the missing image')","default_off_flags":[{"flag":"DSV41_R2","effect":"Master switch for the 2-8 lane; 1 is the only armed value, malformed flag raises at boot. Being present in the repository does NOT turn the lane on.","default":"0 (off)"},{"flag":"VLLM_EXL3_TC_MOE_V2","effect":"Required for the tensor-core expert path on the 48-row graph; the launch command of this recipe arms it explicitly (-e VLLM_EXL3_TC_MOE_V2=1).","default":"0 (off)"},{"flag":"dsv41_draft_tq","effect":"Per-request sampled-drafter flag; NOT turned on by DSV41_R2. The October 26 selection keeps the drafter greedy at batches 2-8 and uses it only for a single temperature-1 request (outside this lane).","default":"off"},{"flag":"DSV41_ARENA_CACHE_DIR","effect":"Optional gate in scripts/run-tp4-tcmoe.sh: unset/empty = zero argv change; when set it adds -v DIR:DIR, -e VLLM_EXL3_ARENA_CACHE=1 and -e VLLM_EXL3_ARENA_CACHE_DIR=DIR.","default":"unset (off)"}],"launch_consistency":"The launch command above arms VLLM_EXL3_TC_MOE_V2=1; DSV41_R2 is NOT set, so the published serve keeps the 2-8 lane default-off until armed. The optional arena-cache group (DSV41_ARENA_CACHE_DIR gate) is NOT in this recipe's launch command \u2014 it stays default-off; add the three flags above to enable it."}},"models":[{"slug":"deepseek-v4-1-flash-exl3-3bpw","name":"DeepSeek-V4.1-Flash-EXL3-3bpw","hf_repo":"Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-3.0bpw","quant":"exl3-3bpw"}],"hardware":[{"slug":"nvidia-cmp-170hxx4","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":4}],"generation":"ampere","tags":["broadwell","dspark","exl3","fp8-kv","host-allreduce","long-context","mtp-speculative","prefix-caching","tensor-core-moe","tp4"],"tasks":["chat","code"],"aa":{"aa_index":39,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash","aa_index_version":null,"aa_methodology_version":"Index v4.3.2 (2026-10-18)","aa_published_fetched_at":"2026-10-20 21:49:11.309778-06","aa_family_slug":"deepseek-v4-1-flash","aa_resolved_slug":"deepseek-v4-1-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-reported T0 (2026-10-17): median of 5 runs incl. the cold sample, temperature 0, top_p 1, thinking off, max 400 tokens; decode = (completion_tokens - 1) / (end - first_token). HBM locked 1728 MHz, 140 W. Baseline is the optimized full-CUDA-graphs+coop+NCCL stack already in service on the same host (172.65 tok/s structured c1, 10.15 s TTFT median) - not an untouched default. KV pool 616,916 tokens (2.35x at 262,144). Needles 9/9 at 32,768 / 131,072 / 261,568 (depths 10/50/90, 'cobalt-heron-4417'). Cold-curve points are /v1/completions, 128 generated tokens, median of 3.\", \"tok_s\": 178.270000000000010231815394945442676544189453125, \"points\": [{\"notes\": \"Growing-context sweep TTFT at 2,080 input tokens (shipped arm).\", \"ttft_ms\": 1780, \"concurrency\": 1, \"context_len\": 2080}, {\"notes\": \"Growing-context sweep TTFT at 255,424 input tokens (shipped arm); 30-turn sweep median 8.37 s, every one of 30 turns faster than baseline.\", \"ttft_ms\": 10900, \"concurrency\": 1, \"context_len\": 255424}, {\"notes\": \"Structured c1 (c1/c4/c8 serving table); 5-run median 178.27 tok/s on the same structured text (hash-identical across arms).\", \"concurrency\": 1, \"context_len\": 2048, \"decode_tok_s\": 178.80000000000001136868377216160297393798828125}, {\"notes\": \"Structured c4 (baseline graphs+coop: 313.8).\", \"concurrency\": 4, \"context_len\": 2048, \"decode_tok_s\": 343.8999999999999772626324556767940521240234375}, {\"notes\": \"Structured c8, +28.5% vs graphs+coop 456.4; gain is the 43-64 row window (moe_out_v2).\", \"concurrency\": 8, \"context_len\": 2048, \"decode_tok_s\": 586.6000000000000227373675443232059478759765625}, {\"notes\": \"Prose c1 (+5.9% vs baseline 86.13 headline). context_len is the tuple marker 2048+10240 (prose row, not structured).\", \"concurrency\": 1, \"context_len\": 12288, \"decode_tok_s\": 93.2999999999999971578290569595992565155029296875}, {\"notes\": \"Prose c4. context_len is the tuple marker 2048+10240 (prose row).\", \"concurrency\": 4, \"context_len\": 12288, \"decode_tok_s\": 165.80000000000001136868377216160297393798828125}, {\"notes\": \"Prose c8, +34.6% vs graphs+coop 193.3. context_len is the tuple marker 2048+10240 (prose row).\", \"concurrency\": 8, \"context_len\": 12288, \"decode_tok_s\": 260.19999999999998863131622783839702606201171875}, {\"notes\": \"Cold /v1/completions at 2,048 prompt tokens (140 W row). context_len is the tuple marker 2048+40960.\", \"concurrency\": 1, \"context_len\": 43008, \"decode_tok_s\": 149.19999999999998863131622783839702606201171875, \"prefill_tok_s\": 1570}, {\"notes\": \"Cold /v1/completions, 8,192 prompt tokens (140 W row).\", \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 146.80000000000001136868377216160297393798828125, \"prefill_tok_s\": 1528}, {\"notes\": \"Cold /v1/completions, 32,768 prompt tokens (140 W row).\", \"concurrency\": 1, \"context_len\": 32768, \"decode_tok_s\": 148.5, \"prefill_tok_s\": 1519}, {\"notes\": \"Cold /v1/completions, 65,536 prompt tokens (140 W row).\", \"concurrency\": 1, \"context_len\": 65536, \"decode_tok_s\": 149.900000000000005684341886080801486968994140625, \"prefill_tok_s\": 1453}, {\"notes\": \"Cold /v1/completions, 131,072 prompt tokens (140 W row).\", \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 156.5, \"prefill_tok_s\": 1329}], \"ttft_ms\": 1780, \"context_len\": 2080}"}],"perf_points":[{"id":2017,"context_len":2048,"prefill_tok_s":null,"decode_tok_s":178.80000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Structured c1 (c1/c4/c8 serving table); 5-run median 178.27 tok/s on the same structured text (hash-identical across arms)."},{"id":2018,"context_len":2048,"prefill_tok_s":null,"decode_tok_s":343.8999999999999772626324556767940521240234375,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"Structured c4 (baseline graphs+coop: 313.8)."},{"id":2019,"context_len":2048,"prefill_tok_s":null,"decode_tok_s":586.6000000000000227373675443232059478759765625,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Structured c8, +28.5% vs graphs+coop 456.4; gain is the 43-64 row window (moe_out_v2)."},{"id":2020,"context_len":2080,"prefill_tok_s":null,"decode_tok_s":null,"ttft_ms":1780,"concurrency":1,"claimed_by":"submitter","notes":"Growing-context sweep TTFT at 2,080 input tokens (shipped arm)."},{"id":2021,"context_len":8192,"prefill_tok_s":1528,"decode_tok_s":146.80000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Cold /v1/completions, 8,192 prompt tokens (140 W row)."},{"id":2022,"context_len":12288,"prefill_tok_s":null,"decode_tok_s":93.2999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Prose c1 (+5.9% vs baseline 86.13 headline). context_len is the tuple marker 2048+10240 (prose row, not structured)."},{"id":2023,"context_len":12288,"prefill_tok_s":null,"decode_tok_s":165.80000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"Prose c4. context_len is the tuple marker 2048+10240 (prose row)."},{"id":2024,"context_len":12288,"prefill_tok_s":null,"decode_tok_s":260.19999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Prose c8, +34.6% vs graphs+coop 193.3. context_len is the tuple marker 2048+10240 (prose row)."},{"id":2025,"context_len":32768,"prefill_tok_s":1519,"decode_tok_s":148.5,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Cold /v1/completions, 32,768 prompt tokens (140 W row)."},{"id":2026,"context_len":43008,"prefill_tok_s":1570,"decode_tok_s":149.19999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Cold /v1/completions at 2,048 prompt tokens (140 W row). context_len is the tuple marker 2048+40960."},{"id":2027,"context_len":65536,"prefill_tok_s":1453,"decode_tok_s":149.900000000000005684341886080801486968994140625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Cold /v1/completions, 65,536 prompt tokens (140 W row)."},{"id":2028,"context_len":131072,"prefill_tok_s":1329,"decode_tok_s":156.5,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Cold /v1/completions, 131,072 prompt tokens (140 W row)."},{"id":2029,"context_len":255424,"prefill_tok_s":null,"decode_tok_s":null,"ttft_ms":10900,"concurrency":1,"claimed_by":"submitter","notes":"Growing-context sweep TTFT at 255,424 input tokens (shipped arm); 30-turn sweep median 8.37 s, every one of 30 turns faster than baseline."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-3.0bpw/tree/c5534b90602b4090980a1c3ded1eb3b4d99a38d5","pin":"c5534b90602b4090980a1c3ded1eb3b4d99a38d5"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE","pin":"e559217ac4f90dd8c2e4820f97b259e584efcf1f"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/Morrowmake/vllm-cmp170hx","pin":"18305948acba3f4c4d912cdd56f2202d2ebd94e8"}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:23.373585-06","created_at":"2026-10-20 21:50:05.71827-06","speeds":{"decode_tok_s":149.19999999999998863131622783839702606201171875,"prefill_tok_s":1486,"leaderboard":"/api/v1/leaderboard.json"}}