diff --git a/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml b/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml index e4eac560e..c7df6746c 100644 --- a/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml +++ b/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml @@ -1,5 +1,5 @@ slug: deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4 -title: 'DeepSeek-V4.1-Flash EXL3 3.05bpw TC-MoE on 4x CMP 170HX TP4' +title: 'DeepSeek-V4.1-Flash EXL3 3bpw TC-MoE on 4x CMP 170HX TP4 (original checkpoint)' engine: name: vllm source_type: fork @@ -7,7 +7,7 @@ engine: git_url: 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE' pinned_commit: cd83ade14c43a42babad39895d75bdd0e3453d49 install_note: "NOT a plain pip install: build the tensor-core MoE extension against the same PyTorch the server imports (TORCH_LIBRARY is not a stable ABI): python3 vllm_exl3/tc_moe/build_standalone.py --out ./out --build-dir ./build -> out/_dsv41_tc_moe_C.abi3.so (measured nvcc 13.0.88 + torch 2.13.0+cu130; build needs no GPU, serve does). Mount the .so + exl3_tc_hook.py into the image, cherry-pick Morrowmake/vllm-cmp170hx @ 18305948 (vllm/sm80, envs.py, platforms/cuda.py, host-allreduce) and wire exl3_tc_hook try_build/try_apply into the EXL3 fused-expert path. No published container image; weights are not included. Startup line 'EXL3 TC MoE layer=... v2=on prefill=on ws=540.4MiB' proves the hook is live." -summary: "DeepSeek-V4.1-Flash served EXL3 (3.02bpw, Mia-AiLab checkpoint) tensor-parallel 4 on four CMP 170HX cards (sm_80, 140 W, HBM locked 1728 MHz) in a 2x Xeon E5-2689 v4 Broadwell host, PCIe Gen2 x16, no NVLink, using Morrowmake's tensor-core MoE kernels + host-staged all-reduce ported onto this checkpoint: K3 on every served layer with a split K2 down-projection, row-rule hybrid (coop rows 1-42, moe_out_v2 43-64, moe_out_prefill 65-2048), one shared 540.4 MiB MoE workspace per GPU, zero-register-spill served kernels. Measured by the author: 178.3 tok/s structured decode c1, 587 tok/s across 8 users, median TTFT 8.37 s from 2K through 255K context (10.9 s at 255,424) against the same host's optimized graphs+coop+NCCL stack at 172.7 / 456 / 10.15 s; needles 9/9 through 256K; KV pool 616,916 tokens." +summary: "DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw — NOT the 2026-10-17 '1017' re-release, which has its own sibling recipe) served EXL3 3bpw tensor-parallel 4 on four CMP 170HX cards (sm_80, 140 W, HBM locked 1728 MHz) in a 2x Xeon E5-2689 v4 Broadwell host, PCIe Gen2 x16, no NVLink, using Morrowmake's tensor-core MoE kernels + host-staged all-reduce ported onto this checkpoint: K3 on every served layer with a split K2 down-projection, row-rule hybrid (coop rows 1-42, moe_out_v2 43-64, moe_out_prefill 65-2048), one shared 540.4 MiB MoE workspace per GPU, zero-register-spill served kernels. Measured by the author: 178.3 tok/s structured decode c1, 587 tok/s across 8 users, median TTFT 8.37 s from 2K through 255K context (10.9 s at 255,424) against the same host's optimized graphs+coop+NCCL stack at 172.7 / 456 / 10.15 s; needles 9/9 through 256K; KV pool 616,916 tokens." license: CC-BY-4.0 model: name: DeepSeek-V4.1-Flash-EXL3-3bpw