diff --git a/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml b/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml index f700f36cf..5640bd33a 100644 --- a/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml +++ b/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml @@ -5,8 +5,8 @@ engine: source_type: fork version: 'upstream vllm + Morrowmake sm80 host-allreduce 18305948' git_url: 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE' - pinned_commit: cd83ade14c43a42babad39895d75bdd0e3453d49 - install_note: "NOT a plain pip install: build the tensor-core MoE extension against the same PyTorch the server imports (TORCH_LIBRARY is not a stable ABI): python3 vllm_exl3/tc_moe/build_standalone.py --out ./out --build-dir ./build -> out/_dsv41_tc_moe_C.abi3.so (measured nvcc 13.0.88 + torch 2.13.0+cu130; build needs no GPU, serve does). Mount the .so + exl3_tc_hook.py into the image, cherry-pick Morrowmake/vllm-cmp170hx @ 18305948 (vllm/sm80, envs.py, platforms/cuda.py, host-allreduce) and wire exl3_tc_hook try_build/try_apply into the EXL3 fused-expert path. No published container image; weights are not included. Startup line 'EXL3 TC MoE layer=... v2=on prefill=on ws=540.4MiB' proves the hook is live." + pinned_commit: e559217ac4f90dd8c2e4820f97b259e584efcf1f + install_note: "NOT a plain pip install: build the tensor-core MoE extension against the same PyTorch the server imports (TORCH_LIBRARY is not a stable ABI): python3 vllm_exl3/tc_moe/build_standalone.py --out ./out --build-dir ./build -> out/_dsv41_tc_moe_C.abi3.so (measured nvcc 13.0.88 + torch 2.13.0+cu130; build needs no GPU, serve does). Mount the .so + exl3_tc_hook.py into the image, cherry-pick Morrowmake/vllm-cmp170hx @ 18305948 (vllm/sm80, envs.py, platforms/cuda.py, host-allreduce) and wire exl3_tc_hook try_build/try_apply into the EXL3 fused-expert path. No published container image; weights are not included. Startup line 'EXL3 TC MoE layer=... v2=on prefill=on ws=540.4MiB' proves the hook is live. Arena cache is opt-in and default-off: run-tp4-tcmoe.sh adds -v :, -e VLLM_EXL3_ARENA_CACHE=1 and -e VLLM_EXL3_ARENA_CACHE_DIR= only when DSV41_ARENA_CACHE_DIR is set; unset (default) the launch carries none of these flags." summary: "DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw — NOT the 2026-10-17 '1017' re-release, which has its own sibling recipe) served EXL3 3bpw tensor-parallel 4 on four CMP 170HX cards (sm_80, 140 W, HBM locked 1728 MHz) in a 2x Xeon E5-2689 v4 Broadwell host, PCIe Gen2 x16, no NVLink, using Morrowmake's tensor-core MoE kernels + host-staged all-reduce ported onto this checkpoint: K3 on every served layer with a split K2 down-projection, row-rule hybrid (coop rows 1-42, moe_out_v2 43-64, moe_out_prefill 65-2048), one shared 540.4 MiB MoE workspace per GPU, zero-register-spill served kernels. Measured by the author: 178.3 tok/s structured decode c1, 587 tok/s across 8 users, median TTFT 8.37 s from 2K through 255K context (10.9 s at 255,424) against the same host's optimized graphs+coop+NCCL stack at 172.7 / 456 / 10.15 s; needles 9/9 through 256K; KV pool 616,916 tokens." license: CC-BY-4.0 model: @@ -61,6 +61,33 @@ config: - tp storage_requirements: note: 'Host RAM: no fixed 384 GB minimum (author machine has 378 GB visible; while serving it uses ~288 GB, ~190 GB of it the pinned Engram tables in system RAM — a 256 GB machine does not have room for that pin). 674 GB checkpoint streams from disk (lazy). ~62.5 GB of each 64 GB card is filled (48 GB cards do not fit).' + serving_selection_2026_10_26: + published_from: "CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE @ e559217ac4f90dd8c2e4820f97b259e584efcf1f (experiments/ds1017-native-mixed-and-split/SERVING-2026-10-26.md + experiments/ds1017-r2-b2-b8/README.md; commits 'Publish the 2-8 user lane and its default-off flags' and 'Note the October 26 serving selection and the missing image')" + measurements: 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX/blob/main/benchmarks/2026-10-26/README.md' + lane: '2-8 user decode lane = experiments/ds1017-r2-b2-b8 (separate source, DEFAULT OFF). The published C1 source stays the one-user bundle in experiments/ds1017-native-mixed-and-split (native mixed experts, split attention, captured K5 drafter).' + band: 'batches of 2-8 users; batch 7 pads onto the 48-row batch-8 graph; draft length stays 5 (6 rows per request). Draft length 7 and the Mia weights are NOT part of this selection.' + hardware: '4x CMP 170HX (sm_80) host as measured; the October 26 run used the 1017 weights on this host class.' + default_off_flags: + - + flag: DSV41_R2 + default: '0 (off)' + effect: 'Master switch for the 2-8 lane; 1 is the only armed value, malformed flag raises at boot. Being present in the repository does NOT turn the lane on.' + - + flag: VLLM_EXL3_TC_MOE_V2 + default: '0 (off)' + effect: 'Required for the tensor-core expert path on the 48-row graph; the launch command of this recipe arms it explicitly (-e VLLM_EXL3_TC_MOE_V2=1).' + - + flag: dsv41_draft_tq + default: 'off' + effect: 'Per-request sampled-drafter flag; NOT turned on by DSV41_R2. The October 26 selection keeps the drafter greedy at batches 2-8 and uses it only for a single temperature-1 request (outside this lane).' + - + flag: DSV41_ARENA_CACHE_DIR + default: 'unset (off)' + effect: 'Optional gate in scripts/run-tp4-tcmoe.sh: unset/empty = zero argv change; when set it adds -v DIR:DIR, -e VLLM_EXL3_ARENA_CACHE=1 and -e VLLM_EXL3_ARENA_CACHE_DIR=DIR.' + lane_defaults: 'DSV41_R2_BATCHES=2-8; DSV41_R2_COMPONENTS=attention,experts,collective,sampler,sampled; DSV41_R2_PAD7=1; DSV41_R2_SPLITS=per batch nearest the one-user 96-CTA shape (empty = default); DSV41_R2_EXPERTS=coop, or tc when tensor-core v2 covers the row count; DSV41_R2_COLLECTIVE=pair at 48 rows, otherwise the host all-reduce size rule; DSV41_R2_CAP_MIB=768 (retained-graph cap checked at startup).' + measured_plan: 'With DSV41_R2=1, DSV41_R2_PAD7=1 and VLLM_EXL3_TC_MOE_V2=1 and no per-batch overrides the defaults are the measured plan: cooperative experts for batches 2-6, tensor-core experts + pair collective at batch 8, batch 7 served by the batch-8 graph. New attention for every batch size; a temperature-0 request stays greedy at every batch size.' + image: "UNAVAILABLE (never published): the author states 'There is still no pullable image' (SERVING-2026-10-26.md). The only image identifier in the stack, sha256:55d2bd9a6664aea2540e3f8e373d784b88420d35c3cedefc9f03923cd687b6f5, is a LOCAL docker image id recorded for the measured serve (1017 repo README.md:93), not a registry reference; the site's image_digest is null. Cloning the repo does not turn the lane on: no image boots the 2-8 user graphs or the sampled drafter yet." + launch_consistency: "The launch command above arms VLLM_EXL3_TC_MOE_V2=1; DSV41_R2 is NOT set, so the published serve keeps the 2-8 lane default-off until armed. The optional arena-cache group (DSV41_ARENA_CACHE_DIR gate) is NOT in this recipe's launch command — it stays default-off; add the three flags above to enable it." performance: claimed_by: submitter metrics: @@ -149,7 +176,7 @@ artifacts: kind: reference pointer_type: git_commit pointer: 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE' - pin: cd83ade14c43a42babad39895d75bdd0e3453d49 + pin: e559217ac4f90dd8c2e4820f97b259e584efcf1f - kind: reference pointer_type: git_commit @@ -158,11 +185,14 @@ artifacts: provenance: credit: - - name: 'Curtis White' + name: CurtisAccelerate url: 'https://github.com/CurtisAccelerate' - name: Morrowmake url: 'https://github.com/Morrowmake/vllm-cmp170hx' + - + name: 'turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)' + url: 'https://github.com/turboderp-org/exllamav3' source_urls: - 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE' - 'https://github.com/Morrowmake/vllm-cmp170hx'