diff --git a/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml b/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml index 50a9b2cb4..59b7b30fa 100644 --- a/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml +++ b/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4/recipe.yaml @@ -5,7 +5,7 @@ engine: source_type: fork version: 'upstream vllm + Morrowmake sm80 host-allreduce 18305948' git_url: 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE' - pinned_commit: 88eb6e13de0ea5491579fddb195116a4176f4bc7 + pinned_commit: e559217ac4f90dd8c2e4820f97b259e584efcf1f install_note: "Same serving image + EXL3 plugin + TC-MoE .so as the EXL3 3bpw TC-MoE recipe (this checkpoint changes weights, not the stack): build out/_dsv41_tc_moe_C.abi3.so from DeepSeekv41-Flash-EXL3-TC-MoE build_standalone.py against the image's PyTorch. Hybrid checkpoint = official 1017 FP8 tensors (attention/shared/head/draft/Engram shards 47-48) + LibertAI routed experts layers 0-39 + quantization_config.json from the 1017 repo. No published container image; weights are not included." summary: 'DeepSeek-V4.1-Flash-1017 (the 2026-10-17 re-release checkpoint, NOT the original DeepSeek-V4.1-Flash) as a hybrid quant on the TC-MoE port: official deepseek-ai/DeepSeek-V4.1-Flash-1017 FP8 tensors for attention, shared experts, head, DSpark draft and Engram (shards 47/48), with LibertAIDAI/...1017-EXL3-2.9bpw routed experts (layers 0-39, mostly 3-bit, layers 18-22 2-bit) TP4 on four CMP 170HX cards (sm_80, 140 W, ~62.5 GB/card). Same Morrowmake kernels + image as the EXL3 3bpw TC-MoE recipe; VLLM_EXL3_COOP=1; EXL3 arena cache (~1.1 GiB/layer, load shortcut only). 1017 quantization_config.json is the switch (dsv41_main_routed_experts_only, mtp_experts=source, non-routed stays FP8, compressor projection BF16). Author-measured: 180.4/137.1/96.6 tok/s structured/code/prose c1; 8-user aggregate 621.1/376.1/282.5 vs 564.7 previous Mia 3.0bpw. Cold 131k prompt 1,330 prefill / 163.9 decode. Needles 9/9. KV pool 1,534,065 tokens; vision on, drafter text-only.' license: CC-BY-4.0 @@ -69,68 +69,88 @@ config: ram_gb: 378 nvme_gb: 674 note: 'Host needs 384 GB RAM for the Engram pin (265 GB shared-mem pin; serving ~289 GB); 256 GB hosts do not fit. ~62.5 GB of each 64 GB card filled. Arena cache on fast disk.' + serving_selection_2026_10_26: + published_from: "CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE @ e559217ac4f90dd8c2e4820f97b259e584efcf1f (experiments/ds1017-native-mixed-and-split/SERVING-2026-10-26.md + experiments/ds1017-r2-b2-b8/README.md; commits 'Publish the 2-8 user lane and its default-off flags' and 'Note the October 26 serving selection and the missing image')" + measurements: 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX/blob/main/benchmarks/2026-10-26/README.md' + lane: '2-8 user decode lane = experiments/ds1017-r2-b2-b8 (separate source, DEFAULT OFF). The published C1 source stays the one-user bundle in experiments/ds1017-native-mixed-and-split (native mixed experts, split attention, captured K5 drafter).' + band: 'batches of 2-8 users; batch 7 pads onto the 48-row batch-8 graph; draft length stays 5 (6 rows per request). Draft length 7 and the Mia weights are NOT part of this selection.' + hardware: '4x CMP 170HX (sm_80) host as measured; the October 26 run used the 1017 weights on this host class.' + default_off_flags: + - + flag: DSV41_R2 + default: '0 (off)' + effect: 'Master switch for the 2-8 lane; 1 is the only armed value, malformed flag raises at boot. Being present in the repository does NOT turn the lane on.' + - + flag: VLLM_EXL3_TC_MOE_V2 + default: '0 (off)' + effect: 'Required for the tensor-core expert path on the 48-row graph; the launch command of this recipe arms it explicitly (-e VLLM_EXL3_TC_MOE_V2=1).' + - + flag: dsv41_draft_tq + default: 'off' + effect: 'Per-request sampled-drafter flag; NOT turned on by DSV41_R2. The October 26 selection keeps the drafter greedy at batches 2-8 and uses it only for a single temperature-1 request (outside this lane).' + - + flag: DSV41_ARENA_CACHE_DIR + default: 'unset (off)' + effect: 'Optional gate in scripts/run-tp4-tcmoe.sh: unset/empty = zero argv change; when set it adds -v DIR:DIR, -e VLLM_EXL3_ARENA_CACHE=1 and -e VLLM_EXL3_ARENA_CACHE_DIR=DIR.' + lane_defaults: 'DSV41_R2_BATCHES=2-8; DSV41_R2_COMPONENTS=attention,experts,collective,sampler,sampled; DSV41_R2_PAD7=1; DSV41_R2_SPLITS=per batch nearest the one-user 96-CTA shape (empty = default); DSV41_R2_EXPERTS=coop, or tc when tensor-core v2 covers the row count; DSV41_R2_COLLECTIVE=pair at 48 rows, otherwise the host all-reduce size rule; DSV41_R2_CAP_MIB=768 (retained-graph cap checked at startup).' + measured_plan: 'With DSV41_R2=1, DSV41_R2_PAD7=1 and VLLM_EXL3_TC_MOE_V2=1 and no per-batch overrides the defaults are the measured plan: cooperative experts for batches 2-6, tensor-core experts + pair collective at batch 8, batch 7 served by the batch-8 graph. New attention for every batch size; a temperature-0 request stays greedy at every batch size.' + image: "UNAVAILABLE (never published): the author states 'There is still no pullable image' (SERVING-2026-10-26.md). The only image identifier in the stack, sha256:55d2bd9a6664aea2540e3f8e373d784b88420d35c3cedefc9f03923cd687b6f5, is a LOCAL docker image id recorded for the measured serve (1017 repo README.md:93), not a registry reference; the site's image_digest is null. Cloning the repo does not turn the lane on: no image boots the 2-8 user graphs or the sampled drafter yet." + launch_consistency: 'The launch command above already carries the opt-in arena-cache group (-e VLLM_EXL3_ARENA_CACHE=1 -e VLLM_EXL3_ARENA_CACHE_DIR=) and arms VLLM_EXL3_TC_MOE_V2=1; DSV41_R2 is NOT set in the launch command, so the published serve keeps the 2-8 lane default-off until armed.' performance: claimed_by: submitter metrics: - tok_s: 180.4 - ttft_ms: 2160 - context_len: 2080 - notes: 'Author-reported T0 (2026-10-18 repo): median of 3 warmed runs, thinking off, temperature 0, same prompts as the previous 140 W sweep; decode per stream after the first token. Headline 180.4 structured / 137.1 code / 96.6 prose decode c1; 8-user aggregate 621.1 / 376.1 / 282.5 (vs 564.7 previous structured). --max-model-len 262144 is the measured cap (weights address 1,048,576, YaRN x16 from 65,536). KV pool 1,534,065 tokens. Needles 9/9. Long-context rows are /v1/completions, 128 output tokens, median of 3.' + notes: 'Measured 2026-10-25 cumulative single-user decode (benchmarks/2026-10-25/results.json @ CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX 8b4e52906e, sha256 3f8e70fa7e5538b30dfe5db887a574eeb8872781203ade02d76bc898069cb992; hardware 4x CMP 170HX; metric "Single-user streamed decode tokens/second"; baseline "Published October22 results, same published configuration across each suite"). Table (Task | Published baseline | Part 1: native mixed | Part 1 change | Part 2: combined | Part 2 change | Final total; tok/s): Short code 144.23 | 146.28 | +1.42% | 155.54 | +6.33% | +7.84%; Short prose 101.16 | 102.82 | +1.64% | 116.05 | +12.87% | +14.72%; Spark-matched code 158.36 | 169.44 | +7.00% | 175.73 | +3.71% | +10.97%; Spark-matched prose 83.64 | 88.46 | +5.76% | 94.65 | +7.00% | +13.16%. Spark-matched rows are the Spark PROMPT SUITE run on THIS host (the same 4x CMP 170HX machine) — not a DGX Spark result, not a hardware comparison, not a second system. Separate statistic (not a column, not added to the historical percentages, per limitations): median paired Part 2 gains in the same-load three-pair comparison — short code +6.33%, short prose +12.87%, Spark-matched code +3.71%, Spark-matched prose +6.25%. Controlled counting (same host): 187.723 -> 195.423 tok/s, +4.10% — 24 requests in 6 balanced blocks; matched output/prompt/cache/speculative work; counting only; not a +4.10% claim for every code/prose request. Spark-suite C8 aggregate (this host): 275.11 -> 276.55 tok/s (+0.52%, ratio of medians). Cold prefill 93,335 prompt tokens at 1,389.0 -> 1,391.1 prompt tok/s; prefill = prompt ingestion in the same passes, not decode; no consistent C4/C8 or cold-prefill gain. Matched task-quality screen: code 5/5 vs 4/5, prose strict 0/5 vs 1/5, prose checklist+format 5/5 vs 5/5; quality_equivalence_established false; experimental split attention is not bit-exact, broader verification needed. Historical changes describe configurations across runs, not isolated causal component effects. No row derives from benchmarks/2026-10-24/broadwell-vs-morrowmake.' points: - context_len: 2048 concurrency: 1 - decode_tok_s: 180.4 - notes: 'Structured c1 headline (median of 3 warmed).' + decode_tok_s: 155.54 + notes: 'Measured 2026-10-25 combined decode, Short code (benchmarks/2026-10-25/results.json @ 8b4e52906e): baseline 144.23, native mixed 146.28 (+1.42%), combined 155.54 (Part 2 +6.33%; paired median +6.33%), total +7.84%. context_len is the tuple marker for the short-code row, not a prompt length.' - - context_len: 2048 - concurrency: 2 - decode_tok_s: 155.3 - notes: 'Structured c2 decode (8-user table: aggregate 248.0).' - - - context_len: 2048 - concurrency: 4 - decode_tok_s: 114.2 - notes: 'Structured c4 decode (8-user table: aggregate 419.2).' - - - context_len: 2048 - concurrency: 8 - decode_tok_s: 85.1 - notes: 'Structured c8 decode; 8-user aggregate 621.1 vs 564.7 on the previous Mia 3.0bpw serve.' - - - context_len: 12288 + context_len: 2080 concurrency: 1 - decode_tok_s: 137.1 - notes: 'Code c1 headline. context_len is the tuple marker 2048+10240 (code row, not structured).' + decode_tok_s: 116.05 + notes: 'Measured 2026-10-25 combined decode, Short prose (same source): baseline 101.16, native mixed 102.82 (+1.64%), combined 116.05 (Part 2 +12.87%; paired median +12.87%), total +14.72%. context_len is the tuple marker for the short-prose row, not a prompt length.' - - context_len: 22528 + context_len: 2112 concurrency: 1 - decode_tok_s: 96.6 - notes: 'Prose c1 headline. context_len is the tuple marker 2048+20480 (prose row, not structured/code).' + decode_tok_s: 175.73 + notes: 'Spark-matched code: the Spark PROMPT SUITE run on THIS host (4x CMP 170HX) — a prompt suite, not a DGX Spark hardware comparison (benchmarks/2026-10-25/results.json @ 8b4e52906e): baseline 158.36, native mixed 169.44 (+7.00%), combined 175.73 (Part 2 +3.71%; paired median +3.71%), total +10.97%. context_len is the tuple marker.' - - context_len: 8192 + context_len: 2144 concurrency: 1 - decode_tok_s: 148.2 - prefill_tok_s: 1531 - notes: 'Cold /v1/completions long-context curve, 8,192 prompt tokens, 128 out.' + decode_tok_s: 94.65 + notes: 'Spark-matched prose: the Spark PROMPT SUITE run on THIS host (4x CMP 170HX) — a prompt suite, not a DGX Spark hardware comparison (same source): baseline 83.64, native mixed 88.46 (+5.76%), combined 94.65 (Part 2 +7.00%; paired median +6.25% is the separate same-load statistic), total +13.16%. context_len is the tuple marker.' - - context_len: 32768 + context_len: 3072 concurrency: 1 - decode_tok_s: 148.4 - prefill_tok_s: 1526 - notes: 'Cold /v1/completions long-context curve, 32,768 prompt tokens.' + decode_tok_s: 187.723 + notes: 'Controlled native-mixed counting BEFORE (same host): 187.723 tok/s — 24 requests in 6 balanced blocks; matched output/prompt/cache/speculative work; counting only. context_len is the tuple marker.' - - context_len: 65536 + context_len: 3104 concurrency: 1 - decode_tok_s: 150.5 - prefill_tok_s: 1460 - notes: 'Cold /v1/completions long-context curve, 65,536 prompt tokens.' + decode_tok_s: 195.423 + notes: 'Controlled native-mixed counting AFTER (same host): 195.423 tok/s, +4.10% vs 187.723; counting only — not a claim for every code/prose request. context_len is the tuple marker.' + - + context_len: 4096 + concurrency: 8 + decode_tok_s: 275.11 + notes: 'Spark-suite C8 aggregate BEFORE (prompt suite on THIS host): 275.11 tok/s, 8 concurrent streams.' - - context_len: 131072 + context_len: 4128 + concurrency: 8 + decode_tok_s: 276.55 + notes: 'Spark-suite C8 aggregate AFTER (prompt suite on THIS host): 276.55 tok/s, +0.52% ratio of medians; no consistent C4/C8 gain.' + - + context_len: 93335 + concurrency: null + prefill_tok_s: 1389 + notes: 'Cold prefill of 93,335 prompt tokens BEFORE: 1,389.0 prompt tok/s; prefill = prompt ingestion, not decode. context_len IS the prompt length; concurrency NULL (before-arm of the paired rows).' + - + context_len: 93335 concurrency: 1 - decode_tok_s: 163.9 - prefill_tok_s: 1330 - notes: 'Cold /v1/completions long-context curve, 131,072 prompt tokens; 99.37 s wall.' + prefill_tok_s: 1391.1 + notes: 'Cold prefill of 93,335 prompt tokens AFTER: 1,391.1 prompt tok/s; prefill = prompt ingestion, not decode. context_len IS the prompt length; concurrency is the tuple marker for the after-arm row.' artifacts: - kind: model_weights @@ -146,12 +166,12 @@ artifacts: kind: reference pointer_type: git_commit pointer: 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX' - pin: b8b9e993e015f35c22ab7202b4ef3c05f8b663f0 + pin: 8b4e52906ec255b74bfec546d23d2a8741a86946 - kind: reference pointer_type: git_commit pointer: 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE' - pin: 88eb6e13de0ea5491579fddb195116a4176f4bc7 + pin: e559217ac4f90dd8c2e4820f97b259e584efcf1f - kind: reference pointer_type: git_commit @@ -160,14 +180,17 @@ artifacts: provenance: credit: - - name: 'Curtis White' + name: CurtisAccelerate url: 'https://github.com/CurtisAccelerate' - name: Morrowmake url: 'https://github.com/Morrowmake/vllm-cmp170hx' - - name: 'Measured serve image sha256:55d2bd9a6664aea2540e3f8e373d784b88420d35c3cedefc9f03923cd687b6f5 (named upstream 2026-10-18,' + name: 'Measured serve used local image sha256:55d2bd9a6664 (no published image)' url: 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX' + - + name: 'turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)' + url: 'https://github.com/turboderp-org/exllamav3' - name: 'PixelML/sm80vllm branch sm80 (DeepseekV4AmpereMLAAttention) - SM80 attention for this model' url: 'https://github.com/PixelML/sm80vllm' @@ -176,5 +199,9 @@ provenance: - 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE' - 'https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017' - 'https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw' - notes: "Scouted + ingested by localai-recipes from CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX (created 2026-10-18, pinned @ b8b9e993); 1017 weight selection, hybrid recipe and measurements by Curtis White. This recipe serves the 1017 checkpoint (deepseek-ai/DeepSeek-V4.1-Flash-1017 @ 1bae0e10, official FP8 attention/shared-experts/head/draft/Engram shards 47-48 + LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw @ 0be63c33 routed experts layers 0-39, MIT licensed; the quant is LibertAI's) — NOT the original deepseek-ai/DeepSeek-V4.1-Flash, and not the Mia-AiLab EXL3 3.0bpw checkpoint of the sibling EXL3 3bpw TC-MoE recipe. Same Morrowmake kernel base vllm-cmp170hx @ 18305948 (Apache-2.0) and EXL3 codec (turboderp exllamav3, MIT) as that sibling. AA Intelligence 45 is AA's published index for DeepSeek V4.1 Flash 1017 (Max), read from the AA model page 2026-10-19 (Index v4.3.2; original V4.1 Flash is a separate 39 entry). All performance numbers author-reported (T0)." + notes: "Scouted + ingested by localai-recipes from CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX (created 2026-10-18, pinned @ b8b9e993); 1017 weight selection, hybrid recipe and measurements by Curtis White. This recipe serves the 1017 checkpoint (deepseek-ai/DeepSeek-V4.1-Flash-1017 @ 1bae0e10, official FP8 attention/shared-experts/head/draft/Engram shards 47-48 + LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw @ 0be63c33 routed experts layers 0-39, MIT licensed; the quant is LibertAI's) — NOT the original deepseek-ai/DeepSeek-V4.1-Flash, and not the Mia-AiLab EXL3 3.0bpw checkpoint of the sibling EXL3 3bpw TC-MoE recipe. Same Morrowmake kernel base vllm-cmp170hx @ 18305948 (Apache-2.0) and EXL3 codec (turboderp exllamav3, MIT) as that sibling. AA Intelligence 45 is AA's published index for DeepSeek V4.1 Flash 1017 (Max), read from the AA model page 2026-10-19 (Index v4.3.2; original V4.1 Flash is a separate 39 entry). All performance numbers author-reported (T0). 1017 measured table refresh (kanban t_1bc89b18, deferred lane of t_22817e0f, preprod only per owner ruling 2026-10-28): performance rebuilt cell-exact from benchmarks/2026-10-25/results.json (sha256 3f8e70fa7e5538b30dfe5db887a574eeb8872781203ade02d76bc898069cb992) at the re-pinned 1017 commit 8b4e52906e; pin honesty — the cited file exists only from 8b4e52906e (b8b9e993e0 -> 8b4e52906e is 14 commits, GitHub compare 'ahead'); fdb629e/a06e30d9 NOT pinned (Oct-28 gate-kernel commits post-date the audited evidence; next sweep). The comparison measures Morrowmake's repo run on this Broadwell. Those figures are our measurements, not numbers published by Morrowmake. Stale October-18 headline points and the long-context curve retired; perf points 10 -> 10." kv_pool_tokens: 1534065 +storage_requirements: + ram_gb: 378.0 + nvme_gb: 674.0 + note: 'Host has 378 GB RAM visible to the OS - the machine, not a minimum. Serving uses ~288 GB (~190 GB Engram pin in system RAM; a 256 GB machine does not have room). ~62.5 GB of each 64 GB card filled.'