diff --git a/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake/recipe.yaml b/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake/recipe.yaml index 7a21d5814..9b7705df5 100644 --- a/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake/recipe.yaml +++ b/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake/recipe.yaml @@ -8,7 +8,7 @@ engine: pinned_commit: a0c57b52376efd67c57735bc949d4bafa22ae29e image_digest: 'ghcr.io/morrowmake/vllm-cmp170hx@sha256:f67d9be47ffbd4a8b84c77fe76bdb3f842187d9b2786fff573e390ae5361bfb0' install_note: 'Runs only as the pinned container image (sm_80 EXL3 + Marlin extensions prebuilt). One command from a fresh clone: git clone https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe && cd dsv41-flash-cmp170hx-recipe && ./start.sh (preflight, image pull, checkpoint download + hybrid build, launch, /health wait; DRY=1 prints the plan; docker compose up runs the same container). Needs: 4x CMP 170HX >=60 GiB each (the cards report 64), PCIe x16 links (warns below x16), NVIDIA driver 580+, Docker + NVIDIA Container Toolkit, ~680 GiB disk for the two checkpoints, ~240 GiB host RAM for Engram-in-RAM (auto falls back to SSD reads), optional P2P-capable driver build. ./start.sh smoke = one chat request and one tool call.' -summary: "DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw export, every other weight from DeepSeek's official FP8 1017 checkpoint (sha256-validated link build, no copies), and the model's ~189 GiB Engram tables in host RAM (VLLM_ENGRAM_PLACEMENT=auto, SSD fallback below ~212 GiB available). 319 tok/s one user structured (287 code / 162 prose), 765 across eight, 3,818 tok/s cold 2K prefill and a 1,247,406-token KV pool at 262,144 context, full-precision BF16 KV. Engine: Morrowmake's vLLM fork @ a0c57b52 on upstream 4e0681ed56 (sm_80 EXL3 + Marlin kernels, custom P2P all-reduce after a startup content check, DSpark speculative decoding 7/5 adaptive), pinned container image, TP4 with whole routed experts per card, optimised for PCIe x16 links. HumanEval 157/164, GSM8K 1,280/1,319, within test noise of the unmodified-checkpoint reference." +summary: "DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw export, every other weight from DeepSeek's official FP8 1017 checkpoint (sha256-validated link build, no copies), and the model's ~189 GiB Engram tables in host RAM (VLLM_ENGRAM_PLACEMENT=auto, SSD fallback below ~212 GiB available). up to 319 tok/s one user structured (287 code / 162 prose), up to 765 across eight, up to 3,818 tok/s cold 2K prefill and a 1,247,406-token KV pool at 262,144 context, full-precision BF16 KV. Engine: Morrowmake's vLLM fork @ a0c57b52 on upstream 4e0681ed56 (sm_80 EXL3 + Marlin kernels, custom P2P all-reduce after a startup content check, DSpark speculative decoding 7/5 adaptive), pinned container image, TP4 with whole routed experts per card, optimised for PCIe x16 links. HumanEval 157/164, GSM8K 1,280/1,319, within test noise of the unmodified-checkpoint reference." license: MIT model: name: DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw-Hybrid @@ -80,78 +80,76 @@ config: - '--tensor-parallel-size 4' - '--pipeline-parallel-size 1' performance: - claimed_by: submitter - metrics: - tok_s: 319.0 - context_len: 262144 - notes: "claimed_by=submitter T0 - the author's own release-check bench (docs/results.md @ pin, measured 2026-10-24 on the release 1.0.0 image, release defaults): 74 SMs/card, mainline cmpunlocker + minimal P2P patch, verified P2P, PCIe x16, 262,144 ctx, Engram tables in host RAM, 180 W/card. Decode: MiaAI-Lab protocol, 400-token completions, temp 0, thinking off, median of 5 runs (c2-c8 = all requests together); structured/code/prose measured on separate boots. c2 totals 360/431/230, c4 545/602/324. Cold prefill = prompt tokens to first token, median of 2. Fallback rows (reference, one boot each): SSD placement 151 c1 / 428 c8 / 3,607 prefill-2K; P2P off 133 c1 / 344 c8 / 3,804 prefill-2K (c2 192, c4 271). KV pool 1,247,406 tokens." - points: - - - context_len: 262144 - concurrency: 1 - decode_tok_s: 319.0 - notes: 'Structured decode c1 headline (TP4, P2P on, Engram in RAM) - median of 5 streaming runs.' - - - context_len: 272384 - concurrency: 1 - decode_tok_s: 287.0 - notes: 'Code decode c1 (TP4). context_len is the tuple marker 262144+10240 (code row, not structured).' - - - context_len: 282624 - concurrency: 1 - decode_tok_s: 162.0 - notes: 'Prose decode c1 (TP4). context_len is the tuple marker 262144+20480 (prose row, not structured/code).' - - - context_len: 262144 - concurrency: 8 - decode_tok_s: 765.0 - notes: 'Aggregate structured decode c8 total (TP4), median of 5. c2 360 / c4 545 in perf notes.' - - - context_len: 272384 - concurrency: 8 - decode_tok_s: 729.0 - notes: 'Aggregate code decode c8 total (TP4). context_len is the tuple marker 262144+10240.' - - - context_len: 282624 - concurrency: 8 - decode_tok_s: 432.0 - notes: 'Aggregate prose decode c8 total (TP4). context_len is the tuple marker 262144+20480.' - - - context_len: 2048 - prefill_tok_s: 3818.0 - notes: 'Cold prefill of a 2,048-token prompt (median of 2, TP4). context_len IS the prompt length; concurrency omitted (NULL).' - - - context_len: 32768 - prefill_tok_s: 3808.0 - notes: 'Cold prefill of a 32,768-token prompt (median of 2, TP4). context_len IS the prompt length; concurrency omitted (NULL).' - - - context_len: 303104 - concurrency: 1 - decode_tok_s: 151.0 - notes: 'REFERENCE FALLBACK: Engram tables from SSD (VLLM_ENGRAM_PLACEMENT=ssd, 48 GiB row cache per card), one boot. context_len is the tuple marker 262144+40960 (SSD row).' - - - context_len: 303104 - concurrency: 8 - decode_tok_s: 428.0 - notes: 'REFERENCE FALLBACK: aggregate c8 total with Engram on SSD. context_len is the tuple marker 262144+40960 (SSD row).' - - - context_len: 40960 - prefill_tok_s: 3607.0 - notes: 'REFERENCE FALLBACK: cold 2,048-token prefill with Engram on SSD. context_len 40960 is the tuple marker (SSD prefill row); concurrency omitted (NULL).' - - - context_len: 323584 - concurrency: 1 - decode_tok_s: 133.0 - notes: 'REFERENCE FALLBACK: P2P off (P2P=off), one boot. context_len is the tuple marker 262144+61440 (P2P-off row).' - - - context_len: 323584 - concurrency: 8 - decode_tok_s: 344.0 - notes: 'REFERENCE FALLBACK: aggregate c8 total with P2P off (c2 192 / c4 271 in perf notes). context_len is the tuple marker 262144+61440 (P2P-off row).' - - - context_len: 51200 - prefill_tok_s: 3804.0 - notes: 'REFERENCE FALLBACK: cold 2,048-token prefill with P2P off. context_len 51200 is the tuple marker (P2P-off prefill row); concurrency omitted (NULL).' + notes: 'claimed_by=submitter T0 - the author''s own release-check bench (docs/results.md @ pin, measured 2026-10-24 on the release 1.0.0 image, release defaults): 74 SMs/card, mainline cmpunlocker + minimal P2P patch, verified P2P, PCIe x16, 262,144 ctx, Engram tables in host RAM, 180 W/card. Decode: MiaAI-Lab protocol, 400-token completions, temp 0, thinking off, median of 5 runs (c2-c8 = all requests together); structured/code/prose measured on separate boots. c2 totals 360/431/230, c4 545/602/324. Cold prefill = prompt tokens to first token, median of 2. Fallback rows (reference, one boot each): SSD placement 151 c1 / 428 c8 / 3,607 prefill-2K; P2P off 133 c1 / 344 c8 / 3,804 prefill-2K (c2 192, c4 271). KV pool 1,247,406 tokens. Author hedged the tables at 7f9023c (docs-only drift, 2026-10-26): "all speeds are up to, in tok/s" - numbers unchanged; the image-cache wording became "carries no compiled kernels" (was "seed: no"), and the first-boot note was reworded.' + tok_s: 319 + points: + - + notes: 'Structured decode c1 headline (TP4, P2P on, Engram in RAM) - median of 5 streaming runs.' + concurrency: 1 + context_len: 262144 + decode_tok_s: 319 + - + notes: 'Code decode c1 (TP4). context_len is the tuple marker 262144+10240 (code row, not structured).' + concurrency: 1 + context_len: 272384 + decode_tok_s: 287 + - + notes: 'Prose decode c1 (TP4). context_len is the tuple marker 262144+20480 (prose row, not structured/code).' + concurrency: 1 + context_len: 282624 + decode_tok_s: 162 + - + notes: 'Aggregate structured decode c8 total (TP4), median of 5. c2 360 / c4 545 in perf notes.' + concurrency: 8 + context_len: 262144 + decode_tok_s: 765 + - + notes: 'Aggregate code decode c8 total (TP4). context_len is the tuple marker 262144+10240.' + concurrency: 8 + context_len: 272384 + decode_tok_s: 729 + - + notes: 'Aggregate prose decode c8 total (TP4). context_len is the tuple marker 262144+20480.' + concurrency: 8 + context_len: 282624 + decode_tok_s: 432 + - + notes: 'Cold prefill of a 2,048-token prompt (median of 2, TP4). context_len IS the prompt length; concurrency omitted (NULL).' + context_len: 2048 + prefill_tok_s: 3818 + - + notes: 'Cold prefill of a 32,768-token prompt (median of 2, TP4). context_len IS the prompt length; concurrency omitted (NULL).' + context_len: 32768 + prefill_tok_s: 3808 + - + notes: 'REFERENCE FALLBACK: Engram tables from SSD (VLLM_ENGRAM_PLACEMENT=ssd, 48 GiB row cache per card), one boot. context_len is the tuple marker 262144+40960 (SSD row).' + concurrency: 1 + context_len: 303104 + decode_tok_s: 151 + - + notes: 'REFERENCE FALLBACK: aggregate c8 total with Engram on SSD. context_len is the tuple marker 262144+40960 (SSD row).' + concurrency: 8 + context_len: 303104 + decode_tok_s: 428 + - + notes: 'REFERENCE FALLBACK: cold 2,048-token prefill with Engram on SSD. context_len 40960 is the tuple marker (SSD prefill row); concurrency omitted (NULL).' + context_len: 40960 + prefill_tok_s: 3607 + - + notes: 'REFERENCE FALLBACK: P2P off (P2P=off), one boot. context_len is the tuple marker 262144+61440 (P2P-off row).' + concurrency: 1 + context_len: 323584 + decode_tok_s: 133 + - + notes: 'REFERENCE FALLBACK: aggregate c8 total with P2P off (c2 192 / c4 271 in perf notes). context_len is the tuple marker 262144+61440 (P2P-off row).' + concurrency: 8 + context_len: 323584 + decode_tok_s: 344 + - + notes: 'REFERENCE FALLBACK: cold 2,048-token prefill with P2P off. context_len 51200 is the tuple marker (P2P-off prefill row); concurrency omitted (NULL).' + context_len: 51200 + prefill_tok_s: 3804 + context_len: 262144 artifacts: - kind: model_weights