DeepSeek-V4.1-Flash-1017 Hybrid EXL3 TP4 4x CMP 170HX Morrowmake

0 sign in to vote

recipe by Morrowmake (recipe, engine fork, measurements)↗, LibertAI - EXL3 2.9-bpw routed-expert quantization↗, DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)↗, cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch↗, MiaAI-Lab - decode benchmark protocol (used as data)↗, Marlin (IST-DASLab + Neural Magic contributors) - kernel basis↗ · listed by localai-recipes

revision r4 · commit 1d43d37b0b · updated 11 h ago · license MIT

chat code vision dspark exl3 speculative sm80 tp4 hybrid-checkpoint engram-ram

DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw export, every other weight from DeepSeek's official FP8 1017 checkpoint (sha256-validated link build, no copies), and the model's ~189 GiB Engram tables in host RAM (VLLM_ENGRAM_PLACEMENT=auto, SSD fallback below ~212 GiB available). up to 319 tok/s one user structured (287 code / 162 prose), up to 765 across eight, up to 3,818 tok/s cold 2K prefill and a 1,247,406-token KV pool at 262,144 context, full-precision BF16 KV. Engine: Morrowmake's vLLM fork @ a0c57b52 on upstream 4e0681ed56 (sm_80 EXL3 + Marlin kernels, custom P2P all-reduce after a startup content check, DSpark speculative decoding 7/5 adaptive), pinned container image, TP4 with whole routed experts per card, optimised for PCIe x16 links. HumanEval 157/164, GSM8K 1,280/1,319, within test noise of the unmodified-checkpoint reference.

Launch command

'docker run -d --name dsv41-flash --user <uid>:<gid> --workdir /cache --gpus all --shm-size 16g --memory <host MemTotal> --memory-swap <host MemTotal>' '-p 127.0.0.1:8000:8000' '-v <MODELS_DIR>/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw:/models/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw:ro' '-v <MODELS_DIR>/DeepSeek-V4.1-Flash-1017:/models/DeepSeek-V4.1-Flash-1017-engram:ro' '-v <MODELS_DIR>/DeepSeek-V4.1-Flash-1017-hybrid-exl3-fp8:/models/DeepSeek-V4.1-Flash-1017-hybrid-exl3-fp8:ro' '-v <CHECKOUT>/cache:/cache' '-v <CHECKOUT>/docker/entry.sh:/recipe/entry.sh:ro (plus boot_check.py, p2p_check.*, engram_placement.sh)' '--env-file <logs/container.env> --entrypoint /bin/bash ghcr.io/morrowmake/vllm-cmp170hx@sha256:f67d9be47ffbd4a8b84c77fe76bdb3f842187d9b2786fff573e390ae5361bfb0 /recipe/entry.sh' './start.sh builds logs/container.env (container_env): LAYOUT=tp4, HOST=0.0.0.0, PORT=8000, cache dirs under /cache, HF_HOME=/cache/huggingface; entry.sh re-decides Engram placement at each start (auto: ram with ~212 GiB available, else ssd) and execs the image'\''s /opt/recipe/serve.sh (LAYOUT=tp4), which runs:' /opt/venv/bin/vllm serve /models/DeepSeek-V4.1-Flash-1017-hybrid-exl3-fp8 '--host 0.0.0.0' '--served-model-name dsv41-flash' '--max-model-len 262144' '--gpu-memory-utilization 0.95' '--max-num-seqs 8' '--max-num-batched-tokens 2048' '--port 8000' '--hf-overrides {"engram_table_dir":"/models/DeepSeek-V4.1-Flash-1017-engram"}' '--load-format exl3' --enable-expert-parallel '--mm-encoder-tp-mode data' '--kv-cache-dtype bfloat16' '--reasoning-parser deepseek_v41' --enable-auto-tool-choice '--tool-call-parser deepseek_v41' '--compilation-config {"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[2,3,4,5,6,8,9,10,12,14,15,16,18,20,21,24,25,28,30,32,35,36,40,42,48]}' '--speculative-config {"method":"dspark","num_speculative_tokens":7,"adaptive_k":{"by_load":[7,5,5,5,5,5,5,5,1,2,3,4,5],"draft_by_load":true},"enable_adaptive_verification":false}' '--prefill-chunk-with-decodes 512' --exclude-tools-when-tool-choice-none '--tensor-parallel-size 4' '--pipeline-parallel-size 1'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake.json

Engine

vllm · source↗ · a0c57b52↗ · ghcr.io/morrowmake/vllm-cmp170hx@sha256:f67d9be47ffbd4a8b84c77fe76bdb3f842187d9b2786fff573e390ae5361bfb0↗

Install note: Runs only as the pinned container image (sm_80 EXL3 + Marlin extensions prebuilt). One command from a fresh clone: git clone https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe && cd dsv41-flash-cmp170hx-recipe && ./start.sh (preflight, image pull, checkpoint download + hybrid build, launch, /health wait; DRY=1 prints the plan; docker compose up runs the same container). Needs: 4x CMP 170HX >=60 GiB each (the cards report 64), PCIe x16 links (warns below x16), NVIDIA driver 580+, Docker + NVIDIA Container Toolkit, ~680 GiB disk for the two checkpoints, ~240 GiB host RAM for Engram-in-RAM (auto falls back to SSD reads), optional P2P-capable driver build. ./start.sh smoke = one chat request and one tool call.

Model

DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw-Hybrid (exl3-2-9bpw-routed-experts-fp8-core-hybrid) · LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw↗
AA intelligence 45 (Artificial Analysis · measured)

Hardware

NVIDIA CMP 170HX 64GB ×4 (Ampere)

Context

max context: 256K tokens

KV pool

1,247,406 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
2,048	—	3818 tok/s	—	—	reported
32,768	—	3808 tok/s	—	—	reported
40,960	—	3607 tok/s	—	—	reported
51,200	—	3804 tok/s	—	—	reported
262,144	1	—	319 tok/s	—	reported
262,144	8	—	765 tok/s	—	reported
272,384	1	—	287 tok/s	—	reported
272,384	8	—	729 tok/s	—	reported
282,624	1	—	162 tok/s	—	reported
282,624	8	—	432 tok/s	—	reported
303,104	1	—	151 tok/s	—	reported
303,104	8	—	428 tok/s	—	reported
323,584	1	—	133 tok/s	—	reported
323,584	8	—	344 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	claimed_by=submitter T0 - the author's own release-check bench (docs/results.md @ pin, measured 2026-10-24 on the release 1.0.0 image, release defaults): 74 SMs/card, mainline cmpunlocker + minimal P2P patch, verified P2P, PCIe x16, 262,144 ctx, Engram tables in host RAM, 180 W/card. Decode: MiaAI-Lab protocol, 400-token completions, temp 0, thinking off, median of 5 runs (c2-c8 = all requests together); structured/code/prose measured on separate boots. c2 totals 360/431/230, c4 545/602/324. Cold prefill = prompt tokens to first token, median of 2. Fallback rows (reference, one boot each): SSD placement 151 c1 / 428 c8 / 3,607 prefill-2K; P2P off 133 c1 / 344 c8 / 3,804 prefill-2K (c2 192, c4 271). KV pool 1,247,406 tokens. Author hedged the tables at 7f9023c (docs-only drift, 2026-10-26): "all speeds are up to, in tok/s" - numbers unchanged; the image-cache wording became "carries no compiled kernels" (was "seed: no"), and the first-boot note was reworded.

notes

claimed_by=submitter T0 - the author's own release-check bench (docs/results.md @ pin, measured 2026-10-24 on the release 1.0.0 image, release defaults): 74 SMs/card, mainline cmpunlocker + minimal P2P patch, verified P2P, PCIe x16, 262,144 ctx, Engram tables in host RAM, 180 W/card. Decode: MiaAI-Lab protocol, 400-token completions, temp 0, thinking off, median of 5 runs (c2-c8 = all requests together); structured/code/prose measured on separate boots. c2 totals 360/431/230, c4 545/602/324. Cold prefill = prompt tokens to first token, median of 2. Fallback rows (reference, one boot each): SSD placement 151 c1 / 428 c8 / 3,607 prefill-2K; P2P off 133 c1 / 344 c8 / 3,804 prefill-2K (c2 192, c4 271). KV pool 1,247,406 tokens. Author hedged the tables at 7f9023c (docs-only drift, 2026-10-26): "all speeds are up to, in tok/s" - numbers unchanged; the image-cache wording became "carries no compiled kernels" (was "seed: no"), and the first-boot note was reworded.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
official FP8 1017 @ 1bae0e1088993b9b82de08f59de5be5a34e42170 - every weight but the routed experts, plus the Engram tables
Weights https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw
EXL3 2.9bpw routed experts @ c178a01284fc5c0e859d8668cffaac81ca8f4042 (docker/prepare.sh pin)
Engine image https://ghcr.io/morrowmake/vllm-cmp170hx@sha256:f67d9be47ffbd4a8b84c77fe76bdb3f842187d9b2786fff573e390ae5361bfb0
release 1.0.0 image - the engine; sm_80 EXL3 + Marlin extensions prebuilt
Engine image https://github.com/Morrowmake/vllm-cmp170hx
fork @ a0c57b52376efd67c57735bc949d4bafa22ae29e on upstream vLLM 4e0681ed56
Reference https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe
recipe repo @ d4d1f525d7323be13dd32ffd594c23c5f73923bf (release 1.0.0 results)
Reference https://github.com/amoghmunikote/cmpunlocker
mainline cmpunlocker + Morrowmake's minimal TRAP31 P2P patch (ready build: github.com/Morrowmake/cmpunlocker)

Revision notes

Community ingest by localai-recipes (owner direct ask 2026-10-24) of Morrowmake's release 1.0.0 recipe repo @ d4d1f525 - HEAD at sweep == HEAD at ingest, drift-free. All perf numbers are the author's own measurements (claimed_by=submitter T0; docs/results.md @ pin, measured 2026-10-24 on the release image with release defaults). NOT a dupe: same-family 4x-CMP-170HX DeepSeek rows are different repos/engines - GNmac-lab Engram-NVMe (own patch stack, Engram streamed from NVMe, TP4+PP4), CurtisAccelerate EXL3 TC-MoE (full-EXL3 experts, no hybrid), kaka86mm PP4 shadow-KV - this is Morrowmake's own engine fork + hybrid EXL3/FP8 + Engram-in-RAM recipe, the first on the site from this repo. CORRECTION to the scout pre-flight note: this repo is SINGLE-LAYOUT TP4 only (start.sh dies on any other LAYOUT; PP=1) - no PP4 lane exists here. Engine: fork @ a0c57b52 on upstream 4e0681ed56, image digest f67d9be4 (release 1.0.0). Weights pinned by the repo's docker/prepare.sh: official FP8 1017 @ 1bae0e10, LibertAI EXL3 2.9bpw @ c178a012; the hybrid build's written files are sha256-validated at every start. P2P is enabled automatically after a startup content check and runs disabled without it (fallback perf rows). Perf-point context_len markers: decode rows base 262144 (+10240 code, +20480 prose, +40960 SSD-fallback, +61440 P2P-off-fallback); prefill rows use the real prompt lengths 2048/32768 (main) and markers 40960/51200 (fallbacks); concurrency omitted on prefill rows (NULL) so no (ctx,conc,claimed_by) dedup collisions. Recipe licence MIT (CITATION.cff).

Revision history

1d43d37b0b recipe: update deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake (r4: /tags) 5 h ago diff
59f85eff01 recipe: update deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake (r3: /provenance/credit) 15 h ago diff
bb2931f496 recipe: update deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake (r2: /summary, /performance) 3 d ago diff
f575158987 recipe: create deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake 6 d ago

Fork

Export

recipe.yaml (raw)