{"meta":{"count":7,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":"deepseek-v4-1-flash-1017","quant":null,"storage":null,"hardware":null,"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T06:49:44+00:00"},"recipes":[{"slug":"deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake","title":"DeepSeek-V4.1-Flash-1017 Hybrid EXL3 TP4 4x CMP 170HX Morrowmake","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw export, every other weight from DeepSeek's official FP8 1017 checkpoint (sha256-validated link build, no copies), and the model's ~189 GiB Engram tables in host RAM (VLLM_ENGRAM_PLACEMENT=auto, SSD fallback below ~212 GiB available). up to 319 tok/s one user structured (287 code / 162 prose), up to 765 across eight, up to 3,818 tok/s cold 2K prefill and a 1,247,406-token KV pool at 262,144 context, full-precision BF16 KV. Engine: Morrowmake's vLLM fork @ a0c57b52 on upstream 4e0681ed56 (sm_80 EXL3 + Marlin kernels, custom P2P all-reduce after a startup content check, DSpark speculative decoding 7/5 adaptive), pinned container image, TP4 with whole routed experts per card, optimised for PCIe x16 links. HumanEval 157/164, GSM8K 1,280/1,319, within test noise of the unmodified-checkpoint reference.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":162,"prefill_tok_s":3804,"updated_at":"2026-10-30 08:13:13.515043-06","commit_sha":"1d43d37b0b9cffa384845e62bcd3af425c04088c","content_hash":"79c2a1c9569e13d38af257c3a7c84e011e8a9cbe","votes":0,"credit":[{"name":"Morrowmake (recipe, engine fork, measurements)","url":"https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe"},{"name":"LibertAI - EXL3 2.9-bpw routed-expert quantization","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"MiaAI-Lab - decode benchmark protocol (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"Marlin (IST-DASLab + Neural Magic contributors) - kernel basis","url":"https://github.com/IST-DASLab/marlin"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake.json","alternate":"/r/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake"}},{"slug":"deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell","title":"DeepSeek-V4.1-Flash-1017 Kraken TP3 3x RTX PRO 6000 Blackwell","summary":"DeepSeek-V4.1-Flash-1017 at TP3 on three 96 GB RTX PRO 6000 Blackwell cards with the full 1M-token context, serving DeepSeek's original weights (FP8 dense + MXFP4 routed experts, nothing requantized). Expert layers 20-33 park in host RAM against a 1.5 GiB promotion reserve; once healthy their hottest experts promote back to GPU (hot-first order from 1017 route counts, lossless FlashInfer CSF scale compression). Spreading that fixed budget across 14 layers instead of 6 shallows each layer's cold tail - the author reports 68.0 -> 72.3 engine steps/s at 32K decode (+6.3%), no quality, KV or prefill cost. Engram layer 1 pinned in host RAM, layer 14 streamed from NVMe. DSpark speculative decoding, 5 drafts verified, NVFP4 drafter head. 1,048,576-token window over a 2,185,581-token KV pool, up to 4 sequences. Measured on image 20261019.3: prefill 13,486/12,245 tok/s (32K/128K), 7,193 at 983K; decode 186.7/340.7 c1/c4 at 32K, 179.2/307.1 at 128K, 474 aggregate on 4 shorts, 271.8 coding peak.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":228.5,"prefill_tok_s":11445.5,"updated_at":"2026-10-29 23:29:37.139691-06","commit_sha":"fe33370ee6a18643174519c8b9e956f05e807711","content_hash":"252307b64eed922ec9291880d6efb1938f670372","votes":0,"credit":[{"name":"BoxTops4Coochie (recipe, TP3 port, measurements)","url":"https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes"},{"name":"infernix - ds41f-tp3 expert-parking design on karmic-kraken","url":"https://github.com/infernix"},{"name":"Local Inference Lab - vLLM karmic-kraken fork + FlashInfer b12x kernels","url":"https://github.com/local-inference-lab"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell.json","alternate":"/r/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell"}},{"slug":"deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81","title":"DeepSeek-V4.1-Flash-1017 Fidelity-First 4x CMP 170HX alanoo81","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four changes: routed experts of layers 18-22 grafted 2-bit to 3-bit (LibertAIDAI EXL3 3.5bpw shards 00020-00024-of-00044 into the 2.9bpw pack, all 40 layers 3-bit); no DSpark drafter (SPEC_TOKENS=0, frees ~2.8 GiB/card for KV); Engram tables genuinely loaded (ENGRAM_STORAGE=ssd from a 200 GiB RAM tmpfs); gpu_memory_utilization 0.98, KV pool 2,717,420 tokens at 262,144 context. Author-measured: 300/300 exact identifier copies, byte-identical repeats; perplexity -1.80% overall / -4.45% French vs the 2.9bpw pack (95% CIs exclude zero); ~105 tok/s decode c1, 449-560 tok/s aggregate c8; cold prefill 2,514 tok/s median. Documents a silent failure: leaving DSV41_ENGRAM_STORAGE unset (host mode) leaves the Engram tables unfilled - layers 1 and 14 act as identity functions, 15/20 copies, temp-0 output drift, no error.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":105,"prefill_tok_s":2514,"updated_at":"2026-10-29 21:57:43.459028-06","commit_sha":"5af1460f4a2dce4f590f6fdfcd108119285e3e7c","content_hash":"67eb1810e7eea071e85e8edab747fd98247dd872","votes":0,"credit":[{"name":"alanoo81 (recipe, graft, measurements)","url":"https://github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx"},{"name":"GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"Curtis White (hybrid layout + quantization_config.json)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)","url":"https://github.com/vcruz305/vllm-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81.json","alternate":"/r/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81"}},{"slug":"deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw TC-MoE on 4x CMP 170HX TP4","summary":"DeepSeek-V4.1-Flash-1017 (the 2026-10-17 re-release checkpoint, NOT the original DeepSeek-V4.1-Flash) as a hybrid quant on the TC-MoE port: official deepseek-ai/DeepSeek-V4.1-Flash-1017 FP8 tensors for attention, shared experts, head, DSpark draft and Engram (shards 47/48), with LibertAIDAI/...1017-EXL3-2.9bpw routed experts (layers 0-39, mostly 3-bit, layers 18-22 2-bit) TP4 on four CMP 170HX cards (sm_80, 140 W, ~62.5 GB/card). Same Morrowmake kernels + image as the EXL3 3bpw TC-MoE recipe; VLLM_EXL3_COOP=1; EXL3 arena cache (~1.1 GiB/layer, load shortcut only). 1017 quantization_config.json is the switch (dsv41_main_routed_experts_only, mtp_experts=source, non-routed stays FP8, compressor projection BF16). Author-measured: 180.4/137.1/96.6 tok/s structured/code/prose c1; 8-user aggregate 621.1/376.1/282.5 vs 564.7 previous Mia 3.0bpw. Cold 131k prompt 1,330 prefill / 163.9 decode. Needles 9/9. KV pool 1,534,065 tokens; vision on, drafter text-only.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":165.599999999999994315658113919198513031005859375,"prefill_tok_s":1390.09999999999990905052982270717620849609375,"updated_at":"2026-10-29 21:32:22.449013-06","commit_sha":"306225cea85cf158c3036c971b031a9c1740075b","content_hash":"59b7b30fa1a9d7fe2197055fbfc51c994b611ccc","votes":0,"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Measured serve used local image sha256:55d2bd9a6664 (no published image)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"PixelML/sm80vllm branch sm80 (DeepseekV4AmpereMLAAttention) - SM80 attention for this model","url":"https://github.com/PixelML/sm80vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4"}},{"slug":"deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx","title":"DeepSeek-V4.1-Flash-1017 Engram-NVMe TP4/PP4 4x CMP 170HX","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards power-capped at 180 W, with the model's 189 GiB Engram n-gram tables streamed from NVMe on demand. Recipe 1.2.0 (author measurements, claimed_by=submitter T0): 355.7/258.4/183.6 tok/s one user (structured/code/prose), 867.6/544.0/440.5 across eight, 2,416 tok/s cold prefill (TP4; PP4 2,311), 1,395,731-token KV pool at 262,144 context (1,270,789 at 1M), HumanEval 154/164 (6 length-capped; McNemar p>=0.5 vs 156/164), GSM8K 1,275/1,319 (TP4). Hybrid checkpoint assembled without copying tensors: LibertAI EXL3 2.9-bpw routed experts (layers 0-39) + official FP8 attention/shared experts/head/DSpark drafter/Engram shards + BF16 embeddings/LM head/vision, fp8_ds_mla KV. Engine: vLLM on the community sm_80 stack (zebgop-ops sm80 sparse attention + DSpark V1 + Engram row store, FlashInfer PCIe IPC all-reduce) plus this recipe's measured keep-or-drop patches. TP4 (default, needs x16 + P2P) and PP4 (x1/x4, no P2P) from one .env switch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":150.599999999999994315658113919198513031005859375,"prefill_tok_s":0,"updated_at":"2026-10-28 23:26:24.539707-06","commit_sha":"1b9a60d70c9e1e1f83c7db45556ac3450d511600","content_hash":"dadcb54c6118ad769c07ea0930690a53cead0c65","votes":0,"credit":[{"name":"GNmac-lab / greencandle (recipe, patches, measurements)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"CurtisAccelerate - TP4 hybrid-checkpoint recipe + tensor-core EXL3 MoE port","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE"},{"name":"zebgop-ops - dsv41reap-pp sm_80 sparse attention, PP shadow KV, DSpark V1, Engram row store","url":"https://github.com/zebgop-ops/dsv41reap-pp"},{"name":"vcruz305 - vllm-exl3 (AGPL-3.0-only)","url":"https://github.com/vcruz305/vllm-exl3"},{"name":"LibertAI - EXL3 2.9-bpw quantization (MIT)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"kaka86mm - dsv41-flash-pp4-170hx (watched sibling)","url":"https://github.com/kaka86mm/dsv41-flash-pp4-170hx"},{"name":"Morrowmake - CMP 170HX recipe format, P2P work, tensor-core MoE kernels","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Mia-AiLab - decode benchmark protocol + prompts (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"FlashInfer - PCIe IPC all-reduce","url":"https://github.com/flashinfer-ai/flashinfer"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx.json","alternate":"/r/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx"}},{"slug":"deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram","title":"DeepSeek-V4.1-Flash-1017 dsx 2.6-bit GPTQ 2x RTX 3090 (Engram)","summary":"DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) quantized to 187.6 GB single GGUF (2.6127 bpw routed-expert average: 5717 PTQ1_0 / 2545 Q2_0 / 7098 Q3_K per-expert GPTQ with allocator-spent bit budget; dense Q8_0 attention/shared expert/head, F16 embedding/compressor/indexer/Engram-wkv, F32 norms/router/mHC), served by dsx, a Rust+CUDA engine built for this checkpoint, on two RTX 3090s with 192 GB RAM. ~26 tok/s decode / ~170 tok/s prefill at --ctx 262144 with all 15,360 routed experts resident. Exact routing: no expert pruning, no router bias - the rank file orders placement only. Engram n-gram tables (~203 GB) are NOT quantized: read at runtime from the official safetensor shards by a prefetch thread. Hottest ~3,300 experts in VRAM across both GPUs (GPU0's set adapts to the conversation, DSX_ADAPT=8 swaps/token), remaining experts computed on 14 CPU threads from a hugepage RAM arena. Quality vs FP8/FP4 teacher (dsx eval, Lucebox 1017 set): KL 0.1972, top-1 82.59%, PPL 9.3626.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":26,"prefill_tok_s":170,"updated_at":"2026-10-28 15:52:50.29183-06","commit_sha":"942f2d4b4ca3b3ca6697019daf09d9615a47b0af","content_hash":"57641e58a944d825b6d88744c42b9451d8ee916a","votes":0,"credit":[{"name":"lxe (dsx engine + 2bit reference quant)","url":"https://github.com/lxe/dsx"},{"name":"icecubetr (2.6-bit GPTQ quant + measurements)","url":"https://huggingface.co/icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit"},{"name":"deepseek-ai (DeepSeek-V4.1-Flash-1017 checkpoint)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"Lucebox (1017 eval set)","url":"https://huggingface.co/Lucebox/DeepSeek-V4.1-Flash-1017-ROCMFP-GGUF"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram.json","alternate":"/r/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram"}},{"slug":"deepseek-v4-1-flash-1017-fp8-tp1xpp2-dspark-2x-dgx-station-gb300","title":"DeepSeek V4.1 Flash 1017 FP8 TP1xPP2 DSpark 2x DGX Station GB300 RoCE vLLM 0.31 overlay","summary":"Pipeline-parallel TP1xPP2 + DSpark serving of DeepSeek V4.1 Flash (1017 official FP8 weights) across two DGX Station GB300 nodes over 400G CX8 RoCE. PP2 splits the 40 layers 20/20 and removes all 84 per-step all-reduces, replacing them with two point-to-point hops: single-stream temp-0 391 tok/s vs TP2 stock 290.5 (+34%), step 12.3 ms -> ~9.4 ms; aggregate soak 323/406/595/974 tok/s at C1/C2/C4/C8 (3.01x C1); GSM8K 97/100, 0 errors; 1M-token context, FP8 KV, Engram host table pinned in Grace RAM on rank 0. Ships the five-file vLLM v0.31.0 source-fix overlay (bind-mounted over stock vllm/vllm-openai:v0.31.0) that makes DSpark work under PP, plus preflight, launch scripts and soak/receipt verifiers. ALL perf numbers author-measured on their own pair (2026-10-25) - not reproduced locally; site has no GB300 hardware. Recipe by ebfio (port by Iris, a GLM-5.3 model, with Eduardo & Kestrel).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":357,"prefill_tok_s":0,"updated_at":"2026-10-27 16:38:42.712204-06","commit_sha":"2acdf45851ff045dc48344004a9ec5fde3af3de4","content_hash":"739e619707a224aa4cd8782dc5ed8d43f68dce86","votes":0,"credit":[{"name":"ebfio - DeepSeek V4.1 Flash PP2+DSpark GB300 pair recipe (port by Iris on 4x RTX Pro 6000, with Eduardo & Kestrel)","url":"https://github.com/ebfio/deepseek-v41-flash-pp2-pair"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-fp8-tp1xpp2-dspark-2x-dgx-station-gb300.json","alternate":"/r/deepseek-v4-1-flash-1017-fp8-tp1xpp2-dspark-2x-dgx-station-gb300"}}]}