{"meta":{"count":22,"page":1,"pages":2,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["dgx-spark","170hx-x4","170hx-x3"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T17:41:38+00:00"},"recipes":[{"slug":"deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake","title":"DeepSeek-V4.1-Flash-1017 Hybrid EXL3 TP4 4x CMP 170HX Morrowmake","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw export, every other weight from DeepSeek's official FP8 1017 checkpoint (sha256-validated link build, no copies), and the model's ~189 GiB Engram tables in host RAM (VLLM_ENGRAM_PLACEMENT=auto, SSD fallback below ~212 GiB available). up to 319 tok/s one user structured (287 code / 162 prose), up to 765 across eight, up to 3,818 tok/s cold 2K prefill and a 1,247,406-token KV pool at 262,144 context, full-precision BF16 KV. Engine: Morrowmake's vLLM fork @ a0c57b52 on upstream 4e0681ed56 (sm_80 EXL3 + Marlin kernels, custom P2P all-reduce after a startup content check, DSpark speculative decoding 7/5 adaptive), pinned container image, TP4 with whole routed experts per card, optimised for PCIe x16 links. HumanEval 157/164, GSM8K 1,280/1,319, within test noise of the unmodified-checkpoint reference.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":162,"prefill_tok_s":3804,"updated_at":"2026-10-30 08:13:13.515043-06","commit_sha":"1d43d37b0b9cffa384845e62bcd3af425c04088c","content_hash":"79c2a1c9569e13d38af257c3a7c84e011e8a9cbe","votes":0,"credit":[{"name":"Morrowmake (recipe, engine fork, measurements)","url":"https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe"},{"name":"LibertAI - EXL3 2.9-bpw routed-expert quantization","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"MiaAI-Lab - decode benchmark protocol (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"Marlin (IST-DASLab + Neural Magic contributors) - kernel basis","url":"https://github.com/IST-DASLab/marlin"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake.json","alternate":"/r/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake"}},{"slug":"glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141","title":"GLM-5.3-Flash on 4x CMP 170HX \u2014 Morrowmake recipe 1.8.0, TP4 DFlash2","summary":"PP4 (pipeline-parallel 4, TP1) as-run on a 4x CMP 170HX rig 2026-09-28 at --gpu-memory-utilization 0.93 after 0.95 OOM-killed warmup (a 72 MiB KDA prefill workspace did not fit in the ~70 MiB left on the last card). Author 1.8.0 measurements below are TP4 unless labeled PP4; KV pool 2,088,960 tokens under PP4 (author-published 1.8.0 figure). Recipe release 1.8.0 2026-10-16 (engine 819146023f, image engine-819146023f) \u2014 updated from 1.7.2; engine rebased on upstream vLLM 4e0681ed56, same context/batch/fair-prefill defaults; 21 engine-switch env names renamed (old names silently ignored at code defaults); container-only install; author re-measured throughput/KV tables at 1.8.0.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":264.3999999999999772626324556767940521240234375,"prefill_tok_s":6264,"updated_at":"2026-10-29 23:29:37.313564-06","commit_sha":"6e47f035b02d04cee06466adb7fb69b21d9d5240","content_hash":"171456f9f739f5c140a9a08da526b95e056ea9fe","votes":1,"credit":[{"name":"Morrowmake","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json","alternate":"/r/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141"}},{"slug":"deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81","title":"DeepSeek-V4.1-Flash-1017 Fidelity-First 4x CMP 170HX alanoo81","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four changes: routed experts of layers 18-22 grafted 2-bit to 3-bit (LibertAIDAI EXL3 3.5bpw shards 00020-00024-of-00044 into the 2.9bpw pack, all 40 layers 3-bit); no DSpark drafter (SPEC_TOKENS=0, frees ~2.8 GiB/card for KV); Engram tables genuinely loaded (ENGRAM_STORAGE=ssd from a 200 GiB RAM tmpfs); gpu_memory_utilization 0.98, KV pool 2,717,420 tokens at 262,144 context. Author-measured: 300/300 exact identifier copies, byte-identical repeats; perplexity -1.80% overall / -4.45% French vs the 2.9bpw pack (95% CIs exclude zero); ~105 tok/s decode c1, 449-560 tok/s aggregate c8; cold prefill 2,514 tok/s median. Documents a silent failure: leaving DSV41_ENGRAM_STORAGE unset (host mode) leaves the Engram tables unfilled - layers 1 and 14 act as identity functions, 15/20 copies, temp-0 output drift, no error.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":105,"prefill_tok_s":2514,"updated_at":"2026-10-29 21:57:43.459028-06","commit_sha":"5af1460f4a2dce4f590f6fdfcd108119285e3e7c","content_hash":"67eb1810e7eea071e85e8edab747fd98247dd872","votes":0,"credit":[{"name":"alanoo81 (recipe, graft, measurements)","url":"https://github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx"},{"name":"GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"Curtis White (hybrid layout + quantization_config.json)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)","url":"https://github.com/vcruz305/vllm-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81.json","alternate":"/r/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81"}},{"slug":"qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g","title":"Qwen3.8-Flash-Next W4A16-AutoRound DGX Spark vLLM MTP4 Iter7g","summary":"Recipe by kyaky. Qwen3.8-Flash-Next W4A16-AutoRound hybrid (INT4 experts + FP8 side layers + INT8 head; 51B-token n-gram PLE table on NVMe mmap) on one DGX Spark (GB10, 121 GB unified memory) on a verified-deterministic vLLM floor (image qwen38-flash-dgx:iter7g-20261015 on the digest-pinned vllm/vllm-openai:qwen38-flash-next base): MoE align arrival-order shim, drafter-Marlin atomicAdd fix, pinned triton config, compile-cache mount. Measured line v19a-w5 = MTP=4 window-5 block verification: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 streams (+9.2% N1 / +4.8% N8 over MTP=3, two boots per arm, same image). Promoted serving line v20-fp8: fp8 KV cache on a 24 GB pool at native 262,144 context, 1,291,120-token capacity. Chinese +31.3% via a 25,862-id (39.5%) Han draft-vocab rebalance (prose-zhen 34.8 to 45.7 tok/s). Stochastic-correctness gate 37/37 vs bit-exact fp64 oracle; per-round raw data + scripts ship in-repo; copy-agentic-1 failure disclosed.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":61.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.107205-06","commit_sha":"dc8982acabf5e6b6417f4871ad5149d61f01755d","content_hash":"e6a2cfdf4903c7ca1b80df3d0f496d3ca3f13840","votes":0,"credit":[{"name":"kyaky","url":"https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json","alternate":"/r/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g"}},{"slug":"qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp","title":"Qwen3.8-Flash-Next NVFP4 DGX Spark GB10 SGLang NEXTN MTP","summary":"Production single-node DGX Spark (Grace Blackwell GB10, SM121, 128GB unified memory) recipe for Qwen3.8-Flash-Next on SGLang: ModelOpt NVFP4 quantization with zero-copy NVMe mmap of the 47.7 GiB ple_table.bin via --ple-offload-embedding (eliminates 51 GiB of unified-memory footprint), custom Triton FP4 prefill kernels + trtllm_mha decode kernels on SM121, NEXTN MTP (3 verification steps / 4 draft tokens) lifting output to 110-152 tok/s, GDN+QSA hybrid (131,072-token served window, 262k native capacity), 96-slot mamba recurrence buffers, sharp froggeric v22.1 Jinja chat template (-18% response length), persistent Triton/Inductor SM121 JIT caches on NVMe, 8-worker parallel NVMe pre-warm (206 shards in 17s) + 16-core Grace OMP. Author-measured vs same-box 27B dense baseline: 110.4-152.8 tok/s decode w/ MTP (baseline 12.1 base / 21.5 MTP), single-stream CoT 25.1-28.8 tok/s (2.62x speculative gain), TTFT ~347ms radix hit / ~465ms cold, 113 GiB / 121 GiB RAM used (~8 GiB headroom), 103.5 GiB","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":110.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.869274-06","commit_sha":"2c5220aa85b5379d7373abd0aa14bb13b1ec993f","content_hash":"bf0c02f95b99494d6720a515e2f8f438f126bc87","votes":0,"credit":[{"name":"Michael Bemler (Bemler Labs)","url":"https://github.com/bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp.json","alternate":"/r/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp"}},{"slug":"qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x","title":"Qwen3.8-Flash-Next FP4-Local 1x DGX Spark vLLM b12x","summary":"Recipe by sojufx. Serves NVIDIA's nvidia/Qwen3.8-Flash-Next-FP4-Local (96 GiB NVFP4 quantization-aware-distilled checkpoint in a new compressed NVFP4 scale format) on one DGX Spark (GB10) via the digest-pinned eugr/spark-vllm-b12x community container (reported vLLM 0.1.dev21830+g8244d1c44.d20261015) - the model card requires community b12x containers until the format lands upstream. FP8 KV cache, exact MTP K=3 with standard rejection sampling, 262,144-token context, 4 scheduler slots, 4096-token batched prefill, disk-backed io_uring PLE, prefix cache with Mamba align, vision plus qwen3_xml tool calling, thinking off by default, localhost-only 127.0.0.1:8001 OpenAI-compatible backend with a deliberate no-replace/no-port-take guard. Measured 2026-10-15/16 on the author's Spark: 66.2 tok/s single-stream sparkDash counting decode, 166.4 tok/s at C4, 41-54 tok/s single-stream on the four-prompt wall-clock suite - different measurement bases, not interchangeable.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":66.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.177524-06","commit_sha":"5fc0d101d8e9b1c6a1419f3d465e21649f4ecbad","content_hash":"be42677d0cc5ca7f71fe72210e756f1622fe3604","votes":0,"credit":[{"name":"sojufx","url":"https://github.com/sojufx/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark-Recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json","alternate":"/r/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x"}},{"slug":"nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k","title":"Nemotron 3.5 Bolt 74B-A9B 1x DGX Spark vLLM b12x LiLiCorr 262K","summary":"NVIDIA Nemotron 3.5 Bolt 74B-A9B-24GB (hybrid Mamba-2/Transformer latent-MoE, 74B with 9B active, IQ2_XXS 2-bit experts, NVFP4 Mamba layers, Q8_0 rest; text+image+video in, 262,144-token window, OpenMDW-1.1) on one DGX Spark (GB10/SM121) with vLLM nightly-77154ca6 (base image vllm/vllm-openai@sha256:1ee85ca8) + b12x 1.5.1 kernels (--linear-backend b12x --moe-backend b12x), FP8 KV cache, gpu-memory-utilization 0.70, max-num-seqs 16, and the NVIDIA LiLiCorr MTP drafter at 7 speculative tokens. Pinned revisions: model c6cc2e8f, drafter 4c6a5613. Measured on the reference box: ~140 tok/s code rewrite / 63 summarise / 50 prose single-stream with the drafter (mean acceptance 4.0 tokens/pass), 31 tok/s without; 74.8/113.4/163.6/201.2 tok/s at 1/4/8/16 streams; KV cache ~6.52M tokens at the 0.70 share. On a 64 GB Spark the lane takes its own memory share 0.60: 667,874 KV tokens (2.55 windows), 110.8 tok/s greedy, 7.1 GiB left at the peak (2026-10-24).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":63,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.717952-06","commit_sha":"2cdb5f11ffa1fb1af8676496fdd631976c3d536d","content_hash":"28887fb2cf3bab1914c803e792898fff92b9afe0","votes":0,"credit":[{"name":"hasso5703","url":"https://github.com/hasso5703"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k.json","alternate":"/r/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k"}},{"slug":"glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4","title":"GLM-5.3-Flash exlr8 EXL3 experts on DGX Spark TP1-4","summary":"GLM-5.3-Flash exlr8: MoE layers are EXL3-trellis units decoded by the custom exlr8 MoE kernel; dense linears stay FP8/NVFP4. One release carries three expert widths (k2/k3/k4 = 2/3/4 bpw), chosen at boot via VLLM_EXLR8_WIDTHS. Reference topology TP=4 on four GB10 boxes; TP=1 serves the whole model (~87 GiB) on one 128 GB DGX Spark / ASUS GX10 via tp1.yaml plus a distinct MENTAT_GROUP. TP=2 and TP=3 also measured. 2026-10-19, clocks TP-locked at 2200 MHz: decode code 128.0 tok/s, prose 72.5 at TP=4; 53.0 / 29.5 at TP=1; prefill 4,393 tok/s at 8K, 4,190 at 32K. Trade vs the shipping NVFP4 build: decode +3-8%, NLL equal or better, KV pool ~+15%, prefill 7.7% lower at Second 1x-Spark GLM-5.3-Flash lane: not the VQ Q4_K GGUF lane (/r/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn), and not a fork of the 2x ATLAS build (/r/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m) whose NVFP4 Flash base this stack overlays.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":65.2000000000000028421709430404007434844970703125,"prefill_tok_s":3438.5,"updated_at":"2026-10-29 21:32:25.385656-06","commit_sha":"2ddd4d836d8efa003f7203ba30597c20dc21e495","content_hash":"79db03010b10026e9788b7c749df25daf5fb38be","votes":0,"credit":[{"name":"kindlingai","url":"https://github.com/kindlingai/glm-5.3-flash-exlr8-gx10"},{"name":"nvidia","url":"https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4.json","alternate":"/r/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4"}},{"slug":"ds4-flash-3x170hx-pp3-dspark-k5","title":"DeepSeek V4 Flash-0731 abliterated on 3x CMP 170HX \u2014 vLLM patched PP3 DSpark K5","summary":"The proven 3-card pipeline-parallel recipe on CMP 170HX (sm_80, 64GB unlocked): PP=3 with manual layer partition 15,16,12 at util 0.98, FP8 KV, block-size 256, DSpark block drafting at exactly 5 speculative tokens, async scheduling, CPU pinning (dedicated physical cores, placeholder in launch args), hard memory cap 64g with swap disallowed. Structured decode 181 tok/s at 99% acceptance. Upstream reference repo: allover326/deepseek-v4-cmp170hx @ 3dd2d88 (Aug 13 2026) \u2014 documents 15,15,13 3-card split, retracted 4-card requirement, INT4 524288 ctx w/ DSV4_LOGITS_ROW_CHUNK=64, +7% decode via c3046d1 rebase patches (full source build required). First-party 15,16,12 numbers on this page are ours.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":129.900000000000005684341886080801486968994140625,"prefill_tok_s":2350,"updated_at":"2026-10-29 21:32:23.860633-06","commit_sha":"24a9ac90156459e4efb49c951590edbea775df38","content_hash":"02d56ece2249f2b3e2b1ee2ffce158e841e20d26","votes":0,"credit":[{"name":"allover326","url":"https://github.com/allover326/deepseek-v4-cmp170hx"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/ds4-flash-3x170hx-pp3-dspark-k5.json","alternate":"/r/ds4-flash-3x170hx-pp3-dspark-k5"}},{"slug":"deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash EXL3 3bpw TC-MoE on 4x CMP 170HX TP4 (original checkpoint)","summary":"DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw \u2014 NOT the 2026-10-17 '1017' re-release, which has its own sibling recipe) served EXL3 3bpw tensor-parallel 4 on four CMP 170HX cards (sm_80, 140 W, HBM locked 1728 MHz) in a 2x Xeon E5-2689 v4 Broadwell host, PCIe Gen2 x16, no NVLink, using Morrowmake's tensor-core MoE kernels + host-staged all-reduce ported onto this checkpoint: K3 on every served layer with a split K2 down-projection, row-rule hybrid (coop rows 1-42, moe_out_v2 43-64, moe_out_prefill 65-2048), one shared 540.4 MiB MoE workspace per GPU, zero-register-spill served kernels. Measured by the author: 178.3 tok/s structured decode c1, 587 tok/s across 8 users, median TTFT 8.37 s from 2K through 255K context (10.9 s at 255,424) against the same host's optimized graphs+coop+NCCL stack at 172.7 / 456 / 10.15 s; needles 9/9 through 256K; KV pool 616,916 tokens.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":149.19999999999998863131622783839702606201171875,"prefill_tok_s":1486,"updated_at":"2026-10-29 21:32:23.373585-06","commit_sha":"dc2dbe7cce66fb0f4d3dfad6b6765ed3c4ea8f74","content_hash":"5640bd33a3ca7d6a71e5a4804e0ec780e0c34fb1","votes":0,"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4"}},{"slug":"deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw TC-MoE on 4x CMP 170HX TP4","summary":"DeepSeek-V4.1-Flash-1017 (the 2026-10-17 re-release checkpoint, NOT the original DeepSeek-V4.1-Flash) as a hybrid quant on the TC-MoE port: official deepseek-ai/DeepSeek-V4.1-Flash-1017 FP8 tensors for attention, shared experts, head, DSpark draft and Engram (shards 47/48), with LibertAIDAI/...1017-EXL3-2.9bpw routed experts (layers 0-39, mostly 3-bit, layers 18-22 2-bit) TP4 on four CMP 170HX cards (sm_80, 140 W, ~62.5 GB/card). Same Morrowmake kernels + image as the EXL3 3bpw TC-MoE recipe; VLLM_EXL3_COOP=1; EXL3 arena cache (~1.1 GiB/layer, load shortcut only). 1017 quantization_config.json is the switch (dsv41_main_routed_experts_only, mtp_experts=source, non-routed stays FP8, compressor projection BF16). Author-measured: 180.4/137.1/96.6 tok/s structured/code/prose c1; 8-user aggregate 621.1/376.1/282.5 vs 564.7 previous Mia 3.0bpw. Cold 131k prompt 1,330 prefill / 163.9 decode. Needles 9/9. KV pool 1,534,065 tokens; vision on, drafter text-only.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":165.599999999999994315658113919198513031005859375,"prefill_tok_s":1390.09999999999990905052982270717620849609375,"updated_at":"2026-10-29 21:32:22.449013-06","commit_sha":"306225cea85cf158c3036c971b031a9c1740075b","content_hash":"59b7b30fa1a9d7fe2197055fbfc51c994b611ccc","votes":0,"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Measured serve used local image sha256:55d2bd9a6664 (no published image)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"PixelML/sm80vllm branch sm80 (DeepseekV4AmpereMLAAttention) - SM80 attention for this model","url":"https://github.com/PixelML/sm80vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4"}},{"slug":"deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx","title":"DeepSeek-V4.1-Flash-1017 Engram-NVMe TP4/PP4 4x CMP 170HX","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards power-capped at 180 W, with the model's 189 GiB Engram n-gram tables streamed from NVMe on demand. Recipe 1.2.0 (author measurements, claimed_by=submitter T0): 355.7/258.4/183.6 tok/s one user (structured/code/prose), 867.6/544.0/440.5 across eight, 2,416 tok/s cold prefill (TP4; PP4 2,311), 1,395,731-token KV pool at 262,144 context (1,270,789 at 1M), HumanEval 154/164 (6 length-capped; McNemar p>=0.5 vs 156/164), GSM8K 1,275/1,319 (TP4). Hybrid checkpoint assembled without copying tensors: LibertAI EXL3 2.9-bpw routed experts (layers 0-39) + official FP8 attention/shared experts/head/DSpark drafter/Engram shards + BF16 embeddings/LM head/vision, fp8_ds_mla KV. Engine: vLLM on the community sm_80 stack (zebgop-ops sm80 sparse attention + DSpark V1 + Engram row store, FlashInfer PCIe IPC all-reduce) plus this recipe's measured keep-or-drop patches. TP4 (default, needs x16 + P2P) and PP4 (x1/x4, no P2P) from one .env switch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":150.599999999999994315658113919198513031005859375,"prefill_tok_s":0,"updated_at":"2026-10-28 23:26:24.539707-06","commit_sha":"1b9a60d70c9e1e1f83c7db45556ac3450d511600","content_hash":"dadcb54c6118ad769c07ea0930690a53cead0c65","votes":0,"credit":[{"name":"GNmac-lab / greencandle (recipe, patches, measurements)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"CurtisAccelerate - TP4 hybrid-checkpoint recipe + tensor-core EXL3 MoE port","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE"},{"name":"zebgop-ops - dsv41reap-pp sm_80 sparse attention, PP shadow KV, DSpark V1, Engram row store","url":"https://github.com/zebgop-ops/dsv41reap-pp"},{"name":"vcruz305 - vllm-exl3 (AGPL-3.0-only)","url":"https://github.com/vcruz305/vllm-exl3"},{"name":"LibertAI - EXL3 2.9-bpw quantization (MIT)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"kaka86mm - dsv41-flash-pp4-170hx (watched sibling)","url":"https://github.com/kaka86mm/dsv41-flash-pp4-170hx"},{"name":"Morrowmake - CMP 170HX recipe format, P2P work, tensor-core MoE kernels","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Mia-AiLab - decode benchmark protocol + prompts (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"FlashInfer - PCIe IPC all-reduce","url":"https://github.com/flashinfer-ai/flashinfer"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx.json","alternate":"/r/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx"}}]}