{"meta":{"count":11,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["170hx-x4","radeon-ai-pro-r9700"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T16:49:53+00:00"},"recipes":[{"slug":"deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake","title":"DeepSeek-V4.1-Flash-1017 Hybrid EXL3 TP4 4x CMP 170HX Morrowmake","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw export, every other weight from DeepSeek's official FP8 1017 checkpoint (sha256-validated link build, no copies), and the model's ~189 GiB Engram tables in host RAM (VLLM_ENGRAM_PLACEMENT=auto, SSD fallback below ~212 GiB available). up to 319 tok/s one user structured (287 code / 162 prose), up to 765 across eight, up to 3,818 tok/s cold 2K prefill and a 1,247,406-token KV pool at 262,144 context, full-precision BF16 KV. Engine: Morrowmake's vLLM fork @ a0c57b52 on upstream 4e0681ed56 (sm_80 EXL3 + Marlin kernels, custom P2P all-reduce after a startup content check, DSpark speculative decoding 7/5 adaptive), pinned container image, TP4 with whole routed experts per card, optimised for PCIe x16 links. HumanEval 157/164, GSM8K 1,280/1,319, within test noise of the unmodified-checkpoint reference.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":162,"prefill_tok_s":3804,"updated_at":"2026-10-30 08:13:13.515043-06","commit_sha":"1d43d37b0b9cffa384845e62bcd3af425c04088c","content_hash":"79c2a1c9569e13d38af257c3a7c84e011e8a9cbe","votes":0,"credit":[{"name":"Morrowmake (recipe, engine fork, measurements)","url":"https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe"},{"name":"LibertAI - EXL3 2.9-bpw routed-expert quantization","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"MiaAI-Lab - decode benchmark protocol (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"Marlin (IST-DASLab + Neural Magic contributors) - kernel basis","url":"https://github.com/IST-DASLab/marlin"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake.json","alternate":"/r/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake"}},{"slug":"qwen3-8-27b-mxfp4-dflash2-1x-radeon-ai-pro-r9700-wsl2-radiance","title":"Qwen3.8-27B MXFP4 DFlash2 1x Radeon AI PRO R9700 WSL2 radiance","summary":"Qwen3.8-27B on a single AMD Radeon AI PRO R9700 (RDNA4 gfx1201, 32 GB) under Windows 11 + WSL2: vLLM radiance 0.9.3 fork pinned by image digest with AMD's official Quark AWQ-MXFP4 checkpoint, DFlash2 FP8 k=7 speculative drafter (59-63% acceptance, 5.1-5.4 tok per verify), hand-written W4A8 GEMM + libr4d FP8 paged-attention/GDN kernels pinned b9e42ab + rx9 patch, WSL pinned-memory fix, explicit context sizing, vision tower skipped. Author-measured 125-134 tok/s decode, 2750-2970 tok/s prefill at 1.9k. The author's defaults moved to a three-sequence long-context shape (max-num-seqs 3, batched tokens 2560, max-model-len 220000) backed by a KV-profiles pin the author labels REPORTED rather than measured, plus an opt-in KV cache subsystem (off|ram|disk) that leaves the serve command byte-identical when off. One-command PowerShell installer, OpenAI-compatible server on :8080.","tier":"community","beta":false,"generation":"amd","decode_tok_s":130.69999999999998863131622783839702606201171875,"prefill_tok_s":2762,"updated_at":"2026-10-29 23:29:38.011666-06","commit_sha":"99b06dc5a6b5393dde004bc60139fe19f74a1aeb","content_hash":"7b9f9a8cd306be8ff298ebb68ff1164fcf07a0ef","votes":0,"credit":[{"name":"mike2153 (recipe, WSL2 installer, benchmarks)","url":"https://github.com/mike2153/mbea-qwen38-dflash"},{"name":"StillDeadcode (vllm-radiance image + libr4d kernels)","url":"https://codeberg.org/StillDeadcode/libr4d"},{"name":"ggz14 (radiance-vllm-mxfp4 fork)","url":"https://codeberg.org/ggz14/radiance-vllm-mxfp4"},{"name":"magiccodingman (radiance RDNA4 kernel lineage)"},{"name":"Inco AI / tcclaviger (DFlash2 FP8 drafter requant)","url":"https://huggingface.co/tcclaviger/Qwen3.8-27B-DFlash2-FP8"},{"name":"AMD (official Qwen3.8-27B Quark AWQ-MXFP4 checkpoint + ROCm librocdxg)","url":"https://huggingface.co/amd/Qwen3.8-27B-Quark-AWQ-MXFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-mxfp4-dflash2-1x-radeon-ai-pro-r9700-wsl2-radiance.json","alternate":"/r/qwen3-8-27b-mxfp4-dflash2-1x-radeon-ai-pro-r9700-wsl2-radiance"}},{"slug":"glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141","title":"GLM-5.3-Flash on 4x CMP 170HX \u2014 Morrowmake recipe 1.8.0, TP4 DFlash2","summary":"PP4 (pipeline-parallel 4, TP1) as-run on a 4x CMP 170HX rig 2026-09-28 at --gpu-memory-utilization 0.93 after 0.95 OOM-killed warmup (a 72 MiB KDA prefill workspace did not fit in the ~70 MiB left on the last card). Author 1.8.0 measurements below are TP4 unless labeled PP4; KV pool 2,088,960 tokens under PP4 (author-published 1.8.0 figure). Recipe release 1.8.0 2026-10-16 (engine 819146023f, image engine-819146023f) \u2014 updated from 1.7.2; engine rebased on upstream vLLM 4e0681ed56, same context/batch/fair-prefill defaults; 21 engine-switch env names renamed (old names silently ignored at code defaults); container-only install; author re-measured throughput/KV tables at 1.8.0.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":264.3999999999999772626324556767940521240234375,"prefill_tok_s":6264,"updated_at":"2026-10-29 23:29:37.313564-06","commit_sha":"6e47f035b02d04cee06466adb7fb69b21d9d5240","content_hash":"171456f9f739f5c140a9a08da526b95e056ea9fe","votes":1,"credit":[{"name":"Morrowmake","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json","alternate":"/r/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141"}},{"slug":"deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81","title":"DeepSeek-V4.1-Flash-1017 Fidelity-First 4x CMP 170HX alanoo81","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four changes: routed experts of layers 18-22 grafted 2-bit to 3-bit (LibertAIDAI EXL3 3.5bpw shards 00020-00024-of-00044 into the 2.9bpw pack, all 40 layers 3-bit); no DSpark drafter (SPEC_TOKENS=0, frees ~2.8 GiB/card for KV); Engram tables genuinely loaded (ENGRAM_STORAGE=ssd from a 200 GiB RAM tmpfs); gpu_memory_utilization 0.98, KV pool 2,717,420 tokens at 262,144 context. Author-measured: 300/300 exact identifier copies, byte-identical repeats; perplexity -1.80% overall / -4.45% French vs the 2.9bpw pack (95% CIs exclude zero); ~105 tok/s decode c1, 449-560 tok/s aggregate c8; cold prefill 2,514 tok/s median. Documents a silent failure: leaving DSV41_ENGRAM_STORAGE unset (host mode) leaves the Engram tables unfilled - layers 1 and 14 act as identity functions, 15/20 copies, temp-0 output drift, no error.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":105,"prefill_tok_s":2514,"updated_at":"2026-10-29 21:57:43.459028-06","commit_sha":"5af1460f4a2dce4f590f6fdfcd108119285e3e7c","content_hash":"67eb1810e7eea071e85e8edab747fd98247dd872","votes":0,"credit":[{"name":"alanoo81 (recipe, graft, measurements)","url":"https://github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx"},{"name":"GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"Curtis White (hybrid layout + quantization_config.json)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)","url":"https://github.com/vcruz305/vllm-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81.json","alternate":"/r/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81"}},{"slug":"deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash EXL3 3bpw TC-MoE on 4x CMP 170HX TP4 (original checkpoint)","summary":"DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw \u2014 NOT the 2026-10-17 '1017' re-release, which has its own sibling recipe) served EXL3 3bpw tensor-parallel 4 on four CMP 170HX cards (sm_80, 140 W, HBM locked 1728 MHz) in a 2x Xeon E5-2689 v4 Broadwell host, PCIe Gen2 x16, no NVLink, using Morrowmake's tensor-core MoE kernels + host-staged all-reduce ported onto this checkpoint: K3 on every served layer with a split K2 down-projection, row-rule hybrid (coop rows 1-42, moe_out_v2 43-64, moe_out_prefill 65-2048), one shared 540.4 MiB MoE workspace per GPU, zero-register-spill served kernels. Measured by the author: 178.3 tok/s structured decode c1, 587 tok/s across 8 users, median TTFT 8.37 s from 2K through 255K context (10.9 s at 255,424) against the same host's optimized graphs+coop+NCCL stack at 172.7 / 456 / 10.15 s; needles 9/9 through 256K; KV pool 616,916 tokens.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":149.19999999999998863131622783839702606201171875,"prefill_tok_s":1486,"updated_at":"2026-10-29 21:32:23.373585-06","commit_sha":"dc2dbe7cce66fb0f4d3dfad6b6765ed3c4ea8f74","content_hash":"5640bd33a3ca7d6a71e5a4804e0ec780e0c34fb1","votes":0,"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4"}},{"slug":"deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw TC-MoE on 4x CMP 170HX TP4","summary":"DeepSeek-V4.1-Flash-1017 (the 2026-10-17 re-release checkpoint, NOT the original DeepSeek-V4.1-Flash) as a hybrid quant on the TC-MoE port: official deepseek-ai/DeepSeek-V4.1-Flash-1017 FP8 tensors for attention, shared experts, head, DSpark draft and Engram (shards 47/48), with LibertAIDAI/...1017-EXL3-2.9bpw routed experts (layers 0-39, mostly 3-bit, layers 18-22 2-bit) TP4 on four CMP 170HX cards (sm_80, 140 W, ~62.5 GB/card). Same Morrowmake kernels + image as the EXL3 3bpw TC-MoE recipe; VLLM_EXL3_COOP=1; EXL3 arena cache (~1.1 GiB/layer, load shortcut only). 1017 quantization_config.json is the switch (dsv41_main_routed_experts_only, mtp_experts=source, non-routed stays FP8, compressor projection BF16). Author-measured: 180.4/137.1/96.6 tok/s structured/code/prose c1; 8-user aggregate 621.1/376.1/282.5 vs 564.7 previous Mia 3.0bpw. Cold 131k prompt 1,330 prefill / 163.9 decode. Needles 9/9. KV pool 1,534,065 tokens; vision on, drafter text-only.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":165.599999999999994315658113919198513031005859375,"prefill_tok_s":1390.09999999999990905052982270717620849609375,"updated_at":"2026-10-29 21:32:22.449013-06","commit_sha":"306225cea85cf158c3036c971b031a9c1740075b","content_hash":"59b7b30fa1a9d7fe2197055fbfc51c994b611ccc","votes":0,"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Measured serve used local image sha256:55d2bd9a6664 (no published image)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"PixelML/sm80vllm branch sm80 (DeepseekV4AmpereMLAAttention) - SM80 attention for this model","url":"https://github.com/PixelML/sm80vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4"}},{"slug":"deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx","title":"DeepSeek-V4.1-Flash-1017 Engram-NVMe TP4/PP4 4x CMP 170HX","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards power-capped at 180 W, with the model's 189 GiB Engram n-gram tables streamed from NVMe on demand. Recipe 1.2.0 (author measurements, claimed_by=submitter T0): 355.7/258.4/183.6 tok/s one user (structured/code/prose), 867.6/544.0/440.5 across eight, 2,416 tok/s cold prefill (TP4; PP4 2,311), 1,395,731-token KV pool at 262,144 context (1,270,789 at 1M), HumanEval 154/164 (6 length-capped; McNemar p>=0.5 vs 156/164), GSM8K 1,275/1,319 (TP4). Hybrid checkpoint assembled without copying tensors: LibertAI EXL3 2.9-bpw routed experts (layers 0-39) + official FP8 attention/shared experts/head/DSpark drafter/Engram shards + BF16 embeddings/LM head/vision, fp8_ds_mla KV. Engine: vLLM on the community sm_80 stack (zebgop-ops sm80 sparse attention + DSpark V1 + Engram row store, FlashInfer PCIe IPC all-reduce) plus this recipe's measured keep-or-drop patches. TP4 (default, needs x16 + P2P) and PP4 (x1/x4, no P2P) from one .env switch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":150.599999999999994315658113919198513031005859375,"prefill_tok_s":0,"updated_at":"2026-10-28 23:26:24.539707-06","commit_sha":"1b9a60d70c9e1e1f83c7db45556ac3450d511600","content_hash":"dadcb54c6118ad769c07ea0930690a53cead0c65","votes":0,"credit":[{"name":"GNmac-lab / greencandle (recipe, patches, measurements)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"CurtisAccelerate - TP4 hybrid-checkpoint recipe + tensor-core EXL3 MoE port","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE"},{"name":"zebgop-ops - dsv41reap-pp sm_80 sparse attention, PP shadow KV, DSpark V1, Engram row store","url":"https://github.com/zebgop-ops/dsv41reap-pp"},{"name":"vcruz305 - vllm-exl3 (AGPL-3.0-only)","url":"https://github.com/vcruz305/vllm-exl3"},{"name":"LibertAI - EXL3 2.9-bpw quantization (MIT)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"kaka86mm - dsv41-flash-pp4-170hx (watched sibling)","url":"https://github.com/kaka86mm/dsv41-flash-pp4-170hx"},{"name":"Morrowmake - CMP 170HX recipe format, P2P work, tensor-core MoE kernels","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Mia-AiLab - decode benchmark protocol + prompts (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"FlashInfer - PCIe IPC all-reduce","url":"https://github.com/flashinfer-ai/flashinfer"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx.json","alternate":"/r/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx"}},{"slug":"qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy","title":"Qwen3.8-Flash-Next-FP8 4x CMP 170HX \u2014 PP4 MTP-4 patched vLLM (g4bwy)","summary":"Recipe by g4bwy. Official Qwen/Qwen3.8-Flash-Next-FP8 on 4x unlocked CMP 170HX (GA100 sm_80, 74 SMs, 64 GB) with PP=4 (layer partition 13,12,12,11) and MTP-4 speculative decoding on pinned vLLM mainline 183625d4f9 (2026-10-13, precompiled wheel 0.32.0rc1.dev34+g183625d4f) plus a 7-patch series applied with git am (tree hash must equal 18e0ec8942f61f82fd3a13bd9d9c6b9ce67ad371; 2 patches adopted from upstream PRs #56444/#58094) that unblocks PP+MTP+PLE on sm_80. 1M context at static YaRN 4.0, PLE n-gram table in pinned host RAM (--engram-config cpu_offload), Mamba align prefix caching with retention 16160 + match-unit 16 (99.8-99.9% hits), NCCL P2P disabled (host-staged wins at PP=4), expandable_segments, 200 W per-card caps, uv venv, no docker. Optional W8A16 int8-dense lane (checkpoint/) +15-20% single-stream.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":183.19999999999998863131622783839702606201171875,"prefill_tok_s":10038,"updated_at":"2026-10-20 21:52:36.721138-06","commit_sha":"540d404493ab67413a71efa0f5e541d18d329313","content_hash":"ab27af4ddefebc602e00353e6d7b640e515f9c4d","votes":0,"credit":[{"name":"g4bwy","url":"https://github.com/g4bwy/qwen3.8-flash-next-cmp-170hx"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy.json","alternate":"/r/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy"}},{"slug":"mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx","title":"MiMo-V2.6-Flash-RL 309B MoE NVFP4 4x CMP 170HX TP4 DFlash 1M ctx (ElectroHobo)","summary":"XiaomiMiMo's MiMo-V2.6-Flash-RL (309B MoE, 15B active, text/image/video/audio input) on four CMP 170HX cards (sm_80) at the full 1,048,576-token context: 322/275/170 tok/s one user (structured/code/prose), 754/738/556 across eight, 2,720 tok/s cold prefill, 1,615,953-token KV pool. NVFP4 weights (ModelOpt mixed precision via Marlin, bf16 activations), bf16 KV, MiMo's own DFlash drafter (7 drafts one user, 7/5/3 by request count), full-decode-step CUDA graph, one engine file bind-mounted (upstream vLLM #58235 vision-sink fix), every weight on the GPUs. Morrowmake's first listed community recipe on vllm-cmp170hx; measured 2026-10-18 under the Morrowmake bench protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":274.8999999999999772626324556767940521240234375,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:18.178415-06","commit_sha":"12a7bfc3d0f95506c73c0dbaf6bb13286ef57281","content_hash":"6388d1808174e278654541bfad590d93a9202607","votes":0,"credit":[{"name":"ElectroHobo (Applied Integrators)","url":"https://github.com/ElectroHobo/mimo-v26-flash-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx fork + engine image + start.sh script set, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"XiaomiMiMo (MiMo-V2.6-Flash-RL base model)","url":"https://huggingface.co/XiaomiMiMo"},{"name":"ProCreations (NVFP4 checkpoint)","url":"https://huggingface.co/ProCreations/MiMo-V2.6-Flash-RL-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx.json","alternate":"/r/mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx"}},{"slug":"deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw","title":"DeepSeek-V4.1-Flash (764B) on 4x CMP 170HX \u2014 PP4, shadow KV, EXL3 2bpw, DSpark K5, 512k ctx, native vision (kaka86mm)","summary":"Four-card Ampere recipe for full DeepSeek-V4.1-Flash (764B, native vision) on 4x CMP 170HX (sm80, Gen2 x4, no P2P \u2014 256 GB VRAM can't hold 510 GB, so: EXL3 2bpw + fp8 KV): PP4 x TP1 partition 10,10,11,9 (rank3 = 9 layers + drafter), shadow KV via dsv41reap-pp overlay (54 files), EXL3 2bpw pack \u2014 engram tables kept FP8 at 203 GB, pinned 189 GiB host RAM (quantizing destroys retrieval), DSpark n=5 (conf=0; truncation is -30% here; ngram is a 7.6 tok/s disaster), 524k ctx, fp8_ds_mla KV, 5.9M-token pool, native ViT, GLOO_SOCKET_IFNAME=lo \u2014 the 09-28 one-line fix for PP gloo via docker bridge (535 ms/send) that collapsed long-context decode. Measured by the author (09-28 v37): decode 60.7 @32K / 58.3 @128K / 54.7 @524K tok/s (slope -10% vs -37% pre-fix), cold prefill 2,771 @32K to 4,061 tok/s @524K (flat), decode-only C8 200.6 / C24 252.1, quality 7/7 + vision 2/2. Patches: img_ids PP relay, engram sm80 fp8, cand-key fix, exl3 shim.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":57.5,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:08.130558-06","commit_sha":"363a35de531734dabec247825b6103dc00425670","content_hash":"49f43cc59d2bfe7c33b889b5fdc5129673136cc3","votes":0,"credit":[{"name":"kaka86mm","url":"https://github.com/kaka86mm/dsv41-flash-pp4-170hx"},{"name":"zebgop-ops (dsv41reap-pp shadow KV overlay)","url":"https://github.com/zebgop-ops/dsv41reap-pp"},{"name":"vcruz305 (vllm-exl3 plugin)","url":"https://github.com/vcruz305/vllm-exl3"},{"name":"turboderp-org (exllamav3)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"wtdcode (vllm-backport)","url":"https://github.com/wtdcode/vllm-backport"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw.json","alternate":"/r/deepseek-v4-1-flash-764b-on-4x-cmp-170hx-pp4-shadow-kv-exl3-2bpw"}},{"slug":"mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t","title":"MiMo-V2.6-Flash-RL (omni) on 4x CMP 170HX \u2014 PP4, official fp8, three-tier KV, v3.3 (kaka86mm)","summary":"Four-card Ampere recipe for XiaomiMiMo MiMo-V2.6-Flash-RL (309B-MoE/15B active, omni: text/image/video/audio in, 1M ctx) on 4x CMP 170HX (64 GB unlocked, Gen2 x4, no P2P \u2014 PP-only). Official 161 GB fp8 checkpoint on wtdcode/vllm-backport sm80: PP4 partition 12,12,12,12, fp8_e4m3 KV (15.5 GiB, 2,061,355-token pool), fp8 o_proj, Marlin W8A16 for fp8 GEMMs, spec decoding OFF (MTP verify kernel cliff), three-tier KV stack (GPU + 100 GB RAM tier via LMCache PR#4410 engine-driven transfers + disk tier). Eight patches baked into image v3.3: exact QKV sharding (0 changed elements audited), MTP loader, fp8\u2192Marlin, ViT sink fix #58235, DiffKV split-KV + spec-3D verify + prefill q-tiling (3.0x) + QK-dot split #59673. Measured by the author (2026-10-04, v3.3): decode 64.5 short / 60 @19K / 56 @67K tok/s, aggregate 483 @16 / 752 @32 streams, 200K cold prefill ~3,650 tok/s (16x via q-tiling). NVFP4 fallback kept (92\u201394 tok/s). All four modalities + tools verified; thinking-off with mm input.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":60,"prefill_tok_s":3650,"updated_at":"2026-10-20 15:34:07.62236-06","commit_sha":"036b9f389cc3965f671623dda781efcd7aae63bc","content_hash":"ac7293fc06185b9b9e23401a6d805f3d10a5cfe0","votes":0,"credit":[{"name":"kaka86mm","url":"https://github.com/kaka86mm/vllm-mimo-170hx"},{"name":"wtdcode (vllm-backport)","url":"https://github.com/wtdcode/vllm-backport"},{"name":"avtc (vllm-backport PR#106)","url":"https://github.com/wtdcode/vllm-backport/pull/106"},{"name":"LMCache PR#4410","url":"https://github.com/LMCache/LMCache/pull/4410"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t.json","alternate":"/r/mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t"}}]}