Working launch recipes for local LLM inference

by Morrowmake (recipe, engine fork, measurements)↗, LibertAI - EXL3 2.9-bpw routed-expert quantization↗, DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)↗, cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch↗, MiaAI-Lab - decode benchmark protocol (used as data)↗, Marlin (IST-DASLab + Neural Magic contributors) - kernel basis↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw expo

by Morrowmake↗ · listed by localai-recipes

PP4 (pipeline-parallel 4, TP1) as-run on a 4x CMP 170HX rig 2026-09-28 at --gpu-memory-utilization 0.93 after 0.95 OOM-killed warmup (a 72 MiB KDA prefill workspace did not fit in

by kindlingai↗, nvidia↗ · listed by localai-recipes

GLM-5.3-Flash exlr8: MoE layers are EXL3-trellis units decoded by the custom exlr8 MoE kernel; dense linears stay FP8/NVFP4. One release carries three expert widths (k2/k3/k4 = 2/3

by knapcio (TP4 profile, v2.1-v2.3 production stack, release gates)↗, MiaAI-Lab (upstream: launcher, Engram store, b12x routing, memory model)↗, 0xSero (original 4x RTX PRO 6000 recipe skeleton boot.py derives from)↗, kpham-sgl (sglang#39187 bounded dense-indexer prefill transient, backported as adapter/indexer_chunked*.py)↗, BBuf + dsv4.1 branch contributors (#39370, #39646, #39648, #39653 decode kernel work in the canary images)↗, rhys101 (SG17 RoCEnante overlay + SG18 prefill TP split from DeepSeek-V4.1-Flash-vLLM-DGX-Spark-8)↗, local-inference-lab — Luke Alonso & Jason (b12x: RoCEnante one-shot RDMA all-reduce + fused MoE kernels)↗, FujitsuPolycom + sparkring contributors (switchless-ring opposite-node paths, path-aware RoCEnante)↗, Saolence (#1/#2/#4/#5/#6: ring as opt-in switch, NCCL/build fixes, first-deployer review)↗, rsync/rchmagos (#8: RoCEnante on the switchless ring, DSV41_ROCE_RING + scripts/ring_mesh/)↗ · listed by localai-recipes

by alanoo81 (recipe, graft, measurements)↗, GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)↗, LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)↗, Curtis White (hybrid layout + quantization_config.json)↗, vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four change

by ElectroHobo↗, Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)↗, klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)↗, halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)↗, inco.ai (DFlash2 drafter, Apache-2.0)↗, Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)↗, Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)↗, MiaAI-Lab (benchmark protocol + category prompts, used as data)↗ · listed by localai-recipes

Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s

by sybil-solutions (image, tiering runtime, omarchy plugin)↗, turboderp (ExLlamaV3 + EXL3 3.05bpw checkpoint)↗ · listed by localai-recipes

320B MoE served on ONE 24 GB RTX 3090 with stock exllamav3 1.5.1 plus process-local monkeypatches: the 12,096 routed experts (42 MoE layers x 288, 8 picked per token per layer) tie

by CurtisAccelerate↗, Morrowmake↗, turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw — NOT the 2026-10-17 '1017' re-release, which has its own sibling r

by CurtisAccelerate↗, Morrowmake↗, Measured serve used local image sha256:55d2bd9a6664 (no published image)↗, turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)↗, PixelML/sm80vllm branch sm80 (DeepseekV4AmpereMLAAttention) - SM80 attention for this model↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (the 2026-10-17 re-release checkpoint, NOT the original DeepSeek-V4.1-Flash) as a hybrid quant on the TC-MoE port: official deepseek-ai/DeepSeek-V4.1-Flash

by GNmac-lab / greencandle (recipe, patches, measurements)↗, CurtisAccelerate - TP4 hybrid-checkpoint recipe + tensor-core EXL3 MoE port↗, zebgop-ops - dsv41reap-pp sm_80 sparse attention, PP shadow KV, DSpark V1, Engram row store↗, vcruz305 - vllm-exl3 (AGPL-3.0-only)↗, LibertAI - EXL3 2.9-bpw quantization (MIT)↗, DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)↗, kaka86mm - dsv41-flash-pp4-170hx (watched sibling)↗, Morrowmake - CMP 170HX recipe format, P2P work, tensor-core MoE kernels↗, Mia-AiLab - decode benchmark protocol + prompts (used as data)↗, FlashInfer - PCIe IPC all-reduce↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards power-capped at 180 W, with the model's 189 GiB Engram n-gram tables streamed from NVMe on demand. Recipe 1.2.0 (author

by lxe (dsx engine + 2bit reference quant)↗, icecubetr (2.6-bit GPTQ quant + measurements)↗, deepseek-ai (DeepSeek-V4.1-Flash-1017 checkpoint)↗, Lucebox (1017 eval set)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) quantized to 187.6 GB single GGUF (2.6127 bpw routed-expert average: 5717 PTQ1_0 / 2545 Q2_0 / 7098 Q3_K per-expert GPTQ with allo

by g4bwy↗ · listed by localai-recipes

Recipe by g4bwy. Official Qwen/Qwen3.8-Flash-Next-FP8 on 4x unlocked CMP 170HX (GA100 sm_80, 74 SMs, 64 GB) with PP=4 (layer partition 13,12,12,11) and MTP-4 speculative decoding o