Working launch recipes for local LLM inference

by Morrowmake (recipe, engine fork, measurements)↗, LibertAI - EXL3 2.9-bpw routed-expert quantization↗, DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)↗, cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch↗, MiaAI-Lab - decode benchmark protocol (used as data)↗, Marlin (IST-DASLab + Neural Magic contributors) - kernel basis↗, Curtis White (CurtisAccelerate) - TC-MoE lane shape + split-expert idea (docs/credits.md)↗, GNmac-lab - FP8 wo_a grouped-GEMM E4M3 decode; target argmax, side-stream mHC, split-expert ideas↗, b12x (local-inference-lab) - copy-engine ring all-reduce design credit↗, ExLlamaV3 (turboderp-org) - EXL3 trellis packing, codebooks, Hadamard conventions (adapted)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.2 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw expo

by BoxTops4Coochie (recipe, Flare TP3 fork, measurements)↗, NVIDIA - Flare runtime + MXFP4 Engram tables↗, Local Inference Lab - FlashInfer b12x kernels↗, DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)↗ · listed by localai-recipes

NVIDIA Flare TP3 recipe for DeepSeek-V4.1-Flash-1017 on 3x 96 GB RTX PRO 6000 Blackwell, full 1M-token context: personal Flare fork (main @ 78b18b1c + source.patch) adds a merged V

by BoxTops4Coochie (recipe, TP3 port, measurements)↗, infernix - ds41f-tp3 expert-parking design on karmic-kraken↗, Local Inference Lab - vLLM karmic-kraken fork + FlashInfer b12x kernels↗, DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 TP3 on three 96 GB RTX PRO 6000 Blackwell cards, full 1M context, DeepSeek's original weights (FP8 dense + MXFP4 experts, nothing requantized). Expert laye

by alexvnesta (server + ExLlamaV3 patches)↗, Flun (original 2x CMP 170HX layer-split recipe)↗, turboderp (ExLlamaV3, EXL3 quant)↗, cmpunlocker (amoghmunikote)↗ · listed by mannydelrio1

Everyday GLM-5.3-Flash setup on 2x CMP 170HX unlocked to 64 GB (PCIe Gen2 x4, 180 W each, 32 GB RAM, 850 W PSU): alexvnesta's ExLlamaV3 server in layer-split mode, MTP head draftin

by AlexBrandes (vLLM CMP170HX fork + plugin + this recipe + measurements)↗, turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 2.60bpw checkpoint)↗, incoai (GLM-5.3-Flash-DFlash2 drafter, CC BY-NC-ND 4.0)↗, Morrowmake (sm_80 GLM-5.3 code lineage via vllm-sm80)↗, Z.ai (GLM-5.3-Flash base model)↗ · listed by localai-recipes

GLM-5.3-Flash (turboderp's EXL3 2.60 bpw pack, sc_2.60bpw) served by AlexBrandes' vLLM fork at PP2/TP1 - VLLM_PP_LAYER_PARTITION=23,22, 23 target layers on card 1 and 22 plus the D

by inordic (recipe, patch series, measurements), 2dameneko (base vLLM build vllm-Qwen3.8-Flash-Next-2xCMP170HX, commit 56e1347), iIIusi0n (PLE-SSD offload recipe and ple_ssd_io.so helper), ukisai (Swift 1.5 W4A16-AWQ checkpoint), doth4580 (DotH MTP head retrained on agent traffic), leonbollerup (Qwen Asgard chat template), AlexBrandes (int4 MTP draft head patch 08, quantize_lm_head.py), vLLM upstream PRs ported: #55601, #60210, #56734, #60615, #58784, #54912, #62344, #65470 · listed by localai-recipes

Qwen3.8-Flash-Next (Swift 1.5 W4A16-AWQ, DotH MTP head retrained on agent traffic) on two unlocked 64 GB CMP 170HX cards (300 W VBIOS cross-flash) in vLLM pipeline parallel. PP2 (2

by CurtisAccelerate↗, Morrowmake↗, turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw — NOT the 2026-10-17 '1017' re-release, which has its own sibling r

by CurtisAccelerate↗, Morrowmake↗, Measured serve used local image sha256:55d2bd9a6664 (no published image)↗, turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)↗, PixelML/sm80vllm branch sm80 (DeepseekV4AmpereMLAAttention) - SM80 attention for this model↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (the 2026-10-17 re-release checkpoint, NOT the original DeepSeek-V4.1-Flash) as a hybrid quant on the TC-MoE port: official deepseek-ai/DeepSeek-V4.1-Flash

by g4bwy↗ · listed by localai-recipes

Recipe by g4bwy. Official Qwen/Qwen3.8-Flash-Next-FP8 on 4x unlocked CMP 170HX (GA100 sm_80, 74 SMs, 64 GB) with PP=4 (partition 13,12,12,11) and MTP-4 speculative decoding on pinn

by alanoo81 (recipe, graft, measurements)↗, GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)↗, LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)↗, Curtis White (hybrid layout + quantization_config.json)↗, vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four change

by ElectroHobo↗, Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)↗, halt95 (W4A16-Merlin checkpoint + companion 3090 engine; not involved in this recipe)↗, inco.ai (DFlash2 drafter, Apache-2.0)↗, Intel (AutoRound INT4 experts) + RadixArk (FP8 n-gram table) + DominikBucko (MTP-head packing)↗, Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)↗, MiaAI-Lab (benchmark protocol + category prompts, used as data)↗ · listed by localai-recipes

Qwen3.8-Flash-Next (125B MoE) INT4-expert W4A16 on two CMP 170HX cards, full 262,144-token context: 241/350/198 tok/s one user (structured/code/prose), 495 tok/s editing code, 895/

by mannydelrio↗, cmpunlocker (amoghmunikote)↗, clee/vllm-sm80↗, wtdcode (vllm-backport)↗ · listed by localai-recipes

Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE) in AWQ W4A16 (leoncca g32 build @ fa56146): PP2 with manual layer partition 25,23 (no P2P on CMP