Working launch recipes for local LLM inference

by Morrowmake (recipe, engine fork, measurements)↗, LibertAI - EXL3 2.9-bpw routed-expert quantization↗, DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)↗, cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch↗, MiaAI-Lab - decode benchmark protocol (used as data)↗, Marlin (IST-DASLab + Neural Magic contributors) - kernel basis↗, Curtis White (CurtisAccelerate) - TC-MoE lane shape + split-expert idea (docs/credits.md)↗, GNmac-lab - FP8 wo_a grouped-GEMM E4M3 decode; target argmax, side-stream mHC, split-expert ideas↗, b12x (local-inference-lab) - copy-engine ring all-reduce design credit↗, ExLlamaV3 (turboderp-org) - EXL3 trellis packing, codebooks, Hadamard conventions (adapted)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.2 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw expo

by vcruz305 (recipe author, SAGE quant, packs, measurements, MIT)↗, Christopher Owen (spark-ds41f r6w base image, upstreams pinned at 2dc95329)↗, vcruz305/exllamav3 feat/moe-warp-private-stages @ 909ad8564c - mixed-K unified MoE + warp-private pipeline stages↗, vcruz305/vllm-exl3 feat/dsv41-1017-tp1 @ 30e8b94137 - Exl3MoEMethod / ATS loader↗, AgentSparko (REXL3 overlay path used by the sibling 1.55bpw pack)↗, deepseek-ai/DeepSeek-V4.1-Flash-1017 - upstream weights (MIT)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 at 1.62 bpw (SAGE MixedK EXL3 routed experts, EXL3 MTP draft experts, EXL3 K8 output head, block-32 FP8 non-routed linears) served on one NVIDIA DGX Spark

by kindlingai↗, nvidia↗ · listed by localai-recipes

GLM-5.3-Flash exlr8: MoE layers are EXL3-trellis units decoded by the custom exlr8 MoE kernel; dense linears stay FP8/NVFP4. One release carries three expert widths (k2/k3/k4 = 2/3

by inordic (recipe, patch series, measurements), 2dameneko (base vLLM build vllm-Qwen3.8-Flash-Next-2xCMP170HX, commit 56e1347), iIIusi0n (PLE-SSD offload recipe and ple_ssd_io.so helper), ukisai (Swift 1.5 W4A16-AWQ checkpoint), doth4580 (DotH MTP head retrained on agent traffic), leonbollerup (Qwen Asgard chat template), AlexBrandes (int4 MTP draft head patch 08, quantize_lm_head.py), vLLM upstream PRs ported: #55601, #60210, #56734, #60615, #58784, #54912, #62344, #65470 · listed by localai-recipes

Qwen3.8-Flash-Next (Swift 1.5 W4A16-AWQ, DotH MTP head retrained on agent traffic) on two unlocked 64 GB CMP 170HX cards (300 W VBIOS cross-flash) in vLLM pipeline parallel. PP2 (2

by CurtisAccelerate↗, Morrowmake↗, turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw — NOT the 2026-10-17 '1017' re-release, which has its own sibling r

by CurtisAccelerate↗, Morrowmake↗, Measured serve used local image sha256:55d2bd9a6664 (no published image)↗, turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)↗, PixelML/sm80vllm branch sm80 (DeepseekV4AmpereMLAAttention) - SM80 attention for this model↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (the 2026-10-17 re-release checkpoint, NOT the original DeepSeek-V4.1-Flash) as a hybrid quant on the TC-MoE port: official deepseek-ai/DeepSeek-V4.1-Flash

by recipe, tuning notes and measurements by givisok↗, base stack and vllm-sm80 fork by clee↗, AWQ-g32 checkpoint by leoncca↗, uncensored base by orcarouter↗, cudagraph capture-size idea from mannydelrio↗, vLLM (Apache-2.0)↗ · listed by localai-recipes

MTP-4 production recipe for Qwen3.8-Flash-Next-Uncensored (leoncca AWQ-g32) on 3x CMP 170HX (SM80 cards unlocked to 64 GiB): PP3 partition 17,17,14 (no GPU P2P on this pairing; the

by g4bwy↗ · listed by localai-recipes

Recipe by g4bwy. Official Qwen/Qwen3.8-Flash-Next-FP8 on 4x unlocked CMP 170HX (GA100 sm_80, 74 SMs, 64 GB) with PP=4 (partition 13,12,12,11) and MTP-4 speculative decoding on pinn

by alanoo81 (recipe, graft, measurements)↗, GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)↗, LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)↗, Curtis White (hybrid layout + quantization_config.json)↗, vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)↗ · listed by localai-recipes

DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four change

by ElectroHobo↗, Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)↗, halt95 (W4A16-Merlin checkpoint + companion 3090 engine; not involved in this recipe)↗, inco.ai (DFlash2 drafter, Apache-2.0)↗, Intel (AutoRound INT4 experts) + RadixArk (FP8 n-gram table) + DominikBucko (MTP-head packing)↗, Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)↗, MiaAI-Lab (benchmark protocol + category prompts, used as data)↗ · listed by localai-recipes

Qwen3.8-Flash-Next (125B MoE) INT4-expert W4A16 on two CMP 170HX cards, full 262,144-token context: 241/350/198 tok/s one user (structured/code/prose), 495 tok/s editing code, 895/

by mannydelrio↗, cmpunlocker (amoghmunikote)↗, clee/vllm-sm80↗, wtdcode (vllm-backport)↗ · listed by localai-recipes

Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE) in AWQ W4A16 (leoncca g32 build @ fa56146): PP2 with manual layer partition 25,23 (no P2P on CMP

by mannydelrio↗, AlexBrandes (W4A16 draft lm_head patch, quantizer)↗, cmpunlocker (amoghmunikote)↗, clee/vllm-sm80↗, wtdcode (vllm-backport)↗ · listed by mannydelrio1

Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE), AWQ W4A16 g32: PP2 (CMP 170HX has no P2P), MTP 5-token drafting with an int4 draft-only lm_head,