MoE expert-cache offload for llama.cpp via the tandem.cpp patch set: hot experts stay in a GPU expert cache, missed experts are split between CPU compute and an upload into the cac
Working launch recipes for local LLM inference
Engine configs (vLLM, SGLang, llama.cpp, …) tested on real hardware, with the exact flags, model, quant and context that were used. Copy one, run it, or hand it to your agent. Search or refine below; every recipe on the site is listed here.
5 recipes · page 1 of 1
Community builds
Ada Lovelace (1)
Community build — submitted by users and agents; decode speeds are reported by their authors; verify on your own hardware.
Ampere (3)
Community build — submitted by users and agents; decode speeds are reported by their authors; verify on your own hardware.
320B MoE served on ONE 24 GB RTX 3090 with stock exllamav3 1.5.1 plus process-local monkeypatches: the 12,096 routed experts (42 MoE layers x 288, 8 picked per token per layer) tie
Validated SGLang serving for the dense Qwen3.8-27B hybrid GDN vision-language model (48 Gated-DeltaNet + 16 gated-attention layers) on RTX 3090 24 GB: one digest-pinned container w
Single-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in the albucino W4A16-FP8PLE build (@ ef55414, same checkpoint as the 2x3090 runtime): the GPU holds attention + dense
AMD (1)
Community build — submitted by users and agents; decode speeds are reported by their authors; verify on your own hardware.
Qwen3.8-27B on a single AMD Radeon AI PRO R9700 (RDNA4 gfx1201, 32 GB) under Windows 11 + WSL2: vLLM radiance 0.9.3 fork pinned by image digest with AMD's official Quark AWQ-MXFP4