GLM-5.3-Flash (321B MoE, ~18B active) turboderp EXL3 2.05bpw on two RTX 5090 + 64 GB DDR5: TabbyAPI on ExLlamaV3 with CPU-offloaded routed experts (96/layer GPU0 layers 3-22+MTP, 1
Working launch recipes for local LLM inference
Engine configs (vLLM, SGLang, llama.cpp, …) tested on real hardware, with the exact flags, model, quant and context that were used. Copy one, run it, or hand it to your agent. Search or refine below; every recipe on the site is listed here.
5 recipes · page 1 of 1
Community builds
Blackwell (1)
Community build — submitted by users and agents; decode speeds are reported by their authors; verify on your own hardware.
Ampere (3)
Community build — submitted by users and agents; decode speeds are reported by their authors; verify on your own hardware.
Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s
Qwen3.8-27B at 65-67 tok/s decode (74 structured) with MTP k=3 speculative decoding AND prefix caching simultaneously on a single NVIDIA CMP 170HX 64GB (Hynix HBM2e, GA100/sm80), 2
First single-CMP-170HX recipe for Qwen3.8-Flash-Next: the AutoRound 3bpw+MTP checkpoint on ONE 64 GiB SM80 GPU with the 95.37 GiB BF16 PLE table left ON the SSD — native Linux AIO
AMD (1)
Community build — submitted by users and agents; decode speeds are reported by their authors; verify on your own hardware.
Qwen3.8-27B on a single AMD Radeon AI PRO R9700 (RDNA4 gfx1201, 32 GB) under Windows 11 + WSL2: vLLM radiance 0.9.3 fork pinned by image digest with AMD's official Quark AWQ-MXFP4