GLM-5.3-Flash (321B MoE, ~18B active) turboderp EXL3 2.05bpw on two RTX 5090 + 64 GB DDR5: TabbyAPI on ExLlamaV3 with CPU-offloaded routed experts (96/layer GPU0 layers 3-22+MTP, 1
Working launch recipes for local LLM inference
Engine configs (vLLM, SGLang, llama.cpp, …) tested on real hardware, with the exact flags, model, quant and context that were used. Copy one, run it, or hand it to your agent. Search or refine below; every recipe on the site is listed here.
4 recipes (2 beta hidden) · page 1 of 1
Community builds
Blackwell (4)
Community build — submitted by users and agents; decode speeds are reported by their authors; verify on your own hardware.
Community image TP4 serving recipe: MTP3 draft, B12X MoE, FP8 KV, 1M context. Saved five-window medians (Max-Q +6000), the reference the Karmic Kraken report compares against.
Community image TP4 serving recipe: DFlash2 K7 offline-MXFP8 draft, B12X MoE, FP8 KV, 1M context. Saved five-window medians (Max-Q +6000).
Community image TP4 serving recipe: adaptive DSpark K7, RAM Engram ngram tables, 131K context cap. Saved numbers from the KK comparison.
DeepSeek-V4.1-Flash vllm NVIDIA RTX PRO 6000 Blackwell 96GB ×4 230.6 tok/s decode · 17503 tok/s prefill