localairecipes launch configs for local inference

GLM-5.3-Flash EXL3 3.05bpw on 1x RTX PRO 6000 - GPU expert cache, host-RAM streaming

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by SSHdotCodes↗, turboderp↗ · listed by localai-recipes

revision r2 · commit b273a7fb8e · updated 10 d ago · license CC-BY-4.0

chat code vision vision long-context exl3 tabbyapi mtp expert-cache moe-expert-offload host-ram-streaming agent-serving

GLM-5.3-Flash (320B, 42x288-expert MoE) served from turboderp's EXL3 3.05bpw checkpoint (116.7 GiB) on ONE 96 GB RTX PRO 6000 Blackwell: a GPU expert cache keeps the 40 most-routed experts static in VRAM plus 148 LFU cache slots on-card, and streams the remaining 100 experts per layer from 91.5 GiB of pinned host RAM by rewriting ExLlamaV3's expert pointer tables in place - stock ExLlamaV3 1.6.0 kernels and unmodified TabbyAPI (pinned commit) behind a front shim. 61.5 tok/s decode short-context, 68 tok/s at 32K-127K in context, 1.9K-2.9K tok/s prefill, 524,288-token context, native vision, MTP drafting; mean KLD 0.0680 vs the original model and 59/60 on a 60-task graded tool-use bench - roughly half the llama.cpp UD-IQ3_XXS fidelity error (0.1258) while beating it on every axis. Replaces the author's own llama.cpp recipe in the same repo.

Launch command

'python3 exl3/run.py start' '# serve commands (python3 exl3/run.py command) - exl3/run.py serve writes .run/config.yml then execs:' 'exl3/.venv/bin/python -u exl3/anthropic_shim.py --port 8098 --upstream http://127.0.0.1:8099 --model glm-5.3-flash' 'exl3/.venv/bin/python -u exl3/tabby_glm.py' '# env set by run.py serve: GLMC_P=40 GLMC_D=148 GLMC_CACHE_DIR=exl3/cache GLMC_TABBY_DIR=exl3/src/tabbyAPI PYTHONUNBUFFERED=1' '# systemd-run --user --unit glm53-exl3 --property=MemorySwapMax=0 --property=KillSignal=SIGINT --property=TimeoutStopSec=60 --property=Restart=no' '# TabbyAPI config (.run/config.yml): network 127.0.0.1:8099 disable_auth api_servers [OAI]; model_dir exl3/models/glm-5.3-flash-exl3/view, model_name glm-5.3-flash, max_seq_len 524288, cache_size 524288, chunk_size 4096, max_batch_size 1, vision true, reasoning true, tool_format auto, template_vars_default {reasoning_effort: high}; draft_model {draft_mode mtp, draft_num_tokens 2}; sampling override_preset glm53 (creator sampling: temp 1.0, top-p 0.95, top-k 0, min-p 0, max output 131072)'

context

apis: 'port 8098: OpenAI chat completions (tools, vision, reasoning split out) from TabbyAPI + Anthropic Messages (/v1/messages, /v1/messages/count_tokens) + OpenAI Responses (/v1/responses) from exl3/anthropic_shim.py translating to chat completions; TabbyAPI itself on 127.0.0.1:8099; loopback only'
vision: 'native vision on (config vision true); measured through api_check.py'
window: '524,288 tokens per request (max_seq_len and cache_size both 524288); with the 512K cache allocated about 3.8 GiB VRAM stays free (about 1.7 GiB at a 128K-token prefill peak)'
reasoning: 'GLM template native levels low/high/max, default reasoning_effort high; sampler_overrides/glm53.yml = creator sampling, request values take precedence'

runtime

cold_start: '55-60 s process start to ready'
launcher_gates: 'exl3/run.py preflight refuses: non-Linux, GPU compute processes active, anything but exactly one RTX PRO 6000 Blackwell with <94,000 MiB free, MemAvailable <104 GiB, missing/wrong-size weights vs model.lock.json, missing cgroup v2; serve() refuses to pin experts unless memory.swap.max reads 0'
verified_public_build: 'Oct 9 fresh-checkout install resolved the same package versions; download.py --verify-only checked all 28 files by size+SHA-256; api_check.py passed 14/14 checks including vision; public-build numbers within noise of the measured install (63.14 vs 61.48 short decode)'

expert_cache

knobs: 'exl3/config.local.json: static_experts (40), cache_slots (148); env read by glm_cache2.py: GLMC_DECAY, GLMC_STAGE, GLMC_COPY_BLOCKS, GLMC_PREFETCH (off - measured net loss, competes with demand misses for PCIe)'
layout: "per MoE layer (42 layers, 288 experts each, top-8 routing, ~9 MiB/expert at 3.05bpw): 40 static experts in VRAM (highest routing counts from exl3/cache/routing-counts.json, summed over 8 prompts / 73,667 routed rows), 148 VRAM slots as an LFU cache, remaining 100 experts in page-locked device-mapped host RAM (91.5 GiB total); glm_cache intercepts the host-resident experts' load"
mechanism: 'decode: ec_manage runs as one CUDA block per layer right after routing - LFU scores decay 0.995, each miss takes the lowest-scoring slot unused by the current call, pointer tables multi_{gate,up,down}.ptrs_trellis rewritten in place, then ec_copy streams missed experts from host RAM at ~49 GB/s over PCIe 5.0 x16; both kernels read the miss count on device so decode never synchronizes with the host; prefill: each chunk stages the uncached experts it needs into one VRAM staging area shared by all layers and points the tables at the staged copies for the call'
policy_data: 'benchmarks/exl3-20261009/cache-policy-simulation.txt: replaying 8 traces, misses/token at 170 slots - static top-170 100.9, LRU 31.5, LFU decay 0.995 29.3, Belady bound 13.6; a dynamic cache matters far more than the choice among dynamic policies'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h.json

Engine

Install note: NOT a pip-only install: exl3/install.sh builds a Python 3.12 venv (torch 2.13.0+cu130 pinned), installs the official ExLlamaV3 1.6.0+cu132/torch2.13 wheel (SHA-256 21e517df8f8e19f07f8fa1075411a55f00689246d7844485c7c36b42b5666034), clones TabbyAPI at pinned commit 2fd6cc76203a66e13042daf7d76e5898b21c1ad8 (run unmodified), and nvcc-builds the expert-cache CUDA extension for sm_120 (CUDA 13.4, driver 615.71 measured). Needs 128 GB RAM (104 GiB free minimum enforced: 91.5 GiB pinned cold experts), cgroup v2 with MemorySwapMax=0 enforced by the launcher, and exl3/download.py fetches 116.7 GiB of weights at HF revision 332ab457b709b7ba30dd9a448be5de03b80a7ac9 with per-file SHA-256 checks.

Model

Hardware

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
1,0241—61.5 tok/s—reported
8,19211942.7 tok/s59.7 tok/s—reported
32,76812620.7 tok/s68.3 tok/s—reported
131,07212851.4 tok/s68 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAll figures are the author's own measurements (benchmarks/exl3-20261009/, driven through the OpenAI streaming API by exl3/bench_api.py: temperature 1.0, top-p 0.95, high reasoning, fresh nonce per request so no prefix reuse; short decode = mean of 5 x 1,024-token runs, depth rows 256 output tokens). llama.cpp UD-IQ3_XXS same-shape baseline from docs/exl3.md: 56.65 short / 53.7-53.9-46.1 depth decode / 597-713-562 prefill / KLD 0.1258. KLD 0.0680 vs BF16 is turboderp's model-card chart (noise floor 0.0138). Graded agent bench: 59/60 simulated-tool tasks with known answers, no malformed tool calls from either runtime.

Downloads

Revision notes

Submitted from the repo at bcc27a7587e0e22e5752b93534e47c7acedb9a1a (2026-10-09 18:06 -0400, 'Make EXL3 3.05bpw with a GPU expert cache the default runtime'). All performance figures are the author's own measurements from benchmarks/exl3-20261009/api-bench-exl3.json (verified verbatim at ingest); the 59/60 agent bench and KLD 0.0680 are from agent-bench-exl3.json and turboderp's model-card chart as documented in docs/exl3.md. Recipe + measurements by SSHdotCodes; EXL3 3.05bpw checkpoint by turboderp; TabbyAPI (theroyallab) run unmodified at a pinned commit, ExLlamaV3 stock 1.6.0 wheel - no engine patches, the cache is process-local pointer-table interception. Runtime replaces the same author's llama.cpp UD-IQ3_XXS recipe of 2026-09-20 (kept in docs/llamacpp.md). Fourth ExLlamaV3 recipe on site (after the 1x RTX 3090 RAM/NVMe tiering, 2x RTX 5090 hot-swap and 2x CMP 170HX entries) but FIRST on Blackwell, and first single-card GLM recipe in the 1x RTX PRO 6000 bucket (prior 6000-bucket GLM entries are NVFP4 4-card lanes). QFN sibling repo (qwen-3.8-flash-next-pro6000, SGLang) intentionally not ingested to avoid a 3rd 1x6000 QFN bucket entry.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)