{"slug":"glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h","title":"GLM-5.3-Flash EXL3 3.05bpw on 1x RTX PRO 6000 - GPU expert cache, host-RAM streaming","summary":"GLM-5.3-Flash (320B, 42x288-expert MoE) served from turboderp's EXL3 3.05bpw checkpoint (116.7 GiB) on ONE 96 GB RTX PRO 6000 Blackwell: a GPU expert cache keeps the 40 most-routed experts static in VRAM plus 148 LFU cache slots on-card, and streams the remaining 100 experts per layer from 91.5 GiB of pinned host RAM by rewriting ExLlamaV3's expert pointer tables in place - stock ExLlamaV3 1.6.0 kernels and unmodified TabbyAPI (pinned commit) behind a front shim. 61.5 tok/s decode short-context, 68 tok/s at 32K-127K in context, 1.9K-2.9K tok/s prefill, 524,288-token context, native vision, MTP drafting; mean KLD 0.0680 vs the original model and 59/60 on a 60-task graded tool-use bench - roughly half the llama.cpp UD-IQ3_XXS fidelity error (0.1258) while beating it on every axis. Replaces the author's own llama.cpp recipe in the same repo.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"b273a7fb8e437bb6978e08e600c43acc1fbb6fda","content_hash":"9a80e9b5da56789795695cd17f70e97159a46050","created_at":"2026-10-20 15:34:15.674077-06","notes":"Submitted from the repo at bcc27a7587e0e22e5752b93534e47c7acedb9a1a (2026-10-09 18:06 -0400, 'Make EXL3 3.05bpw with a GPU expert cache the default runtime'). All performance figures are the author's own measurements from benchmarks/exl3-20261009/api-bench-exl3.json (verified verbatim at ingest); the 59/60 agent bench and KLD 0.0680 are from agent-bench-exl3.json and turboderp's model-card chart as documented in docs/exl3.md. Recipe + measurements by SSHdotCodes; EXL3 3.05bpw checkpoint by turboderp; TabbyAPI (theroyallab) run unmodified at a pinned commit, ExLlamaV3 stock 1.6.0 wheel - no engine patches, the cache is process-local pointer-table interception. Runtime replaces the same author's llama.cpp UD-IQ3_XXS recipe of 2026-09-20 (kept in docs/llamacpp.md). Fourth ExLlamaV3 recipe on site (after the 1x RTX 3090 RAM/NVMe tiering, 2x RTX 5090 hot-swap and 2x CMP 170HX entries) but FIRST on Blackwell, and first single-card GLM recipe in the 1x RTX PRO 6000 bucket (prior 6000-bucket GLM entries are NVFP4 4-card lanes). QFN sibling repo (qwen-3.8-flash-next-pro6000, SGLang) intentionally not ingested to avoid a 3rd 1x6000 QFN bucket entry."},"engine_source":{"type":"release","git_url":"https://github.com/SSHdotCodes/glm-5.3-flash-pro6000","base_engine":"exllamav3","pip_package":"exllamav3==1.6.0","base_version":"ExLlamaV3 1.6.0 + TabbyAPI 2fd6cc7 + expert cache (sm_120)","image_digest":null,"install_note":"NOT a pip-only install: exl3/install.sh builds a Python 3.12 venv (torch 2.13.0+cu130 pinned), installs the official ExLlamaV3 1.6.0+cu132/torch2.13 wheel (SHA-256 21e517df8f8e19f07f8fa1075411a55f00689246d7844485c7c36b42b5666034), clones TabbyAPI at pinned commit 2fd6cc76203a66e13042daf7d76e5898b21c1ad8 (run unmodified), and nvcc-builds the expert-cache CUDA extension for sm_120 (CUDA 13.4, driver 615.71 measured). Needs 128 GB RAM (104 GiB free minimum enforced: 91.5 GiB pinned cold experts), cgroup v2 with MemorySwapMax=0 enforced by the launcher, and exl3/download.py fetches 116.7 GiB of weights at HF revision 332ab457b709b7ba30dd9a448be5de03b80a7ac9 with per-file SHA-256 checks.","pinned_commit":"bcc27a7587e0e22e5752b93534e47c7acedb9a1a"},"credit":[{"name":"SSHdotCodes","url":"https://github.com/SSHdotCodes/glm-5.3-flash-pro6000"},{"name":"turboderp","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["python3 exl3/run.py start","# serve commands (python3 exl3/run.py command) - exl3/run.py serve writes .run/config.yml then execs:","exl3/.venv/bin/python -u exl3/anthropic_shim.py --port 8098 --upstream http://127.0.0.1:8099 --model glm-5.3-flash","exl3/.venv/bin/python -u exl3/tabby_glm.py","# env set by run.py serve: GLMC_P=40 GLMC_D=148 GLMC_CACHE_DIR=exl3/cache GLMC_TABBY_DIR=exl3/src/tabbyAPI PYTHONUNBUFFERED=1","# systemd-run --user --unit glm53-exl3 --property=MemorySwapMax=0 --property=KillSignal=SIGINT --property=TimeoutStopSec=60 --property=Restart=no","# TabbyAPI config (.run/config.yml): network 127.0.0.1:8099 disable_auth api_servers [OAI]; model_dir exl3/models/glm-5.3-flash-exl3/view, model_name glm-5.3-flash, max_seq_len 524288, cache_size 524288, chunk_size 4096, max_batch_size 1, vision true, reasoning true, tool_format auto, template_vars_default {reasoning_effort: high}; draft_model {draft_mode mtp, draft_num_tokens 2}; sampling override_preset glm53 (creator sampling: temp 1.0, top-p 0.95, top-k 0, min-p 0, max output 131072)"],"context":{"apis":"port 8098: OpenAI chat completions (tools, vision, reasoning split out) from TabbyAPI + Anthropic Messages (/v1/messages, /v1/messages/count_tokens) + OpenAI Responses (/v1/responses) from exl3/anthropic_shim.py translating to chat completions; TabbyAPI itself on 127.0.0.1:8099; loopback only","vision":"native vision on (config vision true); measured through api_check.py","window":"524,288 tokens per request (max_seq_len and cache_size both 524288); with the 512K cache allocated about 3.8 GiB VRAM stays free (about 1.7 GiB at a 128K-token prefill peak)","reasoning":"GLM template native levels low/high/max, default reasoning_effort high; sampler_overrides/glm53.yml = creator sampling, request values take precedence"},"runtime":{"cold_start":"55-60 s process start to ready","launcher_gates":"exl3/run.py preflight refuses: non-Linux, GPU compute processes active, anything but exactly one RTX PRO 6000 Blackwell with <94,000 MiB free, MemAvailable <104 GiB, missing/wrong-size weights vs model.lock.json, missing cgroup v2; serve() refuses to pin experts unless memory.swap.max reads 0","verified_public_build":"Oct 9 fresh-checkout install resolved the same package versions; download.py --verify-only checked all 28 files by size+SHA-256; api_check.py passed 14/14 checks including vision; public-build numbers within noise of the measured install (63.14 vs 61.48 short decode)"},"expert_cache":{"knobs":"exl3/config.local.json: static_experts (40), cache_slots (148); env read by glm_cache2.py: GLMC_DECAY, GLMC_STAGE, GLMC_COPY_BLOCKS, GLMC_PREFETCH (off - measured net loss, competes with demand misses for PCIe)","layout":"per MoE layer (42 layers, 288 experts each, top-8 routing, ~9 MiB/expert at 3.05bpw): 40 static experts in VRAM (highest routing counts from exl3/cache/routing-counts.json, summed over 8 prompts / 73,667 routed rows), 148 VRAM slots as an LFU cache, remaining 100 experts in page-locked device-mapped host RAM (91.5 GiB total); glm_cache intercepts the host-resident experts' load","mechanism":"decode: ec_manage runs as one CUDA block per layer right after routing - LFU scores decay 0.995, each miss takes the lowest-scoring slot unused by the current call, pointer tables multi_{gate,up,down}.ptrs_trellis rewritten in place, then ec_copy streams missed experts from host RAM at ~49 GB/s over PCIe 5.0 x16; both kernels read the miss count on device so decode never synchronizes with the host; prefill: each chunk stages the uncached experts it needs into one VRAM staging area shared by all layers and points the tables at the staged copies for the call","policy_data":"benchmarks/exl3-20261009/cache-policy-simulation.txt: replaying 8 traces, misses/token at 170 slots - static top-170 100.9, LRU 31.5, LFU decay 0.995 29.3, Belady bound 13.6; a dynamic cache matters far more than the choice among dynamic policies"},"storage_requirements":{"ram_gb":128}},"models":[{"slug":"glm-5-3-flash-exl3-3-05bpw","name":"GLM-5.3-Flash-EXL3-3.05bpw","hf_repo":"turboderp/GLM-5.3-Flash-exl3","quant":"exl3-3-05bpw"}],"hardware":[{"slug":"nvidia-rtx-pro-6000-blackwell","vendor":"nvidia","card":"rtx-pro-6000-blackwell","card_family":"rtx-pro-6000-blackwell","vram_gb":96,"device_count":1}],"generation":"blackwell","tags":["agent-serving","exl3","expert-cache","host-ram-streaming","long-context","moe-expert-offload","mtp","tabbyapi","vision"],"tasks":["chat","code","vision"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"All figures are the author's own measurements (benchmarks/exl3-20261009/, driven through the OpenAI streaming API by exl3/bench_api.py: temperature 1.0, top-p 0.95, high reasoning, fresh nonce per request so no prefix reuse; short decode = mean of 5 x 1,024-token runs, depth rows 256 output tokens). llama.cpp UD-IQ3_XXS same-shape baseline from docs/exl3.md: 56.65 short / 53.7-53.9-46.1 depth decode / 597-713-562 prefill / KLD 0.1258. KLD 0.0680 vs BF16 is turboderp's model-card chart (noise floor 0.0138). Graded agent bench: 59/60 simulated-tool tasks with known answers, no malformed tool calls from either runtime.\", \"tok_s\": 61.47999999999999687361196265555918216705322265625, \"points\": [{\"notes\": \"short-prompt decode, mean of 5 x 1,024-output-token runs (api-bench-exl3.json short[])\", \"concurrency\": 1, \"context_len\": 1024, \"decode_tok_s\": 61.47999999999999687361196265555918216705322265625}, {\"notes\": \"one request, 256 output tokens, ~7,953 prompt tokens; depth-row decode rates are noisy (80-130 output tokens)\", \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 59.64999999999999857891452847979962825775146484375, \"prefill_tok_s\": 1942.700000000000045474735088646411895751953125}, {\"notes\": \"one request, 256 output tokens, ~31,724 prompt tokens\", \"concurrency\": 1, \"context_len\": 32768, \"decode_tok_s\": 68.340000000000003410605131648480892181396484375, \"prefill_tok_s\": 2620.6999999999998181010596454143524169921875}, {\"notes\": \"one request, 256 output tokens, ~126,801 prompt tokens\", \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 68.030000000000001136868377216160297393798828125, \"prefill_tok_s\": 2851.40000000000009094947017729282379150390625}], \"context_len\": 524288}"}],"perf_points":[{"id":1073,"context_len":1024,"prefill_tok_s":null,"decode_tok_s":61.47999999999999687361196265555918216705322265625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"short-prompt decode, mean of 5 x 1,024-output-token runs (api-bench-exl3.json short[])"},{"id":1074,"context_len":8192,"prefill_tok_s":1942.700000000000045474735088646411895751953125,"decode_tok_s":59.64999999999999857891452847979962825775146484375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"one request, 256 output tokens, ~7,953 prompt tokens; depth-row decode rates are noisy (80-130 output tokens)"},{"id":1075,"context_len":32768,"prefill_tok_s":2620.6999999999998181010596454143524169921875,"decode_tok_s":68.340000000000003410605131648480892181396484375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"one request, 256 output tokens, ~31,724 prompt tokens"},{"id":1076,"context_len":131072,"prefill_tok_s":2851.40000000000009094947017729282379150390625,"decode_tok_s":68.030000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"one request, 256 output tokens, ~126,801 prompt tokens"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3/tree/3.05bpw","pin":"332ab457b709b7ba30dd9a448be5de03b80a7ac9"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/turboderp-org/exllamav3/releases/tag/v1.6.0","pin":"exllamav3-1.6.0+cu132.torch2.13.0 wheel sha256 21e517df8f8e19f07f8fa1075411a55f00689246d7844485c7c36b42b5666034"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/theroyallab/tabbyAPI","pin":"2fd6cc76203a66e13042daf7d76e5898b21c1ad8"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:15.674077-06","created_at":"2026-10-09 20:09:21.425538-06","speeds":{"decode_tok_s":64.7999999999999971578290569595992565155029296875,"prefill_tok_s":2736.09999999999990905052982270717620849609375,"leaderboard":"/api/v1/leaderboard.json"}}