{"slug":"glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw","title":"GLM-5.3-Flash EXL3 2.05bpw on 2x RTX 5090 - TabbyAPI hot-expert swaps, 262K ctx, vision","summary":"GLM-5.3-Flash (321B MoE, ~18B active) turboderp EXL3 2.05bpw on two RTX 5090 + 64 GB DDR5: TabbyAPI on ExLlamaV3 with CPU-offloaded routed experts (96/layer GPU0 layers 3-22+MTP, 102/layer GPU1 23-44), runtime hot-expert swaps, per-GPU expert split, the DSA indexer's per-token rows in a fixed ring, live /metrics token counters, the expert-exchange clock handed to the first text MoE layer so swaps also run at 2-4 streams, MTP head drafting 1 tok/step single-stream, 262,144-token window on an 8-bit KV page pool (~1.6 GiB across 4), vision ON with tower linears pinned in host RAM. Exact-output: no expert skipped, every draft verified. Measured 2026-10-10 (R969, three fresh boots): c1 score 62.9 tok/s (code 59.9/prose 66.7/chat 68.5/html 64.8/edit 53.9), 2/3/4-stream sums 74.4/92.0/98.0. Nine in-repo overlays on the qwen3.8-flash-next-2x-rtx5090 base image.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"4dc491797af361a6a0f4cdedde5f7dbeb918082d","content_hash":"66ee933dc865f3362d1e2880b2ef8e330f331b05","created_at":"2026-10-29 23:29:37.491895-06","notes":"Submitted from the repo at 4e3e0f0f5bfbcf1f963936da89b6a80cda2876b8 (2026-10-09 00:45 +0200). All figures are the author's own measurements (bench/results R927 served arm B-D with R915c/R925 spread arms, R924 prefill arm, R900 memory layout; raw records in bench/results/<date>-<round>/). Hardware note: 60 GB usable of 2x32 GB DDR5-6000 hosts the CPU-offloaded routed experts (26.4 GiB) plus the pinned vision tower. Exact-output claim per author: no expert skipped, every drafted token verified; greedy run-to-run instability documented (R916) with opt-in deterministic flags. First TabbyAPI-served and first RTX-5090 recipe on the site; second ExLlamaV3 entry after Flun's 2x CMP 170HX. UPDATE 2026-10-30 (repo @ 88e112e08a2368e3194232aed1ba5f6375ad3368, 13 commits ahead of the pinned 4e3e0f0f): served configuration moved - image tag gained _ring3_livemetrics2_draftchunk1_textclock1, new INDEX_RING=1, EXL3_MOE_CPU_SPLIT_BY_DEVICE 100,104 -> 96,102, new EXL3_MOE_CPU_SWAP_TEXT_CLOCK=1, new CHUNK=4096 (the recipe previously documented CHUNK=2048 and 100,104 - both stale). Three new overlays are now part of the stack: glm-index-ring-r3 (DSA indexer per-token rows in a fixed ring instead of the KV pool), tabby-livemetrics-r3 (live token counters on /metrics) and textclock-r1 (expert-exchange clock to the first text MoE layer, so swaps also run while 2-4 requests are active). draftchunk-r1 lets chunks above 2,048 tokens boot, which is what makes CHUNK=4096 usable. KV pool restated by the author at ~1.6 GiB (608 bytes per token on each of the 11 attention layers: latent 512, scales 32, pooled indexer keys 64) plus an 8.5 MiB ring per DSA layer, replacing the earlier ~2.3 GiB figure. launch-glm53.sh now HARD-REQUIRES scripts/sampler_overrides/glm53.yml and mounts it read-only at /app/sampler_overrides/glm53.yml - without that file the launch aborts with exit 3. Performance numbers below are refreshed to the author's R969 served arms B1/B2/B3 (2026-10-10, three fresh boots, median of the boots, measured ON the text-clock image): c1 score 62.9, 1 stream 64.8, 2/3/4-stream per stream / sum 37.2/74.4, 30.7/92.0, 24.5/98.0 tok/s; c1 by kind code 59.9 / prose 66.7 / chat 68.5 / html 64.8 / edit 53.9; TTFT 0.48 s c1, 0.75-0.82 / 1.11-1.21 / 1.57-1.70 s at 2/3/4 streams. Prefill remains R959d arm C4a on the image BEFORE the text clock (author states R969 did not measure prefill): 1,973 tok/s @ 32,738 prompt tokens and 2,023 @ 131,051."},"engine_source":{"type":"fork","git_url":"https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090","base_engine":"exllamav3","pip_package":null,"base_version":"dev 5783a93 + TabbyAPI 53da7919","image_digest":null,"install_note":"NOT installable from a stock image: the served image stacks five in-repo overlays (docker/cheapswap-r3, glm-agent-r2-on-cheapswap-r3, mtp-fast-r1, mtp-overhead-r2, glm-splitdev-r2) on the base image tabbyapi:r828-prompt-lookup-r3 (ExLlamaV3 dev 5783a93 with 19 patch sets + TabbyAPI 53da7919, from adrienbrault/qwen3.8-flash-next-2x-rtx5090). Build the overlays from the repo's Dockerfiles, then launch with scripts/glm-daily.env (scripts/launch-glm53.sh writes config.yml, clears image-baked EXL3_* selectors, verifies offload registration; needs the box paths /srv/qwen5090/models/<pack> on NVMe and ~76 GB disk for the checkpoint).","pinned_commit":"88e112e08a2368e3194232aed1ba5f6375ad3368"},"credit":[{"name":"adrienbrault (recipe, five patch overlays, measurements)","url":"https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090"},{"name":"zai-org (GLM-5.3-Flash)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"turboderp (EXL3 2.05bpw quantisation; ExLlamaV3 + CPU MoE offload)","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"},{"name":"theroyallab (TabbyAPI)","url":"https://github.com/theroyallab/tabbyAPI"},{"name":"adrienbrault/qwen3.8-flash-next-2x-rtx5090 (base image: ExLlamaV3 dev 5783a93 patch stack + TabbyAPI 53da7919)","url":"https://github.com/adrienbrault/qwen3.8-flash-next-2x-rtx5090"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"mtp":{"draft":"the model's own MTP head, depth 1 (one token per step), active only while one request is active (EXL3_MTP_MAX_BATCH=1); at 2-4 streams every stream decodes without a draft; MTP_FAST=1 joins the MTP layer's CPU experts to the trunk CPU worker; four host-overhead flags (EXL3_DRAFT_PINNED_STAGING, EXL3_MTP_GPU_DRAFT, EXL3_MTP_GREEDY_ACCEPT, EXL3_MTP_CACHED_REWIND) measure inside arm-to-arm spread (R914)","greedy_note":"EXL3_MTP_GREEDY_ACCEPT applies to greedy requests only"},"launch":["bash scripts/launch-glm53.sh","# env (scripts/glm-daily.env @ 88e112e0): PACK=A OFFLOAD_MODE=split CACHE_TOKENS=262144 MAX_SEQ=262144 CACHE_MODE=8,8 CHUNK=4096 MAX_BATCH=4 SYSMEM_RC_MB=1024","OFFLOAD_N=104 DRAFT=1 DRAFT_N=1 MTP_FAST=1 VISION_OFFLOAD=1 INDEX_RING=1 GPU_SPLIT=31.8,31 CHUNK=4096","GLM_IMG=tabbyapi:cheapswap-r3-agent-r2_mtpfast1_overhead-r2_splitdev2_ring3_livemetrics2_draftchunk1_textclock1 PINNED_ARENA=1 AGENT=1 KEEP_THINKING=1","EXL3_EXTRA='EXL3_MTP_MAX_BATCH=1;EXL3_DRAFT_PINNED_STAGING=1;EXL3_MTP_GPU_DRAFT=1;EXL3_MTP_GREEDY_ACCEPT=1;EXL3_MTP_CACHED_REWIND=1;EXL3_MOE_CPU_SPLIT_BY_DEVICE=96,102;EXL3_MOE_CPU_SWAP_TEXT_CLOCK=1'","SWAP_MODE=exchange SWAP_POLICY=histogram SWAP_INIT_STATS=$PWD/scripts/split-stats-broad-r869.json SWAP_CADENCE=exact SWAP_INTERVAL=64 SWAP_MAX=64 SWAP_SCOPE=global SWAP_FLOOR=4 SWAP_HYST=2.0","# REQUIRED: scripts/sampler_overrides/glm53.yml must exist or launch-glm53.sh aborts (exit 3)","# launch-glm53.sh mounts it read-only at -v \"$HERE/sampler_overrides/glm53.yml\":/app/sampler_overrides/glm53.yml:ro"],"context":{"kv":"one 8-bit KV page pool (cache_mode 8,8) of 262,144 tokens shared by up to 4 concurrent requests, ~2.3 GB (864 bytes/token x 11 attention layers; the other 31 layers hold a fixed-size recurrent state); sysmem_recurrent_cache 1024 MB","vision":"vision on (vision true); VISION_OFFLOAD=1 keeps the ViT tower's linear weights (0.494 GiB) in pinned host RAM, streamed to the GPU per image","window":"262,144 tokens per request (cache_size and max_seq_len both 262144)"},"offload":{"swap":"runtime hot-expert swaps, cheapswap-r3 exchange mode (checkpoint-free GPU<->worker exchange), histogram policy, start from the broad hot set, exact cadence every 64 decode tokens, SWAP_MAX 64, global budget, SWAP_FLOOR 4, hysteresis 2.0","split":"EXL3_MOE_CPU_SPLIT_BY_DEVICE=100,104: 100 of 288 routed experts per MoE layer on the CPU for GPU0's layers (3-22 + MTP 45), 104 for GPU1's (23-44); 8 CPU worker threads, pinned arena (EXL3_MOE_PINNED_ARENA=0 + exchange swaps)","expert_worker":"CPU expert GEMV reads cold experts at 61-62 GB/s of a 63 GB/s ceiling on the 9800X3D (R874); first requests after boot decode slower until placement adapts (R870/R872)"},"runtime":{"ports":"TabbyAPI on 0.0.0.0:8029 (container glm53, --disable-auth true; single box, no other tenants)","clocks":"power limits stock 600/575 W; memory clock offset +4500 MHz, core offset 0 (launcher asserts readback on both GPUs)","tabby_config":"written by scripts/glm53_plan.py: model_name glm53-flash-exl3-2.05bpw-turboderp, backend exllamav3, max_seq_len 262144, cache_size 262144, cache_mode 8,8, max_batch_size 4, tensor_parallel false, gpu_split [31.8, 31.0], gpu_split_auto false, cpu_moe_split_experts 104 (overridden per device to 100,104 by EXL3_MOE_CPU_SPLIT_BY_DEVICE), chunk_size 2048, output_chunking true, vision true, vision_offload true, tool_format glm4_5, reasoning true with reasoning_start/end token (Paperclip) and prompt_template glm53-keep-thinking (KEEP_THINKING=1); draft_model: draft_mode mtp, draft_num_tokens 1, draft_cache_mode Q8, dynamic_draft false; memory: sysmem_recurrent_cache 1024 MB, sysmem_kv_cache 0","swap_hot_stats":"SWAP_INIT_STATS mounts scripts/split-stats-broad-r869.json (static hot-set counts from 32 prompts, R869) as the exchange-swap starting placement","env_sanitization":"the launcher strips every image-baked EXL3_* selector plus TABBY_OUTPUT_CHUNK_TOKENS and re-passes only the GLM offload/swap/MTP settings (scripts/FLAG-DECISIONS.txt records KEEP/DROP per flag with source refs)"},"serving":{"template":"KEEP_THINKING=1 serves scripts/templates/glm53-keep-thinking.jinja: the model's template with clear_thinking defaulting to false so agent follow-ups reuse the prefix cache (R885c: 7,680 of 9,915 prompt tokens reused); requests may force clear_thinking true; missing/unknown reasoning_effort defaults to high","exactness":"no expert is skipped and every drafted token is verified by the full model; greedy output is NOT bit-stable run-to-run unless EXL3_ORDERED_MOE=1 EXL3_FIXED_STREAM_T=8 EXL3_FREEZE_EXCHANGE=1 are added (R916, costs a few percent c1; off in the served config)","tool_calls":"tool_format glm4_5; AGENT=1 (glm-agent-r2 overlay) enables streamed tool-call arguments, GLM tool-call parsing fixes, reasoning-history recovery and SSE keepalive (TABBY_STREAM_TOOLCALLS=1 TABBY_SSE_KEEPALIVE_S=5 TABBY_GLM_TOOL_FIXES=1 TABBY_GLM_FORCING=0)"},"kv_pool_tokens":262144,"storage_requirements":{"ram_gb":64}},"models":[{"slug":"glm-5-3-flash-exl3-2-05bpw","name":"GLM-5.3-Flash-EXL3-2.05bpw","hf_repo":"turboderp/GLM-5.3-Flash-exl3","quant":"exl3-2-05bpw"}],"hardware":[{"slug":"rtx-5090-x2","vendor":"nvidia","card":"RTX 5090","card_family":"rtx-5090","vram_gb":32,"device_count":2}],"generation":"blackwell","tags":["agent-serving","cpu-expert-swaps","exl3","long-context","moe-expert-offload","mtp","tabbyapi","vision"],"tasks":["chat","code","vision"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured 2026-10-10 17:08 UTC on the served configuration with the text clock (R969, arms B1/B2/B3, three fresh boots, results directory 2026-10-10-r969-glm53-textclock-170158). On each boot the requests ran in the order 1,2,3,4 streams starting right after the launcher's boot warmup; each figure is the median of the three boots. 1 stream = the ten c1-score requests (code, prose, chat, html, edit, 2 runs each); 2-4 streams = a different prompt per stream (code, prose, chat, html in that order), sum of the stream rates over the common decode window, median of 2 rounds per boot, and no token is drafted at 2+. Every decode stream is a chat request forced to 1,024 tokens (html 2,048), temperature 0. c1 score (mean of the five kinds' medians) 62.9, range 62.2-63.2 over the boots; 1 stream 64.8, range 63.7-65.6. Draft acceptance at 1 stream 0.59-0.97 per request over the three boots, highest on edits and code. Server side (R959d arm C4a, the engine's per-request counters over its 10 c1-score requests): 15.61 ms per token, 27.19 ms per MTP step at 1.741 tokens per step; four streams 11.01 ms per generated token. Prefill and the server-side counters are from R959d (2026-10-10 06:19 UTC, arm C4a, directory 2026-10-10-r959d-glm53-chunk-confirm-HoWhmg) on the image BEFORE the text clock with the same settings otherwise - R969 did not measure prefill: 1,973 tok/s @ 32,738 prompt tokens, 2,023 @ 131,051. VRAM after warmup on the served config: 31,304 + 31,786 MiB (847 / 365 MiB free).\", \"tok_s\": 62.89999999999999857891452847979962825775146484375, \"ttft_ms\": 440, \"context_len\": 262144}"}],"perf_points":[{"id":2186,"context_len":4096,"prefill_tok_s":null,"decode_tok_s":62.89999999999999857891452847979962825775146484375,"ttft_ms":480,"concurrency":1,"claimed_by":"submitter","notes":"c1 score (mean of the five kinds' medians, R969 median of boots B1/B2/B3); per-kind code 59.9 / prose 66.7 / chat 68.5 / html 64.8 / edit 53.9. The median of the ten individual c1-score requests was 64.8 tok/s, range 63.7-65.6 over the three boots. TTFT 0.48 s (range 0.47-0.49)."},{"id":2187,"context_len":4096,"prefill_tok_s":null,"decode_tok_s":74.400000000000005684341886080801486968994140625,"ttft_ms":785,"concurrency":2,"claimed_by":"submitter","notes":"2 streams: sum over the common decode window, 37.2 per stream, R969 median of 2 rounds per boot; range over boots 74.3-75.5. TTFT 0.75-0.82 s. No token is drafted at 2+ streams."},{"id":2188,"context_len":4096,"prefill_tok_s":null,"decode_tok_s":92,"ttft_ms":1160,"concurrency":3,"claimed_by":"submitter","notes":"3 streams: sum, 30.7 per stream, R969; range over boots 90.3-92.5. TTFT 1.11-1.21 s."},{"id":2189,"context_len":4096,"prefill_tok_s":null,"decode_tok_s":98,"ttft_ms":1635,"concurrency":4,"claimed_by":"submitter","notes":"4 streams: sum, 24.5 per stream, R969; range over boots 95.6-98.8. TTFT 1.57-1.70 s."},{"id":2190,"context_len":32738,"prefill_tok_s":1973,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Cold prefill, R959d arm C4a on the image BEFORE the text clock (author: R969 did not measure prefill); 32,738 prompt tokens."},{"id":2191,"context_len":131051,"prefill_tok_s":2023,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Cold prefill, R959d arm C4a, image before the text clock; 131,051 prompt tokens."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw.json"},"requirements":null,"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090","pin":"4e3e0f0f5bfbcf1f963936da89b6a80cda2876b8"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3","pin":"2.05bpw branch"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/turboderp-org/exllamav3","pin":"dev 5783a93 (base image adrienbrault/qwen3.8-flash-next-2x-rtx5090)"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/theroyallab/tabbyAPI","pin":"53da7919"},{"kind":"reference","pointer_type":"url","pointer":"https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090/blob/main/bench/results/r927-glm53-dflash-window.md","pin":"R927 bench receipt (arm B-D = served config)"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw.json","alternate":"/r/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw","describedby":"/openapi.json"},"updated_at":"2026-10-29 23:29:37.491895-06","created_at":"2026-10-20 21:49:13.845717-06","speeds":{"decode_tok_s":62.89999999999999857891452847979962825775146484375,"prefill_tok_s":2023,"leaderboard":"/api/v1/leaderboard.json"}}