localairecipes launch configs for local inference

GLM-5.3-Flash EXL3 2.05bpw on 2x RTX 5090 - TabbyAPI hot-expert swaps, 262K ctx, vision

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by adrienbrault (recipe, five patch overlays, measurements)↗, zai-org (GLM-5.3-Flash)↗, turboderp (EXL3 2.05bpw quantisation; ExLlamaV3 + CPU MoE offload)↗, theroyallab (TabbyAPI)↗, adrienbrault/qwen3.8-flash-next-2x-rtx5090 (base image: ExLlamaV3 dev 5783a93 patch stack + TabbyAPI 53da7919)↗ · listed by localai-recipes

revision r3 · commit 4dc491797a · updated 19 h ago · license CC-BY-4.0

chat code vision vision long-context exl3 tabbyapi mtp moe-expert-offload agent-serving cpu-expert-swaps

GLM-5.3-Flash (321B MoE, ~18B active) turboderp EXL3 2.05bpw on two RTX 5090 + 64 GB DDR5: TabbyAPI on ExLlamaV3 with CPU-offloaded routed experts (96/layer GPU0 layers 3-22+MTP, 102/layer GPU1 23-44), runtime hot-expert swaps, per-GPU expert split, the DSA indexer's per-token rows in a fixed ring, live /metrics token counters, the expert-exchange clock handed to the first text MoE layer so swaps also run at 2-4 streams, MTP head drafting 1 tok/step single-stream, 262,144-token window on an 8-bit KV page pool (~1.6 GiB across 4), vision ON with tower linears pinned in host RAM. Exact-output: no expert skipped, every draft verified. Measured 2026-10-10 (R969, three fresh boots): c1 score 62.9 tok/s (code 59.9/prose 66.7/chat 68.5/html 64.8/edit 53.9), 2/3/4-stream sums 74.4/92.0/98.0. Nine in-repo overlays on the qwen3.8-flash-next-2x-rtx5090 base image.

Launch command

'bash scripts/launch-glm53.sh' '# env (scripts/glm-daily.env @ 88e112e0): PACK=A OFFLOAD_MODE=split CACHE_TOKENS=262144 MAX_SEQ=262144 CACHE_MODE=8,8 CHUNK=4096 MAX_BATCH=4 SYSMEM_RC_MB=1024' 'OFFLOAD_N=104 DRAFT=1 DRAFT_N=1 MTP_FAST=1 VISION_OFFLOAD=1 INDEX_RING=1 GPU_SPLIT=31.8,31 CHUNK=4096' 'GLM_IMG=tabbyapi:cheapswap-r3-agent-r2_mtpfast1_overhead-r2_splitdev2_ring3_livemetrics2_draftchunk1_textclock1 PINNED_ARENA=1 AGENT=1 KEEP_THINKING=1' 'EXL3_EXTRA='\''EXL3_MTP_MAX_BATCH=1;EXL3_DRAFT_PINNED_STAGING=1;EXL3_MTP_GPU_DRAFT=1;EXL3_MTP_GREEDY_ACCEPT=1;EXL3_MTP_CACHED_REWIND=1;EXL3_MOE_CPU_SPLIT_BY_DEVICE=96,102;EXL3_MOE_CPU_SWAP_TEXT_CLOCK=1'\''' 'SWAP_MODE=exchange SWAP_POLICY=histogram SWAP_INIT_STATS=$PWD/scripts/split-stats-broad-r869.json SWAP_CADENCE=exact SWAP_INTERVAL=64 SWAP_MAX=64 SWAP_SCOPE=global SWAP_FLOOR=4 SWAP_HYST=2.0' '# REQUIRED: scripts/sampler_overrides/glm53.yml must exist or launch-glm53.sh aborts (exit 3)' '# launch-glm53.sh mounts it read-only at -v "$HERE/sampler_overrides/glm53.yml":/app/sampler_overrides/glm53.yml:ro'

mtp

draft: "the model's own MTP head, depth 1 (one token per step), active only while one request is active (EXL3_MTP_MAX_BATCH=1); at 2-4 streams every stream decodes without a draft; MTP_FAST=1 joins the MTP layer's CPU experts to the trunk CPU worker; four host-overhead flags (EXL3_DRAFT_PINNED_STAGING, EXL3_MTP_GPU_DRAFT, EXL3_MTP_GREEDY_ACCEPT, EXL3_MTP_CACHED_REWIND) measure inside arm-to-arm spread (R914)"
greedy_note: 'EXL3_MTP_GREEDY_ACCEPT applies to greedy requests only'

context

kv: 'one 8-bit KV page pool (cache_mode 8,8) of 262,144 tokens shared by up to 4 concurrent requests, ~2.3 GB (864 bytes/token x 11 attention layers; the other 31 layers hold a fixed-size recurrent state); sysmem_recurrent_cache 1024 MB'
vision: "vision on (vision true); VISION_OFFLOAD=1 keeps the ViT tower's linear weights (0.494 GiB) in pinned host RAM, streamed to the GPU per image"
window: '262,144 tokens per request (cache_size and max_seq_len both 262144)'

offload

swap: 'runtime hot-expert swaps, cheapswap-r3 exchange mode (checkpoint-free GPU<->worker exchange), histogram policy, start from the broad hot set, exact cadence every 64 decode tokens, SWAP_MAX 64, global budget, SWAP_FLOOR 4, hysteresis 2.0'
split: "EXL3_MOE_CPU_SPLIT_BY_DEVICE=100,104: 100 of 288 routed experts per MoE layer on the CPU for GPU0's layers (3-22 + MTP 45), 104 for GPU1's (23-44); 8 CPU worker threads, pinned arena (EXL3_MOE_PINNED_ARENA=0 + exchange swaps)"
expert_worker: 'CPU expert GEMV reads cold experts at 61-62 GB/s of a 63 GB/s ceiling on the 9800X3D (R874); first requests after boot decode slower until placement adapts (R870/R872)'

runtime

ports: 'TabbyAPI on 0.0.0.0:8029 (container glm53, --disable-auth true; single box, no other tenants)'
clocks: 'power limits stock 600/575 W; memory clock offset +4500 MHz, core offset 0 (launcher asserts readback on both GPUs)'
tabby_config: 'written by scripts/glm53_plan.py: model_name glm53-flash-exl3-2.05bpw-turboderp, backend exllamav3, max_seq_len 262144, cache_size 262144, cache_mode 8,8, max_batch_size 4, tensor_parallel false, gpu_split [31.8, 31.0], gpu_split_auto false, cpu_moe_split_experts 104 (overridden per device to 100,104 by EXL3_MOE_CPU_SPLIT_BY_DEVICE), chunk_size 2048, output_chunking true, vision true, vision_offload true, tool_format glm4_5, reasoning true with reasoning_start/end token (Paperclip) and prompt_template glm53-keep-thinking (KEEP_THINKING=1); draft_model: draft_mode mtp, draft_num_tokens 1, draft_cache_mode Q8, dynamic_draft false; memory: sysmem_recurrent_cache 1024 MB, sysmem_kv_cache 0'
swap_hot_stats: 'SWAP_INIT_STATS mounts scripts/split-stats-broad-r869.json (static hot-set counts from 32 prompts, R869) as the exchange-swap starting placement'
env_sanitization: 'the launcher strips every image-baked EXL3_* selector plus TABBY_OUTPUT_CHUNK_TOKENS and re-passes only the GLM offload/swap/MTP settings (scripts/FLAG-DECISIONS.txt records KEEP/DROP per flag with source refs)'

serving

template: "KEEP_THINKING=1 serves scripts/templates/glm53-keep-thinking.jinja: the model's template with clear_thinking defaulting to false so agent follow-ups reuse the prefix cache (R885c: 7,680 of 9,915 prompt tokens reused); requests may force clear_thinking true; missing/unknown reasoning_effort defaults to high"
exactness: 'no expert is skipped and every drafted token is verified by the full model; greedy output is NOT bit-stable run-to-run unless EXL3_ORDERED_MOE=1 EXL3_FIXED_STREAM_T=8 EXL3_FREEZE_EXCHANGE=1 are added (R916, costs a few percent c1; off in the served config)'
tool_calls: 'tool_format glm4_5; AGENT=1 (glm-agent-r2 overlay) enables streamed tool-call arguments, GLM tool-call parsing fixes, reasoning-history recovery and SSE keepalive (TABBY_STREAM_TOOLCALLS=1 TABBY_SSE_KEEPALIVE_S=5 TABBY_GLM_TOOL_FIXES=1 TABBY_GLM_FORCING=0)'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw.json

Engine

Install note: NOT installable from a stock image: the served image stacks five in-repo overlays (docker/cheapswap-r3, glm-agent-r2-on-cheapswap-r3, mtp-fast-r1, mtp-overhead-r2, glm-splitdev-r2) on the base image tabbyapi:r828-prompt-lookup-r3 (ExLlamaV3 dev 5783a93 with 19 patch sets + TabbyAPI 53da7919, from adrienbrault/qwen3.8-flash-next-2x-rtx5090). Build the overlays from the repo's Dockerfiles, then launch with scripts/glm-daily.env (scripts/launch-glm53.sh writes config.yml, clears image-baked EXL3_* selectors, verifies offload registration; needs the box paths /srv/qwen5090/models/<pack> on NVMe and ~76 GB disk for the checkpoint).

Model

Hardware

KV pool

  • 256K tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
4,0961—62.9 tok/s480 msreported
4,0962—74.4 tok/s785 msreported
4,0963—92 tok/s1160 msreported
4,0964—98 tok/s1635 msreported
32,73811973 tok/s——reported
131,05112023 tok/s——reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured 2026-10-10 17:08 UTC on the served configuration with the text clock (R969, arms B1/B2/B3, three fresh boots, results directory 2026-10-10-r969-glm53-textclock-170158). On each boot the requests ran in the order 1,2,3,4 streams starting right after the launcher's boot warmup; each figure is the median of the three boots. 1 stream = the ten c1-score requests (code, prose, chat, html, edit, 2 runs each); 2-4 streams = a different prompt per stream (code, prose, chat, html in that order), sum of the stream rates over the common decode window, median of 2 rounds per boot, and no token is drafted at 2+. Every decode stream is a chat request forced to 1,024 tokens (html 2,048), temperature 0. c1 score (mean of the five kinds' medians) 62.9, range 62.2-63.2 over the boots; 1 stream 64.8, range 63.7-65.6. Draft acceptance at 1 stream 0.59-0.97 per request over the three boots, highest on edits and code. Server side (R959d arm C4a, the engine's per-request counters over its 10 c1-score requests): 15.61 ms per token, 27.19 ms per MTP step at 1.741 tokens per step; four streams 11.01 ms per generated token. Prefill and the server-side counters are from R959d (2026-10-10 06:19 UTC, arm C4a, directory 2026-10-10-r959d-glm53-chunk-confirm-HoWhmg) on the image BEFORE the text clock with the same settings otherwise - R969 did not measure prefill: 1,973 tok/s @ 32,738 prompt tokens, 2,023 @ 131,051. VRAM after warmup on the served config: 31,304 + 31,786 MiB (847 / 365 MiB free).

Downloads

Revision notes

Submitted from the repo at 4e3e0f0f5bfbcf1f963936da89b6a80cda2876b8 (2026-10-09 00:45 +0200). All figures are the author's own measurements (bench/results R927 served arm B-D with R915c/R925 spread arms, R924 prefill arm, R900 memory layout; raw records in bench/results/<date>-<round>/). Hardware note: 60 GB usable of 2x32 GB DDR5-6000 hosts the CPU-offloaded routed experts (26.4 GiB) plus the pinned vision tower. Exact-output claim per author: no expert skipped, every drafted token verified; greedy run-to-run instability documented (R916) with opt-in deterministic flags. First TabbyAPI-served and first RTX-5090 recipe on the site; second ExLlamaV3 entry after Flun's 2x CMP 170HX. UPDATE 2026-10-30 (repo @ 88e112e08a2368e3194232aed1ba5f6375ad3368, 13 commits ahead of the pinned 4e3e0f0f): served configuration moved - image tag gained _ring3_livemetrics2_draftchunk1_textclock1, new INDEX_RING=1, EXL3_MOE_CPU_SPLIT_BY_DEVICE 100,104 -> 96,102, new EXL3_MOE_CPU_SWAP_TEXT_CLOCK=1, new CHUNK=4096 (the recipe previously documented CHUNK=2048 and 100,104 - both stale). Three new overlays are now part of the stack: glm-index-ring-r3 (DSA indexer per-token rows in a fixed ring instead of the KV pool), tabby-livemetrics-r3 (live token counters on /metrics) and textclock-r1 (expert-exchange clock to the first text MoE layer, so swaps also run while 2-4 requests are active). draftchunk-r1 lets chunks above 2,048 tokens boot, which is what makes CHUNK=4096 usable. KV pool restated by the author at ~1.6 GiB (608 bytes per token on each of the 11 attention layers: latent 512, scales 32, pooled indexer keys 64) plus an 8.5 MiB ring per DSA layer, replacing the earlier ~2.3 GiB figure. launch-glm53.sh now HARD-REQUIRES scripts/sampler_overrides/glm53.yml and mounts it read-only at /app/sampler_overrides/glm53.yml - without that file the launch aborts with exit 3. Performance numbers below are refreshed to the author's R969 served arms B1/B2/B3 (2026-10-10, three fresh boots, median of the boots, measured ON the text-clock image): c1 score 62.9, 1 stream 64.8, 2/3/4-stream per stream / sum 37.2/74.4, 30.7/92.0, 24.5/98.0 tok/s; c1 by kind code 59.9 / prose 66.7 / chat 68.5 / html 64.8 / edit 53.9; TTFT 0.48 s c1, 0.75-0.82 / 1.11-1.21 / 1.57-1.70 s at 2/3/4 streams. Prefill remains R959d arm C4a on the image BEFORE the text clock (author states R969 did not measure prefill): 1,973 tok/s @ 32,738 prompt tokens and 2,023 @ 131,051.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)