GLM-5.3-Flash EXL3 3.05bpw on 2x RTX 3090 — vLLM + ExLlamaV3 CPU expert offload, 256K (DominikBucko)

community Community submission — verify on your own hardware.

recipe by DominikBucko↗, turboderp-org (EXL3 quant + ExLlamaV3)↗, spele1100 (GLM-5.3 vLLM overlay)↗ · listed by localai-recipes

revision r2 · commit c93c82b83e · updated 10 d ago · license CC-BY-4.0

chat code tensor-parallel long-context expert-offload agents

GLM-5.3-Flash (turboderp EXL3 3.05bpw) tensor-parallel on 2x RTX 3090 24 GB + 128 GB system RAM: vendor vLLM image with the spele1100/vllm-glm53 GLM-5.3 overlay, ExLlamaV3 @ 499890c with EXL3 MoE decode kernels, and CPU expert-offload pools (UVA exact host registers, per-layer LRU expert cache with the hottest 20% pinned, glm53_host.py pool planner, checkpoint lock + k-pool index installer). Bounded 4,096-token thinking budget + preserved reasoning for agent loops, 262,144-token context, prefix caching, OpenAI-compatible API. Docker image + systemd example, reproducible from repro.lock.json pins; measured up to 1,155 tok/s prefill and 47-50 tok/s decode with needles-at-32K/131K/260K + coherence gates passing.

Launch command

vllm serve /model --served-model-name GLM-5.3-Flash --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 --distributed-executor-backend mp --model-impl vllm --quantization exl3 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 1 --max-num-batched-tokens 8192 --enable-chunked-prefill --gpu-memory-utilization 0.94 --kv-cache-memory 3328599040 --kv-cache-dtype auto --offload-backend uva --cpu-offload-gb 64 --cpu-offload-params experts --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}' --enable-prefix-caching --chat-template /opt/glm53/configs/chat-template-bounded.jinja --reasoning-config '{"reasoning_start_str":"<think>","reasoning_end_str":"\n\nI have thought about this enough; I will now act on my best plan.\n</think>"}' --no-enable-flashinfer-autotune --language-model-only --skip-mm-profiling --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45

runtime

env: 'SERVED_MODEL_NAME | MAX_MODEL_LEN | MAX_NUM_SEQS | MAX_NUM_BATCHED_TOKENS | KV_CACHE_MEMORY_BYTES | GPU_MEMORY_UTILIZATION | TENSOR_PARALLEL_SIZE | CUDA_GRAPH_MODE | PREFIX_CACHING | CPU_OFFLOAD_GB | EXL3_DYNAMIC_LRU | EXL3_DYNAMIC_LRU_MAX_LAYER_ID | EXL3_DYNAMIC_LRU_MAX_TOKENS | EXL3_LRU_CAPACITY_PLAN | EXL3_VMM_HOT_RANKINGS | EXL3_VMM_HOT_EXPERTS | EXL3_PINNED_FRAC | VLLM_UVA_OFFLOAD_EXACT_HOST_REGISTER | EXL3_LRU_GATHER_IMPL | EXL3_LRU_GATHER_BLOCK | EXL3_LRU_GATHER_WARPS | EXL3_LRU_FUSED_GATHER | EXL3_LRU_OVERLAP | EXL3_LRU_COPY_STREAM_PRIORITY | EXL3_LRU_HYBRID_PREFILL | EXL3_LRU_SLRU_PROTECTED_PCT | EXL3_LRU_LOG_EVERY | EXL3_PREFETCH_K | EXL3_HYBRID_CPU | EXL3_HYBRID_LAYERS | EXL3_HYBRID_GPU_FRAC | EXL3_HYBRID_ADMIT_CPU | EXL3_HYBRID_FUSED_SUBMIT | EXL3_HYBRID_OVERLAP_GATHER | OMP_NUM_THREADS | EXL3_PREFILL_DMA_STAGE_EXPERTS | EXL3_PREFILL_DMA_MIN_TOKENS | EXL3_PREFILL_DMA_FAT_MODE | EXL3_FUSED_MOE | EXL3_MOE_GEMV | EXL3_BF16_IO | EXL3_FUSED_DECODE_ROUTE_PREP | EXL3_MERGED_QKV | EXL3_MOE_HIT_GROUPS | EXL3_MOE_MISS_GROUPS | EXL3_MOE_MAX_SMS_PER_EXPERT | EXL3_MOE_DECODE_GROUPS | EXL3_MOE_DECODE_GROUPS_MAX_TOKENS | EXL3_MOE_PREFILL_HOT_GROUPS | EXL3_MOE_PREFILL_COLD_GROUPS | EXL3_MOE_PREFILL_HYBRID_GROUPS | EXL3_MOE_ROW_TILE | EXL3_TEMP_ROWS_FUSED | EXL3_FAT_KERNEL | EXL3_FAT_BATCHED | EXL3_FAT_SORTED | EXL3_MULTILINEAR_GATE_UP | EXL3_MULTILINEAR_MAX_ROWS | EXL3_DETERMINISTIC_MOE | VLLM_DISABLE_SHARED_EXPERTS_STREAM | GLM53_FAST_MHC | GLM53_FAST_ROUTER | GLM53_FAST_TOPK | GLM53_SKIP_DUP_ROUTER | GLM53_SPARSE_MLA_KV_SPLITS | VLLM_GLM53_SM86_TILELANG_MHC | VLLM_GLM53_SM86_MLA_QUERY_CHUNK | VLLM_GLM53_INDEXER_HEADS | VLLM_GLM53_INDEXER_WS_BY_SEQS | VLLM_GLM53_EMBED_UVA | VLLM_GLM53_DETERMINISTIC_TOPK | VLLM_KV_CACHE_LAYOUT | VLLM_DEFAULT_THINKING_TOKEN_BUDGET | GLM53_PRESERVE_REASONING_TURNS | CHAT_TEMPLATE | REASONING_CONFIG (verbatim keys from configs/2x3090-256k-agent.env @ 597920a; values in that file)'
build: 'git clone https://github.com/DominikBucko/glm53-flash-2x3090 && cd glm53-flash-2x3090 && scripts/download_model.sh /models/glm53-flash-exl3-3.05bpw (verifies every file against runtime/model/checkpoint.lock.json and links the k-pool aux shard via scripts/install_model.py — the upstream index omits 22 sparse-attention indexer tensors), cp .env.example .env (set MODEL_DIR), make serve (build-image + preflight + scripts/docker_serve.sh)'
image: 'ghcr.io/dominikbucko/glm53-flash-2x3090 (release; validation build sha256:5d6bde38… from docker/Dockerfile @ 597920a) — baked default profile configs/2x3090-256k-agent.env'
container: 'docker run --gpus all --ipc host --cap-add SYS_PTRACE --ulimit memlock=-1 --ulimit stack=67108864 -p 127.0.0.1:8000:8000 -v <MODEL_DIR>:/model:ro; entrypoint /usr/local/bin/glm53-serve loads the baked profile, plans CPU expert pools (python3 /opt/glm53/runtime/glm53_host.py --env), tasksets vLLM to HYBRID_MAIN_CPUS, then execs the argv above; systemd example systemd/glm53-flash.service.example; ready in ~5.5-6 min'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp.json

Engine

vllm · 8e685d19↗ · vllm/vllm-openai:qwen38-flash-next@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8↗

Install note: Not stock vLLM: vendor vllm/vllm-openai:qwen38-flash-next image + spele1100/vllm-glm53 GLM-5.3 overlay @ f2935ce + repo vllm-overlay (build via docker/Dockerfile / scripts/build_overlays.py); turboderp-org/exllamav3 @ 499890c overlay (sm_86) with EXL3 MoE decode kernels; glm_hybrid + glm_fast extensions (CPU expert pools, fused mHC/router/topk). Build: make build-image (docker/Dockerfile); serve: scripts/docker_serve.sh / make serve.

Model

GLM-5.3-Flash-EXL3-3.05bpw (exl3-3-05bpw) · turboderp/GLM-5.3-Flash-exl3↗
AA intelligence 42 (Artificial Analysis · measured)

Hardware

NVIDIA RTX 3090 24GB ×2 (Ampere)

Context

max context: 256K tokens

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
8,192	1	1125 tok/s	50.4 tok/s	7280 ms	reported
131,072	1	1155 tok/s	49.4 tok/s	113530 ms	reported
260,096	1	1106 tok/s	48.2 tok/s	235260 ms	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Repo benchmark report benchmarks/2026-10-12 (scripts/benchmark_serving.py, one request, best of 2, fresh cache_salt, from the published release image; submitter T0). Prefill-256k profile: 8,192+2,048 TTFT 7.28 s / 1,125 input tok/s / 50.4 decode; 131,072+2,048 TTFT 113.5 s / 1,155 input tok/s (best) / 49.4 decode; 260,096+2,048 TTFT 235.3 s / 1,106 input tok/s / 48.2 decode. Agent-256k default profile: 8K 9.6 s TTFT / 859 input / 50.4 decode; 131K 236.2 s / 555 input / 49.3 decode; 260K 506.7 s / 513 input / 48.1 decode. Needles at 32K/131K/260K (exact JSON) + coherence pass in both profiles. Agent profile: 72.9% of prompt tokens cache-hit in a 15-task agent bench, median tool-call turn 7.6 s, real agent turns 37.7 tok/s avg decode.

notes

Repo benchmark report benchmarks/2026-10-12 (scripts/benchmark_serving.py, one request, best of 2, fresh cache_salt, from the published release image; submitter T0). Prefill-256k profile: 8,192+2,048 TTFT 7.28 s / 1,125 input tok/s / 50.4 decode; 131,072+2,048 TTFT 113.5 s / 1,155 input tok/s (best) / 49.4 decode; 260,096+2,048 TTFT 235.3 s / 1,106 input tok/s / 48.2 decode. Agent-256k default profile: 8K 9.6 s TTFT / 859 input / 50.4 decode; 131K 236.2 s / 555 input / 49.3 decode; 260K 506.7 s / 513 input / 48.1 decode. Needles at 32K/131K/260K (exact JSON) + coherence pass in both profiles. Agent profile: 72.9% of prompt tokens cache-hit in a 15-task agent bench, median tool-call turn 7.6 s, real agent turns 37.7 tok/s avg decode.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/turboderp/GLM-5.3-Flash-exl3
332ab457b709b7ba30dd9a448be5de03b80a7ac9 — branch 3.05bpw checkpoint (weights unchanged; k-pool index fix via scripts/install_model.py). Card snapshot albucino/GLM-5.3-Flash-EXL3-3.05bpw @ bdae1c559fc6194093a9dbb1c98df5aabd70d52c
Engine image https://ghcr.io/dominikbucko/glm53-flash-2x3090
ghcr.io/dominikbucko/glm53-flash-2x3090 release image (validation build sha256:5d6bde38…); base runtime vllm/vllm-openai:qwen38-flash-next@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8; overlay base spele1100/vllm-glm53 @ f2935ce
Reference https://github.com/DominikBucko/glm53-flash-2x3090
597920a0c2d3bd0c539d999465710b4b16a888f0 — repo HEAD @ ingest (benchmarks/2026-10-12, repro.lock.json, configs/); exllamav3 pinned 499890c75d20d8e7c9d061f37189ae611a5c9f0b

Revision notes

Ingested by localai-recipes from the public launch-repro repo @ 597920a (Apache-2.0, created+pushed 2026-10-12): launch-repro docker runtime with checksummed overlays, repro.lock.json, in-repo benchmark report (benchmarks/2026-10-12, release_image_validation: kv_cache_tokens 264160, ~21.4 GiB/GPU after load). Performance rows transcribed verbatim from benchmarks/2026-10-12/summary.json measured_runs (best of 2 per shape; both runs' decode ranges kept in notes). Same turboderp 3.05bpw weights pin shared by the site's Flun/SSHdotCodes/sybil GLM EXL3 recipes. Distinct from held saitakarcesme/glm5.3-flash-2x3090 (glm53-flash-offload ExLlamaV3-native stack, ~30 tok/s) — that hold was repo-specific and remains held.

Revision history

0bac45e09a recipe: update glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp (r2: /max_context_len, /storage_requirements) 10 d ago diff
1c29d27283 recipe: create glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp 18 d ago

Fork

Export

recipe.yaml (raw)