GLM-5.3-Flash EXL3 3.05bpw on 1x RTX 3090 — RAM/NVMe expert tiering (near-exact)

community Community submission — verify on your own hardware.

recipe by sybil-solutions (image, tiering runtime, omarchy plugin)↗, turboderp (ExLlamaV3 + EXL3 3.05bpw checkpoint)↗ · listed by localai-recipes

revision r2 · commit 53520bbf5e · updated 21 h ago · license CC-BY-4.0

chat code cpu-offload single-gpu exl3 expert-tiering near-exact nvme-offload omarchy

320B MoE served on ONE 24 GB RTX 3090 with stock exllamav3 1.5.1 plus process-local monkeypatches: the 12,096 routed experts (42 MoE layers x 288, 8 picked per token per layer) tier across VRAM, pinned host RAM and a packed NVMe store. An elastic VRAM cache holds the ~1,300 hottest experts (12.4 GB); evicted experts move into a pinned no-duplicates RAM tier sized from the container cap; a 117 GB O_DIRECT record store covers the rest, with layer-ahead prefetch, a victim write-back ring and an AVX2 CPU lane computing RAM-resident decode picks in place. Modes: fast (all experts in RAM, ~218 GiB free, 28.2 tok/s, KL 0.005), nvme (55 GiB cap, 19.5 tok/s, KL 0.008; 16 GiB cap 14.7 lab / 14.38 sweep), nvme-exact (bit-exact, 8.3 tok/s), and an experimental Intel Arc Pro B70 peer-offload tier (GLM53_B70=1, 26.5 tok/s, KL 0.003 - not lab-run). 131,072-token KV, OpenAI API on port 30000, model id glm-5.3-flash, no output caps. Registry lab six gates pass at 55 GiB (PR #199); 16 GiB reported.

Launch command

docker run -d --name glm53 --gpus '"device=0"' --memory 55g --memory-swap 55g --shm-size 1g --ulimit memlock=-1 -e GLM53_MODE=nvme -e GLM53_EC_MAX_SLOTS=1376 -p 127.0.0.1:30000:30000 -v /data/glm53:/models:ro -v /mnt/nvme/glm53:/nvx:ro ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69

runtime

env: 'GLM53_MODE=nvme (fast | exact | nvme | nvme-exact) | GLM53_EC_MAX_SLOTS=1376 (VRAM expert-cache cap) | --memory 55g --memory-swap 55g sizes the pinned RAM tier from the cgroup cap (16 GiB cap variant: --memory 16g --memory-swap 16g; all-RAM fast: --ulimit memlock=-1 --shm-size 16g, no cap) | GLM53_B70=1 adds the experimental Intel Arc Pro B70 peer-offload expert tier (needs GLM53_MODE=nvme and the b70tier/b70srv.py companion server up first with the shared ring dir mounted; 120 GiB VRAM pair total) | GLM53_MODEL_DIR=/models GLM53_MODEL_REVISION=332ab457b709b7ba30dd9a448be5de03b80a7ac9 PORT=30000 (defaults, verbatim from docker/entrypoint.sh @ 6769b27).'
notes: "docker/entrypoint.sh @ 6769b27 resolves the server argv internally: python3 glm53/serve.py -m /models -cs 131072 --max-batch-size 8 -chunk_size 8192 -ambs 4 --host 0.0.0.0 --port 30000 --served-name glm-5.3-flash (mode nvme adds the shipped S3b2 tuning: GLM53_NV=2 GLM53_NV_CPU=1, 24-slot VRAM victim ring, layer-ahead prefetch, N135 CPU-lane forward, N136 decode switches; the entrypoint die-gates on VRAM < 23,000 MiB and on AVX2+FMA+F16C for the CPU tier, and warns when memlock is capped). One-time store build BEFORE the server: docker run --rm -v /data/glm53:/models:ro -v /mnt/nvme/glm53:/nvx IMG pack-store (117 GB expert store, O_DIRECT sha256 verify of every record + byte-compare of a 300-record sample; GLM53_MODE=nvme* refuses to start without /nvx/glm53_flash_exl3_3.05bpw_experts.bin + .json). Weights: hf download turboderp/GLM-5.3-Flash-exl3 --revision 332ab457 (3.05bpw branch tip, 125.3 GB) or let the entrypoint snapshot_download. Hardware: 1x RTX 3090 PCIe 4.0 x16 (24 GB), EPYC 7443P 24c/48t 8-channel DDR4 (NVMe modes ask for 40+ logical CPUs and AVX2+FMA+F16C), 4x Samsung 9100 PRO 1 TB md RAID0 xfs on PCIe 4.0 x16 (242.6 GB: 125.3 weights + 117.3 store; 16 GiB cap needs the same store, 969-expert tier). Ready in ~80 s at http://127.0.0.1:30000/v1. Alternative: Omarchy Local AI one-click — omarchy plugin add https://github.com/0xSero/omarchy-local-ai --enable, then pick 'GLM-5.3-Flash, EXL3 3.05 bpw (55 GB RAM, experts on NVMe)'."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin.json

Engine

exllamav3 · source↗ · 6769b276↗ · ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69↗

Install note: Image v4.6-nvme is SLSA-attested (built by local-ai-images run 37910016883 from this repo at 0fbc0c5) and installable as-is: needs the NVIDIA container toolkit, ~126 GB free where /models is mounted, and for the nvme modes one pack-store run to build the 117.3 GB expert store on a local NVMe filesystem with O_DIRECT (xfs or ext4). The all-RAM fast mode used the older ghcr.io/0xsero/glm53-flash-offload@sha256:bb633b0b image in the measured runs; the v4 images carry the same fast/exact code paths but those modes were not re-measured in them. ExLlamaV3 1.5.1 built from source in-image for TORCH_CUDA_ARCH_LIST=8.6; sources are NOT modified — every engine adjustment is a process-local monkeypatch enabled by env.

Model

GLM-5.3-Flash-EXL3-3.05bpw (exl3-3-05bpw) · turboderp/GLM-5.3-Flash-exl3↗
AA intelligence 42 (Artificial Analysis · measured)

Hardware

NVIDIA RTX 3090 24GB ×1 (Ampere)

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
8,192	1	—	28.2 tok/s	—	reported
131,072	1	1015 tok/s	20 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Author-measured (lab campaign, docs/results.md @ 6769b27; decode = greedy natural completion, C streams, median of 2 rounds; each row 131,072 KV, 1 GPU). Headline: 55 GiB cap + NVMe decode 19.47 tok/s C1 (v4.6, same-session v4.4 control 17.32).

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/turboderp/GLM-5.3-Flash-exl3
332ab457b709b7ba30dd9a448be5de03b80a7ac9 — 3.05bpw branch tip (docker/entrypoint.sh GLM53_MODEL_REVISION default; 125.3 GB download / ~125 GB; checkpoint by turboderp)
Engine image https://ghcr.io/sybil-solutions/glm53-flash-offload
sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 - v4.6-nvme, SLSA-attested (local-ai-images run 37910016883 from repo 0fbc0c5); all-RAM fast measured on ghcr.io/0xsero image bb633b0b
Reference https://github.com/sybil-solutions/glm-flash-lite
6769b27652a18487758caac45a2b6999dc0a4184 @ HEAD 'Skill pins v4.6-nvme' — docker/entrypoint.sh is the verbatim argv/env source; glm53/serve.py + glm53/nv2.py + glm53/expert_cache.py + glm53/cpu_tier.py are the tiering runtime; raw tables docs/results.md

Revision notes

Submitted 2026-10-10 from lar-scout ingest card t_9d8954b1. Repo @ 6769b27652a18487758caac45a2b6999dc0a4184 (HEAD verified 2026-10-10 via git ls-remote + shallow clone at that sha, MIT license, 241 stars at ingest). Launch argv transcribed verbatim from README 'Docker' run block @ 6769b27; the full serve argv (-cs 131072 --max-batch-size 8 -chunk_size 8192 -ambs 4 --host 0.0.0.0 --port 30000 --served-name glm-5.3-flash) is the docker/entrypoint.sh resolution - flags token-verified against glm53/serve.py argparse (own args port/host/max-batch-size/served-name; -cs/-chunk_size/-ambs/-rcs from exllamav3 model_init.add_args) and the entrypoint's GLM53_MODE case block (mode gates, B70-requires-nvme die, 24 GB VRAM die-gate, cap-sized RAM tier). Env verbatim from docker/entrypoint.sh @ 6769b27. Perf numbers are the repo's measured tables (README + docs/results.md: campaign G067/N119/N129/N137 + registry lab runs 2026-10-09), logged submitter-T0 - lab-accepted 55 GiB arm, reported 16 GiB arm (below the 15 tok/s gate), B70 arm owner-campaign not lab-run; repo documents ~17% session drift, same-session pairs cited where they matter. HF revision 332ab457 is the entrypoint default pin (3.05bpw branch tip), not re-verified against the HF API. Summary trimmed to the 1000-char schema cap at ingest (hardware floors live in config.runtime.notes). Registry recipes (55/16 GB, all-RAM) and the saitakarcesme 2x3090 variant are follow-ons, not filed.

Revision history

53520bbf5e recipe: update glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin (r2: /config/runtime) 15 h ago diff
ce6a4e467b recipe: create glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin 21 d ago

Fork

Export

recipe.yaml (raw)