{"slug":"glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin","title":"GLM-5.3-Flash EXL3 3.05bpw on 1x RTX 3090 \u2014 RAM/NVMe expert tiering (near-exact)","summary":"320B MoE served on ONE 24 GB RTX 3090 with stock exllamav3 1.5.1 plus process-local monkeypatches: the 12,096 routed experts (42 MoE layers x 288, 8 picked per token per layer) tier across VRAM, pinned host RAM and a packed NVMe store. An elastic VRAM cache holds the ~1,300 hottest experts (12.4 GB); evicted experts move into a pinned no-duplicates RAM tier sized from the container cap; a 117 GB O_DIRECT record store covers the rest, with layer-ahead prefetch, a victim write-back ring and an AVX2 CPU lane computing RAM-resident decode picks in place. Modes: fast (all experts in RAM, ~218 GiB free, 28.2 tok/s, KL 0.005), nvme (55 GiB cap, 19.5 tok/s, KL 0.008; 16 GiB cap 14.7 lab / 14.38 sweep), nvme-exact (bit-exact, 8.3 tok/s), and an experimental Intel Arc Pro B70 peer-offload tier (GLM53_B70=1, 26.5 tok/s, KL 0.003 - not lab-run). 131,072-token KV, OpenAI API on port 30000, model id glm-5.3-flash, no output caps. Registry lab six gates pass at 55 GiB (PR #199); 16 GiB reported.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"53520bbf5e06f65db6670e8693efb018f75026c8","content_hash":"65ab07a0787590e7c21331b006d4a13553080d02","created_at":"2026-10-29 21:32:24.549403-06","notes":"Submitted 2026-10-10 from lar-scout ingest card t_9d8954b1. Repo @ 6769b27652a18487758caac45a2b6999dc0a4184 (HEAD verified 2026-10-10 via git ls-remote + shallow clone at that sha, MIT license, 241 stars at ingest). Launch argv transcribed verbatim from README 'Docker' run block @ 6769b27; the full serve argv (-cs 131072 --max-batch-size 8 -chunk_size 8192 -ambs 4 --host 0.0.0.0 --port 30000 --served-name glm-5.3-flash) is the docker/entrypoint.sh resolution - flags token-verified against glm53/serve.py argparse (own args port/host/max-batch-size/served-name; -cs/-chunk_size/-ambs/-rcs from exllamav3 model_init.add_args) and the entrypoint's GLM53_MODE case block (mode gates, B70-requires-nvme die, 24 GB VRAM die-gate, cap-sized RAM tier). Env verbatim from docker/entrypoint.sh @ 6769b27. Perf numbers are the repo's measured tables (README + docs/results.md: campaign G067/N119/N129/N137 + registry lab runs 2026-10-09), logged submitter-T0 - lab-accepted 55 GiB arm, reported 16 GiB arm (below the 15 tok/s gate), B70 arm owner-campaign not lab-run; repo documents ~17% session drift, same-session pairs cited where they matter. HF revision 332ab457 is the entrypoint default pin (3.05bpw branch tip), not re-verified against the HF API. Summary trimmed to the 1000-char schema cap at ingest (hardware floors live in config.runtime.notes). Registry recipes (55/16 GB, all-RAM) and the saitakarcesme 2x3090 variant are follow-ons, not filed."},"engine_source":{"type":"docker_image","git_url":"https://github.com/sybil-solutions/glm-flash-lite","base_engine":"exllamav3","pip_package":null,"base_version":"ExLlamaV3 1.5.1 (stock) + glm53 monkeypatch stack","image_digest":"ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69","install_note":"Image v4.6-nvme is SLSA-attested (built by local-ai-images run 37910016883 from this repo at 0fbc0c5) and installable as-is: needs the NVIDIA container toolkit, ~126 GB free where /models is mounted, and for the nvme modes one pack-store run to build the 117.3 GB expert store on a local NVMe filesystem with O_DIRECT (xfs or ext4). The all-RAM fast mode used the older ghcr.io/0xsero/glm53-flash-offload@sha256:bb633b0b image in the measured runs; the v4 images carry the same fast/exact code paths but those modes were not re-measured in them. ExLlamaV3 1.5.1 built from source in-image for TORCH_CUDA_ARCH_LIST=8.6; sources are NOT modified \u2014 every engine adjustment is a process-local monkeypatch enabled by env.","pinned_commit":"6769b27652a18487758caac45a2b6999dc0a4184"},"credit":[{"name":"sybil-solutions (image, tiering runtime, omarchy plugin)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"turboderp (ExLlamaV3 + EXL3 3.05bpw checkpoint)","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["docker","run","-d","--name","glm53","--gpus","\"device=0\"","--memory","55g","--memory-swap","55g","--shm-size","1g","--ulimit","memlock=-1","-e","GLM53_MODE=nvme","-e","GLM53_EC_MAX_SLOTS=1376","-p","127.0.0.1:30000:30000","-v","/data/glm53:/models:ro","-v","/mnt/nvme/glm53:/nvx:ro","ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69"],"runtime":{"env":"GLM53_MODE=nvme (fast | exact | nvme | nvme-exact) | GLM53_EC_MAX_SLOTS=1376 (VRAM expert-cache cap) | --memory 55g --memory-swap 55g sizes the pinned RAM tier from the cgroup cap (16 GiB cap variant: --memory 16g --memory-swap 16g; all-RAM fast: --ulimit memlock=-1 --shm-size 16g, no cap) | GLM53_B70=1 adds the experimental Intel Arc Pro B70 peer-offload expert tier (needs GLM53_MODE=nvme and the b70tier/b70srv.py companion server up first with the shared ring dir mounted; 120 GiB VRAM pair total) | GLM53_MODEL_DIR=/models GLM53_MODEL_REVISION=332ab457b709b7ba30dd9a448be5de03b80a7ac9 PORT=30000 (defaults, verbatim from docker/entrypoint.sh @ 6769b27).","notes":"docker/entrypoint.sh @ 6769b27 resolves the server argv internally: python3 glm53/serve.py -m /models -cs 131072 --max-batch-size 8 -chunk_size 8192 -ambs 4 --host 0.0.0.0 --port 30000 --served-name glm-5.3-flash (mode nvme adds the shipped S3b2 tuning: GLM53_NV=2 GLM53_NV_CPU=1, 24-slot VRAM victim ring, layer-ahead prefetch, N135 CPU-lane forward, N136 decode switches; the entrypoint die-gates on VRAM < 23,000 MiB and on AVX2+FMA+F16C for the CPU tier, and warns when memlock is capped). One-time store build BEFORE the server: docker run --rm -v /data/glm53:/models:ro -v /mnt/nvme/glm53:/nvx IMG pack-store (117 GB expert store, O_DIRECT sha256 verify of every record + byte-compare of a 300-record sample; GLM53_MODE=nvme* refuses to start without /nvx/glm53_flash_exl3_3.05bpw_experts.bin + .json). Weights: hf download turboderp/GLM-5.3-Flash-exl3 --revision 332ab457 (3.05bpw branch tip, 125.3 GB) or let the entrypoint snapshot_download. Hardware: 1x RTX 3090 PCIe 4.0 x16 (24 GB), EPYC 7443P 24c/48t 8-channel DDR4 (NVMe modes ask for 40+ logical CPUs and AVX2+FMA+F16C), 4x Samsung 9100 PRO 1 TB md RAID0 xfs on PCIe 4.0 x16 (242.6 GB: 125.3 weights + 117.3 store; 16 GiB cap needs the same store, 969-expert tier). Ready in ~80 s at http://127.0.0.1:30000/v1. Alternative: Omarchy Local AI one-click \u2014 omarchy plugin add https://github.com/0xSero/omarchy-local-ai --enable, then pick 'GLM-5.3-Flash, EXL3 3.05 bpw (55 GB RAM, experts on NVMe)'."}},"models":[{"slug":"glm-5-3-flash-exl3-3-05bpw","name":"GLM-5.3-Flash-EXL3-3.05bpw","hf_repo":"turboderp/GLM-5.3-Flash-exl3","quant":"exl3-3-05bpw"}],"hardware":[{"slug":"rtx-3090","vendor":"nvidia","card":"RTX 3090","card_family":"rtx-3090","vram_gb":24,"device_count":1}],"generation":"ampere","tags":["cpu-offload","exl3","expert-tiering","near-exact","nvme-offload","omarchy","single-gpu"],"tasks":["chat","code"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured (lab campaign, docs/results.md @ 6769b27; decode = greedy natural completion, C streams, median of 2 rounds; each row 131,072 KV, 1 GPU). Headline: 55 GiB cap + NVMe decode 19.47 tok/s C1 (v4.6, same-session v4.4 control 17.32).\", \"tok_s\": 19.5, \"points\": [{\"notes\": \"fast mode, all experts in RAM (~218 GiB free), run G067 (previous image bb633b0b); 8k prefill 710 tok/s; C4 33.98 agg; lab run 2026-10-09 on c5a58a40 28.6\", \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 28.14999999999999857891452847979962825775146484375}, {\"notes\": \"nvme mode, 55 GiB cap + NVMe (v4.6-nvme aa74200f, 2026-10-09); C2 19.09 agg, C4 19.75 agg; 32k: 969 prefill / 18.30 decode\", \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 19.469999999999998863131622783839702606201171875, \"prefill_tok_s\": 664}, {\"notes\": \"nvme mode, 16 GiB cap (small-host budget: -rcs 1, 48 readers x 1 MiB); lab run 93,437-token prompt 14.7 tok/s \u2014 reported, below the 15 tok/s speed gate\", \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 14.3800000000000007815970093361102044582366943359375, \"prefill_tok_s\": 650}, {\"notes\": \"nvme-exact mode, bit-exact vs stock exllamav3 (campaign arm S2a); 32k 806 prefill / 7.74 decode\", \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 8.2799999999999993605115378159098327159881591796875, \"prefill_tok_s\": 564}, {\"notes\": \"nvme + 1x Arc Pro B70 peer-offload expert tier (GLM53_B70=1, campaign N137 owner-campaign proof, not lab-run, plugin cannot start companions); 32k prefill 968; decode KL 0.003\", \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 26.5, \"prefill_tok_s\": 658}, {\"notes\": \"registry lab acceptance, 93,437-token prompt (context gate), 55 GB + NVMe recipe accepted \u2014 all six gates passed on digest aa74200f (local-ai-registry PR #199, 2026-10-09)\", \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 20, \"prefill_tok_s\": 1015}], \"context_len\": 131072}"}],"perf_points":[{"id":2045,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":28.14999999999999857891452847979962825775146484375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"fast mode, all experts in RAM (~218 GiB free), run G067 (previous image bb633b0b); 8k prefill 710 tok/s; C4 33.98 agg; lab run 2026-10-09 on c5a58a40 28.6"},{"id":2046,"context_len":131072,"prefill_tok_s":1015,"decode_tok_s":20,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"registry lab acceptance, 93,437-token prompt (context gate), 55 GB + NVMe recipe accepted \u2014 all six gates passed on digest aa74200f (local-ai-registry PR #199, 2026-10-09)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3","pin":"332ab457b709b7ba30dd9a448be5de03b80a7ac9 \u2014 3.05bpw branch tip (docker/entrypoint.sh GLM53_MODEL_REVISION default; 125.3 GB download / ~125 GB; checkpoint by turboderp)"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://ghcr.io/sybil-solutions/glm53-flash-offload","pin":"sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 - v4.6-nvme, SLSA-attested (local-ai-images run 37910016883 from repo 0fbc0c5); all-RAM fast measured on ghcr.io/0xsero image bb633b0b"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/sybil-solutions/glm-flash-lite","pin":"6769b27652a18487758caac45a2b6999dc0a4184 @ HEAD 'Skill pins v4.6-nvme' \u2014 docker/entrypoint.sh is the verbatim argv/env source; glm53/serve.py + glm53/nv2.py + glm53/expert_cache.py + glm53/cpu_tier.py are the tiering runtime; raw tables docs/results.md"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:24.549403-06","created_at":"2026-10-20 21:49:14.118388-06","speeds":{"decode_tok_s":24.10000000000000142108547152020037174224853515625,"prefill_tok_s":1015,"leaderboard":"/api/v1/leaderboard.json"}}