{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x RTX 3090 \u2014 vLLM + ExLlamaV3 CPU expert offload, 256K (DominikBucko)","summary":"GLM-5.3-Flash (turboderp EXL3 3.05bpw) tensor-parallel on 2x RTX 3090 24 GB + 128 GB system RAM: vendor vLLM image with the spele1100/vllm-glm53 GLM-5.3 overlay, ExLlamaV3 @ 499890c with EXL3 MoE decode kernels, and CPU expert-offload pools (UVA exact host registers, per-layer LRU expert cache with the hottest 20% pinned, glm53_host.py pool planner, checkpoint lock + k-pool index installer). Bounded 4,096-token thinking budget + preserved reasoning for agent loops, 262,144-token context, prefix caching, OpenAI-compatible API. Docker image + systemd example, reproducible from repro.lock.json pins; measured up to 1,155 tok/s prefill and 47-50 tok/s decode with needles-at-32K/131K/260K + coherence gates passing.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"c93c82b83e6832d41dd10a7037c607a3f4b53d1b","content_hash":"c807eb1b13116b97fedcab43cef2aad9dca165e7","created_at":"2026-10-20 21:52:36.396387-06","notes":"Ingested by localai-recipes from the public launch-repro repo @ 597920a (Apache-2.0, created+pushed 2026-10-12): launch-repro docker runtime with checksummed overlays, repro.lock.json, in-repo benchmark report (benchmarks/2026-10-12, release_image_validation: kv_cache_tokens 264160, ~21.4 GiB/GPU after load). Performance rows transcribed verbatim from benchmarks/2026-10-12/summary.json measured_runs (best of 2 per shape; both runs' decode ranges kept in notes). Same turboderp 3.05bpw weights pin shared by the site's Flun/SSHdotCodes/sybil GLM EXL3 recipes. Distinct from held saitakarcesme/glm5.3-flash-2x3090 (glm53-flash-offload ExLlamaV3-native stack, ~30 tok/s) \u2014 that hold was repo-specific and remains held."},"engine_source":{"type":"docker_image","git_url":null,"base_engine":"vllm","pip_package":null,"base_version":"0.1.dev20073+g8e685d198 (qwen38-flash-next runtime)","image_digest":"vllm/vllm-openai:qwen38-flash-next@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8","install_note":"Not stock vLLM: vendor vllm/vllm-openai:qwen38-flash-next image + spele1100/vllm-glm53 GLM-5.3 overlay @ f2935ce + repo vllm-overlay (build via docker/Dockerfile / scripts/build_overlays.py); turboderp-org/exllamav3 @ 499890c overlay (sm_86) with EXL3 MoE decode kernels; glm_hybrid + glm_fast extensions (CPU expert pools, fused mHC/router/topk). Build: make build-image (docker/Dockerfile); serve: scripts/docker_serve.sh / make serve.","pinned_commit":"8e685d198"},"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/glm53-flash-2x3090"},{"name":"turboderp-org (EXL3 quant + ExLlamaV3)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"spele1100 (GLM-5.3 vLLM overlay)","url":"https://github.com/spele1100/vllm-glm53"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/model","--served-model-name","GLM-5.3-Flash","--host","0.0.0.0","--port","8000","--tensor-parallel-size","2","--distributed-executor-backend","mp","--model-impl","vllm","--quantization","exl3","--dtype","bfloat16","--max-model-len","262144","--max-num-seqs","1","--max-num-batched-tokens","8192","--enable-chunked-prefill","--gpu-memory-utilization","0.94","--kv-cache-memory","3328599040","--kv-cache-dtype","auto","--offload-backend","uva","--cpu-offload-gb","64","--cpu-offload-params","experts","--compilation-config","{\"mode\":0,\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}","--enable-prefix-caching","--chat-template","/opt/glm53/configs/chat-template-bounded.jinja","--reasoning-config","{\"reasoning_start_str\":\"<think>\",\"reasoning_end_str\":\"\\n\\nI have thought about this enough; I will now act on my best plan.\\n</think>\"}","--no-enable-flashinfer-autotune","--language-model-only","--skip-mm-profiling","--tool-call-parser","glm47","--enable-auto-tool-choice","--reasoning-parser","glm45"],"runtime":{"env":"SERVED_MODEL_NAME | MAX_MODEL_LEN | MAX_NUM_SEQS | MAX_NUM_BATCHED_TOKENS | KV_CACHE_MEMORY_BYTES | GPU_MEMORY_UTILIZATION | TENSOR_PARALLEL_SIZE | CUDA_GRAPH_MODE | PREFIX_CACHING | CPU_OFFLOAD_GB | EXL3_DYNAMIC_LRU | EXL3_DYNAMIC_LRU_MAX_LAYER_ID | EXL3_DYNAMIC_LRU_MAX_TOKENS | EXL3_LRU_CAPACITY_PLAN | EXL3_VMM_HOT_RANKINGS | EXL3_VMM_HOT_EXPERTS | EXL3_PINNED_FRAC | VLLM_UVA_OFFLOAD_EXACT_HOST_REGISTER | EXL3_LRU_GATHER_IMPL | EXL3_LRU_GATHER_BLOCK | EXL3_LRU_GATHER_WARPS | EXL3_LRU_FUSED_GATHER | EXL3_LRU_OVERLAP | EXL3_LRU_COPY_STREAM_PRIORITY | EXL3_LRU_HYBRID_PREFILL | EXL3_LRU_SLRU_PROTECTED_PCT | EXL3_LRU_LOG_EVERY | EXL3_PREFETCH_K | EXL3_HYBRID_CPU | EXL3_HYBRID_LAYERS | EXL3_HYBRID_GPU_FRAC | EXL3_HYBRID_ADMIT_CPU | EXL3_HYBRID_FUSED_SUBMIT | EXL3_HYBRID_OVERLAP_GATHER | OMP_NUM_THREADS | EXL3_PREFILL_DMA_STAGE_EXPERTS | EXL3_PREFILL_DMA_MIN_TOKENS | EXL3_PREFILL_DMA_FAT_MODE | EXL3_FUSED_MOE | EXL3_MOE_GEMV | EXL3_BF16_IO | EXL3_FUSED_DECODE_ROUTE_PREP | EXL3_MERGED_QKV | EXL3_MOE_HIT_GROUPS | EXL3_MOE_MISS_GROUPS | EXL3_MOE_MAX_SMS_PER_EXPERT | EXL3_MOE_DECODE_GROUPS | EXL3_MOE_DECODE_GROUPS_MAX_TOKENS | EXL3_MOE_PREFILL_HOT_GROUPS | EXL3_MOE_PREFILL_COLD_GROUPS | EXL3_MOE_PREFILL_HYBRID_GROUPS | EXL3_MOE_ROW_TILE | EXL3_TEMP_ROWS_FUSED | EXL3_FAT_KERNEL | EXL3_FAT_BATCHED | EXL3_FAT_SORTED | EXL3_MULTILINEAR_GATE_UP | EXL3_MULTILINEAR_MAX_ROWS | EXL3_DETERMINISTIC_MOE | VLLM_DISABLE_SHARED_EXPERTS_STREAM | GLM53_FAST_MHC | GLM53_FAST_ROUTER | GLM53_FAST_TOPK | GLM53_SKIP_DUP_ROUTER | GLM53_SPARSE_MLA_KV_SPLITS | VLLM_GLM53_SM86_TILELANG_MHC | VLLM_GLM53_SM86_MLA_QUERY_CHUNK | VLLM_GLM53_INDEXER_HEADS | VLLM_GLM53_INDEXER_WS_BY_SEQS | VLLM_GLM53_EMBED_UVA | VLLM_GLM53_DETERMINISTIC_TOPK | VLLM_KV_CACHE_LAYOUT | VLLM_DEFAULT_THINKING_TOKEN_BUDGET | GLM53_PRESERVE_REASONING_TURNS | CHAT_TEMPLATE | REASONING_CONFIG (verbatim keys from configs/2x3090-256k-agent.env @ 597920a; values in that file)","build":"git clone https://github.com/DominikBucko/glm53-flash-2x3090 && cd glm53-flash-2x3090 && scripts/download_model.sh /models/glm53-flash-exl3-3.05bpw (verifies every file against runtime/model/checkpoint.lock.json and links the k-pool aux shard via scripts/install_model.py \u2014 the upstream index omits 22 sparse-attention indexer tensors), cp .env.example .env (set MODEL_DIR), make serve (build-image + preflight + scripts/docker_serve.sh)","image":"ghcr.io/dominikbucko/glm53-flash-2x3090 (release; validation build sha256:5d6bde38\u2026 from docker/Dockerfile @ 597920a) \u2014 baked default profile configs/2x3090-256k-agent.env","container":"docker run --gpus all --ipc host --cap-add SYS_PTRACE --ulimit memlock=-1 --ulimit stack=67108864 -p 127.0.0.1:8000:8000 -v <MODEL_DIR>:/model:ro; entrypoint /usr/local/bin/glm53-serve loads the baked profile, plans CPU expert pools (python3 /opt/glm53/runtime/glm53_host.py --env), tasksets vLLM to HYBRID_MAIN_CPUS, then execs the argv above; systemd example systemd/glm53-flash.service.example; ready in ~5.5-6 min"},"max_context_len":262144,"storage_requirements":{"ram_gb":128}},"models":[{"slug":"glm-5-3-flash-exl3-3-05bpw","name":"GLM-5.3-Flash-EXL3-3.05bpw","hf_repo":"turboderp/GLM-5.3-Flash-exl3","quant":"exl3-3-05bpw"}],"hardware":[{"slug":"rtx-3090-x2","vendor":"nvidia","card":"RTX 3090","card_family":"rtx-3090","vram_gb":24,"device_count":2}],"generation":"ampere","tags":["agents","expert-offload","long-context","tensor-parallel"],"tasks":["chat","code"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Repo benchmark report benchmarks/2026-10-12 (scripts/benchmark_serving.py, one request, best of 2, fresh cache_salt, from the published release image; submitter T0). Prefill-256k profile: 8,192+2,048 TTFT 7.28 s / 1,125 input tok/s / 50.4 decode; 131,072+2,048 TTFT 113.5 s / 1,155 input tok/s (best) / 49.4 decode; 260,096+2,048 TTFT 235.3 s / 1,106 input tok/s / 48.2 decode. Agent-256k default profile: 8K 9.6 s TTFT / 859 input / 50.4 decode; 131K 236.2 s / 555 input / 49.3 decode; 260K 506.7 s / 513 input / 48.1 decode. Needles at 32K/131K/260K (exact JSON) + coherence pass in both profiles. Agent profile: 72.9% of prompt tokens cache-hit in a 15-task agent bench, median tool-call turn 7.6 s, real agent turns 37.7 tok/s avg decode.\", \"tok_s\": 50.4200000000000017053025658242404460906982421875, \"points\": [{\"notes\": \"prefill-256k profile, 8,192+2,048, best of 2: TTFT 7.28 s, 1,124.6 input tok/s, 50.42 decode tok/s\", \"ttft_ms\": 7280, \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 50.39999999999999857891452847979962825775146484375, \"prefill_tok_s\": 1125}, {\"notes\": \"prefill-256k profile, 131,072+2,048, best of 2: TTFT 113.53 s, 1,154.5 input tok/s (headline 1,155), 48.9-49.4 decode tok/s\", \"ttft_ms\": 113530, \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 49.39999999999999857891452847979962825775146484375, \"prefill_tok_s\": 1155}, {\"notes\": \"prefill-256k profile, 260,096+2,048, best of 2: TTFT 235.26 s, 1,105.6 input tok/s, 47.9-48.2 decode tok/s\", \"ttft_ms\": 235260, \"concurrency\": 1, \"context_len\": 260096, \"decode_tok_s\": 48.2000000000000028421709430404007434844970703125, \"prefill_tok_s\": 1106}, {\"notes\": \"agent-256k default profile, 8,192+2,048, best of 2: TTFT 9.54 s, 858.8 input tok/s, 50.36 decode tok/s (prefix caching on)\", \"ttft_ms\": 9540, \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 50.39999999999999857891452847979962825775146484375, \"prefill_tok_s\": 859}, {\"notes\": \"agent-256k default profile, 131,072+2,048, best of 2: TTFT 236.24 s, 554.8 input tok/s, 49.3 decode tok/s\", \"ttft_ms\": 236240, \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 49.2999999999999971578290569595992565155029296875, \"prefill_tok_s\": 555}, {\"notes\": \"agent-256k default profile, 260,096+2,048, best of 2: TTFT 506.7 s, 513.3 input tok/s, 48.1 decode tok/s\", \"ttft_ms\": 506700, \"concurrency\": 1, \"context_len\": 260096, \"decode_tok_s\": 48.10000000000000142108547152020037174224853515625, \"prefill_tok_s\": 513}], \"ttft_ms\": 7280, \"context_len\": 262144}"}],"perf_points":[{"id":1615,"context_len":8192,"prefill_tok_s":1125,"decode_tok_s":50.39999999999999857891452847979962825775146484375,"ttft_ms":7280,"concurrency":1,"claimed_by":"submitter","notes":"prefill-256k profile, 8,192+2,048, best of 2: TTFT 7.28 s, 1,124.6 input tok/s, 50.42 decode tok/s"},{"id":1616,"context_len":131072,"prefill_tok_s":1155,"decode_tok_s":49.39999999999999857891452847979962825775146484375,"ttft_ms":113530,"concurrency":1,"claimed_by":"submitter","notes":"prefill-256k profile, 131,072+2,048, best of 2: TTFT 113.53 s, 1,154.5 input tok/s (headline 1,155), 48.9-49.4 decode tok/s"},{"id":1617,"context_len":260096,"prefill_tok_s":1106,"decode_tok_s":48.2000000000000028421709430404007434844970703125,"ttft_ms":235260,"concurrency":1,"claimed_by":"submitter","notes":"prefill-256k profile, 260,096+2,048, best of 2: TTFT 235.26 s, 1,105.6 input tok/s, 47.9-48.2 decode tok/s"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3","pin":"332ab457b709b7ba30dd9a448be5de03b80a7ac9 \u2014 branch 3.05bpw checkpoint (weights unchanged; k-pool index fix via scripts/install_model.py). Card snapshot albucino/GLM-5.3-Flash-EXL3-3.05bpw @ bdae1c559fc6194093a9dbb1c98df5aabd70d52c"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://ghcr.io/dominikbucko/glm53-flash-2x3090","pin":"ghcr.io/dominikbucko/glm53-flash-2x3090 release image (validation build sha256:5d6bde38\u2026); base runtime vllm/vllm-openai:qwen38-flash-next@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8; overlay base spele1100/vllm-glm53 @ f2935ce"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/DominikBucko/glm53-flash-2x3090","pin":"597920a0c2d3bd0c539d999465710b4b16a888f0 \u2014 repo HEAD @ ingest (benchmarks/2026-10-12, repro.lock.json, configs/); exllamav3 pinned 499890c75d20d8e7c9d061f37189ae611a5c9f0b"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp","describedby":"/openapi.json"},"updated_at":"2026-10-20 21:52:36.396387-06","created_at":"2026-10-20 21:49:14.403496-06","speeds":{"decode_tok_s":49.39999999999999857891452847979962825775146484375,"prefill_tok_s":1130.5,"leaderboard":"/api/v1/leaderboard.json"}}