{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-layer-split-mtp3-393k","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX layer split MTP3 393K prompt reuse (mannydelrio)","summary":"Everyday GLM-5.3-Flash setup on 2x CMP 170HX unlocked to 64 GB (PCIe Gen2 x4, 180 W each, 32 GB RAM, 850 W PSU): alexvnesta's ExLlamaV3 server in layer-split mode, MTP head drafting 3 tokens, Q8 KV, 392,960-token context. The prefix cache stays on, so a follow-up turn in a 48K chat starts in about 1 s instead of re-reading the chat. One request at a time. 1 user: 67.4 / 57.8 / 47.3 tok/s structured / code / prose, cold prefill 1,024 tok/s, TTFT 0.26 s. HumanEval 157/164, needle 3/3 up to 219K.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"145311340751aadc7c8c4c8ddce5bb5de6a36ba6","content_hash":"57c79110ce118919cae348f36141db3c4d717877","created_at":"2026-11-06 08:06:58.046163-07","notes":""},"engine_source":{"type":"fork","git_url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp","base_engine":"exllamav3","pip_package":null,"base_version":"alexvnesta server db01c25 + ExLlamaV3 0f685aa (9 patches)","image_digest":null,"install_note":"Source build as in the fork's README (setup_venv.sh, engine/apply_engine.sh, engine/build_native.sh for sm_80). One local change: in service/glm/glm_api.py the MTP guard only accepts -ndt 2; change `args.num_draft_tokens == 2` to `in (2, 3, 4)` to run -ndt 3. Launch with the fork's env exports (see runtime.env).","pinned_commit":"db01c256a8003daf90c932ee25552526ede40f1f"},"credit":[{"name":"alexvnesta (server + ExLlamaV3 patches)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2x CMP 170HX layer-split recipe)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3, EXL3 quant)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"cmpunlocker (amoghmunikote)","url":"https://github.com/amoghmunikote/cmpunlocker"}],"author":{"name":"mannydelrio1","isSite":false},"uploaded_by":"mannydelrio1","config":{"launch":["python","-u","service/glm/glm_api.py","-m","models/GLM-5.3-Flash-exl3-3.05bpw","-cq","8","-cs","458752","--context-cap","392960","-chunk_size","2048","-ambs","1","-gs","59,63","-lv","--q8-staging","--host","127.0.0.1","--port","8012","--recurrent-cache-gib","2","--q8-head-gpu0","--mtp","-ndt","3","--fp16-draft-cache","--recurrent_state_fp32"],"layouts":["pp"],"runtime":{"env":"EXL3_BC_ATTN=1 EXL3_MLA_KIDX_RING=32 EXL3_INT8_GEMV=0 EXL3_GEMV=0 EXL3_DSA_QC_STAGE=1 EXL3_DSA_QC_STAGE_MIN_R=9 EXL3_AUTOSPLIT_MARGIN_MB=128 GLM53_PREFILL_CHUNK_SIZE=2048 GLM53_TP_ACTIVE_HOST=0 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false; PYTHONPATH=native:engine-src:service/glm; GLM53_NATIVE_EXT_SHA256 = sha256 of native/exllamav3_ext*.so; ulimit -n 65536","notes":"Layer split (no -tp): GPU0 59 GB, GPU1 63 GB, output layer on GPU0 (--q8-head-gpu0; with it on GPU1, GPU1 ran out of memory after 5.5 h of traffic). MTP 3 draft tokens measured best: 2 = 59.8/52.8/49.6, 3 = 66.9/57.2/47.0, 4 = 63.2/56.0/42.1 tok/s. Why not TP2: +20% generation but -27% prefill, and on an 850 W PSU TP2's lock-step power swings cut the power, even with clocks capped at 1400 MHz. DFlash2 + hybrid is faster for single answers but has no prompt reuse in layer split."},"kv_pool_tokens":458752,"max_context_len":392960},"models":[{"slug":"glm-5-3-flash-exl3-3-05bpw","name":"GLM-5.3-Flash-EXL3-3.05bpw","hf_repo":"turboderp/GLM-5.3-Flash-exl3","quant":"exl3-3-05bpw"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["exl3","layer-split","long-context","mtp-speculative","prompt-cache","q8-kv"],"tasks":["chat","code"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"points\": [{\"notes\": \"MiaAI-Lab/Morrowmake protocol, thinking off, temp 0, 400 tokens, median of 5. Structured; code 57.8, prose 47.3. Prefill = cold 24K real text, median of 9. 180 W, 2026-11-04.\", \"ttft_ms\": 264, \"concurrency\": 1, \"context_len\": 392960, \"decode_tok_s\": 67.400000000000005684341886080801486968994140625, \"prefill_tok_s\": 1024}]}"}],"perf_points":[{"id":2530,"context_len":392960,"prefill_tok_s":1024,"decode_tok_s":67.400000000000005684341886080801486968994140625,"ttft_ms":264,"concurrency":1,"claimed_by":"submitter","notes":"MiaAI-Lab/Morrowmake protocol, thinking off, temp 0, 400 tokens, median of 5. Structured; code 57.8, prose 47.3. Prefill = cold 24K real text, median of 9. 180 W, 2026-11-04."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-layer-split-mtp3-393k.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-layer-split-mtp3-393k.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3","pin":"332ab457b709b7ba30dd9a448be5de03b80a7ac9 (3.05bpw branch)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp","pin":"db01c256a8003daf90c932ee25552526ede40f1f"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/Flun/glm53-flash-cmp170hx-exl3","pin":"5933112fdb858c960dfa1a68f507c3a98a9f1201"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-layer-split-mtp3-393k.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-layer-split-mtp3-393k","describedby":"/openapi.json"},"updated_at":"2026-11-06 08:06:58.046163-07","created_at":"2026-11-06 08:06:58.046163-07","speeds":{"decode_tok_s":67.400000000000005684341886080801486968994140625,"prefill_tok_s":1024,"leaderboard":"/api/v1/leaderboard.json"}}