{"slug":"glm-5-3-flash-exl3-2-60bpw-pp2-on-2x-cmp-170hx-dflash2-k3","title":"GLM-5.3-Flash EXL3 2.60bpw PP2 on 2x CMP 170HX DFlash2 k3","summary":"GLM-5.3-Flash (turboderp's EXL3 2.60 bpw pack, sc_2.60bpw) served by AlexBrandes' vLLM fork at PP2/TP1 - VLLM_PP_LAYER_PARTITION=23,22, 23 target layers on card 1 and 22 plus the DFlash2 drafter (3 draft tokens) on card 2 - on two CMP 170HX 64 GB cards. sm_80 tensor-core EXL3 paths on for prefill and small-row decode; PP scheduler spreads decodes and never mixes prefill+decode tokens in one step; FULL_DECODE_ONLY CUDA graphs. 225,280-token context, up to 4 sequences, block size 4608. Image built locally from the repo Dockerfile at three pins (upstream vLLM a0d7356b98 + branch 2a4e1379 + plugin 947ef7c), 17 min, no GPU. Author-measured 2026-11-05: structured C1 83.0 / code 76.5 / prose 57.8 tok/s, aggregate C4 191.9, cold prefill 1,741-1,852 tok/s, HumanEval 142/164, GSM8K 1228/1319, needle 16/16, vision 7/7. Recipe, engine and measurements by AlexBrandes; checkpoint MIT, drafter CC BY-NC-ND 4.0.","license":"Apache-2.0","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"bf9a8d56d0f0b251fb91b6b715f34e45d1d3c977","content_hash":"31644446ea58e9ff384107270c37632a05a3ab32","created_at":"2026-11-05 18:58:15.961295-07","notes":"Ingested from the lar-scout digest 2026-11-06 QUEUE; serve.sh + results.json transcribed verbatim at pinned commit db1af3228a1bd764bf9ba2c12ae9780c13819cb3 (branch plugin; branch HEAD has since moved to d5b2c6920 with docs-only changes - recipe dir byte-identical at both). Results fingerprint (results.json @ the pin): measured_source_sha256 fb920d8ccc1977f48c2e352fa4511bf291fa462bc696e79d738de8f74d31bc10, EXL3 extension sha256 316dccf116412a8c24f834613a569deecea20dae2f8705fff9003957de8a521f, measured image id sha256:ee218f38789644c2b42d8bcbc79b4dc18a5bb25cc7c33ca5491486dcd8d7156c. Measurement flags --enable-per-request-metrics --cudagraph-metrics and VLLM_KV_PLAN_LOG=1 are benchlib instrumentation beside serve.sh, not part of the serving profile. Card's engine 'vllm-exllamav3' is not in the site enum; recorded as vllm fork + ExLlamaV3-derived EXL3 plugin kernels per the fork-on-base-engine precedent. Drafter license CC BY-NC-ND 4.0 restricts commercial use. Not re-measured by the site; all figures are the author's T0 claims from results.json."},"engine_source":{"type":"fork","git_url":"https://github.com/AlexBrandes/vllm","base_engine":"vllm","pip_package":null,"base_version":"0.33.0rc1.dev659+ga0d7356b9","image_digest":null,"install_note":"NOT installable from a published image: build from a checkout of the fork's plugin branch - docker build -t glm53-flash-exl3:947ef7c recipes/glm53-flash-exl3 (base nvidia/cuda:13.0.3-devel-ubuntu24.04 pinned by digest; upstream vLLM prebuilt wheel for a0d7356b98 + the branch's Python files + the vllm-cmp170hx plugin with its EXL3 extension compiled against the image PyTorch, TORCH_CUDA_ARCH_LIST=8.0; every download sha256-checked; build needs network+CPU/RAM not a GPU, ~17 min). Engine = three pins: upstream a0d7356b98e929c17051bac44f5a086e4117974d, branch 2a4e1379093a84fd0852c197bb76c09a34a0473f, plugin 947ef7c7e1a96135521dce27cf953178816da42e. EXL3 quant/load-format and the PP/EXL3 kernels are registered by the plugin, not stock vLLM.","pinned_commit":"2a4e1379093a84fd0852c197bb76c09a34a0473f"},"credit":[{"name":"AlexBrandes (vLLM CMP170HX fork + plugin + this recipe + measurements)","url":"https://github.com/AlexBrandes/vllm"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 2.60bpw checkpoint)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"Morrowmake (sm_80 GLM-5.3 code lineage via vllm-sm80)","url":"https://github.com/Morrowmake/vllm-sm80"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/model","--host","0.0.0.0","--port","8000","--served-model-name","glm-5.3-flash-exl3-2.60bpw","--quantization","exl3","--load-format","exl3","--dtype","bfloat16","--pipeline-parallel-size","2","--tensor-parallel-size","1","--mm-encoder-tp-mode","data","--max-model-len","225280","--max-num-seqs","4","--max-num-batched-tokens","2048","--prefill-chunk-with-decodes","512","--block-size","4608","--gpu-memory-utilization","0.97","--enable-auto-tool-choice","--tool-call-parser","glm47","--reasoning-parser","glm47","--enable-prompt-tokens-details","--compilation-config","{\"cudagraph_mode\": \"FULL_DECODE_ONLY\"}","--speculative-config","{\"method\": \"dflash\", \"model\": \"/draft\", \"num_speculative_tokens\": 3, \"quantization\": null, \"draft_load_config\": {\"load_format\": \"safetensors\"}}","--hf-overrides","{\"text_config\": {\"aux_hidden_state_tensor\": \"stream_mean\"}}"],"layouts":["pp"],"runtime":{"env":"serve.sh exports (verbatim @ db1af3228): VLLM_CACHE_ROOT=/cache/vllm TRITON_CACHE_DIR=/cache/triton CUDA_CACHE_PATH=/cache/cuda VLLM_PP_LAYER_PARTITION=23,22 (23 target layers on card 1, 22 + drafter on card 2) VLLM_PP_SPREAD_DECODES=1 VLLM_PP_HOMOGENEOUS_STEPS=1 VLLM_SM80_KDA_PREFILL=1 VLLM_EXL3_MOE_PREFILL_TC=1 VLLM_EXL3_DENSE_PREFILL_TC=1 VLLM_EXL3_DENSE_PREFILL_ONE_PRODUCT=float VLLM_EXL3_MOE_DECODE_TC=1 VLLM_EXL3_MOE_DECODE_TC_MODE=full VLLM_EXL3_DENSE_DECODE_TC=1 VLLM_EXL3_DENSE_DECODE_TC_FAMILIES=all VLLM_EXL3_DENSE_DECODE_SPLITK=1 VLLM_EXL3_DECODE_VARIANT=off. Run (README step 3): docker build -t glm53-flash-exl3:947ef7c recipes/glm53-flash-exl3, then docker run --rm --gpus all --shm-size 16g --user $(id -u):$(id -g) -e HOME=/cache -e HF_HUB_OFFLINE=1 -e CUDA_VISIBLE_DEVICES=0,1 -p 127.0.0.1:8000:8000 -v $MODEL_DIR:/model:ro -v $DRAFT_DIR:/draft:ro -v $CACHE_DIR:/cache -v $PWD/recipes/glm53-flash-exl3/chat_template.jinja:/model/chat_template.jinja:ro glm53-flash-exl3:947ef7c (the jinja mount must replace the tokenizer's own template, not --chat-template: the GLM reasoning parser reads that one). Weights: hf download turboderp/GLM-5.3-Flash-exl3 --revision 24781209f2862a939cc21e01d256c943bd1a58a3; hf download incoai/GLM-5.3-Flash-DFlash2 --revision bf582e4eacc1810f76656d1811693ff6c6737d2a.","notes":"Engine is three pins per the Dockerfile: upstream vLLM a0d7356b98e929c17051bac44f5a086e4117974d (prebuilt wheel from wheels.vllm.ai), AlexBrandes/vllm branch 2a4e1379093a84fd0852c197bb76c09a34a0473f (Python files over the wheel), vllm-cmp170hx plugin 947ef7c7e1a96135521dce27cf953178816da42e (EXL3 extension compiled at build time, TORCH_CUDA_ARCH_LIST=8.0). Measured image id sha256:ee218f38789644c2b42d8bcbc79b4dc18a5bb25cc7c33ca5491486dcd8d7156c (docker build --no-cache); rebuilds get their own image identity. serve.sh ends in exec vllm serve ... \"$@\"; extra argv (none used for the measured runs) would append here."},"max_context_len":225280,"storage_requirements":{"note":"README requirements: ~110 GB NVMe checkpoint + 2.3 GB drafter + 8.3 GB engine image + compilation caches; NVIDIA driver 580+ (CUDA 13.0), hf CLI.","nvme_gb":121}},"models":[{"slug":"glm-5-3-flash-exl3-2-60bpw","name":"GLM-5.3-Flash-EXL3-2.60bpw","hf_repo":"turboderp/GLM-5.3-Flash-exl3","quant":"exl3-2-60bpw"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["dflash2","docker-build","exl3","long-context","pipeline-parallel","speculative","vision"],"tasks":["chat","code","vision"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured 2026-11-05 on the rig (2x CMP 170HX, 74 SMs/card, 170 W/card, PCIe Gen2 x4, no P2P) with benchlib's SparkDash-compatible public-v1 battery (SparkDash 3e6d4ef: greedy, thinking off, 400 output tokens, engine-direct); values are medians of 3 complete runs (run IDs public-v1.20261105T192046Z.a455953f / T205324Z.696b161a / T210946Z.6cca7682). C2-C8 are aggregate across that many requests (max-num-seqs 4, so C8 queues). Structured C1 83.0 / code 76.5 / JSON 70.9 / prose 57.8 tok/s. Quality same session: HumanEval pass@1 142/164 chat (134/164 completion), GSM8K 1228/1319 chat (1190/1319 completion), needle 16/16 at 4K/16K/32K/64K, 225279+1 tokens accepted / 225280 refused, image+video 7/7. Full per-cell data in results.json @ db1af3228.\", \"tok_s\": 82.953000000000002955857780762016773223876953125, \"points\": [{\"notes\": \"median of 3 runs, SparkDash public-v1 protocol (greedy, thinking off, 400 output tokens), structured battery; same-concurrency variants: code 76.5, JSON 70.9, prose 57.8 tok/s (aggregate at C1). results.json decode @ 2a4e1379. context_len 1024 is the short-prompt marker for concurrency-decode rows.\", \"concurrency\": 1, \"context_len\": 1024, \"decode_tok_s\": 82.953000000000002955857780762016773223876953125}, {\"notes\": \"median of 3 runs, SparkDash public-v1 protocol (greedy, thinking off, 400 output tokens), structured battery; same-concurrency variants: code 109.9, JSON 113.2, prose 92.4 tok/s (aggregate at C2). results.json decode @ 2a4e1379. context_len 1024 is the short-prompt marker for concurrency-decode rows.\", \"concurrency\": 2, \"context_len\": 1024, \"decode_tok_s\": 122.45900000000000318323145620524883270263671875}, {\"notes\": \"median of 3 runs, SparkDash public-v1 protocol (greedy, thinking off, 400 output tokens), structured battery; same-concurrency variants: code 151.4, JSON 164.6, prose 139.6 tok/s (aggregate at C4). results.json decode @ 2a4e1379. context_len 1024 is the short-prompt marker for concurrency-decode rows.\", \"concurrency\": 4, \"context_len\": 1024, \"decode_tok_s\": 191.90199999999998681232682429254055023193359375}, {\"notes\": \"median of 3 runs, SparkDash public-v1 protocol (greedy, thinking off, 400 output tokens), structured battery; same-concurrency variants: code 137.0, JSON 156.5, prose 123.0 tok/s (aggregate at C8). results.json decode @ 2a4e1379. context_len 1024 is the short-prompt marker for concurrency-decode rows.\", \"concurrency\": 8, \"context_len\": 1024, \"decode_tok_s\": 161.143000000000000682121026329696178436279296875}, {\"notes\": \"code battery, C1 aggregate, median of 3 runs; context_len is the tuple marker 1024+10240. Full variant table on the concurrency-1 structured point.\", \"concurrency\": 1, \"context_len\": 11264, \"decode_tok_s\": 76.4539999999999935198502498678863048553466796875}, {\"notes\": \"code battery, C2 aggregate, median of 3 runs; context_len is the tuple marker 1024+10240. Full variant table on the concurrency-2 structured point.\", \"concurrency\": 2, \"context_len\": 11264, \"decode_tok_s\": 109.8960000000000007958078640513122081756591796875}, {\"notes\": \"code battery, C4 aggregate, median of 3 runs; context_len is the tuple marker 1024+10240. Full variant table on the concurrency-4 structured point.\", \"concurrency\": 4, \"context_len\": 11264, \"decode_tok_s\": 151.441000000000002501110429875552654266357421875}, {\"notes\": \"code battery, C8 aggregate, median of 3 runs; context_len is the tuple marker 1024+10240. Full variant table on the concurrency-8 structured point.\", \"concurrency\": 8, \"context_len\": 11264, \"decode_tok_s\": 137.049000000000006593836587853729724884033203125}, {\"notes\": \"JSON battery, C1 aggregate, median of 3 runs; context_len is the tuple marker 1024+20480. Full variant table on the concurrency-1 structured point.\", \"concurrency\": 1, \"context_len\": 21504, \"decode_tok_s\": 70.8900000000000005684341886080801486968994140625}, {\"notes\": \"JSON battery, C2 aggregate, median of 3 runs; context_len is the tuple marker 1024+20480. Full variant table on the concurrency-2 structured point.\", \"concurrency\": 2, \"context_len\": 21504, \"decode_tok_s\": 113.1680000000000063664629124104976654052734375}, {\"notes\": \"JSON battery, C4 aggregate, median of 3 runs; context_len is the tuple marker 1024+20480. Full variant table on the concurrency-4 structured point.\", \"concurrency\": 4, \"context_len\": 21504, \"decode_tok_s\": 164.578000000000002955857780762016773223876953125}, {\"notes\": \"JSON battery, C8 aggregate, median of 3 runs; context_len is the tuple marker 1024+20480. Full variant table on the concurrency-8 structured point.\", \"concurrency\": 8, \"context_len\": 21504, \"decode_tok_s\": 156.513000000000005229594535194337368011474609375}, {\"notes\": \"prose battery, C1 aggregate, median of 3 runs; context_len is the tuple marker 1024+30720. Full variant table on the concurrency-1 structured point.\", \"concurrency\": 1, \"context_len\": 31744, \"decode_tok_s\": 57.820999999999997953636921010911464691162109375}, {\"notes\": \"prose battery, C2 aggregate, median of 3 runs; context_len is the tuple marker 1024+30720. Full variant table on the concurrency-2 structured point.\", \"concurrency\": 2, \"context_len\": 31744, \"decode_tok_s\": 92.4180000000000063664629124104976654052734375}, {\"notes\": \"prose battery, C4 aggregate, median of 3 runs; context_len is the tuple marker 1024+30720. Full variant table on the concurrency-4 structured point.\", \"concurrency\": 4, \"context_len\": 31744, \"decode_tok_s\": 139.640999999999991132426657713949680328369140625}, {\"notes\": \"prose battery, C8 aggregate, median of 3 runs; context_len is the tuple marker 1024+30720. Full variant table on the concurrency-8 structured point.\", \"concurrency\": 8, \"context_len\": 31744, \"decode_tok_s\": 123.0090000000000003410605131648480892181396484375}, {\"notes\": \"cold prefill median of 3 unique-prompt runs @ 4096 tokens (results.json prefill; prompt tokens 4126) @ 2a4e1379.\", \"context_len\": 4096, \"prefill_tok_s\": 1741.339999999999918145476840436458587646484375}, {\"notes\": \"cold prefill median of 3 unique-prompt runs @ 8192 tokens (results.json prefill; prompt tokens 8222) @ 2a4e1379.\", \"context_len\": 8192, \"prefill_tok_s\": 1851.5399999999999636202119290828704833984375}, {\"notes\": \"cold prefill median of 3 unique-prompt runs @ 16384 tokens (results.json prefill; prompt tokens 16416) @ 2a4e1379.\", \"context_len\": 16384, \"prefill_tok_s\": 1842.1300000000001091393642127513885498046875}, {\"notes\": \"cold prefill median of 3 unique-prompt runs @ 32768 tokens (results.json prefill; prompt tokens 32798) @ 2a4e1379.\", \"context_len\": 32768, \"prefill_tok_s\": 1835.549999999999954525264911353588104248046875}, {\"notes\": \"cold prefill median of 3 unique-prompt runs @ 65536 tokens (results.json prefill; prompt tokens 65566) @ 2a4e1379.\", \"context_len\": 65536, \"prefill_tok_s\": 1822.76999999999998181010596454143524169921875}, {\"notes\": \"cold prefill median of 3 unique-prompt runs @ 131072 tokens (results.json prefill; prompt tokens 131103) @ 2a4e1379.\", \"context_len\": 131072, \"prefill_tok_s\": 1802.660000000000081854523159563541412353515625}, {\"notes\": \"single-stream decode after a 4096-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379.\", \"concurrency\": 1, \"context_len\": 4096, \"decode_tok_s\": 59.05799999999999982946974341757595539093017578125}, {\"notes\": \"single-stream decode after a 8192-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379.\", \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 59.01700000000000301270119962282478809356689453125}, {\"notes\": \"single-stream decode after a 16384-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379.\", \"concurrency\": 1, \"context_len\": 16384, \"decode_tok_s\": 60.75800000000000267164068645797669887542724609375}, {\"notes\": \"single-stream decode after a 32768-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379.\", \"concurrency\": 1, \"context_len\": 32768, \"decode_tok_s\": 58.078000000000002955857780762016773223876953125}, {\"notes\": \"single-stream decode after a 65536-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379.\", \"concurrency\": 1, \"context_len\": 65536, \"decode_tok_s\": 56.08500000000000085265128291212022304534912109375}, {\"notes\": \"single-stream decode after a 131072-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379.\", \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 58.95100000000000051159076974727213382720947265625}], \"context_len\": 225280}"}],"perf_points":[{"id":2454,"context_len":1024,"prefill_tok_s":null,"decode_tok_s":82.953000000000002955857780762016773223876953125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"median of 3 runs, SparkDash public-v1 protocol (greedy, thinking off, 400 output tokens), structured battery; same-concurrency variants: code 76.5, JSON 70.9, prose 57.8 tok/s (aggregate at C1). results.json decode @ 2a4e1379. context_len 1024 is the short-prompt marker for concurrency-decode rows."},{"id":2455,"context_len":1024,"prefill_tok_s":null,"decode_tok_s":122.45900000000000318323145620524883270263671875,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"median of 3 runs, SparkDash public-v1 protocol (greedy, thinking off, 400 output tokens), structured battery; same-concurrency variants: code 109.9, JSON 113.2, prose 92.4 tok/s (aggregate at C2). results.json decode @ 2a4e1379. context_len 1024 is the short-prompt marker for concurrency-decode rows."},{"id":2456,"context_len":1024,"prefill_tok_s":null,"decode_tok_s":191.90199999999998681232682429254055023193359375,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"median of 3 runs, SparkDash public-v1 protocol (greedy, thinking off, 400 output tokens), structured battery; same-concurrency variants: code 151.4, JSON 164.6, prose 139.6 tok/s (aggregate at C4). results.json decode @ 2a4e1379. context_len 1024 is the short-prompt marker for concurrency-decode rows."},{"id":2457,"context_len":1024,"prefill_tok_s":null,"decode_tok_s":161.143000000000000682121026329696178436279296875,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"median of 3 runs, SparkDash public-v1 protocol (greedy, thinking off, 400 output tokens), structured battery; same-concurrency variants: code 137.0, JSON 156.5, prose 123.0 tok/s (aggregate at C8). results.json decode @ 2a4e1379. context_len 1024 is the short-prompt marker for concurrency-decode rows."},{"id":2470,"context_len":4096,"prefill_tok_s":1741.339999999999918145476840436458587646484375,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill median of 3 unique-prompt runs @ 4096 tokens (results.json prefill; prompt tokens 4126) @ 2a4e1379."},{"id":2476,"context_len":4096,"prefill_tok_s":null,"decode_tok_s":59.05799999999999982946974341757595539093017578125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode after a 4096-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379."},{"id":2471,"context_len":8192,"prefill_tok_s":1851.5399999999999636202119290828704833984375,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill median of 3 unique-prompt runs @ 8192 tokens (results.json prefill; prompt tokens 8222) @ 2a4e1379."},{"id":2477,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":59.01700000000000301270119962282478809356689453125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode after a 8192-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379."},{"id":2458,"context_len":11264,"prefill_tok_s":null,"decode_tok_s":76.4539999999999935198502498678863048553466796875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"code battery, C1 aggregate, median of 3 runs; context_len is the tuple marker 1024+10240. Full variant table on the concurrency-1 structured point."},{"id":2459,"context_len":11264,"prefill_tok_s":null,"decode_tok_s":109.8960000000000007958078640513122081756591796875,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"code battery, C2 aggregate, median of 3 runs; context_len is the tuple marker 1024+10240. Full variant table on the concurrency-2 structured point."},{"id":2460,"context_len":11264,"prefill_tok_s":null,"decode_tok_s":151.441000000000002501110429875552654266357421875,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"code battery, C4 aggregate, median of 3 runs; context_len is the tuple marker 1024+10240. Full variant table on the concurrency-4 structured point."},{"id":2461,"context_len":11264,"prefill_tok_s":null,"decode_tok_s":137.049000000000006593836587853729724884033203125,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"code battery, C8 aggregate, median of 3 runs; context_len is the tuple marker 1024+10240. Full variant table on the concurrency-8 structured point."},{"id":2472,"context_len":16384,"prefill_tok_s":1842.1300000000001091393642127513885498046875,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill median of 3 unique-prompt runs @ 16384 tokens (results.json prefill; prompt tokens 16416) @ 2a4e1379."},{"id":2478,"context_len":16384,"prefill_tok_s":null,"decode_tok_s":60.75800000000000267164068645797669887542724609375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode after a 16384-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379."},{"id":2462,"context_len":21504,"prefill_tok_s":null,"decode_tok_s":70.8900000000000005684341886080801486968994140625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"JSON battery, C1 aggregate, median of 3 runs; context_len is the tuple marker 1024+20480. Full variant table on the concurrency-1 structured point."},{"id":2463,"context_len":21504,"prefill_tok_s":null,"decode_tok_s":113.1680000000000063664629124104976654052734375,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"JSON battery, C2 aggregate, median of 3 runs; context_len is the tuple marker 1024+20480. Full variant table on the concurrency-2 structured point."},{"id":2464,"context_len":21504,"prefill_tok_s":null,"decode_tok_s":164.578000000000002955857780762016773223876953125,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"JSON battery, C4 aggregate, median of 3 runs; context_len is the tuple marker 1024+20480. Full variant table on the concurrency-4 structured point."},{"id":2465,"context_len":21504,"prefill_tok_s":null,"decode_tok_s":156.513000000000005229594535194337368011474609375,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"JSON battery, C8 aggregate, median of 3 runs; context_len is the tuple marker 1024+20480. Full variant table on the concurrency-8 structured point."},{"id":2466,"context_len":31744,"prefill_tok_s":null,"decode_tok_s":57.820999999999997953636921010911464691162109375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"prose battery, C1 aggregate, median of 3 runs; context_len is the tuple marker 1024+30720. Full variant table on the concurrency-1 structured point."},{"id":2467,"context_len":31744,"prefill_tok_s":null,"decode_tok_s":92.4180000000000063664629124104976654052734375,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"prose battery, C2 aggregate, median of 3 runs; context_len is the tuple marker 1024+30720. Full variant table on the concurrency-2 structured point."},{"id":2468,"context_len":31744,"prefill_tok_s":null,"decode_tok_s":139.640999999999991132426657713949680328369140625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"prose battery, C4 aggregate, median of 3 runs; context_len is the tuple marker 1024+30720. Full variant table on the concurrency-4 structured point."},{"id":2469,"context_len":31744,"prefill_tok_s":null,"decode_tok_s":123.0090000000000003410605131648480892181396484375,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"prose battery, C8 aggregate, median of 3 runs; context_len is the tuple marker 1024+30720. Full variant table on the concurrency-8 structured point."},{"id":2473,"context_len":32768,"prefill_tok_s":1835.549999999999954525264911353588104248046875,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill median of 3 unique-prompt runs @ 32768 tokens (results.json prefill; prompt tokens 32798) @ 2a4e1379."},{"id":2479,"context_len":32768,"prefill_tok_s":null,"decode_tok_s":58.078000000000002955857780762016773223876953125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode after a 32768-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379."},{"id":2474,"context_len":65536,"prefill_tok_s":1822.76999999999998181010596454143524169921875,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill median of 3 unique-prompt runs @ 65536 tokens (results.json prefill; prompt tokens 65566) @ 2a4e1379."},{"id":2480,"context_len":65536,"prefill_tok_s":null,"decode_tok_s":56.08500000000000085265128291212022304534912109375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode after a 65536-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379."},{"id":2475,"context_len":131072,"prefill_tok_s":1802.660000000000081854523159563541412353515625,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill median of 3 unique-prompt runs @ 131072 tokens (results.json prefill; prompt tokens 131103) @ 2a4e1379."},{"id":2481,"context_len":131072,"prefill_tok_s":null,"decode_tok_s":58.95100000000000051159076974727213382720947265625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode after a 131072-token prompt (benchlib depth_decode addition, median of 3) @ 2a4e1379."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-2-60bpw-pp2-on-2x-cmp-170hx-dflash2-k3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exl3-2-60bpw-pp2-on-2x-cmp-170hx-dflash2-k3.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3","pin":"24781209f2862a939cc21e01d256c943bd1a58a3 - sc_2.60bpw branch (results.json checkpoint): 2.5-bit routed experts, 6-bit dense/attention, LM head + vision tower; ~110 GB; keeps GLM-5.3-Flash's MIT license"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2","pin":"bf582e4eacc1810f76656d1811693ff6c6737d2a - BF16 drafter, stream_mean auxiliary state, CC BY-NC-ND 4.0 (non-commercial, no redistribution)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/AlexBrandes/vllm","pin":"2a4e1379093a84fd0852c197bb76c09a34a0473f - measured vLLM branch commit; recipe dir recipes/glm53-flash-exl3 @ db1af3228a1bd764bf9ba2c12ae9780c13819cb3 (branch plugin)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/vllm-project/vllm","pin":"a0d7356b98e929c17051bac44f5a086e4117974d - upstream vLLM base commit; prebuilt wheel from wheels.vllm.ai (sha256 5e293d985d...)"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/AlexBrandes/vllm","pin":"947ef7c7e1a96135521dce27cf953178816da42e - vllm-cmp170hx plugin commit (EXL3 extension, sm_80 GLM-5.3 code derived from Morrowmake's vllm-sm80); measured image id sha256:ee218f38789644c2... (local build, not published)"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-2-60bpw-pp2-on-2x-cmp-170hx-dflash2-k3.json","alternate":"/r/glm-5-3-flash-exl3-2-60bpw-pp2-on-2x-cmp-170hx-dflash2-k3","describedby":"/openapi.json"},"updated_at":"2026-11-05 18:58:15.961295-07","created_at":"2026-11-05 18:58:15.961295-07","speeds":{"decode_tok_s":59,"prefill_tok_s":1822.799999999999954525264911353588104248046875,"leaderboard":"/api/v1/leaderboard.json"}}