diff --git a/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw/recipe.yaml b/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw/recipe.yaml index 91b482a4d..66ee933dc 100644 --- a/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw/recipe.yaml +++ b/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw/recipe.yaml @@ -5,9 +5,9 @@ engine: version: 'dev 5783a93 + TabbyAPI 53da7919' source_type: fork git_url: 'https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090' - pinned_commit: 4e3e0f0f5bfbcf1f963936da89b6a80cda2876b8 + pinned_commit: 88e112e08a2368e3194232aed1ba5f6375ad3368 install_note: "NOT installable from a stock image: the served image stacks five in-repo overlays (docker/cheapswap-r3, glm-agent-r2-on-cheapswap-r3, mtp-fast-r1, mtp-overhead-r2, glm-splitdev-r2) on the base image tabbyapi:r828-prompt-lookup-r3 (ExLlamaV3 dev 5783a93 with 19 patch sets + TabbyAPI 53da7919, from adrienbrault/qwen3.8-flash-next-2x-rtx5090). Build the overlays from the repo's Dockerfiles, then launch with scripts/glm-daily.env (scripts/launch-glm53.sh writes config.yml, clears image-baked EXL3_* selectors, verifies offload registration; needs the box paths /srv/qwen5090/models/ on NVMe and ~76 GB disk for the checkpoint)." -summary: "GLM-5.3-Flash (321B MoE, ~18B active) turboderp EXL3 2.05bpw (76 GB) on two RTX 5090 (64 GB VRAM) + 64 GB DDR5: TabbyAPI on ExLlamaV3 with CPU-offloaded routed experts (100/layer on GPU0's layers 3-22+MTP, 104/layer on GPU1's 23-44), runtime hot-expert swaps (exchange mode from a static hot set), per-GPU expert split, MTP head drafting 1 tok/step single-stream with a one-request draft cap, 262,144-token window on an 8-bit KV page pool (~2.3 GB shared by 4 concurrent), vision ON with tower linears pinned in host RAM. Exact-output serving: no expert skipped, every draft verified. Measured 2026-10-08 (R927): c1 decode 64.7 tok/s (code 60.0/prose 69.2/chat 72.7/html 66.6/edit 54.9), 4-stream sum 93.5 tok/s, TTFT 0.44 s c1; five in-repo patch overlays on the qwen3.8-flash-next-2x-rtx5090 base image." +summary: "GLM-5.3-Flash (321B MoE, ~18B active) turboderp EXL3 2.05bpw on two RTX 5090 + 64 GB DDR5: TabbyAPI on ExLlamaV3 with CPU-offloaded routed experts (96/layer GPU0 layers 3-22+MTP, 102/layer GPU1 23-44), runtime hot-expert swaps, per-GPU expert split, the DSA indexer's per-token rows in a fixed ring, live /metrics token counters, the expert-exchange clock handed to the first text MoE layer so swaps also run at 2-4 streams, MTP head drafting 1 tok/step single-stream, 262,144-token window on an 8-bit KV page pool (~1.6 GiB across 4), vision ON with tower linears pinned in host RAM. Exact-output: no expert skipped, every draft verified. Measured 2026-10-10 (R969, three fresh boots): c1 score 62.9 tok/s (code 59.9/prose 66.7/chat 68.5/html 64.8/edit 53.9), 2/3/4-stream sums 74.4/92.0/98.0. Nine in-repo overlays on the qwen3.8-flash-next-2x-rtx5090 base image." model: name: GLM-5.3-Flash-EXL3-2.05bpw family: glm-5-3-flash @@ -40,11 +40,13 @@ tags: config: launch: - 'bash scripts/launch-glm53.sh' - - '# env (scripts/glm-daily.env @ 4e3e0f0f): PACK=A OFFLOAD_MODE=split CACHE_TOKENS=262144 MAX_SEQ=262144 CACHE_MODE=8,8 CHUNK=2048 MAX_BATCH=4 SYSMEM_RC_MB=1024' - - 'OFFLOAD_N=104 DRAFT=1 DRAFT_N=1 MTP_FAST=1 VISION_OFFLOAD=1 GPU_SPLIT=31.8,31' - - 'GLM_IMG=tabbyapi:cheapswap-r3-agent-r2_mtpfast1_overhead-r2_splitdev2 PINNED_ARENA=1 AGENT=1 KEEP_THINKING=1' - - "EXL3_EXTRA='EXL3_MTP_MAX_BATCH=1;EXL3_DRAFT_PINNED_STAGING=1;EXL3_MTP_GPU_DRAFT=1;EXL3_MTP_GREEDY_ACCEPT=1;EXL3_MTP_CACHED_REWIND=1;EXL3_MOE_CPU_SPLIT_BY_DEVICE=100,104'" + - '# env (scripts/glm-daily.env @ 88e112e0): PACK=A OFFLOAD_MODE=split CACHE_TOKENS=262144 MAX_SEQ=262144 CACHE_MODE=8,8 CHUNK=4096 MAX_BATCH=4 SYSMEM_RC_MB=1024' + - 'OFFLOAD_N=104 DRAFT=1 DRAFT_N=1 MTP_FAST=1 VISION_OFFLOAD=1 INDEX_RING=1 GPU_SPLIT=31.8,31 CHUNK=4096' + - 'GLM_IMG=tabbyapi:cheapswap-r3-agent-r2_mtpfast1_overhead-r2_splitdev2_ring3_livemetrics2_draftchunk1_textclock1 PINNED_ARENA=1 AGENT=1 KEEP_THINKING=1' + - "EXL3_EXTRA='EXL3_MTP_MAX_BATCH=1;EXL3_DRAFT_PINNED_STAGING=1;EXL3_MTP_GPU_DRAFT=1;EXL3_MTP_GREEDY_ACCEPT=1;EXL3_MTP_CACHED_REWIND=1;EXL3_MOE_CPU_SPLIT_BY_DEVICE=96,102;EXL3_MOE_CPU_SWAP_TEXT_CLOCK=1'" - 'SWAP_MODE=exchange SWAP_POLICY=histogram SWAP_INIT_STATS=$PWD/scripts/split-stats-broad-r869.json SWAP_CADENCE=exact SWAP_INTERVAL=64 SWAP_MAX=64 SWAP_SCOPE=global SWAP_FLOOR=4 SWAP_HYST=2.0' + - '# REQUIRED: scripts/sampler_overrides/glm53.yml must exist or launch-glm53.sh aborts (exit 3)' + - '# launch-glm53.sh mounts it read-only at -v "$HERE/sampler_overrides/glm53.yml":/app/sampler_overrides/glm53.yml:ro' runtime: tabby_config: 'written by scripts/glm53_plan.py: model_name glm53-flash-exl3-2.05bpw-turboderp, backend exllamav3, max_seq_len 262144, cache_size 262144, cache_mode 8,8, max_batch_size 4, tensor_parallel false, gpu_split [31.8, 31.0], gpu_split_auto false, cpu_moe_split_experts 104 (overridden per device to 100,104 by EXL3_MOE_CPU_SPLIT_BY_DEVICE), chunk_size 2048, output_chunking true, vision true, vision_offload true, tool_format glm4_5, reasoning true with reasoning_start/end token (Paperclip) and prompt_template glm53-keep-thinking (KEEP_THINKING=1); draft_model: draft_mode mtp, draft_num_tokens 1, draft_cache_mode Q8, dynamic_draft false; memory: sysmem_recurrent_cache 1024 MB, sysmem_kv_cache 0' ports: 'TabbyAPI on 0.0.0.0:8029 (container glm53, --disable-auth true; single box, no other tenants)' @@ -69,39 +71,45 @@ config: performance: claimed_by: submitter metrics: - tok_s: 64.7 + tok_s: 62.9 ttft_ms: 440 context_len: 262144 - notes: 'Author-measured 2026-10-08 21:43 UTC on the served configuration (R927 control arm B-D, scripts/glm-daily.env unchanged). c1 decode by content kind, chat requests forced to 1,024 tokens (html 2,048), temperature 0: code 60.0, prose 69.2, chat 72.7, html 66.6, edit 54.9 tok/s; c1 score 64.7. 4-stream sum 93.5 tok/s (39.5/stream, distinct prompts, no drafting at c2/c4); c2 78.9 (39.5/stream). TTFT 0.44 s c1, 0.74-0.78 s c2, 1.55-1.74 s c4. Server side 15.60 ms/token, 26.74 ms per MTP step at 1.714 tokens/step; draft acceptance 0.57-0.97. Run-to-run spread arms (R915c/R925): c1 64.9-65.3, c4 sum 90.0-92.0. Cold prefill not measured on the served image; R924 arms with its flag off measured 1,940-2,018 tok/s @ ~8,200 prompt tokens and 2,029-2,060 @ ~32,700 on the adjacent image. VRAM after warmup 31,304 + 31,786 MiB (847 / 365 MiB free). MTP accepts 0.57-0.97 per kind; decode rate moves with text while drafting.' + notes: "Author-measured 2026-10-10 17:08 UTC on the served configuration with the text clock (R969, arms B1/B2/B3, three fresh boots, results directory 2026-10-10-r969-glm53-textclock-170158). On each boot the requests ran in the order 1,2,3,4 streams starting right after the launcher's boot warmup; each figure is the median of the three boots. 1 stream = the ten c1-score requests (code, prose, chat, html, edit, 2 runs each); 2-4 streams = a different prompt per stream (code, prose, chat, html in that order), sum of the stream rates over the common decode window, median of 2 rounds per boot, and no token is drafted at 2+. Every decode stream is a chat request forced to 1,024 tokens (html 2,048), temperature 0. c1 score (mean of the five kinds' medians) 62.9, range 62.2-63.2 over the boots; 1 stream 64.8, range 63.7-65.6. Draft acceptance at 1 stream 0.59-0.97 per request over the three boots, highest on edits and code. Server side (R959d arm C4a, the engine's per-request counters over its 10 c1-score requests): 15.61 ms per token, 27.19 ms per MTP step at 1.741 tokens per step; four streams 11.01 ms per generated token. Prefill and the server-side counters are from R959d (2026-10-10 06:19 UTC, arm C4a, directory 2026-10-10-r959d-glm53-chunk-confirm-HoWhmg) on the image BEFORE the text clock with the same settings otherwise - R969 did not measure prefill: 1,973 tok/s @ 32,738 prompt tokens, 2,023 @ 131,051. VRAM after warmup on the served config: 31,304 + 31,786 MiB (847 / 365 MiB free)." points: - context_len: 4096 concurrency: 1 - decode_tok_s: 64.7 - ttft_ms: 440 - notes: 'c1 score (mean of code/prose/chat/html/edit medians), R927 arm B-D; per-kind 60.0/69.2/72.7/66.6/54.9' + decode_tok_s: 62.9 + ttft_ms: 480 + notes: "c1 score (mean of the five kinds' medians, R969 median of boots B1/B2/B3); per-kind code 59.9 / prose 66.7 / chat 68.5 / html 64.8 / edit 53.9. The median of the ten individual c1-score requests was 64.8 tok/s, range 63.7-65.6 over the three boots. TTFT 0.48 s (range 0.47-0.49)." - context_len: 4096 concurrency: 2 - decode_tok_s: 78.9 - ttft_ms: 760 - notes: 'c2 sum of stream rates, distinct prompts, per stream 39.5; TTFT 0.74-0.78 s' + decode_tok_s: 74.4 + ttft_ms: 785 + notes: '2 streams: sum over the common decode window, 37.2 per stream, R969 median of 2 rounds per boot; range over boots 74.3-75.5. TTFT 0.75-0.82 s. No token is drafted at 2+ streams.' + - + context_len: 4096 + concurrency: 3 + decode_tok_s: 92.0 + ttft_ms: 1160 + notes: '3 streams: sum, 30.7 per stream, R969; range over boots 90.3-92.5. TTFT 1.11-1.21 s.' - context_len: 4096 concurrency: 4 - decode_tok_s: 93.5 - ttft_ms: 1645 - notes: 'c4 sum of stream rates, distinct prompts, per stream 23.4; TTFT 1.55-1.74 s' + decode_tok_s: 98.0 + ttft_ms: 1635 + notes: '4 streams: sum, 24.5 per stream, R969; range over boots 95.6-98.8. TTFT 1.57-1.70 s.' - - context_len: 8200 + context_len: 32738 concurrency: 1 - prefill_tok_s: 1979 - notes: 'cold prefill 1,940-2,018 tok/s @ ~8,200 prompt tokens measured on the R924 adjacent-image arms (flag off), not on the served image' + prefill_tok_s: 1973.0 + notes: 'Cold prefill, R959d arm C4a on the image BEFORE the text clock (author: R969 did not measure prefill); 32,738 prompt tokens.' - - context_len: 32700 + context_len: 131051 concurrency: 1 - prefill_tok_s: 2044 - notes: 'cold prefill 2,029-2,060 tok/s @ ~32,700 prompt tokens, same adjacent-image arms' + prefill_tok_s: 2023.0 + notes: 'Cold prefill, R959d arm C4a, image before the text clock; 131,051 prompt tokens.' artifacts: - kind: reference @@ -150,7 +158,7 @@ provenance: - 'https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090/blob/main/scripts/glm-daily.env' - 'https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090/blob/main/scripts/FLAG-DECISIONS.txt' - 'https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090/blob/main/bench/results/r927-glm53-dflash-window.md' - notes: "Submitted from the repo at 4e3e0f0f5bfbcf1f963936da89b6a80cda2876b8 (2026-10-09 00:45 +0200). All figures are the author's own measurements (bench/results R927 served arm B-D with R915c/R925 spread arms, R924 prefill arm, R900 memory layout; raw records in bench/results/-/). Hardware note: 60 GB usable of 2x32 GB DDR5-6000 hosts the CPU-offloaded routed experts (26.4 GiB) plus the pinned vision tower. Exact-output claim per author: no expert skipped, every drafted token verified; greedy run-to-run instability documented (R916) with opt-in deterministic flags. First TabbyAPI-served and first RTX-5090 recipe on the site; second ExLlamaV3 entry after Flun's 2x CMP 170HX." + notes: "Submitted from the repo at 4e3e0f0f5bfbcf1f963936da89b6a80cda2876b8 (2026-10-09 00:45 +0200). All figures are the author's own measurements (bench/results R927 served arm B-D with R915c/R925 spread arms, R924 prefill arm, R900 memory layout; raw records in bench/results/-/). Hardware note: 60 GB usable of 2x32 GB DDR5-6000 hosts the CPU-offloaded routed experts (26.4 GiB) plus the pinned vision tower. Exact-output claim per author: no expert skipped, every drafted token verified; greedy run-to-run instability documented (R916) with opt-in deterministic flags. First TabbyAPI-served and first RTX-5090 recipe on the site; second ExLlamaV3 entry after Flun's 2x CMP 170HX. UPDATE 2026-10-30 (repo @ 88e112e08a2368e3194232aed1ba5f6375ad3368, 13 commits ahead of the pinned 4e3e0f0f): served configuration moved - image tag gained _ring3_livemetrics2_draftchunk1_textclock1, new INDEX_RING=1, EXL3_MOE_CPU_SPLIT_BY_DEVICE 100,104 -> 96,102, new EXL3_MOE_CPU_SWAP_TEXT_CLOCK=1, new CHUNK=4096 (the recipe previously documented CHUNK=2048 and 100,104 - both stale). Three new overlays are now part of the stack: glm-index-ring-r3 (DSA indexer per-token rows in a fixed ring instead of the KV pool), tabby-livemetrics-r3 (live token counters on /metrics) and textclock-r1 (expert-exchange clock to the first text MoE layer, so swaps also run while 2-4 requests are active). draftchunk-r1 lets chunks above 2,048 tokens boot, which is what makes CHUNK=4096 usable. KV pool restated by the author at ~1.6 GiB (608 bytes per token on each of the 11 attention layers: latent 512, scales 32, pooled indexer keys 64) plus an 8.5 MiB ring per DSA layer, replacing the earlier ~2.3 GiB figure. launch-glm53.sh now HARD-REQUIRES scripts/sampler_overrides/glm53.yml and mounts it read-only at /app/sampler_overrides/glm53.yml - without that file the launch aborts with exit 3. Performance numbers below are refreshed to the author's R969 served arms B1/B2/B3 (2026-10-10, three fresh boots, median of the boots, measured ON the text-clock image): c1 score 62.9, 1 stream 64.8, 2/3/4-stream per stream / sum 37.2/74.4, 30.7/92.0, 24.5/98.0 tok/s; c1 by kind code 59.9 / prose 66.7 / chat 68.5 / html 64.8 / edit 53.9; TTFT 0.48 s c1, 0.75-0.82 / 1.11-1.21 / 1.57-1.70 s at 2/3/4 streams. Prefill remains R959d arm C4a on the image BEFORE the text clock (author states R969 did not measure prefill): 1,973 tok/s @ 32,738 prompt tokens and 2,023 @ 131,051." kv_pool_tokens: 262144 storage_requirements: ram_gb: 64.0