diff --git a/recipes/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots/recipe.yaml b/recipes/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots/recipe.yaml index 47d75b087..85d4b82e8 100644 --- a/recipes/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots/recipe.yaml +++ b/recipes/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots/recipe.yaml @@ -4,9 +4,9 @@ engine: name: exllamav3 source_type: fork git_url: 'https://github.com/alexvnesta/glm53-flash-cmp170hx-tp' - pinned_commit: fa0adf1899015f692ddce30391ee7e740a0910c9 - install_note: '[base_version 0f685aa + row32 four-slot profile] Source build required; no image. Follow https://github.com/alexvnesta/glm53-flash-cmp170hx-tp/blob/fa0adf1899015f692ddce30391ee7e740a0910c9/profiles/community-four-active/README.md. Set up the pinned venv; apply_engine.sh; install gguf==0.17.1; profiles/community-four-active/build.sh /ABS/profile; get_models.sh using the prepared engine/native directories; obtain the gated GLP44 vector under your own account; launch run.sh with GLM53_COMMUNITY_PROFILE and GLM_WEIGHTLESS_VECTOR. Rebuilt extension hashes vary with build paths. Two 64 GiB cards, CUDA 13.3, Python 3.13 and the pinned Torch environment were used.' -summary: 'Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records low reasoning effort, selected per request with model GLM-5.3-Flash-low. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens.' + pinned_commit: db01c256a8003daf90c932ee25552526ede40f1f + install_note: '[base_version 0f685aa + row32 four-slot profile] Source build required; no image. Follow https://github.com/alexvnesta/glm53-flash-cmp170hx-tp/blob/db01c256a8003daf90c932ee25552526ede40f1f/profiles/community-four-active/README.md. Set up the pinned venv; apply_engine.sh; install gguf==0.17.1; profiles/community-four-active/build.sh /ABS/profile; get_models.sh using the prepared engine/native directories; obtain the gated GLP44 vector under your own account; launch run.sh with GLM53_COMMUNITY_PROFILE and GLM_WEIGHTLESS_VECTOR. Rebuilt extension hashes vary with build paths. Two 64 GiB cards, CUDA 13.3, Python 3.13 and the pinned Torch environment were used.' +summary: 'Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records low reasoning effort, selected per request with model GLM-5.3-Flash-low. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).' license: CC-BY-4.0 model: name: GLM-5.3-Flash-EXL3-3.05bpw @@ -33,7 +33,7 @@ config: launch: - ./profiles/community-four-active/run.sh active_kv: 'GPU-resident Q8 pool 655360 shared tokens, 392960 cap including output per request; no active RAM paging' - report_url: 'https://github.com/alexvnesta/glm53-flash-cmp170hx-tp/blob/fa0adf1899015f692ddce30391ee7e740a0910c9/docs/COMMUNITY_BENCHMARK_2026-10-17.md' + report_url: 'https://github.com/alexvnesta/glm53-flash-cmp170hx-tp/blob/db01c256a8003daf90c932ee25552526ede40f1f/docs/COMMUNITY_BENCHMARK_2026-10-17.md' weightless: alpha: 2 layers: 1-44 @@ -51,11 +51,55 @@ config: reasoning_effort: low performance: claimed_by: submitter - metrics: - tok_s: 41.721784744869 - notes: 'Headline tok_s: one agent, short prompt (78 input tokens). Community prose workload, low effort, temperature 0, 30 s steady-state windows, one run per cell, output cap 2048, immediate backfill and ignore-EOS. All rates are whole-rig. Points use actual input tokens; the run files keep the nominal targets. Prefill not measured. Not comparable with structured-task numbers for the same fork, including the two-slot listing. Points at context_len 62: structured tasks (arrays, a short function, a JSON copy), glm53 four-slot trial 2026-10-16; self-reported, no run file, not comparable with the prose points. The attached run file measures prose only. https://github.com/alexvnesta/glm53-flash-cmp170hx-tp/blob/fa0adf1899015f692ddce30391ee7e740a0910c9/docs/FOUR_ACTIVE_2026-10-16.md' - points: { } - context_len: 78 + tok_s: 41.721784744869 + context_len: 78 + notes: "Headline tok_s: one agent, short prompt (78 input tokens). Community prose workload, low effort, temperature 0, 30 s steady-state windows, one run per cell, output cap 2048, immediate backfill and ignore-EOS. All rates are whole-rig. Points use actual input tokens; the run files keep the nominal targets. Prefill not measured. Not comparable with structured-task numbers for the same fork, including the two-slot listing. Points at context_len 62: structured tasks (arrays, a short function, a JSON copy), glm53 four-slot trial 2026-10-16; self-reported, no run file, not comparable with the prose points. The attached run file measures prose only. https://github.com/alexvnesta/glm53-flash-cmp170hx-tp/blob/fa0adf1899015f692ddce30391ee7e740a0910c9/docs/FOUR_ACTIVE_2026-10-16.md [t_71325891: 9 aggregate points added from the author's own COMMUNITY_BENCHMARK_2026-10-17.md tables at fa0adf18 — previously this revision carried tok_s in metrics with an empty points list. Numbers are the author's; no measurement was performed by us.]" + points: + - + context_len: 78 + concurrency: 1 + decode_tok_s: 41.72 + notes: 'Low effort; 1 submitting agent(s); nominal context 0 (short prompt); aggregate whole-rig tok/s from exact native completion counts over a 30 s steady-state window (temperature 0, output cap 2048, ignore-EOS); actual input 78 tokens. Source: alexvnesta/glm53-flash-cmp170hx-tp docs/COMMUNITY_BENCHMARK_2026-10-17.md @ fa0adf18, "Short prompt" table.' + - + context_len: 78 + concurrency: 2 + decode_tok_s: 62.83 + notes: 'Low effort; 2 submitting agent(s); nominal context 0 (short prompt); aggregate whole-rig tok/s from exact native completion counts over a 30 s steady-state window (temperature 0, output cap 2048, ignore-EOS); actual input 78 tokens. Source: alexvnesta/glm53-flash-cmp170hx-tp docs/COMMUNITY_BENCHMARK_2026-10-17.md @ fa0adf18, "Short prompt" table.' + - + context_len: 78 + concurrency: 4 + decode_tok_s: 82.7 + notes: 'Low effort; 4 submitting agent(s); nominal context 0 (short prompt); aggregate whole-rig tok/s from exact native completion counts over a 30 s steady-state window (temperature 0, output cap 2048, ignore-EOS); actual input 78 tokens. Source: alexvnesta/glm53-flash-cmp170hx-tp docs/COMMUNITY_BENCHMARK_2026-10-17.md @ fa0adf18, "Short prompt" table.' + - + context_len: 32319 + concurrency: 1 + decode_tok_s: 47.6 + notes: 'Low effort; 1 submitting agent(s); nominal context 32768; aggregate whole-rig tok/s from exact native completion counts over a 30 s steady-state window (temperature 0, output cap 2048, ignore-EOS); actual input 32319 tokens. Source: alexvnesta/glm53-flash-cmp170hx-tp docs/COMMUNITY_BENCHMARK_2026-10-17.md @ fa0adf18, "32K nominal input context" table.' + - + context_len: 32319 + concurrency: 2 + decode_tok_s: 44.73 + notes: 'Low effort; 2 submitting agent(s); nominal context 32768; aggregate whole-rig tok/s from exact native completion counts over a 30 s steady-state window (temperature 0, output cap 2048, ignore-EOS); actual input 32319 tokens. Source: alexvnesta/glm53-flash-cmp170hx-tp docs/COMMUNITY_BENCHMARK_2026-10-17.md @ fa0adf18, "32K nominal input context" table.' + - + context_len: 32319 + concurrency: 4 + decode_tok_s: 68.18 + notes: 'Low effort; 4 submitting agent(s); nominal context 32768; aggregate whole-rig tok/s from exact native completion counts over a 30 s steady-state window (temperature 0, output cap 2048, ignore-EOS); actual input 32319 tokens. Source: alexvnesta/glm53-flash-cmp170hx-tp docs/COMMUNITY_BENCHMARK_2026-10-17.md @ fa0adf18, "32K nominal input context" table.' + - + context_len: 128886 + concurrency: 1 + decode_tok_s: 43.37 + notes: 'Low effort; 1 submitting agent(s); nominal context 131072; aggregate whole-rig tok/s from exact native completion counts over a 30 s steady-state window (temperature 0, output cap 2048, ignore-EOS); actual input 128886 tokens. Source: alexvnesta/glm53-flash-cmp170hx-tp docs/COMMUNITY_BENCHMARK_2026-10-17.md @ fa0adf18, "128K nominal input context" table.' + - + context_len: 128886 + concurrency: 2 + decode_tok_s: 43.38 + notes: 'Low effort; 2 submitting agent(s); nominal context 131072; aggregate whole-rig tok/s from exact native completion counts over a 30 s steady-state window (temperature 0, output cap 2048, ignore-EOS); actual input 128886 tokens. Source: alexvnesta/glm53-flash-cmp170hx-tp docs/COMMUNITY_BENCHMARK_2026-10-17.md @ fa0adf18, "128K nominal input context" table.' + - + context_len: 128886 + concurrency: 4 + decode_tok_s: 56.09 + notes: 'Low effort; 4 submitting agent(s); nominal context 131072; aggregate whole-rig tok/s from exact native completion counts over a 30 s steady-state window (temperature 0, output cap 2048, ignore-EOS); actual input 128886 tokens. Source: alexvnesta/glm53-flash-cmp170hx-tp docs/COMMUNITY_BENCHMARK_2026-10-17.md @ fa0adf18, "128K nominal input context" table.' artifacts: - kind: model_weights