localairecipes launch configs for local inference

GLM-5.3-Flash max effort on 2x CMP170HX EXL3 TP2 four slots

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)↗, Flun (original 2-card layer-split server + recipe this fork starts from)↗, turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)↗, incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)↗, 0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)↗, Morrowmake (content-verified P2P check idea + benchmark prompt style)↗, bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)↗, Z.ai (GLM-5.3-Flash base model)↗, msuiche (Weightless GLP-44 vector and specification)↗ · listed by alex.v.nesta

revision r5 · commit 663174ab27 · updated 19 h ago · license CC-BY-4.0

chat code vision tensor-parallel dflash2 exl3 q8-kv concurrent-sessions

Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records max reasoning effort, selected per request with model GLM-5.3-Flash-max. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).

Launch command

./profiles/community-four-active/run.sh

runtime

notes: 'RECURRENT_STATE_DTYPE sets the recurrent-state storage dtype: fp32 default (launch adds --recurrent_state_fp32); fp16 opts into the upstream FP16 recurrent storage patch (0005-upstream-fp16-recurrent-state); accepts only fp16|fp32, anything else exits 2 at launch.'

active_kv

'GPU-resident Q8 pool 655360 shared tokens, 392960 cap including output per request; no active RAM paging'

report_url

'https://github.com/alexvnesta/glm53-flash-cmp170hx-tp/blob/db01c256a8003daf90c932ee25552526ede40f1f/docs/COMMUNITY_BENCHMARK_2026-10-17.md'

weightless

alpha: 2
layers: 1-44
vector_sha256: 0ccce6b748f87da81505ff2bc3ca82429110940b6c0def60063d304785ccc00c
vector_revision: ef85b0169fd25a3c0223eedddee124cf6b1580dc

environment

GLM_WEIGHTLESS_VECTOR: /absolute/path/to/GLM-5.3-Flash-abliterated-cyber-GLP-44-L1-44-a2.gguf
RECURRENT_STATE_DTYPE: fp32
GLM53_COMMUNITY_PROFILE: /absolute/path/to/community-profile
GLM_PERSISTENT_CACHE_DIR: /absolute/path/to/session-cache
GLM_PERSISTENT_CACHE_GIB: '50'
GLM_PERSISTENT_CACHE_MIN_FREE_GIB: '16'

client_model

GLM-5.3-Flash-max

native_layout

'4 active sequences, row32 verifier, K7 DFlash2 plus hybrid15, native broadcast/gather with NCCL reductions'

request_defaults

reasoning_effort: max

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json

Engine

Install note: Source build required; no image. Follow https://github.com/alexvnesta/glm53-flash-cmp170hx-tp/blob/db01c256a8003daf90c932ee25552526ede40f1f/profiles/community-four-active/README.md. Set up the pinned venv; apply_engine.sh; install gguf==0.17.1; profiles/community-four-active/build.sh /ABS/profile; get_models.sh using the prepared engine/native directories; obtain the gated GLP44 vector under your own account; launch run.sh with GLM53_COMMUNITY_PROFILE and GLM_WEIGHTLESS_VECTOR. Rebuilt extension hashes vary with build paths. Two 64 GiB cards, CUDA 13.3, Python 3.13 and the pinned Torch environment were used.

Model

Hardware

KV pool

  • 640K tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
781—41.5 tok/s—reported
781—41.5 tok/s331.8 msreported view run file variant: GLM-5.3-Flash
782—65.4 tok/s—reported
784—101.2 tok/s—reported
32,3191—46.9 tok/s—reported
32,3192—46.4 tok/s—reported
32,3194—64.1 tok/s—reported
128,8781—49.8 tok/s—reported
128,8782—51.8 tok/s—reported
128,8784—68 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesHeadline tok_s: one agent, short prompt (78 input tokens). Community prose workload, max effort, temperature 0, 30 s steady-state windows, one run per cell, output cap 2048, immediate backfill and ignore-EOS. All rates are whole-rig. Points use actual input tokens; the run files keep the nominal targets. Prefill not measured. Not comparable with structured-task numbers for the same fork, including the two-slot listing.

Downloads

Revision notes

Four-slot configuration of the fork behind the site-owned two-slot listing; this prose workload is not comparable with that listing's numbers. Exact measured source profile is packaged under profiles/community-four-active. Engine stays GPU-resident; Weightless steering vector enabled. Effort is selected per API request via the documented model alias. Inference speed does not establish accuracy or intelligence; no local AA score is asserted. PROD SYNC (kanban t_3ad93f24, owner go Casey 2026-10-28): replay of QC-approved preprod revision r3 content_hash 0b2f1599d710d60751862fce4aa071073828d53a (parent QC t_4c7cb164 APPROVE); prod pre-state content_hash a05d0aa17b178237398ef71a96d99829c5659f82 captured for rollback; perf points carried unchanged (prod lineage).

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)