localairecipes launch configs for local inference

GLM-5.3-Flash exlr8 EXL3 experts on DGX Spark TP1-4

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by kindlingai↗, nvidia↗ · listed by localai-recipes

revision r4 · commit 2ddd4d836d · updated 21 h ago · license MIT

chat code vision

GLM-5.3-Flash exlr8: MoE layers are EXL3-trellis units decoded by the custom exlr8 MoE kernel; dense linears stay FP8/NVFP4. One release carries three expert widths (k2/k3/k4 = 2/3/4 bpw), chosen at boot via VLLM_EXLR8_WIDTHS. Reference topology TP=4 on four GB10 boxes; TP=1 serves the whole model (~87 GiB) on one 128 GB DGX Spark / ASUS GX10 via tp1.yaml plus a distinct MENTAT_GROUP. TP=2 and TP=3 also measured. 2026-10-19, clocks TP-locked at 2200 MHz: decode code 128.0 tok/s, prose 72.5 at TP=4; 53.0 / 29.5 at TP=1; prefill 4,393 tok/s at 8K, 4,190 at 32K. Trade vs the shipping NVFP4 build: decode +3-8%, NLL equal or better, KV pool ~+15%, prefill 7.7% lower at Second 1x-Spark GLM-5.3-Flash lane: not the VQ Q4_K GGUF lane (/r/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn), and not a fork of the 2x ATLAS build (/r/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m) whose NVFP4 Flash base this stack overlays.

Launch command

'sudo env MODELS_ROOT=/mnt/nfs/models ./glm53f up -d' 'sudo env TP=2 VLLM_EXLR8_WIDTHS=k3 ./glm53f up -d' 'sudo env TP=1 VLLM_EXLR8_WIDTHS=k2 MENTAT_GROUP=glm53-solo ./glm53f -f experimental/compose/tp1.yaml up -d' 'sudo env TP=3 VLLM_EXLR8_WIDTHS=k4 MODEL_DIR=/models/nas/glm-5.3-flash/exlr8-release-tp3 ./glm53f -f experimental/compose/tp3.yaml up -d'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4.json

Engine

Install note: Source build: docker build -f image/Dockerfile . The overlay sets --quantization exlr8 --load-format exlr8 over the NVFP4 Flash base image. The vllm engine tree arrives via the repo symlink vllm -> .submodules/spark-agent/vllm (submodule mmastrac/spark-agent); the fork revision a2e163cb05 is no longer resolvable in any public repo.

Model

Hardware

Context

  • max context: 512K tokens

KV pool

  • 6,459,582 tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
8,192—4393 tok/s——reported
8,192—2609 tok/s——reported
32,768—4190 tok/s——reported
32,768—2687 tok/s——reported
524,2881—128 tok/s—reported
532,4801—72.5 tok/s—reported
540,6721—105.8 tok/s—reported
548,8641—57.8 tok/s—reported
557,0561—84.2 tok/s—reported
565,2481—50.1 tok/s—reported
573,4401—53 tok/s—reported
581,6321—29.5 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesHeadline 53.0 tok/s = TP=1 single-box k2 code decode, the topology that makes this a distinct single-box lane. Transcribed matrix (README.md, author campaign 2026-10-19, clocks TP-locked 2200 MHz), decode code / prose: TP=1 53.0 / 29.5, TP=2 84.2 / 50.1, TP=3 105.8 / 57.8, TP=4 128.0 / 72.5. Cold prefill 8K / 32K: TP=1 495 / 523, TP=2 2,609 / 2,687, TP=3 3,002 / 3,391, TP=4 4,393 / 4,190. Figures are the repo's own receipts, not re-measured here.

Downloads

Revision notes

Upstream pinned at main @ 6f5ede7f2d2675fadbb9bb216a34b28386547c27 (re-read live 2026-10-23). Figures from README.md (author campaign 2026-10-19, clocks TP-locked at 2200 MHz); stack and knob detail from NOTES.md, KNOBS.md, experimental/compose/exlr8.yaml and tp1.yaml at that sha. corrections_to_card: (1) card called this the first single-DGX-Spark GLM-5.3-Flash recipe; that is wrong -- glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn already carries a 1x Spark lane, so this is a SECOND single-box lane on a different weight path (EXL3-trellis experts vs the VQ Q4_K GGUF lane); cross-linked to that recipe and to glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m. (2) the card's perf table put 2,609 in the TP=2 decode-prose column; README gives 2,609 as TP=2 cold prefill at 8K and 50.1 tok/s for TP=2 decode prose. (3) card asserted MIT for the recipe repo; GitHub reports no license for it and there is no LICENSE file at the pinned sha -- MIT is asserted here for the redistributed expert units and the base NVFP4 weights, not the recipe repo. (4) no upstream release tag exists; the reference is the pinned commit.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)