GLM-5.3-Flash QAD (CSF) on 2x RTX PRO 6000 Blackwell — TP2/DCP2 MTP3

community Community submission — verify on your own hardware.

recipe by local-inference-lab↗ · listed by localai-recipes

revision r2 · commit 379e6111fe · updated 10 d ago · license CC-BY-4.0

code chat fp8-kv 1m-ctx mtp-speculative tensor-parallel vision mtp3

Local Inference Lab production two-card serving config: TP2/DCP2, 8 concurrent seqs, MTP3 on B12X, lossless-CSF NVFP4 MoE (MXFP8-CSF-QAD weights, 188.87 GB), online MXFP8 attention and shared experts, NVFP4 MTP experts, FP8 KV, host embeddings, MXFP8 vision, 1,048,576 context at 0.985 GPU memory. Reported KV pool 2,053,877 tokens at 0.984 memory with the 4096-token batch budget; guild-benchmarked at 207/488/672 tok/s at 1/4/8 users, about 68% of the 4-card TP4 reference on the same checkpoint.

Launch command

docker run -d --init --network host --ipc host -v glm53-qad-2x6000-runtime-cache:/cache -v glm53-qad-2x6000-hf-cache:/root/.cache/huggingface -e MODEL=local-inference-lab/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD -e CACHE_MODE=vram -e KV_CACHE_QUANT=fp8_ds_mla -e TP=2 -e DCP=2 -e PORT=8000 -e MAX_MODEL_LEN=1048576 -e MAX_NUM_SEQS=8 -e MAX_NUM_BATCHED_TOKENS=4096 -e GPU_MEMORY_UTILIZATION=0.985 -e SPECULATOR=mtp -e MTP_DEPTH=3 -e HOST_EMBEDDINGS=1 -e VISION=MXFP8 ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261003-a717f13e26c631e3

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3.json

Engine

vllm · source↗ · ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261003-a717f13e26c631e3↗

Model

GLM-5.3-Flash (nvfp4-csf-mxfp8-attn-qad) · local-inference-lab/GLM-5.3-Flash-NVFP4↗
AA intelligence 42 (Artificial Analysis · measured)

Hardware

NVIDIA RTX PRO 6000 Blackwell 96GB ×2

Context

max context: 1M tokens

KV pool

2,053,877 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
1	1	—	207 tok/s	—	reported
1	4	—	488 tok/s	—	reported
1	8	—	672 tok/s	—	reported
1,048,576	—	—	—	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Claimed by local-inference-lab daily summaries (2026-10-02/03): TP2 207/488/672 tok/s at 1/4/8 users on the same QAD checkpoint as the TP4 reference (306/694/962), about 68% of four-card speed; MMLU-Pro 168/200 (200q, p=1.0 vs TP4) and 82.6% (1000q); salted-needle retrieval pass at a 1,040,355-token prompt (249 s cold); KV pool 2,053,877 tokens at 0.984 memory with batch 4096; step-5500 QAD weight update on beta a717f13e measured +5% MTP-normalized decode and +25% with MTP.

notes

Claimed by local-inference-lab daily summaries (2026-10-02/03): TP2 207/488/672 tok/s at 1/4/8 users on the same QAD checkpoint as the TP4 reference (306/694/962), about 68% of four-card speed; MMLU-Pro 168/200 (200q, p=1.0 vs TP4) and 82.6% (1000q); salted-needle retrieval pass at a 1,040,355-token prompt (249 s cold); KV pool 2,053,877 tokens at 0.984 memory with batch 4096; step-5500 QAD weight update on beta a717f13e measured +5% MTP-normalized decode and +25% with MTP.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/local-inference-lab/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD
fd660d51d1fc3caae26a4bf31b7451475bbb9bdc

Revision notes

Ingested by lar-scout (kanban t_1a24b45b) from the local-inference-lab #daily-summaries 2026-10-04T08:25Z summary of the 10-03 production config. Launch argv is the guild env summary expressed with this image's documented env knobs (SPECULATOR=mtp + MTP_DEPTH=3 verified from the LIL launcher's own release posts R29-R32; MTP_TOKENS in the scout card was a reconstruction error). ENGINE verification: image tag karmic-kraken-beta-20261003-a717f13e26c631e3 confirmed present on ghcr (matches beta commit a717f13e in the summary). Weights artifact reconciled: guild serves local-inference-lab/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD (main @ fd660d51d1fc..., 188.87 GB); the card's revision pin cfd47bd7680e... is the head of the base repo's mtp-bf16 branch (verified) and is kept as upstream lineage, not the weights pin. Hardware: guild post says 2x RTX PRO 6000 WS 96GB; site carries only Max-Q variants of this card, so a plain WS 2x profile is created at ingest. gpu-memory-utilization 0.985 is the guild production value as reported (existing 4x recipes use 0.93).

Revision history

9756a84667 recipe: update glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3 (r2: /max_context_len, /kv_pool_tokens) 10 d ago diff
49a0177355 recipe: create glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3 25 d ago

Fork

Export

recipe.yaml (raw)