GLM-5.3-Flash on 4x RTX PRO 6000 Blackwell \x{2014} TP4 MTP3

community Community submission — verify on your own hardware.

recipe by local-inference-lab↗ · listed by localai-recipes

revision r2 · commit 6dd7e5b0fb · updated 10 d ago · license CC-BY-4.0

Community image TP4 serving recipe: MTP3 draft, B12X MoE, FP8 KV, 1M context. Saved five-window medians (Max-Q +6000), the reference the Karmic Kraken report compares against.

Launch command

docker run -d --init --network host --ipc host -v jovian-judgement-r33-runtime-cache:/cache -v jovian-judgement-huggingface-cache:/root/.cache/huggingface -e MODEL=local-inference-lab/GLM-5.3-Flash-NVFP4 -e CACHE_MODE=vram -e KV_CACHE_QUANT=fp8_ds_mla -e TP=4 -e DCP=1 -e PORT=8000 -e MAX_MODEL_LEN=1048576 -e MAX_NUM_SEQS=32 -e MAX_NUM_BATCHED_TOKENS=4096 -e PREFILL_SCHEDULE_INTERVAL=1 -e PREFILL_COMPUTE_SHARE=0.4 -e GPU_MEMORY_UTILIZATION=0.93 -e SPECULATOR=mtp -e MTP_DEPTH=3 localinferencelab/vllm:jovian-judgement-community-20260911-r35

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-4x6000-jj-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-4x6000-jj-mtp3.json

Engine

vllm · source↗ · localinferencelab/vllm:jovian-judgement-community-20260911-r35↗

Model

GLM-5.3-Flash (nvfp4) · local-inference-lab/GLM-5.3-Flash-NVFP4↗
AA intelligence 42 (Artificial Analysis · measured)

Hardware

NVIDIA RTX PRO 6000 Blackwell 96GB ×4

Context

max context: 1M tokens

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
1	1	—	248.9 tok/s	—	reported
1	8	—	873.8 tok/s	—	reported
32,768	1	13514 tok/s	—	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Revision notes

Saved r35 numbers from the Karmic Kraken comparison table (same-box Max-Q +6000 conditions). Launch argv transcribed from the r35 community deployment page (docker run, TP4/DCP1, GPU_MEM 0.93, fp8_ds_mla KV).

Revision history

9a2bb537ad recipe: update glm-5-3-flash-4x6000-jj-mtp3 (r2: /max_context_len) 10 d ago diff
bfdf9daa3e history: re-squash corpus to a single root commit after the slug reconciliation (prior tip kept at refs/lar-archive/pre-single-slug-resquash) 25 d ago

Fork

Export

recipe.yaml (raw)