{"slug":"glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3","title":"GLM-5.3-Flash QAD (CSF) on 2x RTX PRO 6000 Blackwell \u2014 TP2/DCP2 MTP3","summary":"Local Inference Lab production two-card serving config: TP2/DCP2, 8 concurrent seqs, MTP3 on B12X, lossless-CSF NVFP4 MoE (MXFP8-CSF-QAD weights, 188.87 GB), online MXFP8 attention and shared experts, NVFP4 MTP experts, FP8 KV, host embeddings, MXFP8 vision, 1,048,576 context at 0.985 GPU memory. Reported KV pool 2,053,877 tokens at 0.984 memory with the 4096-token batch budget; guild-benchmarked at 207/488/672 tok/s at 1/4/8 users, about 68% of the 4-card TP4 reference on the same checkpoint.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"379e6111fef42f62bf472d1b559f4514dd06c1b6","content_hash":"fc89c71f949991e16656968e86cdcababa686f4a","created_at":"2026-10-20 15:34:06.136201-06","notes":"Ingested by lar-scout (kanban t_1a24b45b) from the local-inference-lab #daily-summaries 2026-10-04T08:25Z summary of the 10-03 production config. Launch argv is the guild env summary expressed with this image's documented env knobs (SPECULATOR=mtp + MTP_DEPTH=3 verified from the LIL launcher's own release posts R29-R32; MTP_TOKENS in the scout card was a reconstruction error). ENGINE verification: image tag karmic-kraken-beta-20261003-a717f13e26c631e3 confirmed present on ghcr (matches beta commit a717f13e in the summary). Weights artifact reconciled: guild serves local-inference-lab/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD (main @ fd660d51d1fc..., 188.87 GB); the card's revision pin cfd47bd7680e... is the head of the base repo's mtp-bf16 branch (verified) and is kept as upstream lineage, not the weights pin. Hardware: guild post says 2x RTX PRO 6000 WS 96GB; site carries only Max-Q variants of this card, so a plain WS 2x profile is created at ingest. gpu-memory-utilization 0.985 is the guild production value as reported (existing 4x recipes use 0.93)."},"engine_source":{"type":"docker_image","git_url":"https://github.com/local-inference-lab/blackwell-llm-docker","base_engine":"vllm","pip_package":null,"base_version":"karmic-kraken beta 20261003 (LIL community image)","image_digest":"ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261003-a717f13e26c631e3","install_note":null,"pinned_commit":null},"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/blackwell-llm-docker"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["docker","run","-d","--init","--network","host","--ipc","host","-v","glm53-qad-2x6000-runtime-cache:/cache","-v","glm53-qad-2x6000-hf-cache:/root/.cache/huggingface","-e","MODEL=local-inference-lab/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD","-e","CACHE_MODE=vram","-e","KV_CACHE_QUANT=fp8_ds_mla","-e","TP=2","-e","DCP=2","-e","PORT=8000","-e","MAX_MODEL_LEN=1048576","-e","MAX_NUM_SEQS=8","-e","MAX_NUM_BATCHED_TOKENS=4096","-e","GPU_MEMORY_UTILIZATION=0.985","-e","SPECULATOR=mtp","-e","MTP_DEPTH=3","-e","HOST_EMBEDDINGS=1","-e","VISION=MXFP8","ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261003-a717f13e26c631e3"],"kv_pool_tokens":2053877,"max_context_len":1048576},"models":[{"slug":"glm-5-3-flash","name":"GLM-5.3-Flash","hf_repo":"local-inference-lab/GLM-5.3-Flash-NVFP4","quant":"nvfp4-csf-mxfp8-attn-qad"}],"hardware":[{"slug":"nvidia-rtx-pro-6000-blackwell-max-qx2","vendor":"nvidia","card":"RTX PRO 6000 Blackwell Max-Q","card_family":"rtx-pro-6000-blackwell","vram_gb":96,"device_count":2}],"generation":"blackwell","tags":["1m-ctx","fp8-kv","mtp3","mtp-speculative","tensor-parallel","vision"],"tasks":["chat","code"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":"Index v4.3.2","aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Claimed by local-inference-lab daily summaries (2026-10-02/03): TP2 207/488/672 tok/s at 1/4/8 users on the same QAD checkpoint as the TP4 reference (306/694/962), about 68% of four-card speed; MMLU-Pro 168/200 (200q, p=1.0 vs TP4) and 82.6% (1000q); salted-needle retrieval pass at a 1,040,355-token prompt (249 s cold); KV pool 2,053,877 tokens at 0.984 memory with batch 4096; step-5500 QAD weight update on beta a717f13e measured +5% MTP-normalized decode and +25% with MTP.\", \"tok_s\": 207, \"points\": [{\"notes\": \"Claimed TP2 C1 decode (10-02 guild summary)\", \"concurrency\": 1, \"context_len\": 1, \"decode_tok_s\": 207}, {\"notes\": \"Claimed TP2 aggregate C4\", \"concurrency\": 4, \"context_len\": 1, \"decode_tok_s\": 488}, {\"notes\": \"Claimed TP2 aggregate C8\", \"concurrency\": 8, \"context_len\": 1, \"decode_tok_s\": 672}, {\"notes\": \"Salted-needle retrieval pass at 1,040,355-token prompt, both facts retrieved in 249 s cold\", \"context_len\": 1048576}]}"}],"perf_points":[{"id":1009,"context_len":1,"prefill_tok_s":null,"decode_tok_s":207,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Claimed TP2 C1 decode (10-02 guild summary)"},{"id":1010,"context_len":1,"prefill_tok_s":null,"decode_tok_s":488,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"Claimed TP2 aggregate C4"},{"id":1011,"context_len":1,"prefill_tok_s":null,"decode_tok_s":672,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Claimed TP2 aggregate C8"},{"id":1012,"context_len":1048576,"prefill_tok_s":null,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"Salted-needle retrieval pass at 1,040,355-token prompt, both facts retrieved in 249 s cold"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/local-inference-lab/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD","pin":"fd660d51d1fc3caae26a4bf31b7451475bbb9bdc"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3.json","alternate":"/r/glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:06.136201-06","created_at":"2026-10-06 00:19:40.41371-06","speeds":{"decode_tok_s":207,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}