{"slug":"glm-5-3-flash-4x6000-kk-mtp3","title":"GLM-5.3-Flash on 4x RTX PRO 6000 Blackwell \u2014 Karmic Kraken TP4 MTP3","summary":"Karmic Kraken TP4 serving recipe: MTP3 draft, b12x MoE/attention, FP8 KV, 1M context. Five-window benchmark medians from the rtx6kpro qualification run.","license":"CC-BY-4.0","tier":"community","beta":true,"revision":{"no":3,"commit_sha":"606fcc125ac02bd004f6dec43cd3b6460b3747bf","content_hash":"f26ef8300e00a402ee4f1e8f84cfd1313c243538","created_at":"2026-10-20 15:34:03.65139-06","notes":"Benchmark-qualified configuration from local-inference-lab/rtx6kpro (Karmic Kraken serving report). Five warmed 30-second windows per cell on RTX PRO 6000 Blackwell Max-Q; decode ctx-0 temp 1. Launch argv transcribed verbatim from the repo's unified Docker guide (same composition). Measured: C1 281.3 tok/s, C8 897.6 tok/s, 32K prefill 13816 tok/s."},"engine_source":{"type":"docker_image","git_url":"https://github.com/local-inference-lab/vllm","base_engine":"vllm","base_version":"karmic-kraken-beta (vLLM cfdecfaa4)","image_digest":"ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20260922-3221ccacf71002ea","pinned_commit":"cfdecfaa4739ee3141c9745997cd8a83f31745a4"},"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["serve","local-inference-lab/GLM-5.3-Flash-NVFP4","--additional-config","'{\"glm53_kda_decode_backend\":\"auto\",\"kda_prefill_backend\":\"b12x\"}'","--attention-backend","B12X","--block-size","256","--compilation-config","'{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}'","--cp-kv-cache-interleave-size","4","--cudagraph-capture-sizes","1","2","4","8","16","32","40","48","64","96","128","192","256","--dcp-kv-cache-interleave-size","4","--decode-context-parallel-size","1","--default-chat-template-kwargs","'{\"reasoning_effort\":\"high\",\"clear_thinking\":false}'","--dtype","bfloat16","--enable-auto-tool-choice","--enable-chunked-prefill","--no-enable-flashinfer-autotune","--enable-prefix-caching","--enable-prompt-tokens-details","--gpu-memory-utilization","0.93","--host","0.0.0.0","--kv-cache-dtype","fp8","--linear-backend","b12x","--load-format","instanttensor","--mamba-cache-mode","align","--max-cudagraph-capture-size","256","--max-model-len","1048576","--max-num-batched-tokens","4096","--max-num-seqs","32","--max-parallel-prefills","1","--moe-backend","b12x","--override-generation-config","'{\"temperature\":1.0,\"top_p\":0.95}'","--pipeline-parallel-size","1","--port","8000","--prefill-compute-share","0.4","--prefill-schedule-interval","1","--quantization","modelopt_mixed","--reasoning-parser","glm45","--recurrent-checkpoint-policy","request_boundaries","--served-model-name","GLM-5.3-Flash-NVFP4","--speculative-config","'{\"method\":\"mtp\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"standard\",\"moe_backend\":\"marlin\",\"attention_backend\":\"B12X\",\"num_speculative_tokens\":3}'","--tensor-parallel-size","4","--tool-call-parser","glm47"],"max_context_len":1048576},"models":[],"hardware":[],"generation":null,"tags":[],"tasks":[],"aa":null,"performance":[{"claimed_by":"submitter","metrics":"{\"tok_s\": 281.3}"},{"claimed_by":"submitter","metrics":"{\"notes\": \"Benchmark-qualified medians from local-inference-lab/rtx6kpro (five warmed 30-s windows).\", \"tok_s\": 281.3, \"points\": [{\"notes\": \"Karmic Kraken 5-window median, ctx-0 decode, temp 1 (verifier 112.1 steps/s)\", \"concurrency\": 1, \"context_len\": 1, \"decode_tok_s\": 281.3}, {\"notes\": \"Karmic Kraken 5-window median, ctx-0 decode, temp 1 (verifier 364.8 steps/s)\", \"concurrency\": 8, \"context_len\": 1, \"decode_tok_s\": 897.6}], \"context_len\": 1048576}"},{"claimed_by":"submitter","metrics":"[]"}],"perf_points":[{"id":994,"context_len":1,"prefill_tok_s":null,"decode_tok_s":281.30000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Karmic Kraken 5-window median, ctx-0 decode, temp 1 (verifier 112.1 steps/s)"},{"id":995,"context_len":1,"prefill_tok_s":null,"decode_tok_s":897.6000000000000227373675443232059478759765625,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Karmic Kraken 5-window median, ctx-0 decode, temp 1 (verifier 364.8 steps/s)"},{"id":996,"context_len":32768,"prefill_tok_s":13816,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Karmic Kraken 5-window median, unique uncached nominal-32K requests, client TTFT"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-4x6000-kk-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-4x6000-kk-mtp3.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[],"links":{"self":"/api/v1/recipes/glm-5-3-flash-4x6000-kk-mtp3.json","alternate":"/r/glm-5-3-flash-4x6000-kk-mtp3","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:03.65139-06","created_at":"2026-09-28 15:47:09.028845-06"}