{"slug":"glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2","title":"GLM-5.3-Flash NVFP4 4x DGX Spark switchless ring TP4 DFlash2","summary":"Recipe by Alex Ellis / OpenFaaS Ltd. GLM-5.3-Flash (320B-A18B, NVFP4) served tensor-parallel 4 across four DGX Spark (GB10, sm_121) nodes cabled as a closed switchless RoCE ring: two rails per node wired to two neighbours, the non-adjacent hop relayed through a neighbour \u2014 no switch on the data path or the procurement path. Collectives use canonical switchless-nccl v0.0.1 (NCCL 2.30.7 + two Apache-2.0 switchless patches + OpenFaaS hardening patch), LD_PRELOADed with pinned library sha256 and NCCL_ALGO=Ring; not SparkRing/SIRCL, distinct from the site's switched 4x Spark SGLang lane. vLLM on the public tonyd2wild sm121-v11-dflash2 image: TP4, marlin MoE, FP8 E4M3 KV at 12 GiB per rank (pool 1,576,246 tokens = 6.01x window), DFlash2 drafter k=7, glm47 tool + glm45 reasoning parsers, max-num-seqs 6, workers (ranks 3,2,1) headless first, then head rank 0 opens the single OpenAI-compatible endpoint on :8000. Same fabric also served GLM-5.2 (EXL3, QuantTrio) and DeepSeek-V4-Flash at TP4.","license":"MIT","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"3ec203d32cde9b249d3e88f040a8d0c79650c3db","content_hash":"9cf1d9bb8001b5e721793f15e201aee0b5916f6d","created_at":"2026-10-20 21:52:36.421715-06","notes":"Scout-discovered (lar-scout, card t_f36bc073): long-lived prominent repo (1566 stars, created 2026-08-29) with zero prior digests; ingested at live HEAD 9ad50c53 (card snapshot 07cbcb41 was stale). Transcribed at pinned sha: docs/recipe.md serve argv + env, scripts/rank-launcher.sh docker run + container flags, docs/nccl-build.md switchless-nccl v0.0.1 pins (archive b4a68638\u2026, lib 78cb8387\u2026). Card's nvidia/GLM-5.3-Flash-NVFP4 hf_repo guess corrected to LibertAIDAI/GLM-5.3-Flash-NVFP4 @ 9e0d74e3 (README fixed-weights table + RigMark appliance.model). Distinguish from knapcio t_ad3aebd3 switched 4x Spark SGLang lane: this is a switchless closed RoCE ring (two rails/node, non-adjacent hop relayed via neighbour) with patched canonical NCCL, not SparkRing/SIRCL. Performance from README tables + data/rigmark/glm53-nccl-v0.0.1-tp4-20260912.json at pinned sha (release-qualified 2026-09-12 run; image in that receipt was the local radixark build, launcher ships public tonyd2wild image)."},"engine_source":{"type":"docker_image","git_url":"https://github.com/alexellis/glm-5.3-flash-4x-dgx-spark-switchless","base_engine":"vllm","pip_package":null,"base_version":"0.1.dev20051+g487ecf187","image_digest":"ghcr.io/tonyd2wild/vllm-glm53-flash:sm121-v11-dflash2","install_note":"Image digested as ghcr.io/tonyd2wild/vllm-glm53-flash@sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6; requires switchless-nccl v0.0.1 (NCCL 2.30.7, lib sha256 78cb83871792ec57d763d142e4cae26fc754ae284bcc81dcb2a7d50e17d4fa57) installed on every node and LD_PRELOAD + VLLM_NCCL_SO_PATH set; launch order is workers (ranks 3,2,1 headless) then head rank 0.","pinned_commit":"9ad50c53840ec734f6df0379218e8c38e0b02c47"},"credit":[{"name":"Alex Ellis / OpenFaaS Ltd","url":"https://github.com/alexellis/glm-5.3-flash-4x-dgx-spark-switchless"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["docker run -d --name glm53_tp4 --restart no","--cap-add IPC_LOCK --ulimit memlock=-1:-1","--network host --ipc host --shm-size 32g --gpus all","--device /dev/infiniband:/dev/infiniband","-v /model:ro -v /draft:ro -v /opt/patched-nccl:ro -v /cache","-e LD_PRELOAD=/opt/patched-nccl/libnccl.so.2","-e VLLM_NCCL_SO_PATH=/opt/patched-nccl/libnccl.so.2","-e NCCL_SKIP_TREE_CONNECT=1 -e NCCL_SWITCHLESS_RING_ONLY=1","-e NCCL_SOCKET_IFNAME=<mgmt-if> -e GLOO_SOCKET_IFNAME=<mgmt-if> -e VLLM_HOST_IP=<mgmt-ip>","-e NCCL_NET=IB -e NCCL_IB_DISABLE=0 -e NCCL_IB_HCA=<pair-hca>,<cross-hca>","-e NCCL_IB_GID_INDEX=3 -e NCCL_IB_SUBNET_PREFIX_LEN=24 -e NCCL_IB_SUBNET_AWARE_ROUTING=1","-e NCCL_IB_MERGE_NICS=0","-e NCCL_ALGO=Ring -e NCCL_PROTO=LL,LL128,Simple -e NCCL_P2P_LEVEL=SYS","-e NCCL_MIN_NCHANNELS=4 -e NCCL_MAX_NCHANNELS=4 -e NCCL_CROSS_NIC=1 -e NCCL_CUMEM_ENABLE=0","-e NCCL_IGNORE_CPU_AFFINITY=1","-e VLLM_ONE_GPU_PER_NODE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True","-e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1","-e NODE_RANK=<0-3> -e MASTER_ADDR=<head-mgmt-ip>","-e TORCH_CUDA_ARCH_LIST=12.1a -e FLASHINFER_CUDA_ARCH_LIST=12.1a","ghcr.io/tonyd2wild/vllm-glm53-flash:sm121-v11-dflash2","/model","--served-model-name glm-5.3-flash --trust-remote-code","--tensor-parallel-size 4 --nnodes 4 --node-rank <0-3>","--master-addr <head-mgmt-ip> --master-port <29520>","--gpu-memory-utilization 0.85 --max-model-len 262144","--max-num-seqs 6 --block-size 2304 --moe-backend marlin","--limit-mm-per-prompt {\"image\":16}","--kv-cache-dtype fp8_e4m3 --kv-cache-memory 12884901888","--speculative-config {\"method\":\"dflash\",\"model\":\"/draft\",\"num_speculative_tokens\":7}","--tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45","--chat-template /opt/glm53/chat_template.jinja","--default-chat-template-kwargs {\"reasoning_effort\":\"max\"}","--distributed-executor-backend mp","--host 0.0.0.0 --port 8000 (rank 0) | --headless (ranks 1-3)"],"layouts":["tp"],"kv_pool_tokens":1576246,"max_context_len":262144,"storage_requirements":{"note":"Checkpoint ~182 GiB + drafter + JIT/torch.compile cache per node.","nvme_gb":190}},"models":[{"slug":"glm-5-3-flash-nvfp4","name":"GLM-5.3-Flash-NVFP4","hf_repo":"LibertAIDAI/GLM-5.3-Flash-NVFP4","quant":"nvfp4-w4a16-modelopt-0-45"}],"hardware":[{"slug":"dgx-spark-x4","vendor":"nvidia","card":"DGX Spark (GB10, unified memory)","card_family":"dgx-spark","vram_gb":128,"device_count":4}],"generation":"blackwell","tags":["agent-serving","dflash2","dgx-spark","fp8-kv","gb10","long-context","marlin-moe","nvfp4","roce","sm121","speculative","switchless","tensor-parallel","vision"],"tasks":["chat","code","vision"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"RigMark protocol 1.0.0 (repo d8353e93b274) release-receipt glm53-nccl-v0.0.1-tp4-20260912: completed code decode median 71.3 tok/s (5/5 gates), prose 31.2, structured ceiling 109.2; TTFT 361 ms code median. Real agentic daily-driver decode ~45 tok/s typical (up to ~100 warm). Cold 64K prefill 1965 tok/s; warm replay 36075 tok/s.\", \"tok_s\": 71.2999999999999971578290569595992565155029296875, \"points\": [{\"notes\": \"Concurrency-1 per-stream decode median 45.558 tok/s (RigMark c1 rounds, code workload).\", \"ttft_ms\": 352, \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 45.60000000000000142108547152020037174224853515625}, {\"notes\": \"Aggregate e2e median 72.456 tok/s at c2 (code workload); per-stream 42.2 median.\", \"concurrency\": 2, \"context_len\": 262144, \"decode_tok_s\": 72.5}, {\"notes\": \"Aggregate e2e median 110.814 tok/s at c4 (code workload); per-stream 30.2 median.\", \"concurrency\": 4, \"context_len\": 262144, \"decode_tok_s\": 110.7999999999999971578290569595992565155029296875}, {\"notes\": \"Cold prefill median 1978 tok/s at 8192 (RigMark prefill receipts).\", \"ttft_ms\": 4141, \"context_len\": 8192, \"prefill_tok_s\": 1978.09999999999990905052982270717620849609375}, {\"notes\": \"Cold prefill median 1972 tok/s at 32768.\", \"ttft_ms\": 16620, \"context_len\": 32768, \"prefill_tok_s\": 1971.59999999999990905052982270717620849609375}, {\"notes\": \"Cold prefill median 1965 tok/s at 65536.\", \"ttft_ms\": 33358, \"context_len\": 65536, \"prefill_tok_s\": 1964.700000000000045474735088646411895751953125}, {\"notes\": \"Warm 64K replay median 36075 tok/s (prefill_depths 65536 warm_replay); single-stream decode 31.2 prose median for context.\", \"ttft_ms\": 1817, \"concurrency\": 1, \"context_len\": 65536, \"decode_tok_s\": 31.199999999999999289457264239899814128875732421875, \"prefill_tok_s\": 36074.9000000000014551915228366851806640625}, {\"notes\": \"Daily-driver agentic traffic ~45 tok/s typical; deepest single prompt served 122K tokens of the 262K window.\", \"concurrency\": 1, \"context_len\": 122880, \"decode_tok_s\": 45}], \"ttft_ms\": 361, \"context_len\": 262144}"}],"perf_points":[{"id":1618,"context_len":8192,"prefill_tok_s":1978.09999999999990905052982270717620849609375,"decode_tok_s":null,"ttft_ms":4141,"concurrency":null,"claimed_by":"submitter","notes":"Cold prefill median 1978 tok/s at 8192 (RigMark prefill receipts)."},{"id":1619,"context_len":32768,"prefill_tok_s":1971.59999999999990905052982270717620849609375,"decode_tok_s":null,"ttft_ms":16620,"concurrency":null,"claimed_by":"submitter","notes":"Cold prefill median 1972 tok/s at 32768."},{"id":1621,"context_len":65536,"prefill_tok_s":1964.700000000000045474735088646411895751953125,"decode_tok_s":null,"ttft_ms":33358,"concurrency":null,"claimed_by":"submitter","notes":"Cold prefill median 1965 tok/s at 65536."},{"id":1620,"context_len":65536,"prefill_tok_s":36074.9000000000014551915228366851806640625,"decode_tok_s":31.199999999999999289457264239899814128875732421875,"ttft_ms":1817,"concurrency":1,"claimed_by":"submitter","notes":"Warm 64K replay median 36075 tok/s (prefill_depths 65536 warm_replay); single-stream decode 31.2 prose median for context."},{"id":1622,"context_len":122880,"prefill_tok_s":null,"decode_tok_s":45,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Daily-driver agentic traffic ~45 tok/s typical; deepest single prompt served 122K tokens of the 262K window."},{"id":1623,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":45.60000000000000142108547152020037174224853515625,"ttft_ms":352,"concurrency":1,"claimed_by":"submitter","notes":"Concurrency-1 per-stream decode median 45.558 tok/s (RigMark c1 rounds, code workload)."},{"id":1624,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":72.5,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"Aggregate e2e median 72.456 tok/s at c2 (code workload); per-stream 42.2 median."},{"id":1625,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":110.7999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"Aggregate e2e median 110.814 tok/s at c4 (code workload); per-stream 30.2 median."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/LibertAIDAI/GLM-5.3-Flash-NVFP4","pin":"9e0d74e3cef17f634e84fb8e2223707e02616290"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2","pin":null},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://ghcr.io/tonyd2wild/vllm-glm53-flash@sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6","pin":"sm121-v11-dflash2"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/alexellis/switchless-nccl","pin":"v0.0.1"}],"links":{"self":"/api/v1/recipes/glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2.json","alternate":"/r/glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2","describedby":"/openapi.json"},"updated_at":"2026-10-20 21:52:36.421715-06","created_at":"2026-10-20 21:49:14.69659-06","speeds":{"decode_tok_s":45,"prefill_tok_s":1971.59999999999990905052982270717620849609375,"leaderboard":"/api/v1/leaderboard.json"}}