GLM-5.3-Flash NVFP4 4x DGX Spark switchless ring TP4 DFlash2

community Community submission — verify on your own hardware.

recipe by Alex Ellis / OpenFaaS Ltd↗ · listed by localai-recipes

revision r2 · commit 3ec203d32c · updated 10 d ago · license MIT

chat code vision fp8-kv tensor-parallel vision dflash2 long-context speculative dgx-spark gb10 sm121 roce agent-serving marlin-moe nvfp4 switchless

Recipe by Alex Ellis / OpenFaaS Ltd. GLM-5.3-Flash (320B-A18B, NVFP4) served tensor-parallel 4 across four DGX Spark (GB10, sm_121) nodes cabled as a closed switchless RoCE ring: two rails per node wired to two neighbours, the non-adjacent hop relayed through a neighbour — no switch on the data path or the procurement path. Collectives use canonical switchless-nccl v0.0.1 (NCCL 2.30.7 + two Apache-2.0 switchless patches + OpenFaaS hardening patch), LD_PRELOADed with pinned library sha256 and NCCL_ALGO=Ring; not SparkRing/SIRCL, distinct from the site's switched 4x Spark SGLang lane. vLLM on the public tonyd2wild sm121-v11-dflash2 image: TP4, marlin MoE, FP8 E4M3 KV at 12 GiB per rank (pool 1,576,246 tokens = 6.01x window), DFlash2 drafter k=7, glm47 tool + glm45 reasoning parsers, max-num-seqs 6, workers (ranks 3,2,1) headless first, then head rank 0 opens the single OpenAI-compatible endpoint on :8000. Same fabric also served GLM-5.2 (EXL3, QuantTrio) and DeepSeek-V4-Flash at TP4.

Launch command

'docker run -d --name glm53_tp4 --restart no' '--cap-add IPC_LOCK --ulimit memlock=-1:-1' '--network host --ipc host --shm-size 32g --gpus all' '--device /dev/infiniband:/dev/infiniband' '-v /model:ro -v /draft:ro -v /opt/patched-nccl:ro -v /cache' '-e LD_PRELOAD=/opt/patched-nccl/libnccl.so.2' '-e VLLM_NCCL_SO_PATH=/opt/patched-nccl/libnccl.so.2' '-e NCCL_SKIP_TREE_CONNECT=1 -e NCCL_SWITCHLESS_RING_ONLY=1' '-e NCCL_SOCKET_IFNAME=<mgmt-if> -e GLOO_SOCKET_IFNAME=<mgmt-if> -e VLLM_HOST_IP=<mgmt-ip>' '-e NCCL_NET=IB -e NCCL_IB_DISABLE=0 -e NCCL_IB_HCA=<pair-hca>,<cross-hca>' '-e NCCL_IB_GID_INDEX=3 -e NCCL_IB_SUBNET_PREFIX_LEN=24 -e NCCL_IB_SUBNET_AWARE_ROUTING=1' '-e NCCL_IB_MERGE_NICS=0' '-e NCCL_ALGO=Ring -e NCCL_PROTO=LL,LL128,Simple -e NCCL_P2P_LEVEL=SYS' '-e NCCL_MIN_NCHANNELS=4 -e NCCL_MAX_NCHANNELS=4 -e NCCL_CROSS_NIC=1 -e NCCL_CUMEM_ENABLE=0' '-e NCCL_IGNORE_CPU_AFFINITY=1' '-e VLLM_ONE_GPU_PER_NODE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True' '-e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1' '-e NODE_RANK=<0-3> -e MASTER_ADDR=<head-mgmt-ip>' '-e TORCH_CUDA_ARCH_LIST=12.1a -e FLASHINFER_CUDA_ARCH_LIST=12.1a' ghcr.io/tonyd2wild/vllm-glm53-flash:sm121-v11-dflash2 /model '--served-model-name glm-5.3-flash --trust-remote-code' '--tensor-parallel-size 4 --nnodes 4 --node-rank <0-3>' '--master-addr <head-mgmt-ip> --master-port <29520>' '--gpu-memory-utilization 0.85 --max-model-len 262144' '--max-num-seqs 6 --block-size 2304 --moe-backend marlin' '--limit-mm-per-prompt {"image":16}' '--kv-cache-dtype fp8_e4m3 --kv-cache-memory 12884901888' '--speculative-config {"method":"dflash","model":"/draft","num_speculative_tokens":7}' '--tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45' '--chat-template /opt/glm53/chat_template.jinja' '--default-chat-template-kwargs {"reasoning_effort":"max"}' '--distributed-executor-backend mp' '--host 0.0.0.0 --port 8000 (rank 0) | --headless (ranks 1-3)'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2.json

Engine

vllm · source↗ · 9ad50c53↗ · ghcr.io/tonyd2wild/vllm-glm53-flash:sm121-v11-dflash2↗

Install note: Image digested as ghcr.io/tonyd2wild/vllm-glm53-flash@sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6; requires switchless-nccl v0.0.1 (NCCL 2.30.7, lib sha256 78cb83871792ec57d763d142e4cae26fc754ae284bcc81dcb2a7d50e17d4fa57) installed on every node and LD_PRELOAD + VLLM_NCCL_SO_PATH set; launch order is workers (ranks 3,2,1 headless) then head rank 0.

Model

GLM-5.3-Flash-NVFP4 (nvfp4-w4a16-modelopt-0-45) · LibertAIDAI/GLM-5.3-Flash-NVFP4↗
AA intelligence 42 (Artificial Analysis · measured)

Hardware

NVIDIA DGX Spark 128GB ×4 (Blackwell)

Context

max context: 256K tokens

KV pool

1,576,246 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
8,192	—	1978.1 tok/s	—	4141 ms	reported
32,768	—	1971.6 tok/s	—	16620 ms	reported
65,536	—	1964.7 tok/s	—	33358 ms	reported
65,536	1	36074.9 tok/s	31.2 tok/s	1817 ms	reported
122,880	1	—	45 tok/s	—	reported
262,144	1	—	45.6 tok/s	352 ms	reported
262,144	2	—	72.5 tok/s	—	reported
262,144	4	—	110.8 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	RigMark protocol 1.0.0 (repo d8353e93b274) release-receipt glm53-nccl-v0.0.1-tp4-20260912: completed code decode median 71.3 tok/s (5/5 gates), prose 31.2, structured ceiling 109.2; TTFT 361 ms code median. Real agentic daily-driver decode ~45 tok/s typical (up to ~100 warm). Cold 64K prefill 1965 tok/s; warm replay 36075 tok/s.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/LibertAIDAI/GLM-5.3-Flash-NVFP4
9e0d74e3cef17f634e84fb8e2223707e02616290
Drafter https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2
Engine image https://ghcr.io/tonyd2wild/vllm-glm53-flash@sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6
sm121-v11-dflash2
Reference https://github.com/alexellis/switchless-nccl
v0.0.1

Revision notes

Scout-discovered (lar-scout, card t_f36bc073): long-lived prominent repo (1566 stars, created 2026-08-29) with zero prior digests; ingested at live HEAD 9ad50c53 (card snapshot 07cbcb41 was stale). Transcribed at pinned sha: docs/recipe.md serve argv + env, scripts/rank-launcher.sh docker run + container flags, docs/nccl-build.md switchless-nccl v0.0.1 pins (archive b4a68638…, lib 78cb8387…). Card's nvidia/GLM-5.3-Flash-NVFP4 hf_repo guess corrected to LibertAIDAI/GLM-5.3-Flash-NVFP4 @ 9e0d74e3 (README fixed-weights table + RigMark appliance.model). Distinguish from knapcio t_ad3aebd3 switched 4x Spark SGLang lane: this is a switchless closed RoCE ring (two rails/node, non-adjacent hop relayed via neighbour) with patched canonical NCCL, not SparkRing/SIRCL. Performance from README tables + data/rigmark/glm53-nccl-v0.0.1-tp4-20260912.json at pinned sha (release-qualified 2026-09-12 run; image in that receipt was the local radixark build, launcher ships public tonyd2wild image).

Revision history

fadac70226 recipe: update glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2 (r3: /kv_pool_tokens) 10 d ago diff
aa74c8a2e0 recipe: update glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2 (r2: /max_context_len) 11 d ago diff
8a52bc8dc4 recipe: create glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2 15 d ago

Fork

Export

recipe.yaml (raw)