diff --git a/recipes/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m/recipe.yaml b/recipes/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m/recipe.yaml index 0f57f4da6..82ec05ee0 100644 --- a/recipes/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m/recipe.yaml +++ b/recipes/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m/recipe.yaml @@ -80,72 +80,70 @@ config: env: "Install layer (install/, image tag 246a3a948ab9 = git tree @ 3c7d671): profile 4x1m environment arms the full RC4.2 decode stack — NCCL Ring/Simple over the CX-7 (NCCL_IB_HCA + NCCL_SOCKET_IFNAME set per node from the detected fabric port, FABRIC_SIBLINGS=1 adds the RoCE device behind the cable's other PCIe domain), ATLAS_RDMA_ALLREDUCE=1 + ATLAS_RDMA_ONESHOT=1 + ATLAS_RDMA_ONESHOT_SM_STAGE=1 + ATLAS_RDMA_PAIR_CHAIN=1 + ATLAS_GLM_CMD_RDMA=1 for the direct RDMA all-reduce, ATLAS_GLM_KV_SHARD=1 + ATLAS_GLM_KV_SHARD_COMPACT=1 when --kv-shard, DFlash2 (ATLAS_GLM_DFLASH=1, ATLAS_DFLASH_OPTION_B=1, ATLAS_DFLASH_ADAPTIVE_WIDTH=1 with ATLAS_DFLASH_CONF_WIDTH=1 + ATLAS_DFLASH_CONF_PROBE=256, ATLAS_DFLASH2_EDGE=0.6, ATLAS_DFLASH_MULTI_DRAFT_CAP=7, ATLAS_GLM_DRAFT_TP=1 + ATLAS_GLM_DRAFT_TP_BATCH=1 + ATLAS_GLM_DRAFT_TP_CTX=1 + ATLAS_GLM_DRAFT_TP_TOPK=1, ATLAS_DFLASH_ASYNC_UPLOADS=1, ATLAS_DFLASH_MXFP8=1 + ATLAS_DFLASH_NVFP4_HEAD=1 + ATLAS_DFLASH_NVFP4_TC=1 + ATLAS_DFLASH_CONV_TWIN=nvfp4), canonical verify (ATLAS_GLM_CANONICAL_VERIFY-class ATLAS_GLM_STRICT_SPEC=1 for structured outputs, ATLAS_GLM_VERIFY_VOCAB_SPLIT=1, ATLAS_GLM_LONG_BATCH_VERIFY=1), prefix cache (ATLAS_GLM_PC_EVICT=1 + ATLAS_GLM_PC_BRANCH=1, Marconi prefill-only ATLAS_MARCONI_PREFILL_ONLY=1, ATLAS_GLM_WARM_SKIP_CACHED=1 + ATLAS_GLM_WARM_CHUNK_RUN=1), NVFP4 kernels (ATLAS_NVFP4_FUSED_SILU_QUANT=1, ATLAS_NVFP4_PREQUANT_MOE=1, ATLAS_NVFP4_VECSCALE=1, ATLAS_UNIFIED_MOE_LAYOUT=1) and the RC4 fusions (ATLAS_GLM_MOE_ROUTE_FUSE=1, ATLAS_GLM_MLA_FUSE=1, ATLAS_GLM_DENSE_TP_SPLIT=1, ATLAS_GLM_KDA_COMMIT_CTAS=96, ATLAS_GLM_MXFP8_TILED=1, ATLAS_PDL=1) — ~220 ATLAS_*/NCCL_* keys total, transcribed in full into install/profiles/4x1m.json @ 3c7d671 (values win over ambient flags: serve.py strips ATLAS_* from the container env and re-applies the profile's own). Ranks: docker run --network host --ipc host --shm-size 32g --memory 114g --gpus all --device /dev/infiniband --cap-add IPC_LOCK --cap-add SYS_NICE --ulimit memlock=-1:-1 --security-opt no-new-privileges=true (start-node.sh); --cap-add SYS_ADMIN only with DISPLAY_CARVEOUT=1. NVMe tier env (serve.py disk_prefix_cache): ATLAS_KV_NVME_DIR=/prefix-cache/kv + ATLAS_KV_NVME_GB=, ATLAS_GLM_NVME_FAST=1, ATLAS_SSM_TIER=1 + ATLAS_SSM_TIER_UNIFIED=1 + ATLAS_SSM_TIER_SWAP_DIR=/prefix-cache/ssm + ATLAS_SSM_TIER_DISK_GB= + ATLAS_SSM_TIER_SLOTS=2." notes: "SparkGLM RC4.2 (Enntity/sparkglm @ 3c7d671, RC4.2 merge): GLM-5.3-Flash NVFP4 on two DGX Sparks joined by one 200G ConnectX-7 cable, served by the Atlas engine (Enntity/atlas sparkglm/atlas-20261012-rc4.2-layered @ dfb9c26a, tree 89e392a3) with tensor + expert parallelism over RDMA (RoCE v2, both PCIe domains of the cable), DFlash2 speculative decoding (gamma 8, window 2048), FP8-latent shared KV pool with prefix caching, up to 4 concurrent requests with per-request 1M context (4x1m profile), OpenAI-compatible API on 127.0.0.1:8893. ./start.sh on rank 0 drives everything: pulls/builds the image (tag = git tree of install/), downloads both pinned checkpoints once (207 GB) and mirrors them over the cable, converts the 864 NVIDIA MTP matrices to NVFP4 in a verified overlay per node (install/convert.sh, ~1 min), then starts rank 1 over ssh and rank 0 and waits for health (~2.5 min). Optional knobs (default off unless set in .env): DISPLAY_CARVEOUT=1 lends each GB10's 2 GiB display carveout to the KV pool (server drops CAP_SYS_ADMIN after export); PREFIX_CACHE_DIR(+PREFIX_CACHE_GB, 16-100 GiB per node) keeps evicted prefix-cache entries on each node's disk; KV_SHARD=1 (start-node --kv-shard) splits the MLA latent KV across the ranks (~1.8x pool); SPARKGLM_GPU_MEMORY_UTILIZATION 0.80-0.95 overrides the profile's 0.88. Decode is concurrency-invariant (canonical verify): greedy output at c4 equals c1 4/4. Rerun ./start.sh to restart; ./start.sh stop|status|logs|build|download|convert." performance: - claimed_by: submitter - metrics: - tok_s: 72.3 - context_len: 1048576 - notes: "Headline = RigMark code decode c1 median (rig-decode.json); see points for the full transcribed matrix. RigMark cells on the 8x128k profile, single-stream rows on production 4x1m; all rows from the repo's own receipts, not re-measured." - points: - - - context_len: 131072 - concurrency: 1 - decode_tok_s: 72.3 - notes: 'code decode c1, median of 3 (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' - - - context_len: 131072 - concurrency: 1 - decode_tok_s: 38.3 - notes: 'prose decode c1, median of 3 (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' - - - context_len: 131072 - concurrency: 1 - decode_tok_s: 91.7 - notes: 'structured decode c1, median of 3 (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' - - - context_len: 8192 - concurrency: 1 - ttft_ms: 3293 - notes: 'RigMark cold prefill TTFT 8K (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' - - - context_len: 32768 - concurrency: 1 - ttft_ms: 12474 - notes: 'RigMark cold prefill TTFT 32K (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' - - - context_len: 65536 - concurrency: 1 - ttft_ms: 23912 - notes: 'RigMark cold prefill TTFT 64K (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' - - - context_len: 32768 - concurrency: 2 - ttft_ms: 2802 - notes: 'RigMark staggered arrivals prefill-first: newcomer TTFT at c2, 32K depth (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' - - - context_len: 32768 - concurrency: 4 - ttft_ms: 3537 - notes: 'RigMark staggered arrivals prefill-first: newcomer TTFT at c4, 32K depth (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' - - - context_len: 131072 - concurrency: 1 - decode_tok_s: 47.0 - notes: 'RigMark short code c1 aggregate end-to-end (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' - - - context_len: 131072 - concurrency: 4 - decode_tok_s: 70.9 - notes: 'RigMark short code c4 aggregate end-to-end (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' - - - context_len: 1048576 - concurrency: 1 - decode_tok_s: 45.57 - notes: 'single-stream prose decode, 12 prompts x 256 tokens (production settings: 4x1m profile, 0.91, display carveout, KV shard, disk prefix cache 48 GB; thinking off, greedy, raw/multi-rc4_on.json)' - - - context_len: 1048576 - concurrency: 1 - decode_tok_s: 64.91 - notes: 'single-stream code decode, 12 prompts x 256 tokens (production settings: 4x1m profile, 0.91, display carveout, KV shard, disk prefix cache 48 GB; thinking off, greedy, raw/multi-rc4_on.json)' + tok_s: 72.3 + context_len: 1048576 + notes: "Headline = RigMark code decode c1 median (rig-decode.json); see points for the full transcribed matrix. RigMark cells on the 8x128k profile, single-stream rows on production 4x1m; all rows from the repo's own receipts, not re-measured." + points: + - + context_len: 131072 + concurrency: 1 + decode_tok_s: 72.3 + notes: 'code decode c1, median of 3 (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md); context_len is the tuple marker 131072+0 because the table keys one row per (context_len, concurrency)' + - + context_len: 151552 + concurrency: 1 + decode_tok_s: 38.3 + notes: 'prose decode c1, median of 3 (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md); context_len is the tuple marker 131072+20480 because the table keys one row per (context_len, concurrency)' + - + context_len: 172032 + concurrency: 1 + decode_tok_s: 91.7 + notes: 'structured decode c1, median of 3 (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md); context_len is the tuple marker 131072+40960 because the table keys one row per (context_len, concurrency)' + - + context_len: 8192 + concurrency: 1 + ttft_ms: 3293 + notes: 'RigMark cold prefill TTFT 8K (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' + - + context_len: 32768 + concurrency: 1 + ttft_ms: 12474 + notes: 'RigMark cold prefill TTFT 32K (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' + - + context_len: 65536 + concurrency: 1 + ttft_ms: 23912 + notes: 'RigMark cold prefill TTFT 64K (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' + - + context_len: 32768 + concurrency: 2 + ttft_ms: 2802 + notes: 'RigMark staggered arrivals prefill-first: newcomer TTFT at c2, 32K depth (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' + - + context_len: 32768 + concurrency: 4 + ttft_ms: 3537 + notes: 'RigMark staggered arrivals prefill-first: newcomer TTFT at c4, 32K depth (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' + - + context_len: 131072 + concurrency: 1 + decode_tok_s: 47.0 + notes: 'RigMark short code c1 aggregate end-to-end (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' + - + context_len: 131072 + concurrency: 4 + decode_tok_s: 70.9 + notes: 'RigMark short code c4 aggregate end-to-end (RigMark (40fabcaf fork w/ staggered-arrival suite, seed 20260905, reasoning_effort low, thinking on), 8x128k profile per bench/RIGMARK.md)' + - + context_len: 1048576 + concurrency: 1 + decode_tok_s: 45.57 + notes: 'single-stream prose decode, 12 prompts x 256 tokens (production settings: 4x1m profile, 0.91, display carveout, KV shard, disk prefix cache 48 GB; thinking off, greedy, raw/multi-rc4_on.json)' + - + context_len: 1069056 + concurrency: 1 + decode_tok_s: 64.91 + notes: 'single-stream code decode, 12 prompts x 256 tokens (production settings: 4x1m profile, 0.91, display carveout, KV shard, disk prefix cache 48 GB; thinking off, greedy, raw/multi-rc4_on.json); context_len is the tuple marker 1048576+20480 because the table keys one row per (context_len, concurrency)' artifacts: - kind: model_weights