{"slug":"motif-3-nvfp4-314b-2x-dgx-spark-tp2-vllm-fork-fp8-dense-mtp","title":"Motif-3 NVFP4 314B 2x DGX Spark TP2 vLLM Fork FP8 Dense MTP","summary":"Recipe by sigjhl. Motif-Technologies/Motif-3-NVFP4 (314B total / 13.2B active MoE, 262k context, MTP) served on two NVIDIA DGX Spark (GB10, sm_121, 128 GB each) with tensor parallel 2 over the Spark-to-Spark RoCE link, using Motif's own vLLM fork (v0.20.2-based) rebuilt for SM12x. Motif's published images target Hopper/Blackwell datacenter parts (SM90/SM100), so the fork needs a patch ladder on GB10: SM12x build of grouped_poly_norm_nvfp4_quant for the NVFP4 MoE path, apache-tvm-ffi for an xgrammar import crash, ring-aware NCCL 2.32.3 for QSFP-ring ibv_modify_qp timeouts, a sliding-window FA2 V-pad fix, and an online FP8 W8A8 layer that moves the dense bf16 GEMMs and lm_head off the bf16 path (45% of a decode step). Author-measured: ~19 tok/s single-request decode (15-17.6 without the FP8 layer), MTP acceptance ~1.7-1.8, ~7 min startup.","license":"Apache-2.0","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"f5df89017fe3603ec395e1c9a14fc8fc1a72adf5","content_hash":"d9f2833eef2cec94069f6cf94ffe121d5756ab72","created_at":"2026-11-04 22:13:43.141228-07","notes":"Ingested from https://github.com/sigjhl/motif3-dgx-spark @ 91ee567c61229cbe75536901053f0061d0fc8835 (created 2026-11-04, Apache-2.0, 0 stars, branch main). Submitter/listed_by = localai-recipes; recipe author sigjhl (credited in credit). Launch argv transcribed from scripts/node.sh + scripts/config.sh at the pin (README is prose). NEW model family 'motif-3' seeded via sql/migrations/0077_motif3_family.sql (0046/0068 precedent; major=true pending=false, upstream official spelling 'Motif-3'). Corrections to the scout card: (1) the card's 4th artifacts row pointer 'eugr/spark-vllm:nightly-20261016 (ring-aware NCCL 2.32.3 source image)' is not a URL - the schema pointer pattern requires https?://|magnet:|git, restated as the Docker Hub URL with the tag in pin; (2) hardware vram_gb given as 256 (per-pair total) - submitted 128 per device to match the live dgx-spark-x2 profile row and sibling dgx-spark recipes; (3) title reworded (card title slugifies to exactly 64 with a mid-word cut 'fp8-dens'; reworded so every distinctive token survives the 64-cut, per t_4e22cc26 precedent). Repo HEAD verified == pin 2026-11-05; HF revision 79f2fad1f8229f5db8a7ad08bbd24a20d6fb0bff resolvable."},"engine_source":{"type":"docker_image","git_url":"https://github.com/sigjhl/motif3-dgx-spark","base_engine":"vllm","pip_package":null,"base_version":"v0.20.2-motif3-sm121-r5","image_digest":null,"install_note":"Image is BUILT locally, not pulled: scripts/build.sh clones MotifTechnologies/vllm at 4cd9eb4129883565e69d508038d783d59ee01867 (branch spark-sm121, v0.20.2 based), applies patches/0001 SM12x grouped_poly_norm_nvfp4_quant, then docker-builds motif-vllm:0.20.2-motif3-sm121 plus layers r2..r5 (r2 apache-tvm-ffi, r3 ring-aware NCCL 2.32.3 from eugr/spark-vllm:nightly-20261016, r4 SWA V-pad, r5 online FP8 W8A8 dense+lm_head) - about 1-2 h at MAX_JOBS=20, ~60 GB disk; then docker save | ssh docker load onto the worker. Base image nvcr.io/nvidia/pytorch:26.02-py3, torch 2.11.0 cu130, TORCH_CUDA_ARCH_LIST=12.1a. Checkpoint is 187 GB and must be rsynced whole (HF caches keep data in a shared blobs/ store). HF Xet must stay disabled (HF_HUB_DISABLE_XET=1). Requires two Sparks joined by QSFP (or a ring) with RoCE working, passwordless ssh from head to worker over that link, Docker with the NVIDIA runtime, and the repo at the SAME PATH on both nodes (MOTIF_DIR). Keep ~10 GB free per node.","pinned_commit":"91ee567c61229cbe75536901053f0061d0fc8835"},"credit":[{"name":"sigjhl (recipe author, SM12x patch ladder, FP8 dense layer, measurements, Apache-2.0)","url":"https://github.com/sigjhl/motif3-dgx-spark"},{"name":"Motif Technologies (Motif-3-NVFP4 weights + vLLM fork base @ 4cd9eb41)"},{"name":"eugr (spark-vllm nightly-20261016 - ring-aware NCCL 2.32.3)"},{"name":"apache-tvm-ffi (xgrammar load_lib_module fix)"},{"name":"NVIDIA (nvcr.io/nvidia/pytorch:26.02-py3 base image)"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"env":["HEAD_IP=192.168.0.1","WORKER_IP=192.168.4.1","WORKER_SSH=spark-2-cl","API_HOST=127.0.0.1","API_PORT=8000","NET_IF=enp1s0f0np0","IB_HCA=rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1","IMG=motif-vllm:0.20.2-motif3-sm121-r5","REV=79f2fad1f8229f5db8a7ad08bbd24a20d6fb0bff","MAXLEN=262144","GPU_UTIL=0.80","MTP=1","MAX_SEQS=4","HF_HUB_OFFLINE=1","HF_HUB_DISABLE_XET=1","HF_MODULES_CACHE=/root/.cache/vllm/hf_modules","MOTIF_MHC_TILELANG=0","LD_LIBRARY_PATH=/usr/local/cuda/compat/lib.real:/usr/local/lib/python3.12/dist-packages/torch/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64","NCCL_SOCKET_IFNAME=enp1s0f0np0","GLOO_SOCKET_IFNAME=enp1s0f0np0","NCCL_IB_HCA=rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1","NCCL_IB_SUBNET_AWARE_ROUTING=1","NCCL_IB_MERGE_NICS=0","NCCL_NET_PLUGIN=none","NCCL_NET=IB","NCCL_IGNORE_CPU_AFFINITY=1"],"launch":["# launch authority AT 91ee567c61229cbe75536901053f0061d0fc8835: scripts/node.sh (head: node.sh 0; worker: node.sh 1, normally started by scripts/start.sh on the head over ssh)","docker run -d --name motif3 --gpus all --network host --ipc host --shm-size 16g --device /dev/infiniband --ulimit memlock=-1 --ulimit stack=67108864 --cap-add IPC_LOCK --security-opt seccomp=unconfined --memory 112g","-v $HOME/.cache/huggingface:/root/.cache/huggingface:ro -v $HOME/.cache/motif3/vllm:/root/.cache/vllm -v $HOME/.cache/motif3/triton:/root/.triton -v $HOME/.cache/motif3/flashinfer:/root/.cache/flashinfer -v $HOME/.cache/motif3/tilelang:/root/.tilelang -v $HOME/.cache/motif3/inductor:/tmp/torchinductor_root -v $HOME/.cache/motif3/nv:/root/.nv","-e HF_HUB_OFFLINE=1 -e HF_HUB_DISABLE_XET=1 -e HF_MODULES_CACHE=/root/.cache/vllm/hf_modules -e MOTIF_MHC_TILELANG=0 -e LD_LIBRARY_PATH=/usr/local/cuda/compat/lib.real:/usr/local/lib/python3.12/dist-packages/torch/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64","-e VLLM_HOST_IP=$IP (head HEAD_IP=192.168.0.1 / worker WORKER_IP=192.168.4.1) -e NCCL_SOCKET_IFNAME=enp1s0f0np0 -e GLOO_SOCKET_IFNAME=enp1s0f0np0","-e NCCL_IB_HCA=rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1 -e NCCL_IB_SUBNET_AWARE_ROUTING=1 -e NCCL_IB_MERGE_NICS=0 -e NCCL_NET_PLUGIN=none -e NCCL_NET=IB -e NCCL_IGNORE_CPU_AFFINITY=1 ${EXTRA_ENV:-}","motif-vllm:0.20.2-motif3-sm121-r5","vllm serve /root/.cache/huggingface/hub/models--Motif-Technologies--Motif-3-NVFP4/snapshots/79f2fad1f8229f5db8a7ad08bbd24a20d6fb0bff --served-model-name Motif-Technologies/Motif-3-NVFP4 --trust-remote-code","--quantization modelopt_nvfp4 --dtype bfloat16 --tensor-parallel-size 2 --nnodes 2 --node-rank $RANK --master-addr 192.168.0.1 --master-port 29511","--gpu-memory-utilization 0.80 --max-model-len 262144 --max-num-seqs 4 --block-size 128","--enable-auto-tool-choice --tool-call-parser motif --reasoning-parser motif","--speculative-config '{\"model\": \"/root/.cache/huggingface/hub/models--Motif-Technologies--Motif-3-NVFP4/snapshots/79f2fad1f8229f5db8a7ad08bbd24a20d6fb0bff\", \"num_speculative_tokens\": 1}' (MTP=1; MTP=0 drops this flag)","# head adds: --host 127.0.0.1 --port 8000; worker adds: --headless (worker started via ssh spark-2-cl 'node.sh 1')"],"layouts":["tp"],"runtime":{"notes":"Rank 0 (head) is the API node, rank 1 is --headless; scripts/start.sh ssh-forwards the knobs and starts both (node.sh IS the launch authority; README launch text is prose). MOTIF_MHC_TILELANG=0 is REQUIRED: the default mHC path calls DeepGEMM which is SM90/SM100 only. LD_LIBRARY_PATH must start with /usr/local/cuda/compat/lib.real because the image ships CUDA 13.1 while the Spark driver is CUDA 13.0 and FA2 is shipped as sm_80 PTX (its JIT fails with cudaErrorUnsupportedPtxVersion otherwise). HF_MODULES_CACHE must be on a writable path (the HF cache mount is read-only). The fork forces expert parallel (192 of 384 experts per node); on SM12x MLA runs on TRITON_MLA so CUDA graphs are PIECEWISE only. --network host with API_HOST bound explicitly - do not use 0.0.0.0 on a shared LAN. FP8 dense layer is off with EXTRA_ENV='-e MOTIF_FP8_DENSE=0 -e MOTIF_FP8_LM_HEAD=0' (costs 15-17.6 vs ~19 tok/s). Thinking is controlled per request via chat_template_kwargs.enable_thinking plus thinking_token_budget (server default 80% of remaining space)."},"kv_pool_tokens":2600000,"max_context_len":262144},"models":[{"slug":"motif-3-nvfp4","name":"Motif-3-NVFP4","hf_repo":"Motif-Technologies/Motif-3-NVFP4","quant":"nvfp4-modelopt"}],"hardware":[{"slug":"dgx-spark-x2","vendor":"nvidia","card":"DGX Spark (GB10, unified memory)","card_family":"dgx-spark","vram_gb":128,"device_count":2}],"generation":"blackwell","tags":["dgx-spark","fp8-dense","gb10","modelopt","mtp","multinode","nvfp4","roce","sm121"],"tasks":["chat","code"],"aa":null,"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured on 2x DGX Spark, tested 2026-11-04 on DGX OS 7.6.0 / driver 580.178.04 (README Results table + tools/bench.py, decode 1 request: ~19 tok/s with the r5 FP8 layer, 15-17.6 tok/s without it i.e. EXTRA_ENV MOTIF_FP8_DENSE=0 MOTIF_FP8_LM_HEAD=0; TTFT ~0.3 s; MTP acceptance length ~1.7-1.8). Weights in memory 85.6 GiB per node (NVFP4 experts + FP8 dense). KV pool ~2.6M tokens at --gpu-memory-utilization 0.80. Startup ~7 min (5 min weight load, 2 min compile/graph capture). Free memory when idle ~7-8 GB per node. Pre-r5 profile of a 114 ms decode step at 91% GPU busy: dense bf16 GEMMs 45%, NCCL all-reduce 22% (isolated all-reduce latency 24-38 us), NVFP4 expert grouped GEMM 21%, lm_head 4%. No concurrency or prefill table published - single-request decode only; bench.py uses a ~20-token prompt (context_len on the point = the 262144 serve context).\", \"tok_s\": 19, \"points\": [{\"notes\": \"README Results / tools/bench.py c=1 streamed decode with the r5 FP8 dense+lm_head layer (~20-token prompt, max_tokens 256); 15-17.6 tok/s without it; MTP acceptance 1.7-1.8\", \"ttft_ms\": 300, \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 19}], \"ttft_ms\": 300, \"context_len\": 262144}"}],"perf_points":[{"id":2442,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":19,"ttft_ms":300,"concurrency":1,"claimed_by":"submitter","notes":"README Results / tools/bench.py c=1 streamed decode with the r5 FP8 dense+lm_head layer (~20-token prompt, max_tokens 256); 15-17.6 tok/s without it; MTP acceptance 1.7-1.8"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/motif-3-nvfp4-314b-2x-dgx-spark-tp2-vllm-fork-fp8-dense-mtp.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/motif-3-nvfp4-314b-2x-dgx-spark-tp2-vllm-fork-fp8-dense-mtp.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/sigjhl/motif3-dgx-spark","pin":"91ee567c61229cbe75536901053f0061d0fc8835"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/Motif-Technologies/Motif-3-NVFP4","pin":"79f2fad1f8229f5db8a7ad08bbd24a20d6fb0bff"},{"kind":"other","pointer_type":"git_commit","pointer":"https://github.com/MotifTechnologies/vllm","pin":"4cd9eb4129883565e69d508038d783d59ee01867"},{"kind":"other","pointer_type":"url","pointer":"https://hub.docker.com/r/eugr/spark-vllm","pin":"nightly-20261016 (ring-aware NCCL 2.32.3 source image)"}],"links":{"self":"/api/v1/recipes/motif-3-nvfp4-314b-2x-dgx-spark-tp2-vllm-fork-fp8-dense-mtp.json","alternate":"/r/motif-3-nvfp4-314b-2x-dgx-spark-tp2-vllm-fork-fp8-dense-mtp","describedby":"/openapi.json"},"updated_at":"2026-11-04 22:13:43.141228-07","created_at":"2026-11-04 22:13:43.141228-07","speeds":{"decode_tok_s":19,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}