diff --git a/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k/recipe.yaml b/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k/recipe.yaml index bf3c1ba09..0a79437dc 100644 --- a/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k/recipe.yaml +++ b/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k/recipe.yaml @@ -6,9 +6,9 @@ engine: source_type: docker_image image_digest: 'vllm/vllm-openai@sha256:1ee85ca80e0fc110017d79ea2081a0cbef9a208a05d7f22724801bcf6a956620' git_url: 'https://github.com/hasso5703/dgx-spark-qwen38' - pinned_commit: a1139fafaaa4585c60242419a750df7e60a219c8 - install_note: 'Builds its own image: vllm/vllm-openai nightly-77154ca6 (digest above) plus the b12x 1.5.1 wheel (sha256 8731e235...53a3d, pip install --no-deps). b12x prints an FP6-plugin hook warning at every start; Bolt uses no FP6 and nothing follows from it. FlashInfer kernels ship in the image, so it starts offline.' -summary: 'NVIDIA Nemotron 3.5 Bolt 74B-A9B-24GB (hybrid Mamba-2/Transformer latent-MoE, 74B with 9B active, IQ2_XXS 2-bit experts, NVFP4 Mamba layers, Q8_0 rest; text+image+video in, 262,144-token window, OpenMDW-1.1) on one DGX Spark (GB10/SM121) with vLLM nightly-77154ca6 (base image vllm/vllm-openai@sha256:1ee85ca8) + b12x 1.5.1 kernels (--linear-backend b12x --moe-backend b12x), FP8 KV cache, gpu-memory-utilization 0.70, max-num-seqs 16, and the NVIDIA LiLiCorr MTP drafter at 7 speculative tokens. Pinned revisions: model c6cc2e8f, drafter 4c6a5613. Measured on the reference box: ~140 tok/s code rewrite / 63 summarise / 50 prose single-stream with the drafter (mean acceptance 4.0 tokens/pass), 31 tok/s without; 74.8/113.4/163.6/201.2 tok/s at 1/4/8/16 streams; KV cache ~6.52M tokens at the 0.70 share.' + pinned_commit: 9e63ad54e2c0cacb489811f5eb243181cb4bddd5 + install_note: "Builds its own image: vllm/vllm-openai nightly-77154ca6 (digest above) plus the b12x 1.5.1 wheel (sha256 8731e235...53a3d, pip install --no-deps). b12x prints an FP6-plugin hook warning at every start; Bolt uses no FP6 and nothing follows from it. FlashInfer kernels ship in the image, so it starts offline. Known upstream exposure on this build, not fixed in it: a structured-output request can end vLLM's engine process (GHSA-85xf-c7hm-whqw, fixed in vLLM 0.30.0; D06 of the 2026-10-24 security review, open upstream at v1.40.3 - this lane's image is built on nightly-77154ca6 and does not carry the fix). It takes a valid key. Rebuild on a vLLM >= 0.30.0 nightly when NVIDIA's card moves." +summary: 'NVIDIA Nemotron 3.5 Bolt 74B-A9B-24GB (hybrid Mamba-2/Transformer latent-MoE, 74B with 9B active, IQ2_XXS 2-bit experts, NVFP4 Mamba layers, Q8_0 rest; text+image+video in, 262,144-token window, OpenMDW-1.1) on one DGX Spark (GB10/SM121) with vLLM nightly-77154ca6 (base image vllm/vllm-openai@sha256:1ee85ca8) + b12x 1.5.1 kernels (--linear-backend b12x --moe-backend b12x), FP8 KV cache, gpu-memory-utilization 0.70, max-num-seqs 16, and the NVIDIA LiLiCorr MTP drafter at 7 speculative tokens. Pinned revisions: model c6cc2e8f, drafter 4c6a5613. Measured on the reference box: ~140 tok/s code rewrite / 63 summarise / 50 prose single-stream with the drafter (mean acceptance 4.0 tokens/pass), 31 tok/s without; 74.8/113.4/163.6/201.2 tok/s at 1/4/8/16 streams; KV cache ~6.52M tokens at the 0.70 share. On a 64 GB Spark the lane takes its own memory share 0.60: 667,874 KV tokens (2.55 windows), 110.8 tok/s greedy, 7.1 GiB left at the peak (2026-10-24).' license: OpenMDW-1.1 model: name: Nemotron-3.5-Bolt-74B-A9B-24GB @@ -92,6 +92,7 @@ config: - '--enable-auto-tool-choice' - '--tool-call-parser' - qwen3_coder + - '--enable-prompt-tokens-details' - '--reasoning-parser-plugin' - /root/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB/snapshots/c6cc2e8ffa353403213d6b9b18aaf8eb684d705a/ultra_v3_reasoning_parser.py - '--reasoning-parser' @@ -194,8 +195,8 @@ artifacts: - kind: reference pointer_type: git_commit - pointer: 'https://github.com/hasso5703/dgx-spark-qwen38/commit/a1139fafaaa4585c60242419a750df7e60a219c8' - pin: a1139fafaaa4585c60242419a750df7e60a219c8 + pointer: 'https://github.com/hasso5703/dgx-spark-qwen38/commit/9e63ad54e2c0cacb489811f5eb243181cb4bddd5' + pin: 9e63ad54e2c0cacb489811f5eb243181cb4bddd5 provenance: credit: - @@ -205,6 +206,6 @@ provenance: - 'https://github.com/hasso5703/dgx-spark-qwen38' - 'https://github.com/hasso5703/dgx-spark-qwen38/blob/main/docs/nemotron-lane.md' - 'https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB' - notes: "Recipe by hasso5703 (dgx-spark-qwen38 nemotron lane, docs/nemotron-lane.md + install-nemotron.sh + qwen38-nemotron.service.template @ a1139fafaa, v1.35.1 2026-10-17). Submitter/listed_by = localai-recipes. NVIDIA rewrote both HF repos' history on publication; the pinned model+drafter revisions are author-blob-verified equal to the first-measured ones (lane doc). Launch argv transcribed from the pinned service template; box port/proxy/key discipline shared with the already-ingested qwen38 lanes. All numbers author-reported, not re-measured (T0)." + notes: "Recipe by hasso5703 (dgx-spark-qwen38 nemotron lane, docs/nemotron-lane.md + install-nemotron.sh + qwen38-nemotron.service.template @ a1139fafaa, v1.35.1 2026-10-17). Submitter/listed_by = localai-recipes. NVIDIA rewrote both HF repos' history on publication; the pinned model+drafter revisions are author-blob-verified equal to the first-measured ones (lane doc). Launch argv transcribed from the pinned service template; box port/proxy/key discipline shared with the already-ingested qwen38 lanes. All numbers author-reported, not re-measured (T0). PIN REFRESH (kanban t_0485af4e, owner go 2026-10-28): lane files (docs/nemotron-lane.md, install-nemotron.sh, qwen38-nemotron.service.template) re-audited @ 9e63ad54e2c0cacb489811f5eb243181cb4bddd5 (v1.40.3, GitHub HEAD verified); pinned_commit + reference artifact bumped a1139faf..9e63ad54e2 (ancestor verified); launch re-transcribed — single delta --enable-prompt-tokens-details (5ce1a01, #49); 64 GB values added to the summary per 1827802 (v1.40.0, measured on a 64 GB budget 2026-10-24); a new 64 GB box ends its install on Bolt (bef405e) and the 64 GB refusal list (d4ceccb, 9adafb6) does not touch Bolt; perf points untouched; GSM8K not published on this slug (8f72c7e no-op). PROXY SECURITY (kanban t_5be2157a, owner go 2026-10-28 Tier 2/3 preprod): F5 - proxy statement added: clients reach the lane through the box's keepalive proxy on :30001 with the box's key, as model nemotron-3.5-bolt; the engine itself binds 127.0.0.1:30000 (ENGINE_BIND=0.0.0.0 opens it to the network); the key reaches vLLM through /out/engine-secrets.yaml (mode 0600) with vllm_engine_auth.RequireKey middleware, so /health and /metrics answer without the key by design; qwen38-nemotron.service.template re-checked at 9e63ad54e2 (/out mounted read-only, --enable-prompt-tokens-details added). F4 - torch distributed store caveat: PyTorch's distributed store binds every interface on a random port no setting narrows (D05, open upstream at v1.40.3), takes no key; firewall the box to the ports you use before anything else (SECURITY.md's ufw block). F6 - GHSA-85xf-c7hm-whqw caveat added to the engine install note (D06; fixed in vLLM 0.30.0, no lane moved upstream)." max_context_len: 262144 kv_pool_tokens: 6520000