Nemotron 3.5 Bolt 74B-A9B 1x DGX Spark vLLM b12x LiLiCorr 262K

community Community submission — verify on your own hardware.

recipe by hasso5703↗ · listed by localai-recipes

revision r4 · commit 2cdb5f11ff · updated 21 h ago · license OpenMDW-1.1

chat code vision fp8-kv vision multimodal speculative-decoding gb10 mtp prefix-caching 262k b12x video-input

NVIDIA Nemotron 3.5 Bolt 74B-A9B-24GB (hybrid Mamba-2/Transformer latent-MoE, 74B with 9B active, IQ2_XXS 2-bit experts, NVFP4 Mamba layers, Q8_0 rest; text+image+video in, 262,144-token window, OpenMDW-1.1) on one DGX Spark (GB10/SM121) with vLLM nightly-77154ca6 (base image vllm/vllm-openai@sha256:1ee85ca8) + b12x 1.5.1 kernels (--linear-backend b12x --moe-backend b12x), FP8 KV cache, gpu-memory-utilization 0.70, max-num-seqs 16, and the NVIDIA LiLiCorr MTP drafter at 7 speculative tokens. Pinned revisions: model c6cc2e8f, drafter 4c6a5613. Measured on the reference box: ~140 tok/s code rewrite / 63 summarise / 50 prose single-stream with the drafter (mean acceptance 4.0 tokens/pass), 31 tok/s without; 74.8/113.4/163.6/201.2 tok/s at 1/4/8/16 streams; KV cache ~6.52M tokens at the 0.70 share. On a 64 GB Spark the lane takes its own memory share 0.60: 667,874 KV tokens (2.55 windows), 110.8 tok/s greedy, 7.1 GiB left at the peak (2026-10-24).

Launch command

vllm serve nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB --revision c6cc2e8ffa353403213d6b9b18aaf8eb684d705a --served-model-name nemotron-3.5-bolt --trust-remote-code --max-model-len 262144 --max-num-seqs 16 --kv-cache-dtype fp8 --gpu-memory-utilization 0.70 --linear-backend b12x --moe-backend b12x --attention-backend FLASHINFER --mamba-backend flashinfer --mamba-ssm-cache-dtype bfloat16 --mamba-cache-dtype bfloat16 --seed 42 --block-size 64 --prefix-match-unit 64 --enable-mamba-shared-prefix-checkpoint --limit-mm-per-prompt.image 16 --limit-mm-per-prompt.video 2 --video-pruning-rate 0.5 --media-io-kwargs.video.fps 2 --media-io-kwargs.video.num_frames 256 --allowed-media-domains media.invalid --enable-auto-tool-choice --tool-call-parser qwen3_coder --enable-prompt-tokens-details --reasoning-parser-plugin /root/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB/snapshots/c6cc2e8ffa353403213d6b9b18aaf8eb684d705a/ultra_v3_reasoning_parser.py --reasoning-parser ultra_v3 --speculative-config.method dflash --speculative-config.model nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr --speculative-config.revision 4c6a5613b6d887194a3a5493e3a18a5b3930bb86 --speculative-config.num_speculative_tokens 7 --speculative-config.rejection_sample_method standard --speculative-config.attention_backend FLASHINFER --speculative-config.draft_load_config.load_format auto --config /out/engine-secrets.yaml --middleware vllm_engine_auth.RequireKey --host 127.0.0.1 --port 30000

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k.json

Engine

vllm · source↗ · 9e63ad54↗ · vllm/vllm-openai@sha256:1ee85ca80e0fc110017d79ea2081a0cbef9a208a05d7f22724801bcf6a956620↗

Install note: Builds its own image: vllm/vllm-openai nightly-77154ca6 (digest above) plus the b12x 1.5.1 wheel (sha256 8731e235...53a3d, pip install --no-deps). b12x prints an FP6-plugin hook warning at every start; Bolt uses no FP6 and nothing follows from it. FlashInfer kernels ship in the image, so it starts offline. Known upstream exposure on this build, not fixed in it: a structured-output request can end vLLM's engine process (GHSA-85xf-c7hm-whqw, fixed in vLLM 0.30.0; D06 of the 2026-10-24 security review, open upstream at v1.40.3 - this lane's image is built on nightly-77154ca6 and does not carry the fix). It takes a valid key. Rebuild on a vLLM >= 0.30.0 nightly when NVIDIA's card moves.

Model

Nemotron-3.5-Bolt-74B-A9B-24GB (iq2-xxs-experts-nvfp4-mamba-q8-0-rest) · nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB↗

Hardware

NVIDIA DGX Spark 128GB ×1 (Blackwell)

Context

max context: 256K tokens

KV pool

6,520,000 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
10,240	1	—	140 tok/s	—	reported
20,480	1	—	63 tok/s	—	reported
30,720	1	—	50 tok/s	—	reported
40,960	1	—	31 tok/s	—	reported
51,200	1	—	74.8 tok/s	—	reported
61,440	4	—	113.4 tok/s	—	reported
71,680	8	—	163.6 tok/s	—	reported
81,920	16	—	201.2 tok/s	—	reported
92,160	16	—	178 tok/s	—	reported
262,144	1	—	—	192000 ms	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Reference-box measurements 2026-10-15/16 (docs/nemotron-lane.md @ a1139faf), real workloads with reasoning off, median of 3. points[].context_len is a workload marker (N + offset), not a served-context limit; the served window is 262,144.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB
c6cc2e8ffa353403213d6b9b18aaf8eb684d705a
Drafter https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr
4c6a5613b6d887194a3a5493e3a18a5b3930bb86
Engine image https://hub.docker.com/r/vllm/vllm-openai
nightly-77154ca6 + b12x 1.5.1
Reference https://github.com/hasso5703/dgx-spark-qwen38/commit/9e63ad54e2c0cacb489811f5eb243181cb4bddd5
9e63ad54e2c0cacb489811f5eb243181cb4bddd5

Revision notes

Recipe by hasso5703 (dgx-spark-qwen38 nemotron lane, docs/nemotron-lane.md + install-nemotron.sh + qwen38-nemotron.service.template @ a1139fafaa, v1.35.1 2026-10-17). Submitter/listed_by = localai-recipes. NVIDIA rewrote both HF repos' history on publication; the pinned model+drafter revisions are author-blob-verified equal to the first-measured ones (lane doc). Launch argv transcribed from the pinned service template; box port/proxy/key discipline shared with the already-ingested qwen38 lanes. All numbers author-reported, not re-measured (T0). PIN REFRESH (kanban t_0485af4e, owner go 2026-10-28): lane files (docs/nemotron-lane.md, install-nemotron.sh, qwen38-nemotron.service.template) re-audited @ 9e63ad54e2c0cacb489811f5eb243181cb4bddd5 (v1.40.3, GitHub HEAD verified); pinned_commit + reference artifact bumped a1139faf..9e63ad54e2 (ancestor verified); launch re-transcribed — single delta --enable-prompt-tokens-details (5ce1a01, #49); 64 GB values added to the summary per 1827802 (v1.40.0, measured on a 64 GB budget 2026-10-24); a new 64 GB box ends its install on Bolt (bef405e) and the 64 GB refusal list (d4ceccb, 9adafb6) does not touch Bolt; perf points untouched; GSM8K not published on this slug (8f72c7e no-op). PROXY SECURITY (kanban t_5be2157a, owner go 2026-10-28 Tier 2/3 preprod): F5 - proxy statement added: clients reach the lane through the box's keepalive proxy on :30001 with the box's key, as model nemotron-3.5-bolt; the engine itself binds 127.0.0.1:30000 (ENGINE_BIND=0.0.0.0 opens it to the network); the key reaches vLLM through /out/engine-secrets.yaml (mode 0600) with vllm_engine_auth.RequireKey middleware, so /health and /metrics answer without the key by design; qwen38-nemotron.service.template re-checked at 9e63ad54e2 (/out mounted read-only, --enable-prompt-tokens-details added). F4 - torch distributed store caveat: PyTorch's distributed store binds every interface on a random port no setting narrows (D05, open upstream at v1.40.3), takes no key; firewall the box to the ports you use before anything else (SECURITY.md's ufw block). F6 - GHSA-85xf-c7hm-whqw caveat added to the engine install note (D06; fixed in vLLM 0.30.0, no lane moved upstream).

Revision history

2cdb5f11ff recipe: update nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k (r4: /artifacts) 15 h ago diff
382ec73b1f recipe: update nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k (r3: /config/launch, /engine_source, /summary, /notes, /artifacts reference pin — prod sync of QC-approved preprod r5 0a 44 h ago diff
39f49ea41c recipe: update nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k (r3: /kv_pool_tokens) 10 d ago diff
df8598a1c5 recipe: update nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k (r2: /max_context_len) 11 d ago diff
ffe4985534 recipe: create nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k 13 d ago

Fork

Export

recipe.yaml (raw)