{"slug":"nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k","title":"Nemotron 3.5 Bolt 74B-A9B 1x DGX Spark vLLM b12x LiLiCorr 262K","summary":"NVIDIA Nemotron 3.5 Bolt 74B-A9B-24GB (hybrid Mamba-2/Transformer latent-MoE, 74B with 9B active, IQ2_XXS 2-bit experts, NVFP4 Mamba layers, Q8_0 rest; text+image+video in, 262,144-token window, OpenMDW-1.1) on one DGX Spark (GB10/SM121) with vLLM nightly-77154ca6 (base image vllm/vllm-openai@sha256:1ee85ca8) + b12x 1.5.1 kernels (--linear-backend b12x --moe-backend b12x), FP8 KV cache, gpu-memory-utilization 0.70, max-num-seqs 16, and the NVIDIA LiLiCorr MTP drafter at 7 speculative tokens. Pinned revisions: model c6cc2e8f, drafter 4c6a5613. Measured on the reference box: ~140 tok/s code rewrite / 63 summarise / 50 prose single-stream with the drafter (mean acceptance 4.0 tokens/pass), 31 tok/s without; 74.8/113.4/163.6/201.2 tok/s at 1/4/8/16 streams; KV cache ~6.52M tokens at the 0.70 share. On a 64 GB Spark the lane takes its own memory share 0.60: 667,874 KV tokens (2.55 windows), 110.8 tok/s greedy, 7.1 GiB left at the peak (2026-10-24).","license":"OpenMDW-1.1","tier":"community","beta":false,"revision":{"no":4,"commit_sha":"2cdb5f11ffa1fb1af8676496fdd631976c3d536d","content_hash":"28887fb2cf3bab1914c803e792898fff92b9afe0","created_at":"2026-10-29 21:32:26.717952-06","notes":"Recipe by hasso5703 (dgx-spark-qwen38 nemotron lane, docs/nemotron-lane.md + install-nemotron.sh + qwen38-nemotron.service.template @ a1139fafaa, v1.35.1 2026-10-17). Submitter/listed_by = localai-recipes. NVIDIA rewrote both HF repos' history on publication; the pinned model+drafter revisions are author-blob-verified equal to the first-measured ones (lane doc). Launch argv transcribed from the pinned service template; box port/proxy/key discipline shared with the already-ingested qwen38 lanes. All numbers author-reported, not re-measured (T0). PIN REFRESH (kanban t_0485af4e, owner go 2026-10-28): lane files (docs/nemotron-lane.md, install-nemotron.sh, qwen38-nemotron.service.template) re-audited @ 9e63ad54e2c0cacb489811f5eb243181cb4bddd5 (v1.40.3, GitHub HEAD verified); pinned_commit + reference artifact bumped a1139faf..9e63ad54e2 (ancestor verified); launch re-transcribed \u2014 single delta --enable-prompt-tokens-details (5ce1a01, #49); 64 GB values added to the summary per 1827802 (v1.40.0, measured on a 64 GB budget 2026-10-24); a new 64 GB box ends its install on Bolt (bef405e) and the 64 GB refusal list (d4ceccb, 9adafb6) does not touch Bolt; perf points untouched; GSM8K not published on this slug (8f72c7e no-op). PROXY SECURITY (kanban t_5be2157a, owner go 2026-10-28 Tier 2/3 preprod): F5 - proxy statement added: clients reach the lane through the box's keepalive proxy on :30001 with the box's key, as model nemotron-3.5-bolt; the engine itself binds 127.0.0.1:30000 (ENGINE_BIND=0.0.0.0 opens it to the network); the key reaches vLLM through /out/engine-secrets.yaml (mode 0600) with vllm_engine_auth.RequireKey middleware, so /health and /metrics answer without the key by design; qwen38-nemotron.service.template re-checked at 9e63ad54e2 (/out mounted read-only, --enable-prompt-tokens-details added). F4 - torch distributed store caveat: PyTorch's distributed store binds every interface on a random port no setting narrows (D05, open upstream at v1.40.3), takes no key; firewall the box to the ports you use before anything else (SECURITY.md's ufw block). F6 - GHSA-85xf-c7hm-whqw caveat added to the engine install note (D06; fixed in vLLM 0.30.0, no lane moved upstream)."},"engine_source":{"type":"docker_image","git_url":"https://github.com/hasso5703/dgx-spark-qwen38","base_engine":"vllm","pip_package":null,"base_version":"nightly-77154ca6 + b12x 1.5.1","image_digest":"vllm/vllm-openai@sha256:1ee85ca80e0fc110017d79ea2081a0cbef9a208a05d7f22724801bcf6a956620","install_note":"Builds its own image: vllm/vllm-openai nightly-77154ca6 (digest above) plus the b12x 1.5.1 wheel (sha256 8731e235...53a3d, pip install --no-deps). b12x prints an FP6-plugin hook warning at every start; Bolt uses no FP6 and nothing follows from it. FlashInfer kernels ship in the image, so it starts offline. Known upstream exposure on this build, not fixed in it: a structured-output request can end vLLM's engine process (GHSA-85xf-c7hm-whqw, fixed in vLLM 0.30.0; D06 of the 2026-10-24 security review, open upstream at v1.40.3 - this lane's image is built on nightly-77154ca6 and does not carry the fix). It takes a valid key. Rebuild on a vLLM >= 0.30.0 nightly when NVIDIA's card moves.","pinned_commit":"9e63ad54e2c0cacb489811f5eb243181cb4bddd5"},"credit":[{"name":"hasso5703","url":"https://github.com/hasso5703"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB","--revision","c6cc2e8ffa353403213d6b9b18aaf8eb684d705a","--served-model-name","nemotron-3.5-bolt","--trust-remote-code","--max-model-len","262144","--max-num-seqs","16","--kv-cache-dtype","fp8","--gpu-memory-utilization","0.70","--linear-backend","b12x","--moe-backend","b12x","--attention-backend","FLASHINFER","--mamba-backend","flashinfer","--mamba-ssm-cache-dtype","bfloat16","--mamba-cache-dtype","bfloat16","--seed","42","--block-size","64","--prefix-match-unit","64","--enable-mamba-shared-prefix-checkpoint","--limit-mm-per-prompt.image","16","--limit-mm-per-prompt.video","2","--video-pruning-rate","0.5","--media-io-kwargs.video.fps","2","--media-io-kwargs.video.num_frames","256","--allowed-media-domains","media.invalid","--enable-auto-tool-choice","--tool-call-parser","qwen3_coder","--enable-prompt-tokens-details","--reasoning-parser-plugin","/root/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB/snapshots/c6cc2e8ffa353403213d6b9b18aaf8eb684d705a/ultra_v3_reasoning_parser.py","--reasoning-parser","ultra_v3","--speculative-config.method","dflash","--speculative-config.model","nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr","--speculative-config.revision","4c6a5613b6d887194a3a5493e3a18a5b3930bb86","--speculative-config.num_speculative_tokens","7","--speculative-config.rejection_sample_method","standard","--speculative-config.attention_backend","FLASHINFER","--speculative-config.draft_load_config.load_format","auto","--config","/out/engine-secrets.yaml","--middleware","vllm_engine_auth.RequireKey","--host","127.0.0.1","--port","30000"],"kv_pool_tokens":6520000,"max_context_len":262144,"storage_requirements":{"note":"26 GB checkpoint+drafter weights (USB SSD on the reference box) + 25 GB image.","ssd_gb":51}},"models":[{"slug":"nemotron-3-5-bolt-74b-a9b-24gb","name":"Nemotron-3.5-Bolt-74B-A9B-24GB","hf_repo":"nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB","quant":"iq2-xxs-experts-nvfp4-mamba-q8-0-rest"}],"hardware":[{"slug":"dgx-spark","vendor":"nvidia","card":"DGX Spark (GB10, unified memory)","card_family":"dgx-spark","vram_gb":128,"device_count":1}],"generation":"blackwell","tags":["262k","b12x","fp8-kv","gb10","mtp","multimodal","prefix-caching","speculative-decoding","video-input","vision"],"tasks":["chat","code","vision"],"aa":null,"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Reference-box measurements 2026-10-15/16 (docs/nemotron-lane.md @ a1139faf), real workloads with reasoning off, median of 3. points[].context_len is a workload marker (N + offset), not a served-context limit; the served window is 262,144.\", \"tok_s\": 140, \"points\": [{\"notes\": \"code-rewrite workload, single stream, drafter on (mean acceptance 4.0 tok/pass); marker = workload 1 + 10240 offset\", \"concurrency\": 1, \"context_len\": 10240, \"decode_tok_s\": 140}, {\"notes\": \"summarise workload, single stream, drafter on; marker = workload 2 + 20480 offset\", \"concurrency\": 1, \"context_len\": 20480, \"decode_tok_s\": 63}, {\"notes\": \"prose workload, single stream, drafter on; marker = workload 3 + 30720 offset\", \"concurrency\": 1, \"context_len\": 30720, \"decode_tok_s\": 50}, {\"notes\": \"drafter off, single stream (any text); marker = workload 4 + 40960 offset\", \"concurrency\": 1, \"context_len\": 40960, \"decode_tok_s\": 31}, {\"notes\": \"aggregate tok/s at 1 stream, drafter on, gmu 0.70 (median of 3 passes); marker = streams 1 + 51200\", \"concurrency\": 1, \"context_len\": 51200, \"decode_tok_s\": 74.7999999999999971578290569595992565155029296875}, {\"notes\": \"aggregate tok/s at 4 streams, drafter on, gmu 0.70 (median of 3); marker = streams 4 + 61440\", \"concurrency\": 4, \"context_len\": 61440, \"decode_tok_s\": 113.400000000000005684341886080801486968994140625}, {\"notes\": \"aggregate tok/s at 8 streams, drafter on, gmu 0.70 (median of 3); marker = streams 8 + 71680\", \"concurrency\": 8, \"context_len\": 71680, \"decode_tok_s\": 163.599999999999994315658113919198513031005859375}, {\"notes\": \"aggregate tok/s at 16 streams, drafter on, gmu 0.70 (median of 3); marker = streams 16 + 81920\", \"concurrency\": 16, \"context_len\": 81920, \"decode_tok_s\": 201.19999999999998863131622783839702606201171875}, {\"notes\": \"aggregate tok/s at 16 streams, drafter OFF (one pass, gmu 0.45); marker = streams 16 no-draft + 92160\", \"concurrency\": 16, \"context_len\": 92160, \"decode_tok_s\": 178}, {\"notes\": \"first token after ~192 s at a 252,000-token prompt (~1300-1800 tok/s prefill)\", \"ttft_ms\": 192000, \"concurrency\": 1, \"context_len\": 262144}], \"context_len\": 262144}"}],"perf_points":[{"id":2102,"context_len":10240,"prefill_tok_s":null,"decode_tok_s":140,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"code-rewrite workload, single stream, drafter on (mean acceptance 4.0 tok/pass); marker = workload 1 + 10240 offset"},{"id":2103,"context_len":20480,"prefill_tok_s":null,"decode_tok_s":63,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"summarise workload, single stream, drafter on; marker = workload 2 + 20480 offset"},{"id":2104,"context_len":30720,"prefill_tok_s":null,"decode_tok_s":50,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"prose workload, single stream, drafter on; marker = workload 3 + 30720 offset"},{"id":2105,"context_len":40960,"prefill_tok_s":null,"decode_tok_s":31,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"drafter off, single stream (any text); marker = workload 4 + 40960 offset"},{"id":2106,"context_len":51200,"prefill_tok_s":null,"decode_tok_s":74.7999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"aggregate tok/s at 1 stream, drafter on, gmu 0.70 (median of 3 passes); marker = streams 1 + 51200"},{"id":2107,"context_len":61440,"prefill_tok_s":null,"decode_tok_s":113.400000000000005684341886080801486968994140625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"aggregate tok/s at 4 streams, drafter on, gmu 0.70 (median of 3); marker = streams 4 + 61440"},{"id":2108,"context_len":71680,"prefill_tok_s":null,"decode_tok_s":163.599999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate tok/s at 8 streams, drafter on, gmu 0.70 (median of 3); marker = streams 8 + 71680"},{"id":2109,"context_len":81920,"prefill_tok_s":null,"decode_tok_s":201.19999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":16,"claimed_by":"submitter","notes":"aggregate tok/s at 16 streams, drafter on, gmu 0.70 (median of 3); marker = streams 16 + 81920"},{"id":2110,"context_len":92160,"prefill_tok_s":null,"decode_tok_s":178,"ttft_ms":null,"concurrency":16,"claimed_by":"submitter","notes":"aggregate tok/s at 16 streams, drafter OFF (one pass, gmu 0.45); marker = streams 16 no-draft + 92160"},{"id":2111,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":null,"ttft_ms":192000,"concurrency":1,"claimed_by":"submitter","notes":"first token after ~192 s at a 252,000-token prompt (~1300-1800 tok/s prefill)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB","pin":"c6cc2e8ffa353403213d6b9b18aaf8eb684d705a"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr","pin":"4c6a5613b6d887194a3a5493e3a18a5b3930bb86"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/vllm/vllm-openai","pin":"nightly-77154ca6 + b12x 1.5.1"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/hasso5703/dgx-spark-qwen38/commit/9e63ad54e2c0cacb489811f5eb243181cb4bddd5","pin":"9e63ad54e2c0cacb489811f5eb243181cb4bddd5"}],"links":{"self":"/api/v1/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k.json","alternate":"/r/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:26.717952-06","created_at":"2026-10-20 21:49:15.596056-06","speeds":{"decode_tok_s":63,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}