{"slug":"qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x","title":"Qwen3.8-Flash-Next FP4-Local 1x DGX Spark vLLM b12x","summary":"Recipe by sojufx. Serves NVIDIA's nvidia/Qwen3.8-Flash-Next-FP4-Local (96 GiB NVFP4 quantization-aware-distilled checkpoint in a new compressed NVFP4 scale format) on one DGX Spark (GB10) via the digest-pinned eugr/spark-vllm-b12x community container (reported vLLM 0.1.dev21830+g8244d1c44.d20261015) - the model card requires community b12x containers until the format lands upstream. FP8 KV cache, exact MTP K=3 with standard rejection sampling, 262,144-token context, 4 scheduler slots, 4096-token batched prefill, disk-backed io_uring PLE, prefix cache with Mamba align, vision plus qwen3_xml tool calling, thinking off by default, localhost-only 127.0.0.1:8001 OpenAI-compatible backend with a deliberate no-replace/no-port-take guard. Measured 2026-10-15/16 on the author's Spark: 66.2 tok/s single-stream sparkDash counting decode, 166.4 tok/s at C4, 41-54 tok/s single-stream on the four-prompt wall-clock suite - different measurement bases, not interchangeable.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"5fc0d101d8e9b1c6a1419f3d465e21649f4ecbad","content_hash":"be42677d0cc5ca7f71fe72210e756f1622fe3604","created_at":"2026-10-29 21:32:27.177524-06","notes":"Ingest t_816f74e2: all launch flags transcribed from start.sh and benchmarks from benchmarks/RESULTS.md at live repo HEAD bb8c9d0e70d9dc840a9351fb8ae0e73e265bd715 (2026-10-16, verified via git ls-remote this run). Corrections to the scout card: its pinned commit 8ea8164 is not present in the live repo history (repo re-published/force-pushed; live history has 2 commits, 394c910 then bb8c9d0); the card said the README only references the digest, but start.sh inlines it (resolved: sha256:03a956...bb3595); model revision b9ce18a6 verified via HF API (96,486,615,005 params; license nvidia-open-model-agreement - read upstream terms before download; recipe code MIT). Image digest pinned in start.sh, not stated in the README. TTFT table in RESULTS.md is K=4 only (not remeasured at K=3). No TokenV3 overlay / reduced draft vocabulary - plain FP4-Local lane, distinct from ttlogan's Uncensored ModelOpt lane and LIL QAD lanes. Container restart unless-stopped; localhost-only bind, no backend API key by design - gateway must authenticate."},"engine_source":{"type":"docker_image","git_url":"https://github.com/sojufx/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark-Recipe","base_engine":"vllm","pip_package":null,"base_version":"0.1.dev21830+g8244d1c44.d20261015","image_digest":"docker.io/eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595","install_note":"NOT stock vLLM: requires the digest-pinned eugr/spark-vllm-b12x community container - the checkpoint's compressed NVFP4 scale format is unsupported upstream (model card requires community b12x containers until features land upstream). bash download.sh pulls the pinned image and HF snapshot; bash start.sh launches (refuses to replace an existing container or take an occupied port). Privileged-ish container: --ipc host --network host, seccomp/apparmor unconfined, shm 16g. First start ~14 min incl. compile and warm-up. Read upstream terms before downloading; DGX Spark GB10 host required (CUTE_DSL_ARCH=sm_121a).","pinned_commit":"bb8c9d0e70d9dc840a9351fb8ae0e73e265bd715"},"credit":[{"name":"sojufx","url":"https://github.com/sojufx/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark-Recipe"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["docker run -d --pull never --name vllm-nvidia-fp4-local --init --restart unless-stopped","--gpus all --ipc host --network host --shm-size=16g","--ulimit memlock=-1 --ulimit stack=67108864","--security-opt seccomp=unconfined --security-opt apparmor=unconfined","-v \"$HOME/.cache/huggingface:/root/.cache/huggingface\"","-v \"$HOME/.cache/nvidia-fp4-local/vllm:/root/.cache/vllm\"","-v \"$HOME/.cache/nvidia-fp4-local/flashinfer:/root/.cache/flashinfer\"","-v \"$HOME/.cache/nvidia-fp4-local/triton:/root/.triton\"","-v \"$HOME/.cache/nvidia-fp4-local/b12x:/root/.cache/b12x\"","-e HF_HUB_OFFLINE=1 -e VLLM_PLE_TABLE_MEMORY=disk -e B12X_DISK_BACKEND=io_uring","-e CUTE_DSL_ARCH=sm_121a -e SAFETENSORS_FAST_GPU=1","-e VLLM_WORKER_MULTIPROC_METHOD=spawn -e VLLM_SSM_CONV_STATE_LAYOUT=DS","-e VLLM_USE_AOT_COMPILE=1 -e VLLM_USE_MEGA_AOT_ARTIFACT=1 -e VLLM_USE_V2_MODEL_RUNNER=1","-e B12X_POLICY_MODE=auto -e VLLM_MXFP8_LM_HEAD=1","-e VLLM_B12X_MOE_FP4_LAYER_MAX_INPUT_SCALE=w13 -e VLLM_B12X_MOE_FP4_FORCE_A16=1","-e B12X_W4A16_A4_PREFILL=1","--entrypoint /usr/local/bin/vllm eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595","serve /root/.cache/huggingface/hub/models--nvidia--Qwen3.8-Flash-Next-FP4-Local/snapshots/b9ce18a6887e18c089fdea8509a33e63429a3300","--host 127.0.0.1 --port 8001 --served-model-name qwen38-flash-next-fp4-local","--tensor-parallel-size 1 --pipeline-parallel-size 1","--mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill","--dtype bfloat16 --kv-cache-dtype fp8 --quantization modelopt_mixed","--block-size 16 --load-format b12x","--max-model-len 262144 --max-num-seqs 4 --max-num-batched-tokens 4096","--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3,\"rejection_sample_method\":\"standard\"}'","--gdn-decode-kernel b12x --linear-backend b12x --moe-backend b12x","--no-enable-flashinfer-autotune --mm-encoder-tp-mode data","--reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice","--default-chat-template-kwargs '{\"enable_thinking\":false}'","--compilation-config '{\"pass_config\":{\"fuse_act_quant\":true}}'","--gpu-memory-utilization 0.8"],"max_context_len":262144,"storage_requirements":{"note":"Checkpoint download ~102 GB (HF usedStorage 102,252,102,010 B); README says allow additional space for image and compile caches.","ssd_gb":102}},"models":[{"slug":"qwen3-8-flash-next-fp4-local","name":"Qwen3.8-Flash-Next-FP4-Local","hf_repo":"nvidia/Qwen3.8-Flash-Next-FP4-Local","quant":"nvfp4"}],"hardware":[{"slug":"dgx-spark","vendor":"nvidia","card":"DGX Spark (GB10, unified memory)","card_family":"dgx-spark","vram_gb":128,"device_count":1}],"generation":"blackwell","tags":["262k","b12x","fp4-local","fp8-kv","localhost-only","long-context","modelopt","mtp3","mtp-speculative","nvfp4","prefix-caching","speculative","tool-calling","vision"],"tasks":["chat","code","vision"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Two distinct measured bases (benchmarks/RESULTS.md @ bb8c9d0, recorded 2026-10-15/16), all loopback, thinking off, no failed requests. sparkDash 1.8.6 structured counting decode: aggregate + per-stream decode tok/s over the post-first-token window, 400-token cap, temperature 0, top_p 1, 32-token warm-up; short-prompt band marker context_len=2048; counting favors speculative decoding and is not a quality benchmark. Four-prompt wall-clock suite (256-token cap, temperature 0, prefill included): C1 40.8/48.8/54.3/44.2 and C4 aggregate 94.5/124.6/125.1/104.1 tok/s for agent-tool/code-edit/generic-coding/production-review prompts - NOT equivalent to the sparkDash rows and not indexed as points (source states no per-prompt context lengths). Follow-up TTFT reported only at K=4 (4.16 s cold, 0.22 s follow-ups), not remeasured at K=3.\", \"tok_s\": 66.1700000000000017053025658242404460906982421875, \"points\": [{\"notes\": \"sparkDash 1.8.6 structured counting decode, aggregate tok/s over the post-first-token window (not wall time), 400-token cap, temperature 0, top_p 1, thinking off, 32-token warm-up; short-prompt band marker context_len=2048. Not comparable to the four-prompt suite.\", \"ttft_ms\": 42.1700000000000017053025658242404460906982421875, \"concurrency\": 1, \"context_len\": 2048, \"decode_tok_s\": 66.1700000000000017053025658242404460906982421875}, {\"notes\": \"sparkDash 1.8.6 structured counting decode, aggregate tok/s over the post-first-token window (not wall time), 400-token cap, temperature 0, top_p 1, thinking off, 32-token warm-up; short-prompt band marker context_len=2048. Not comparable to the four-prompt suite.\", \"concurrency\": 2, \"context_len\": 2048, \"decode_tok_s\": 106.2699999999999960209606797434389591217041015625}, {\"notes\": \"sparkDash 1.8.6 structured counting decode, aggregate tok/s over the post-first-token window (not wall time), 400-token cap, temperature 0, top_p 1, thinking off, 32-token warm-up; short-prompt band marker context_len=2048. Not comparable to the four-prompt suite.\", \"concurrency\": 4, \"context_len\": 2048, \"decode_tok_s\": 166.3600000000000136424205265939235687255859375}], \"ttft_ms\": 42.1700000000000017053025658242404460906982421875, \"context_len\": 2048}"}],"perf_points":[{"id":2120,"context_len":2048,"prefill_tok_s":null,"decode_tok_s":66.1700000000000017053025658242404460906982421875,"ttft_ms":42.1700000000000017053025658242404460906982421875,"concurrency":1,"claimed_by":"submitter","notes":"sparkDash 1.8.6 structured counting decode, aggregate tok/s over the post-first-token window (not wall time), 400-token cap, temperature 0, top_p 1, thinking off, 32-token warm-up; short-prompt band marker context_len=2048. Not comparable to the four-prompt suite."},{"id":2121,"context_len":2048,"prefill_tok_s":null,"decode_tok_s":106.2699999999999960209606797434389591217041015625,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"sparkDash 1.8.6 structured counting decode, aggregate tok/s over the post-first-token window (not wall time), 400-token cap, temperature 0, top_p 1, thinking off, 32-token warm-up; short-prompt band marker context_len=2048. Not comparable to the four-prompt suite."},{"id":2122,"context_len":2048,"prefill_tok_s":null,"decode_tok_s":166.3600000000000136424205265939235687255859375,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"sparkDash 1.8.6 structured counting decode, aggregate tok/s over the post-first-token window (not wall time), 400-token cap, temperature 0, top_p 1, thinking off, 32-token warm-up; short-prompt band marker context_len=2048. Not comparable to the four-prompt suite."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local/tree/b9ce18a6887e18c089fdea8509a33e63429a3300","pin":"b9ce18a6887e18c089fdea8509a33e63429a3300"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/eugr/spark-vllm-b12x","pin":"sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json","alternate":"/r/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:27.177524-06","created_at":"2026-10-20 21:49:17.622605-06","speeds":{"decode_tok_s":66.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}