Qwen3.8-Flash-Next FP4-Local 1x DGX Spark vLLM b12x

community Community submission — verify on your own hardware.

recipe by sojufx↗ · listed by localai-recipes

revision r3 · commit 5fc0d101d8 · updated 21 h ago · license CC-BY-4.0

chat code vision fp8-kv mtp-speculative vision mtp3 long-context speculative prefix-caching nvfp4 262k b12x fp4-local localhost-only modelopt tool-calling

Recipe by sojufx. Serves NVIDIA's nvidia/Qwen3.8-Flash-Next-FP4-Local (96 GiB NVFP4 quantization-aware-distilled checkpoint in a new compressed NVFP4 scale format) on one DGX Spark (GB10) via the digest-pinned eugr/spark-vllm-b12x community container (reported vLLM 0.1.dev21830+g8244d1c44.d20261015) - the model card requires community b12x containers until the format lands upstream. FP8 KV cache, exact MTP K=3 with standard rejection sampling, 262,144-token context, 4 scheduler slots, 4096-token batched prefill, disk-backed io_uring PLE, prefix cache with Mamba align, vision plus qwen3_xml tool calling, thinking off by default, localhost-only 127.0.0.1:8001 OpenAI-compatible backend with a deliberate no-replace/no-port-take guard. Measured 2026-10-15/16 on the author's Spark: 66.2 tok/s single-stream sparkDash counting decode, 166.4 tok/s at C4, 41-54 tok/s single-stream on the four-prompt wall-clock suite - different measurement bases, not interchangeable.

Launch command

'docker run -d --pull never --name vllm-nvidia-fp4-local --init --restart unless-stopped' '--gpus all --ipc host --network host --shm-size=16g' '--ulimit memlock=-1 --ulimit stack=67108864' '--security-opt seccomp=unconfined --security-opt apparmor=unconfined' '-v "$HOME/.cache/huggingface:/root/.cache/huggingface"' '-v "$HOME/.cache/nvidia-fp4-local/vllm:/root/.cache/vllm"' '-v "$HOME/.cache/nvidia-fp4-local/flashinfer:/root/.cache/flashinfer"' '-v "$HOME/.cache/nvidia-fp4-local/triton:/root/.triton"' '-v "$HOME/.cache/nvidia-fp4-local/b12x:/root/.cache/b12x"' '-e HF_HUB_OFFLINE=1 -e VLLM_PLE_TABLE_MEMORY=disk -e B12X_DISK_BACKEND=io_uring' '-e CUTE_DSL_ARCH=sm_121a -e SAFETENSORS_FAST_GPU=1' '-e VLLM_WORKER_MULTIPROC_METHOD=spawn -e VLLM_SSM_CONV_STATE_LAYOUT=DS' '-e VLLM_USE_AOT_COMPILE=1 -e VLLM_USE_MEGA_AOT_ARTIFACT=1 -e VLLM_USE_V2_MODEL_RUNNER=1' '-e B12X_POLICY_MODE=auto -e VLLM_MXFP8_LM_HEAD=1' '-e VLLM_B12X_MOE_FP4_LAYER_MAX_INPUT_SCALE=w13 -e VLLM_B12X_MOE_FP4_FORCE_A16=1' '-e B12X_W4A16_A4_PREFILL=1' '--entrypoint /usr/local/bin/vllm eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595' 'serve /root/.cache/huggingface/hub/models--nvidia--Qwen3.8-Flash-Next-FP4-Local/snapshots/b9ce18a6887e18c089fdea8509a33e63429a3300' '--host 127.0.0.1 --port 8001 --served-model-name qwen38-flash-next-fp4-local' '--tensor-parallel-size 1 --pipeline-parallel-size 1' '--mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill' '--dtype bfloat16 --kv-cache-dtype fp8 --quantization modelopt_mixed' '--block-size 16 --load-format b12x' '--max-model-len 262144 --max-num-seqs 4 --max-num-batched-tokens 4096' '--speculative-config '\''{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"standard"}'\''' '--gdn-decode-kernel b12x --linear-backend b12x --moe-backend b12x' '--no-enable-flashinfer-autotune --mm-encoder-tp-mode data' '--reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice' '--default-chat-template-kwargs '\''{"enable_thinking":false}'\''' '--compilation-config '\''{"pass_config":{"fuse_act_quant":true}}'\''' '--gpu-memory-utilization 0.8'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json

Engine

vllm · source↗ · bb8c9d0e↗ · docker.io/eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595↗

Install note: NOT stock vLLM: requires the digest-pinned eugr/spark-vllm-b12x community container - the checkpoint's compressed NVFP4 scale format is unsupported upstream (model card requires community b12x containers until features land upstream). bash download.sh pulls the pinned image and HF snapshot; bash start.sh launches (refuses to replace an existing container or take an occupied port). Privileged-ish container: --ipc host --network host, seccomp/apparmor unconfined, shm 16g. First start ~14 min incl. compile and warm-up. Read upstream terms before downloading; DGX Spark GB10 host required (CUTE_DSL_ARCH=sm_121a).

Model

Qwen3.8-Flash-Next-FP4-Local (nvfp4) · nvidia/Qwen3.8-Flash-Next-FP4-Local↗
AA intelligence 40 (Artificial Analysis · measured)

Hardware

NVIDIA DGX Spark 128GB ×1 (Blackwell)

Context

max context: 256K tokens

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
2,048	1	—	66.2 tok/s	42.2 ms	reported
2,048	2	—	106.3 tok/s	—	reported
2,048	4	—	166.4 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Two distinct measured bases (benchmarks/RESULTS.md @ bb8c9d0, recorded 2026-10-15/16), all loopback, thinking off, no failed requests. sparkDash 1.8.6 structured counting decode: aggregate + per-stream decode tok/s over the post-first-token window, 400-token cap, temperature 0, top_p 1, 32-token warm-up; short-prompt band marker context_len=2048; counting favors speculative decoding and is not a quality benchmark. Four-prompt wall-clock suite (256-token cap, temperature 0, prefill included): C1 40.8/48.8/54.3/44.2 and C4 aggregate 94.5/124.6/125.1/104.1 tok/s for agent-tool/code-edit/generic-coding/production-review prompts - NOT equivalent to the sparkDash rows and not indexed as points (source states no per-prompt context lengths). Follow-up TTFT reported only at K=4 (4.16 s cold, 0.22 s follow-ups), not remeasured at K=3.

notes

Two distinct measured bases (benchmarks/RESULTS.md @ bb8c9d0, recorded 2026-10-15/16), all loopback, thinking off, no failed requests. sparkDash 1.8.6 structured counting decode: aggregate + per-stream decode tok/s over the post-first-token window, 400-token cap, temperature 0, top_p 1, 32-token warm-up; short-prompt band marker context_len=2048; counting favors speculative decoding and is not a quality benchmark. Four-prompt wall-clock suite (256-token cap, temperature 0, prefill included): C1 40.8/48.8/54.3/44.2 and C4 aggregate 94.5/124.6/125.1/104.1 tok/s for agent-tool/code-edit/generic-coding/production-review prompts - NOT equivalent to the sparkDash rows and not indexed as points (source states no per-prompt context lengths). Follow-up TTFT reported only at K=4 (4.16 s cold, 0.22 s follow-ups), not remeasured at K=3.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local/tree/b9ce18a6887e18c089fdea8509a33e63429a3300
b9ce18a6887e18c089fdea8509a33e63429a3300
Engine image https://hub.docker.com/r/eugr/spark-vllm-b12x
sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595

Revision notes

Ingest t_816f74e2: all launch flags transcribed from start.sh and benchmarks from benchmarks/RESULTS.md at live repo HEAD bb8c9d0e70d9dc840a9351fb8ae0e73e265bd715 (2026-10-16, verified via git ls-remote this run). Corrections to the scout card: its pinned commit 8ea8164 is not present in the live repo history (repo re-published/force-pushed; live history has 2 commits, 394c910 then bb8c9d0); the card said the README only references the digest, but start.sh inlines it (resolved: sha256:03a956...bb3595); model revision b9ce18a6 verified via HF API (96,486,615,005 params; license nvidia-open-model-agreement - read upstream terms before download; recipe code MIT). Image digest pinned in start.sh, not stated in the README. TTFT table in RESULTS.md is K=4 only (not remeasured at K=3). No TokenV3 overlay / reduced draft vocabulary - plain FP4-Local lane, distinct from ttlogan's Uncensored ModelOpt lane and LIL QAD lanes. Container restart unless-stopped; localhost-only bind, no backend API key by design - gateway must authenticate.

Revision history

5fc0d101d8 recipe: update qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x (r3: /artifacts) 15 h ago diff
15cd3fef01 recipe: update qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x (r2: /max_context_len) 11 d ago diff
1f26a38e3b recipe: create qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x 15 d ago

Fork

Export

recipe.yaml (raw)