Qwen3.8-Flash-Next NVFP4 1x DGX Spark vLLM 71.3 c1 (salient-data)

Launch command

docker run --rm -i --name qwen38-flash-next --gpus all --network host --ipc host --shm-size 32g --ulimit memlock=-1:-1 --memory 112g --memory-swap 112g -v '$HF_HUB_CACHE/models--nvidia--Qwen3.8-Flash-Next-NVFP4:/hfrepo:ro' -v '$HOME/.cache/qwen38-next-dgx-recipe:/root/.cache' -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e VLLM_PLE_FILE_GATHER_WORKERS=64 -e QWEN4EXP_DRAFT_VOCAB=65536 -e VLLM_USE_V2_MODEL_RUNNER=1 -e QWEN38_FP8_DENSE=gdn,attn,shared,mtp,lm_head --entrypoint vllm ghcr.io/salient-data/qwen38-next-dgx@sha256:d55a2740651064f6c8aba20bc9ada104ae9df0a793cc87d0c170d312aa052b4d serve /hfrepo/snapshots/fc694b54fb0174e0913e6adf86691ef85a4ead47 --host 127.0.0.1 --port 8000 --served-model-name qwen3.8-flash-next --trust-remote-code --quantization modelopt --tensor-parallel-size 1 --max-model-len 262144 --max-num-seqs 6 --gpu-memory-utilization 0.80 --max-num-batched-tokens 4096 --engram-config '{"host_file_gather": true}' --compilation-config '{"mode": 0, "cudagraph_mode": "FULL_DECODE_ONLY"}' --no-enable-flashinfer-autotune --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml --default-chat-template-kwargs '{"enable_thinking": false}' --kv-cache-dtype fp8_e4m3 --use-replayssm --kv-cache-dtype-skip-layers 48 --mamba-ssm-cache-dtype bfloat16 --enable-prefix-caching --speculative-config '{"method": "mtp", "num_speculative_tokens": 6, "draft_lm_head_quantization": "nvfp4", "draft_confidence_threshold": 0.7, "draft_confidence_fallback_depth": 3, "draft_token_map": "/root/.cache/draft-token-maps/d22ff5232d12f6935add32541a2a447976349217e5b125669fab0e92e3e2a20b.json", "draft_token_map_dynamic_rows": 16384, "draft_token_map_dynamic_rank": 256, "draft_token_map_quantization": "nvfp4", "draft_token_map_max_rows": 1}'

runtime

env: 'Same vllm serve args (recipe/serve.args + recipe/serve.env) for the prod and dev paths; launch.sh adds --host/--port from BIND/PORT env (127.0.0.1:8000 default, BIND=0.0.0.0 exposes unauthenticated unless VLLM_API_KEY is set). Drafting: up to 6 tokens single-stream with early stop when the drafter top-1 probability falls below 0.7 (draft_confidence_fallback_depth 3 with 2-6 concurrent requests). Dense BF16 layers cast to FP8 block-128 (197 linears, QWEN38_FP8_DENSE=gdn,attn,shared,mtp,lm_head). Draft head: context-aware sparse head (32,766 fixed token ids + 16,384 rows chosen per context, NVFP4; token map d22ff523... copied into the cache mount by launch.sh). MTP-layer KV BF16. Prefix caching ON (shortens TTFT; author estimates at most 1-2% of the 6-stream aggregate). --max-num-seqs 6.'
notes: "Launch argv transcribed from recipe/serve.args + serve.env + launch.sh @ 7432036e7d (identity.json in results/2026-10-22-8e2beaa3/ records the measured run's exact args/env). launch.sh also: warms the server with two known-answer requests so first real answers are not slowed by kernel compilation (WARMUP=0 skips), runs a memory watch sampling every 10 s that warns when any other process grows 2 GiB+ (MEMWATCH=0 skips), and verifies after load (./recipe/verify.sh; known-answer checks battery). Thinking off by default; per-request enable_thinking=true. -- overrides: ./recipe/launch.sh -- --flag value changes one run's args. API is a standard vllm serve OpenAI endpoint. Results dir per build results/2026-10-22-8e2beaa3/ (harness raw + quality records, incl. harness-confirm/ and the 2026-10-25 harness-clean/ first-time run with gsm8k.json and boots.txt); tools for the fixed checks not yet published."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data.json

Install requirements

• Docker (or a compatible OCI container runtime)
• NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

• Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
• GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
• Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
• Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

• Weights https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4
  fc694b54fb0174e0913e6adf86691ef85a4ead47 (repo-wide pin; model-files.sha256 pins the small files; an FP8_PB_WO alias was added to tonyd2wild's stack to load it)
• Engine image https://ghcr.io/salient-data/qwen38-next-dgx@sha256:d55a2740651064f6c8aba20bc9ada104ae9df0a793cc87d0c170d312aa052b4d
  PROD_IMAGE in recipe/pins.env @ 7432036e7d (registry digest, cosign-signed, SPDX SBOM); image id sha256:3eda66c8163064d664465a803ca09fb66ab626814b6bba7e58a6fd97a3cde282
• Reference https://github.com/salient-data/qwen38-next-dgx-recipe
  HEAD 7432036e7d549ed266d16832c43ea960f2cb7d55 (main, 2026-10-25) at revision 2026-10-28; tag build-2026-10-22-8e2beaa3 @ eed3f7a47f97; 14 patches in recipe/patches/PROVENANCE.md
• Reference https://github.com/tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark
  bench harness bench_categories.py @ 6ad1c8f15cbab1ababd2048e8e5f94094dbfc4a0 (sha256 5d9b6954...e0d3), fetched and run unchanged by bench/run-third-party.sh; tonyd2wild's stack is also the comparator home

Revision notes

Community ingest of salient-data's recipe (Apache-2.0), lar-scout ingest t_5d21dce4 2026-10-22. Attribution: submitter/listed_by localai-recipes; recipe by Jeremy Newhouse / salient-data; announced in local-inference-lab #qwen-38-flash-next 2026-10-21 17:48Z. All perf numbers author-measured T0 with a named third-party harness and published raw data per build (results/); the only non-author numbers are tonyd2wild's published comparator rows (flagged in-line, different host/date, revision unstated). The recipe is explicitly outside the model card's support statement (B200/B300 only; GB10 unlisted). WATCH: author says a private inference engine beats this vLLM build and may publish findings/PRs after Flare testing (17:52Z follow-up). corrections_to_card: the bench-harness repo is tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark (card artifact named the DeepSeek-V4.1-Flash repo - that is the comparator-stack home; the harness lives in the QFN repo @ 6ad1c8f); serve --max-model-len is 262144 (card prose used 131072 - that is the needle-verified depth, kept for perf points); driver pin is 580.178.04 open kernel module; README carries a generated verified-date block (re-check the generated:verified date + CHANGELOG on every drift pass). BUILD REVISION (kanban t_976fa730, owner go 2026-10-28 Casey 'for 1 we should update' via t_b21a5b73): recipe re-pinned f309019de1..7432036e7d (build 2026-10-22-8e2beaa3, commit #7 2026-10-25, live origin HEAD verified 2026-10-28). Patch series renumbered to the 14-patch set at the pin: DELETED 0005-vllm58737-fuse-the-mamba-align-block-table + 0006-vllm59260-warm-up-top-k-top-p-and-greedy; NEW 0013-vllm62841-avoid-re-marking-committed-prompt-tail-boundaries + 0014-draft-head-per-batch; RENUMBERED 0005-quiet-boot 0006-mtp-shard-filter 0007-vllm54912-widen-the-qsa-raw-key-ring 0008-vllm60068-confidence-stop-for-autoregressive-drafting 0009-skinny-gemm-rows 0010-vllm58863-recoverssm-for-qwen-gdn-and 0011-vllm59740-context-aware-sparse-lm-head 0012-moe-batched-load (patches.sha256 byte-verified 14/14; PROVENANCE.md applies at vLLM 8e2beaa3bff595577a069d49d96bde9074294ec2). serve.args/serve.env re-transcribed verbatim @ 7432036e7d: QWEN4EXP_DRAFT_VOCAB 0->65536; speculative-config adds draft_lm_head_quantization=nvfp4 + draft_token_map_max_rows=1 (launch argv delta 3 of 86 tokens, positions 35/42/85). PROD_IMAGE d55a27406510 (image id 3eda66c8...), base digest 167ccd86; engine vLLM 0.32.1rc1.dev197+g8e2beaa3b / torch 2.14.0+cu130 / FlashInfer 0.7.2. Measured table republished from results/2026-10-22-8e2beaa3/summary.md (69.3 c1 / 41.0 prose / 91.9 c6; 2/2 perf points preserved, values re-measured on the new build); quality set unchanged vs quality.json at the pin (GSM8K 99/100, arith 12/12, ten-task 9/10, needle 20/20, canary 0 failures). memory-budget.md changed at the pin: weights 73.51->73.57 GiB, KV pool 981,604-1,039,008 -> 1,002,985-1,094,166 tokens on four boots -> kv_pool_tokens 981604->1002985. HF pins resolve: nvidia/Qwen3.8-Flash-Next-NVFP4 @ fc694b54 HTTP 200 (2026-10-28); draft token map d22ff523... self-hash sha256==filename verified. Title/slug unchanged (legacy 71.3 in the slug is a historical name, not a current claim).

Revision history

• 85e2e4120e recipe: update qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data (r3: /provenance/credit) 45 h ago diff
• 663a7b322a recipe: update qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data (r2: /engine, /artifacts, /config/launch, /config/runtime/notes, /summary, /kv_pool_tokens, /performance, /notes — buil 45 h ago diff
• 605c776c07 recipe: create qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data 9 d ago

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)