Qwen3.8-27B SGLang NVFP4 DFlash2 on 1x DGX Spark (deterministic)

community Community submission — verify on your own hardware.

recipe by hasso5703↗ · listed by localai-recipes

revision r3 · commit 307fa2334d · updated 2 d ago · license MIT

chat code single-gpu dflash2 long-context speculative yarn-1m dgx-spark gb10 sm121 agent-serving deterministic-decode

First dense Qwen3.8-27B recipe on 1x DGX Spark (GB10, 128 GB unified): SGLang v0.5.22 release image + NVFP4 target + DFlash2 speculative decoding (NVFP4 calibrated 5-layer draft, depth 16), served by a boot-persistent systemd unit from a one-command installer. Author-measured 68.3 tok/s greedy median single stream (bench.sh, thinking on; v1.39.0 A/B 2026-10-23), 81.7 tok/s at 4 streams (serving-bench), 135-148 tok/s at 8 streams and 258 at 32. Fully pinned (image digest, checkpoint revisions, draft depth) and reproducible to the decimal across boots via --disable-flashinfer-autotune ("The boot lottery", BENCHMARKS.md); 1M-token YaRN context by default, native 262144 as fallback; OpenAI- and Anthropic-compatible API on :30001 through the box's keepalive proxy (the engine itself answers on 127.0.0.1:30000, on the box only), with opencode wired out of the box.

Launch command

python3 -m sglang.launch_server --context-length 1010000 --model-path RadixArk/Qwen3.8-27B-NVFP4 --revision 52d1adc5f38aa5ebf099c29ed7025ba34cfbb854 --tp-size 1 --served-model-name qwen3.8-27b --mem-fraction-static 0.76 --attention-backend flashinfer --chunked-prefill-size 8192 --disable-prefill-cuda-graph --cuda-graph-max-bs-decode 8 --disable-flashinfer-autotune --speculative-algorithm DFLASH --speculative-draft-model-path maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal --speculative-draft-model-revision bd7a934213c47a9e7ef69eef36bb3325f47fd1f1 --speculative-num-draft-tokens 16 --speculative-draft-model-quantization modelopt_fp4 --mamba-radix-cache-strategy extra_buffer --mamba-ssm-dtype bfloat16 --max-mamba-cache-size 96 --max-running-requests 8 --enable-torch-compile --torch-compile-max-bs 4 --num-continuous-decode-steps 2 --sleep-on-idle --enable-metrics --enable-cache-report --reasoning-parser qwen3 --tool-call-parser qwen3_coder --chat-template /out/chat-template-sglang.jinja --config /out/engine-secrets.yaml --host 127.0.0.1 --port 30000

runtime

env: 'HF_HUB_OFFLINE=1 | SGLANG_ENABLE_REQUEST_HEADER_OVERRIDES=1 | SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 (1m unit) | SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION=0 (GET /health generates nothing) | TORCHINDUCTOR_CACHE_DIR=/cache/inductor - docker run --rm --name qwen38-sglang --gpus all --memory 100g --memory-swap 100g --shm-size 16g --network host --ipc=host; mounts: HF cache -> /root/.cache/huggingface, ~/.config/qwen38 -> /out:ro (chat-template-sglang.jinja, engine-secrets.yaml), sglang-cache -> /cache'
notes: "Launch argv = qwen38-sglang-1m.service.template (the installer default since v1.12.1) with pins substituted, re-transcribed verbatim at repo HEAD 9e63ad54e2 (2026-10-26, release v1.40.3); keepalive proxy :30001 in front for agent CLIs, and it is the box's door from the network (proxy v6.47 at repo HEAD; PROXY_BIND defaults to 0.0.0.0, install.sh:800; the engine answers on 127.0.0.1:30000, on the box only). Its guards, added in proxy v6.46 (upstream v1.40.1): a network peer gets 64 KiB for its request line and headers in all (431 past it), 30 s without a byte while its body arrives, 413 for any body on /health, /ready or /metrics, and those three routes only - /health_generate needs the key. The box's key is the proxy port's gate, applied before a body is read; the bounds are upstream's, per CHANGELOG v1.40.1 and SECURITY.md. Delta vs the 3849269 transcription: --trust-remote-code dropped (no target or drafter ships checkpoint code; MODEL_CHOICE=custom needs the flag back in a drop-in) and --enable-cache-report added (#49, usage.prompt_tokens_details.cached_tokens). Native-262144 fallback: qwen38-sglang.service.template (same flags, --context-length dropped, mem-fraction 0.50) or foreground ./run.sh. Stock target shown; MODEL_CHOICE switches uncensored/fp8 targets (fp8 adds --kv-cache-dtype fp8_e4m3) and the flash lane serves Qwen3.8-Flash-Next 176B on the same box. ENGINE_BIND=127.0.0.1 default (0.0.0.0 opens to the network). That keeps the API port on the box, not the whole engine: PyTorch's distributed store binds every interface on a random port no setting narrows (still open upstream at v1.40.3, D05 of the 2026-10-24 review), takes no key, and a hostile connection can disturb the engine. Firewall the box to the ports you use before anything else (SECURITY.md's ufw block: allow lo, tailscale0, 22/tcp and 30001/tcp only, then default deny incoming). API key delivered via engine-secrets.yaml file merge, never argv."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic.json

Engine

sglang · source↗ · 9e63ad54↗ · lmsysorg/sglang@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae↗

Install note: One-command install on stock DGX OS (docker + NVIDIA container toolkit): curl get.sh | bash (or ./install.sh from a clone) installs image + checkpoints + boot-persistent systemd service and proves it serves; ./install.sh --no-service && ./run.sh for foreground. ~85 GB disk for the 27B lane (45 GB HF cache + 40 GB docker image); first boot 7-9 min (torch.compile + CUDA graph capture, cached), warm boots 5-7 min. MODEL_REV / DRAFT2_REPO / DRAFT2_REV / DRAFT2_TOKENS env overrides re-pin per run; tagged releases checkout cleanly (v1.1-v1.40.3).

Model

Qwen3.8-27B (nvfp4) · RadixArk/Qwen3.8-27B-NVFP4↗
AA intelligence 34

Hardware

NVIDIA DGX Spark 128GB ×1 (Blackwell)

Context

max context: 1,010,000 tokens

KV pool

473,664 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
262,144	4	—	81.7 tok/s	—	reported
262,144	8	—	141.5 tok/s	—	reported
262,144	32	—	258 tok/s	—	reported
1,010,000	1	—	68.3 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	All numbers author-measured on the reference DGX Spark - never third-party. 68.3 tok/s = midpoint of the two v0.5.22 ./bench.sh greedy-median runs 67.0 and 69.6 on the reference box 2026-10-23 (CHANGELOG v1.39.0 A/B, both images one night; v0.5.21 same night 63.3/60.5/69.0); serving-bench 1-stream output 38.8/38.5/38.8 tok/s, draft accepted length 5.05; needle still exact at 120,196 and 299,901 prompt tokens (113 s, 478 s); GSM8K quick-eval 244/250 on v0.5.22. Campaign history: 71.4 (2026-09-17 release-image campaign, CHANGELOG v1.14.0), 69.8 prior image, 65.3 (v1.9), 50 (v1.2 BF16 draft D8). c4 81.65 = midpoint of the v0.5.22 serving-bench 4-stream outputs 81.7/81.6 (v0.5.21 77.5/77.8/74.7, same A/B) - serving-bench output tok/s, a different instrument from bench.sh. c8/c32 rows are aggregate decode carried over from the v1.2 battery (README: the draft only helps concurrency); c32 needs --max-running-requests 32 and was still climbing.

notes

All numbers author-measured on the reference DGX Spark - never third-party. 68.3 tok/s = midpoint of the two v0.5.22 ./bench.sh greedy-median runs 67.0 and 69.6 on the reference box 2026-10-23 (CHANGELOG v1.39.0 A/B, both images one night; v0.5.21 same night 63.3/60.5/69.0); serving-bench 1-stream output 38.8/38.5/38.8 tok/s, draft accepted length 5.05; needle still exact at 120,196 and 299,901 prompt tokens (113 s, 478 s); GSM8K quick-eval 244/250 on v0.5.22. Campaign history: 71.4 (2026-09-17 release-image campaign, CHANGELOG v1.14.0), 69.8 prior image, 65.3 (v1.9), 50 (v1.2 BF16 draft D8). c4 81.65 = midpoint of the v0.5.22 serving-bench 4-stream outputs 81.7/81.6 (v0.5.21 77.5/77.8/74.7, same A/B) - serving-bench output tok/s, a different instrument from bench.sh. c8/c32 rows are aggregate decode carried over from the v1.2 battery (README: the draft only helps concurrency); c32 needs --max-running-requests 32 and was still climbing.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/RadixArk/Qwen3.8-27B-NVFP4
52d1adc5f38aa5ebf099c29ed7025ba34cfbb854 - STOCK_REV pinned in install.sh @ 3849269; NVFP4 release weights, lossless vs Q8 reference + live canaries per README
Drafter https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal
bd7a934213c47a9e7ef69eef36bb3325f47fd1f1 - DRAFT2_REV pinned in install.sh @ 3849269; 5-layer NVFP4 draft, calibrated activation scales, served modelopt_fp4 at depth 16; BF16 fallback z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c documented in install.sh
Engine image https://hub.docker.com/r/lmsysorg/sglang
sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae - = tag v0.5.22 (2026-10-16), IMAGE pin in install.sh @ 9e63ad54e2; official release image (retired overlay record: flash-sglang/ATTRIBUTION.md, CI-checked MANIFEST.sha256)
Reference https://github.com/hasso5703/dgx-spark-qwen38
9e63ad54e2c0cacb489811f5eb243181cb4bddd5 - recipe repo HEAD (2026-10-26, release v1.40.3); BENCHMARKS.md "The boot lottery" determinism methodology

Revision notes

Recipe by hasso5703 (dgx-spark-qwen38, MIT); DFlash2 NVFP4-RTNcal draft quantization credit MiaAI-Lab + r0b0tlab per BENCHMARKS.md; retired flash-lane overlay provenance in flash-sglang/ATTRIBUTION.md (CI-checked MANIFEST.sha256, 6 files). Ingested from repo HEAD 3849269 (2026-10-13); SGLang lane move re-pins and re-transcribes every launch flag verbatim at repo HEAD 9e63ad54e2 (2026-10-26, release v1.40.3) by localai-recipes: SGLang v0.5.22 (image digest e3a4f060, 2026-10-16; CHANGELOG v1.39.0). Numbers are author-measured T0: 68.3 greedy median = midpoint of the v0.5.22 runs 67.0/69.6 (CHANGELOG v1.39.0 A/B, reference box 2026-10-23); c4 81.65 = midpoint of serving-bench 4-stream 81.7/81.6 (same A/B); c8 135-148 (midpoint 141.5) and c32 258 carried over from the v1.2 battery per README (re-measure with ./bench-matrix.sh). Determinism: --disable-flashinfer-autotune + frozen battery, reproducible across boots ('The boot lottery'). Not re-run on our hardware (no GB10 box). Install footprint ~85 GB for the 27B lane; first boot 7-9 min, warm 5-7 min. Complements the Oct 13 27B-on-4x-RTX-3090 ingest on a different hardware family. PROXY SECURITY (kanban t_5be2157a, owner go 2026-10-28 Tier 2/3 preprod): F1 - summary corrected: client API is :30001 through the keepalive proxy, the engine answers on 127.0.0.1:30000 box-only (README.md:103/200 @ 9e63ad54e2). F2 - proxy sentence expanded: HEAD proxy v6.47; v6.46 added HEADER_BYTES_MAX 64 KiB/431, BODY_STALL_S 30s, 413-on-body for /health /ready /metrics, exact ENGINE_OPEN_ROUTES so /health_generate needs the key; PROXY_BIND default 0.0.0.0 (install.sh:800). F4 - torch distributed store caveat added (D05: binds every interface, no setting narrows it, open at v1.40.3; firewall per SECURITY.md ufw block). Audit draft adaptation: the drafted F2 opener 'transcribed at repo HEAD 3849269' was NOT re-applied because r3 already re-transcribes the launch at 9e63ad54e2 (F3 fix) - only the proxy sentence was expanded.

Revision history

d6fd296dbc recipe: update qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic (r3: /engine, /artifacts, /config/launch, /config/runtime/env, /config/runtime/notes, /performance, /artifacts/engine_imag 44 h ago diff
73daf7c432 recipe: update qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic (r2: /max_context_len, /kv_pool_tokens) 10 d ago diff
de161d5b52 recipe: create qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic 17 d ago

Fork

Export

recipe.yaml (raw)