{"slug":"qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic","title":"Qwen3.8-27B SGLang NVFP4 DFlash2 on 1x DGX Spark (deterministic)","summary":"First dense Qwen3.8-27B recipe on 1x DGX Spark (GB10, 128 GB unified): SGLang v0.5.22 release image + NVFP4 target + DFlash2 speculative decoding (NVFP4 calibrated 5-layer draft, depth 16), served by a boot-persistent systemd unit from a one-command installer. Author-measured 68.3 tok/s greedy median single stream (bench.sh, thinking on; v1.39.0 A/B 2026-10-23), 81.7 tok/s at 4 streams (serving-bench), 135-148 tok/s at 8 streams and 258 at 32. Fully pinned (image digest, checkpoint revisions, draft depth) and reproducible to the decimal across boots via --disable-flashinfer-autotune (\"The boot lottery\", BENCHMARKS.md); 1M-token YaRN context by default, native 262144 as fallback; OpenAI- and Anthropic-compatible API on :30001 through the box's keepalive proxy (the engine itself answers on 127.0.0.1:30000, on the box only), with opencode wired out of the box.","license":"MIT","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"307fa2334d111e38258e27fa9e690c44e96b441b","content_hash":"7d8b3e291e1b8abfa9db4ded3e90cd601b8bba78","created_at":"2026-10-28 16:49:58.824279-06","notes":"Recipe by hasso5703 (dgx-spark-qwen38, MIT); DFlash2 NVFP4-RTNcal draft quantization credit MiaAI-Lab + r0b0tlab per BENCHMARKS.md; retired flash-lane overlay provenance in flash-sglang/ATTRIBUTION.md (CI-checked MANIFEST.sha256, 6 files). Ingested from repo HEAD 3849269 (2026-10-13); SGLang lane move re-pins and re-transcribes every launch flag verbatim at repo HEAD 9e63ad54e2 (2026-10-26, release v1.40.3) by localai-recipes: SGLang v0.5.22 (image digest e3a4f060, 2026-10-16; CHANGELOG v1.39.0). Numbers are author-measured T0: 68.3 greedy median = midpoint of the v0.5.22 runs 67.0/69.6 (CHANGELOG v1.39.0 A/B, reference box 2026-10-23); c4 81.65 = midpoint of serving-bench 4-stream 81.7/81.6 (same A/B); c8 135-148 (midpoint 141.5) and c32 258 carried over from the v1.2 battery per README (re-measure with ./bench-matrix.sh). Determinism: --disable-flashinfer-autotune + frozen battery, reproducible across boots ('The boot lottery'). Not re-run on our hardware (no GB10 box). Install footprint ~85 GB for the 27B lane; first boot 7-9 min, warm 5-7 min. Complements the Oct 13 27B-on-4x-RTX-3090 ingest on a different hardware family. PROXY SECURITY (kanban t_5be2157a, owner go 2026-10-28 Tier 2/3 preprod): F1 - summary corrected: client API is :30001 through the keepalive proxy, the engine answers on 127.0.0.1:30000 box-only (README.md:103/200 @ 9e63ad54e2). F2 - proxy sentence expanded: HEAD proxy v6.47; v6.46 added HEADER_BYTES_MAX 64 KiB/431, BODY_STALL_S 30s, 413-on-body for /health /ready /metrics, exact ENGINE_OPEN_ROUTES so /health_generate needs the key; PROXY_BIND default 0.0.0.0 (install.sh:800). F4 - torch distributed store caveat added (D05: binds every interface, no setting narrows it, open at v1.40.3; firewall per SECURITY.md ufw block). Audit draft adaptation: the drafted F2 opener 'transcribed at repo HEAD 3849269' was NOT re-applied because r3 already re-transcribes the launch at 9e63ad54e2 (F3 fix) - only the proxy sentence was expanded."},"engine_source":{"type":"docker_image","git_url":"https://github.com/hasso5703/dgx-spark-qwen38","base_engine":"sglang","pip_package":null,"base_version":"v0.5.22","image_digest":"lmsysorg/sglang@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae","install_note":"One-command install on stock DGX OS (docker + NVIDIA container toolkit): curl get.sh | bash (or ./install.sh from a clone) installs image + checkpoints + boot-persistent systemd service and proves it serves; ./install.sh --no-service && ./run.sh for foreground. ~85 GB disk for the 27B lane (45 GB HF cache + 40 GB docker image); first boot 7-9 min (torch.compile + CUDA graph capture, cached), warm boots 5-7 min. MODEL_REV / DRAFT2_REPO / DRAFT2_REV / DRAFT2_TOKENS env overrides re-pin per run; tagged releases checkout cleanly (v1.1-v1.40.3).","pinned_commit":"9e63ad54e2c0cacb489811f5eb243181cb4bddd5"},"credit":[{"name":"hasso5703","url":"https://github.com/hasso5703"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["python3","-m","sglang.launch_server","--context-length","1010000","--model-path","RadixArk/Qwen3.8-27B-NVFP4","--revision","52d1adc5f38aa5ebf099c29ed7025ba34cfbb854","--tp-size","1","--served-model-name","qwen3.8-27b","--mem-fraction-static","0.76","--attention-backend","flashinfer","--chunked-prefill-size","8192","--disable-prefill-cuda-graph","--cuda-graph-max-bs-decode","8","--disable-flashinfer-autotune","--speculative-algorithm","DFLASH","--speculative-draft-model-path","maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal","--speculative-draft-model-revision","bd7a934213c47a9e7ef69eef36bb3325f47fd1f1","--speculative-num-draft-tokens","16","--speculative-draft-model-quantization","modelopt_fp4","--mamba-radix-cache-strategy","extra_buffer","--mamba-ssm-dtype","bfloat16","--max-mamba-cache-size","96","--max-running-requests","8","--enable-torch-compile","--torch-compile-max-bs","4","--num-continuous-decode-steps","2","--sleep-on-idle","--enable-metrics","--enable-cache-report","--reasoning-parser","qwen3","--tool-call-parser","qwen3_coder","--chat-template","/out/chat-template-sglang.jinja","--config","/out/engine-secrets.yaml","--host","127.0.0.1","--port","30000"],"runtime":{"env":"HF_HUB_OFFLINE=1 | SGLANG_ENABLE_REQUEST_HEADER_OVERRIDES=1 | SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 (1m unit) | SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION=0 (GET /health generates nothing) | TORCHINDUCTOR_CACHE_DIR=/cache/inductor - docker run --rm --name qwen38-sglang --gpus all --memory 100g --memory-swap 100g --shm-size 16g --network host --ipc=host; mounts: HF cache -> /root/.cache/huggingface, ~/.config/qwen38 -> /out:ro (chat-template-sglang.jinja, engine-secrets.yaml), sglang-cache -> /cache","notes":"Launch argv = qwen38-sglang-1m.service.template (the installer default since v1.12.1) with pins substituted, re-transcribed verbatim at repo HEAD 9e63ad54e2 (2026-10-26, release v1.40.3); keepalive proxy :30001 in front for agent CLIs, and it is the box's door from the network (proxy v6.47 at repo HEAD; PROXY_BIND defaults to 0.0.0.0, install.sh:800; the engine answers on 127.0.0.1:30000, on the box only). Its guards, added in proxy v6.46 (upstream v1.40.1): a network peer gets 64 KiB for its request line and headers in all (431 past it), 30 s without a byte while its body arrives, 413 for any body on /health, /ready or /metrics, and those three routes only - /health_generate needs the key. The box's key is the proxy port's gate, applied before a body is read; the bounds are upstream's, per CHANGELOG v1.40.1 and SECURITY.md. Delta vs the 3849269 transcription: --trust-remote-code dropped (no target or drafter ships checkpoint code; MODEL_CHOICE=custom needs the flag back in a drop-in) and --enable-cache-report added (#49, usage.prompt_tokens_details.cached_tokens). Native-262144 fallback: qwen38-sglang.service.template (same flags, --context-length dropped, mem-fraction 0.50) or foreground ./run.sh. Stock target shown; MODEL_CHOICE switches uncensored/fp8 targets (fp8 adds --kv-cache-dtype fp8_e4m3) and the flash lane serves Qwen3.8-Flash-Next 176B on the same box. ENGINE_BIND=127.0.0.1 default (0.0.0.0 opens to the network). That keeps the API port on the box, not the whole engine: PyTorch's distributed store binds every interface on a random port no setting narrows (still open upstream at v1.40.3, D05 of the 2026-10-24 review), takes no key, and a hostile connection can disturb the engine. Firewall the box to the ports you use before anything else (SECURITY.md's ufw block: allow lo, tailscale0, 22/tcp and 30001/tcp only, then default deny incoming). API key delivered via engine-secrets.yaml file merge, never argv."},"kv_pool_tokens":473664,"max_context_len":1010000},"models":[{"slug":"qwen3-8-27b","name":"Qwen3.8-27B","hf_repo":"RadixArk/Qwen3.8-27B-NVFP4","quant":"nvfp4"}],"hardware":[{"slug":"dgx-spark","vendor":"nvidia","card":"DGX Spark (GB10, unified memory)","card_family":"dgx-spark","vram_gb":128,"device_count":1}],"generation":"blackwell","tags":["agent-serving","deterministic-decode","dflash2","dgx-spark","gb10","long-context","single-gpu","sm121","speculative","yarn-1m"],"tasks":["chat","code"],"aa":{"aa_index":34,"aa_provenance":"published","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-07 18:00:00-06","aa_family_slug":"qwen-3-8-27b","aa_resolved_slug":"qwen-3-8-27b"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"All numbers author-measured on the reference DGX Spark - never third-party. 68.3 tok/s = midpoint of the two v0.5.22 ./bench.sh greedy-median runs 67.0 and 69.6 on the reference box 2026-10-23 (CHANGELOG v1.39.0 A/B, both images one night; v0.5.21 same night 63.3/60.5/69.0); serving-bench 1-stream output 38.8/38.5/38.8 tok/s, draft accepted length 5.05; needle still exact at 120,196 and 299,901 prompt tokens (113 s, 478 s); GSM8K quick-eval 244/250 on v0.5.22. Campaign history: 71.4 (2026-09-17 release-image campaign, CHANGELOG v1.14.0), 69.8 prior image, 65.3 (v1.9), 50 (v1.2 BF16 draft D8). c4 81.65 = midpoint of the v0.5.22 serving-bench 4-stream outputs 81.7/81.6 (v0.5.21 77.5/77.8/74.7, same A/B) - serving-bench output tok/s, a different instrument from bench.sh. c8/c32 rows are aggregate decode carried over from the v1.2 battery (README: the draft only helps concurrency); c32 needs --max-running-requests 32 and was still climbing.\", \"tok_s\": 68.3, \"context_len\": 1010000}"}],"perf_points":[{"id":1965,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":81.650000000000005684341886080801486968994140625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"serving-bench output tok/s at 4 streams, midpoint of the author-measured 81.7/81.6 on v0.5.22 (v0.5.21 77.5/77.8/74.7) - CHANGELOG v1.39.0 A/B 2026-10-23; different instrument from bench.sh"},{"id":1962,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":141.5,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate decode at 8 concurrent streams, midpoint of the author-measured 135-148 range (v1.2 DFlash2 battery carry-over, README); deterministic stack"},{"id":1963,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":258,"ttft_ms":null,"concurrency":32,"claimed_by":"submitter","notes":"aggregate at 32 streams (--max-running-requests 32), still climbing at c32 per BENCHMARKS.md; v1.2 battery carry-over via README"},{"id":1964,"context_len":1010000,"prefill_tok_s":null,"decode_tok_s":68.2999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"./bench.sh greedy median, thinking on, 1m context (mem-fraction 0.76) - v0.5.22, midpoint of runs 67.0/69.6 on the reference box 2026-10-23 (CHANGELOG v1.39.0); BENCHMARKS.md 'The boot lottery'"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/RadixArk/Qwen3.8-27B-NVFP4","pin":"52d1adc5f38aa5ebf099c29ed7025ba34cfbb854 - STOCK_REV pinned in install.sh @ 3849269; NVFP4 release weights, lossless vs Q8 reference + live canaries per README"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal","pin":"bd7a934213c47a9e7ef69eef36bb3325f47fd1f1 - DRAFT2_REV pinned in install.sh @ 3849269; 5-layer NVFP4 draft, calibrated activation scales, served modelopt_fp4 at depth 16; BF16 fallback z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c documented in install.sh"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/lmsysorg/sglang","pin":"sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae - = tag v0.5.22 (2026-10-16), IMAGE pin in install.sh @ 9e63ad54e2; official release image (retired overlay record: flash-sglang/ATTRIBUTION.md, CI-checked MANIFEST.sha256)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/hasso5703/dgx-spark-qwen38","pin":"9e63ad54e2c0cacb489811f5eb243181cb4bddd5 - recipe repo HEAD (2026-10-26, release v1.40.3); BENCHMARKS.md \"The boot lottery\" determinism methodology"}],"links":{"self":"/api/v1/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic.json","alternate":"/r/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic","describedby":"/openapi.json"},"updated_at":"2026-10-28 16:49:58.824279-06","created_at":"2026-10-20 21:49:17.065017-06","speeds":{"decode_tok_s":68.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}