diff --git a/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic/recipe.yaml b/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic/recipe.yaml index 0fa924a8e..7d8b3e291 100644 --- a/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic/recipe.yaml +++ b/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic/recipe.yaml @@ -2,13 +2,13 @@ slug: qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic title: 'Qwen3.8-27B SGLang NVFP4 DFlash2 on 1x DGX Spark (deterministic)' engine: name: sglang - version: v0.5.21 + version: v0.5.22 source_type: docker_image git_url: 'https://github.com/hasso5703/dgx-spark-qwen38' - pinned_commit: 3849269bd1c7dbc6682b7b0c941368216f4ab781 - image_digest: 'lmsysorg/sglang@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469' - install_note: 'One-command install on stock DGX OS (docker + NVIDIA container toolkit): curl get.sh | bash (or ./install.sh from a clone) installs image + checkpoints + boot-persistent systemd service and proves it serves; ./install.sh --no-service && ./run.sh for foreground. ~85 GB disk for the 27B lane (45 GB HF cache + 40 GB docker image); first boot 7-9 min (torch.compile + CUDA graph capture, cached), warm boots 5-7 min. MODEL_REV / DRAFT2_REPO / DRAFT2_REV / DRAFT2_TOKENS env overrides re-pin per run; tagged releases checkout cleanly (v1.1-v1.11.1 at ingest).' -summary: 'First dense Qwen3.8-27B recipe on 1x DGX Spark (GB10, 128 GB unified): SGLang v0.5.21 release image + NVFP4 target + DFlash2 speculative decoding (NVFP4 calibrated 5-layer draft, depth 16), served by a boot-persistent systemd unit from a one-command installer. Author-measured 71.4 tok/s greedy median single stream (bench.sh, thinking on), 135-148 tok/s at 8 streams and 258 at 32. Fully pinned (image digest, checkpoint revisions, draft depth) and reproducible to the decimal across boots via --disable-flashinfer-autotune ("The boot lottery", BENCHMARKS.md); 1M-token YaRN context by default, native 262144 as fallback; OpenAI- and Anthropic-compatible API on :30000 with opencode wired out of the box.' + pinned_commit: 9e63ad54e2c0cacb489811f5eb243181cb4bddd5 + image_digest: 'lmsysorg/sglang@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae' + install_note: 'One-command install on stock DGX OS (docker + NVIDIA container toolkit): curl get.sh | bash (or ./install.sh from a clone) installs image + checkpoints + boot-persistent systemd service and proves it serves; ./install.sh --no-service && ./run.sh for foreground. ~85 GB disk for the 27B lane (45 GB HF cache + 40 GB docker image); first boot 7-9 min (torch.compile + CUDA graph capture, cached), warm boots 5-7 min. MODEL_REV / DRAFT2_REPO / DRAFT2_REV / DRAFT2_TOKENS env overrides re-pin per run; tagged releases checkout cleanly (v1.1-v1.40.3).' +summary: 'First dense Qwen3.8-27B recipe on 1x DGX Spark (GB10, 128 GB unified): SGLang v0.5.22 release image + NVFP4 target + DFlash2 speculative decoding (NVFP4 calibrated 5-layer draft, depth 16), served by a boot-persistent systemd unit from a one-command installer. Author-measured 68.3 tok/s greedy median single stream (bench.sh, thinking on; v1.39.0 A/B 2026-10-23), 81.7 tok/s at 4 streams (serving-bench), 135-148 tok/s at 8 streams and 258 at 32. Fully pinned (image digest, checkpoint revisions, draft depth) and reproducible to the decimal across boots via --disable-flashinfer-autotune ("The boot lottery", BENCHMARKS.md); 1M-token YaRN context by default, native 262144 as fallback; OpenAI- and Anthropic-compatible API on :30001 through the box''s keepalive proxy (the engine itself answers on 127.0.0.1:30000, on the box only), with opencode wired out of the box.' license: MIT model: name: Qwen3.8-27B @@ -42,7 +42,6 @@ config: - python3 - '-m' - sglang.launch_server - - '--trust-remote-code' - '--context-length' - '1010000' - '--model-path' @@ -88,6 +87,7 @@ config: - '2' - '--sleep-on-idle' - '--enable-metrics' + - '--enable-cache-report' - '--reasoning-parser' - qwen3 - '--tool-call-parser' @@ -101,20 +101,25 @@ config: - '--port' - '30000' runtime: - env: 'HF_HUB_OFFLINE=1 | SGLANG_ENABLE_REQUEST_HEADER_OVERRIDES=1 | SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 (1m unit) | TORCHINDUCTOR_CACHE_DIR=/cache/inductor - docker run --rm --name qwen38-sglang --gpus all --memory 100g --memory-swap 100g --shm-size 16g --network host --ipc=host; mounts: HF cache -> /root/.cache/huggingface, ~/.config/qwen38 -> /out (chat-template-sglang.jinja, engine-secrets.yaml), sglang-cache -> /cache' - notes: 'Launch argv = qwen38-sglang-1m.service.template (the installer default since v1.12.1) with pins substituted, transcribed at repo HEAD 3849269; keepalive proxy :30001 in front for agent CLIs. Native-262144 fallback: qwen38-sglang.service.template (same flags, --context-length dropped, mem-fraction 0.50) or foreground ./run.sh. Stock target shown; MODEL_CHOICE switches uncensored/fp8 targets (fp8 adds --kv-cache-dtype fp8_e4m3) and the flash lane serves Qwen3.8-Flash-Next 176B on the same box. ENGINE_BIND=127.0.0.1 default (0.0.0.0 opens to the network). API key delivered via engine-secrets.yaml file merge, never argv.' + env: 'HF_HUB_OFFLINE=1 | SGLANG_ENABLE_REQUEST_HEADER_OVERRIDES=1 | SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 (1m unit) | SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION=0 (GET /health generates nothing) | TORCHINDUCTOR_CACHE_DIR=/cache/inductor - docker run --rm --name qwen38-sglang --gpus all --memory 100g --memory-swap 100g --shm-size 16g --network host --ipc=host; mounts: HF cache -> /root/.cache/huggingface, ~/.config/qwen38 -> /out:ro (chat-template-sglang.jinja, engine-secrets.yaml), sglang-cache -> /cache' + notes: "Launch argv = qwen38-sglang-1m.service.template (the installer default since v1.12.1) with pins substituted, re-transcribed verbatim at repo HEAD 9e63ad54e2 (2026-10-26, release v1.40.3); keepalive proxy :30001 in front for agent CLIs, and it is the box's door from the network (proxy v6.47 at repo HEAD; PROXY_BIND defaults to 0.0.0.0, install.sh:800; the engine answers on 127.0.0.1:30000, on the box only). Its guards, added in proxy v6.46 (upstream v1.40.1): a network peer gets 64 KiB for its request line and headers in all (431 past it), 30 s without a byte while its body arrives, 413 for any body on /health, /ready or /metrics, and those three routes only - /health_generate needs the key. The box's key is the proxy port's gate, applied before a body is read; the bounds are upstream's, per CHANGELOG v1.40.1 and SECURITY.md. Delta vs the 3849269 transcription: --trust-remote-code dropped (no target or drafter ships checkpoint code; MODEL_CHOICE=custom needs the flag back in a drop-in) and --enable-cache-report added (#49, usage.prompt_tokens_details.cached_tokens). Native-262144 fallback: qwen38-sglang.service.template (same flags, --context-length dropped, mem-fraction 0.50) or foreground ./run.sh. Stock target shown; MODEL_CHOICE switches uncensored/fp8 targets (fp8 adds --kv-cache-dtype fp8_e4m3) and the flash lane serves Qwen3.8-Flash-Next 176B on the same box. ENGINE_BIND=127.0.0.1 default (0.0.0.0 opens to the network). That keeps the API port on the box, not the whole engine: PyTorch's distributed store binds every interface on a random port no setting narrows (still open upstream at v1.40.3, D05 of the 2026-10-24 review), takes no key, and a hostile connection can disturb the engine. Firewall the box to the ports you use before anything else (SECURITY.md's ufw block: allow lo, tailscale0, 22/tcp and 30001/tcp only, then default deny incoming). API key delivered via engine-secrets.yaml file merge, never argv." performance: claimed_by: submitter metrics: - tok_s: 71.4 + tok_s: 68.3 context_len: 1010000 - notes: 'All numbers author-measured on the reference DGX Spark with ./bench.sh (streaming TTFT-separated, sampling pinned) - never third-party. 71.4 tok/s = greedy median, thinking on, 1m mode (mem-fraction 0.76), release-image campaign 2026-09-17 (CHANGELOG v1.14.0); campaign history: 69.8 on the prior image, 65.3 (v1.9, native 262144, mem 0.50), 50 (v1.2, BF16 draft D8). Free prose is the slow case on any drafter (v1.9 probes: code 64-65, reasoning 65-66, math 57-71, prose 23-25). c8/c32 rows are aggregate decode carried over from the v1.2 battery (README: the draft only helps concurrency); c32 needs --max-running-requests 32 and was still climbing.' + notes: 'All numbers author-measured on the reference DGX Spark - never third-party. 68.3 tok/s = midpoint of the two v0.5.22 ./bench.sh greedy-median runs 67.0 and 69.6 on the reference box 2026-10-23 (CHANGELOG v1.39.0 A/B, both images one night; v0.5.21 same night 63.3/60.5/69.0); serving-bench 1-stream output 38.8/38.5/38.8 tok/s, draft accepted length 5.05; needle still exact at 120,196 and 299,901 prompt tokens (113 s, 478 s); GSM8K quick-eval 244/250 on v0.5.22. Campaign history: 71.4 (2026-09-17 release-image campaign, CHANGELOG v1.14.0), 69.8 prior image, 65.3 (v1.9), 50 (v1.2 BF16 draft D8). c4 81.65 = midpoint of the v0.5.22 serving-bench 4-stream outputs 81.7/81.6 (v0.5.21 77.5/77.8/74.7, same A/B) - serving-bench output tok/s, a different instrument from bench.sh. c8/c32 rows are aggregate decode carried over from the v1.2 battery (README: the draft only helps concurrency); c32 needs --max-running-requests 32 and was still climbing.' points: - context_len: 1010000 concurrency: 1 - decode_tok_s: 71.4 - notes: "./bench.sh greedy median, thinking on, 1m context (mem-fraction 0.76) - release-image campaign 2026-09-17; BENCHMARKS.md 'The boot lottery'" + decode_tok_s: 68.3 + notes: "./bench.sh greedy median, thinking on, 1m context (mem-fraction 0.76) - v0.5.22, midpoint of runs 67.0/69.6 on the reference box 2026-10-23 (CHANGELOG v1.39.0); BENCHMARKS.md 'The boot lottery'" + - + context_len: 262144 + concurrency: 4 + decode_tok_s: 81.65 + notes: 'serving-bench output tok/s at 4 streams, midpoint of the author-measured 81.7/81.6 on v0.5.22 (v0.5.21 77.5/77.8/74.7) - CHANGELOG v1.39.0 A/B 2026-10-23; different instrument from bench.sh' - context_len: 262144 concurrency: 8 @@ -140,12 +145,12 @@ artifacts: kind: engine_image pointer_type: docker_digest pointer: 'https://hub.docker.com/r/lmsysorg/sglang' - pin: 'sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 - = tag v0.5.21 (2026-10-01), IMAGE pin in install.sh @ 3849269; official release image (retired overlay record: flash-sglang/ATTRIBUTION.md, CI-checked MANIFEST.sha256)' + pin: 'sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae - = tag v0.5.22 (2026-10-16), IMAGE pin in install.sh @ 9e63ad54e2; official release image (retired overlay record: flash-sglang/ATTRIBUTION.md, CI-checked MANIFEST.sha256)' - kind: reference pointer_type: git_commit pointer: 'https://github.com/hasso5703/dgx-spark-qwen38' - pin: "3849269bd1c7dbc6682b7b0c941368216f4ab781 - recipe repo HEAD at ingest (2026-10-14); BENCHMARKS.md 'The boot lottery' determinism methodology" + pin: '9e63ad54e2c0cacb489811f5eb243181cb4bddd5 - recipe repo HEAD (2026-10-26, release v1.40.3); BENCHMARKS.md "The boot lottery" determinism methodology' provenance: credit: - @@ -154,6 +159,6 @@ provenance: source_urls: - 'https://github.com/hasso5703/dgx-spark-qwen38' - 'https://github.com/hasso5703/dgx-spark-qwen38/blob/main/BENCHMARKS.md' - notes: "Recipe by hasso5703 (dgx-spark-qwen38, MIT); DFlash2 NVFP4-RTNcal draft quantization credit MiaAI-Lab + r0b0tlab per BENCHMARKS.md; retired flash-lane overlay provenance in flash-sglang/ATTRIBUTION.md (CI-checked MANIFEST.sha256, 6 files). Ingested from repo HEAD 3849269 (2026-10-13) by localai-recipes; every pin transcribed from install.sh at that sha and each launch flag verified against the unit templates + run.sh. Numbers are author-measured T0: 71.4 greedy median from the 2026-09-17 release-image campaign (CHANGELOG v1.14.0, 1m mode, needle exact at 300,108 prompt tokens); c8 135-148 and c32 258 carried over from the v1.2 battery per README (re-measure with ./bench-matrix.sh). Determinism: --disable-flashinfer-autotune + frozen battery, reproducible across boots ('The boot lottery'). Not re-run on our hardware (no GB10 box). Install footprint ~85 GB for the 27B lane; first boot 7-9 min, warm 5-7 min. Complements the Oct 13 27B-on-4x-RTX-3090 ingest on a different hardware family." + notes: "Recipe by hasso5703 (dgx-spark-qwen38, MIT); DFlash2 NVFP4-RTNcal draft quantization credit MiaAI-Lab + r0b0tlab per BENCHMARKS.md; retired flash-lane overlay provenance in flash-sglang/ATTRIBUTION.md (CI-checked MANIFEST.sha256, 6 files). Ingested from repo HEAD 3849269 (2026-10-13); SGLang lane move re-pins and re-transcribes every launch flag verbatim at repo HEAD 9e63ad54e2 (2026-10-26, release v1.40.3) by localai-recipes: SGLang v0.5.22 (image digest e3a4f060, 2026-10-16; CHANGELOG v1.39.0). Numbers are author-measured T0: 68.3 greedy median = midpoint of the v0.5.22 runs 67.0/69.6 (CHANGELOG v1.39.0 A/B, reference box 2026-10-23); c4 81.65 = midpoint of serving-bench 4-stream 81.7/81.6 (same A/B); c8 135-148 (midpoint 141.5) and c32 258 carried over from the v1.2 battery per README (re-measure with ./bench-matrix.sh). Determinism: --disable-flashinfer-autotune + frozen battery, reproducible across boots ('The boot lottery'). Not re-run on our hardware (no GB10 box). Install footprint ~85 GB for the 27B lane; first boot 7-9 min, warm 5-7 min. Complements the Oct 13 27B-on-4x-RTX-3090 ingest on a different hardware family. PROXY SECURITY (kanban t_5be2157a, owner go 2026-10-28 Tier 2/3 preprod): F1 - summary corrected: client API is :30001 through the keepalive proxy, the engine answers on 127.0.0.1:30000 box-only (README.md:103/200 @ 9e63ad54e2). F2 - proxy sentence expanded: HEAD proxy v6.47; v6.46 added HEADER_BYTES_MAX 64 KiB/431, BODY_STALL_S 30s, 413-on-body for /health /ready /metrics, exact ENGINE_OPEN_ROUTES so /health_generate needs the key; PROXY_BIND default 0.0.0.0 (install.sh:800). F4 - torch distributed store caveat added (D05: binds every interface, no setting narrows it, open at v1.40.3; firewall per SECURITY.md ufw block). Audit draft adaptation: the drafted F2 opener 'transcribed at repo HEAD 3849269' was NOT re-applied because r3 already re-transcribes the launch at 9e63ad54e2 (F3 fix) - only the proxy sentence was expanded." max_context_len: 1010000 kv_pool_tokens: 473664