diff --git a/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-mtp-213-262k/recipe.yaml b/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-mtp-213-262k/recipe.yaml deleted file mode 100644 index 2a0a1fd94..000000000 --- a/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-mtp-213-262k/recipe.yaml +++ /dev/null @@ -1,147 +0,0 @@ -slug: qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-mtp-213-262k -title: 'Qwen3.8-Flash-Next NVFP4 DGX Spark GB10 SGLang MTP-213 262K' -engine: - name: sglang - version: 'image commit d91c3682 (qwen4 source 73a255206f)' - source_type: fork - git_url: 'https://github.com/Felliks/qwen38-flash-next-one-dgx-spark' - pinned_commit: 5187a06c6c44a894ab6df2c7f07e005e1f8e66b9 - image_digest: 'lmsysorg/sglang@sha256:14ed582518584c5c830206b5318a2c2769e68229c3422e48a28b952b3a888bd4' - install_note: "Fully pinned per repo 'Pinned components' table: base image lmsysorg/sglang @ sha256:14ed582518584c5c830206b5318a2c2769e68229c3422e48a28b952b3a888bd4 (image commit d91c3682b0b429e4c70df63cd57f819588ce29b0, qwen4 experimental source 73a255206f916366c8d26d4022f82ddfb0ab558d) + the repo's 2 source-hash-guarded patches (patch_qwen4_ple_disk_cache.py, patch_sm121_qsa_sdpa.py - the build FAILS if upstream files no longer match expected SHA-256) + prepare_ple.py mmap build w/ 256-row byte verification. Local image tag qwen38-flash-next-gx10:73a255-ple-disk-v3. Requires aarch64 + compute_cap 12.1 (run-spark.sh require_gb10 gate) and >=230GB free NVMe." -summary: "Native-262K-context DGX Spark recipe for RadixArk/Qwen3.8-Flash-Next-NVFP4, validated on ASUS Ascent GX10 (same GB10 / 128GB unified memory as NVIDIA DGX Spark): the NVFP4 checkpoint is 135.24 GB and stock PLE host offload still lands the whole 51.2 GB table in host memory - but host memory and VRAM are the SAME 128 GB pool on GB10 - so this recipe adds two source-hash-guarded patches to a pinned SGLang image: (1) verified raw FP8 mmap keeps the full PLE table on NVMe with a bounded 256 MiB four-way pinned row cache for decode lookups (large prefill lookups bypass it), (2) QSA decode uses a PyTorch SDPA fallback on SM121 because the pinned FlashAttention-4 varlen path does not compile on GB10. prepare step converts the checkpoint's 128 FP8 PLE shards into a contiguous 51.2 GB mmap and byte-verifies 256 deterministic rows against the safetensors source. Official SGLang MTP-213 profile (2 speculative steps / top-k 1 / 3 draft tokens). Author-measured 2026-08-26 (benchmarks/asus-ascent-gx" -model: - name: Qwen3.8-Flash-Next-NVFP4 - family: qwen3-8-flash-next - hf_repo: RadixArk/Qwen3.8-Flash-Next-NVFP4 - hf_revision: 7b719225242aacd3dbd3f9407468c2ee9a9d2594 - quant: nvfp4 -hardware: - - - vendor: nvidia - card: 'DGX Spark (GB10, unified memory)' - vram_gb: 128 - device_count: 1 -tasks: - - chat - - code -tags: - - felliks - - sglang - - dgx-spark - - gb10 - - sm121 - - asus-ascent-gx10 - - nvme-ple - - mtp-speculative - - eager-decode - - bf16-kv -config: - launch: - - sglang - - serve - - '--model-path' - - RadixArk/Qwen3.8-Flash-Next-NVFP4 - - '--revision' - - 7b719225242aacd3dbd3f9407468c2ee9a9d2594 - - '--served-model-name' - - qwen38-flash-next-nvfp4-mtp - - '--trust-remote-code' - - '--host' - - 0.0.0.0 - - '--port' - - '30000' - - '--quantization' - - modelopt_fp4 - - '--fp4-gemm-backend' - - flashinfer_cutlass - - '--page-size' - - '64' - - '--mamba-radix-cache-strategy' - - extra_buffer - - '--mamba-track-interval' - - '64' - - '--max-mamba-cache-size' - - '20' - - '--mamba-ssm-dtype' - - float32 - - '--chunked-prefill-size' - - '4096' - - '--max-running-requests' - - '4' - - '--max-total-tokens' - - '524288' - - '--context-length' - - '262144' - - '--mem-fraction-static' - - '0.95' - - '--allow-auto-truncate' - - '--ple-offload-embedding' - - '--reasoning-parser' - - qwen3 - - '--tool-call-parser' - - qwen3_coder - - '--preferred-sampling-params' - - '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' - - '--disable-prefill-cuda-graph' - - '--cuda-graph-backend-decode' - - disabled - - '--disable-flashinfer-autotune' - - '--speculative-algorithm' - - NEXTN - - '--speculative-num-steps' - - '2' - - '--speculative-eagle-topk' - - '1' - - '--speculative-num-draft-tokens' - - '3' - runtime: - env: 'SGLANG_QWEN4_PLE_DISK_CACHE_PATH=/ple-cache/ple-fp8.raw | SGLANG_QWEN4_PLE_DISK_CACHE_MANIFEST=/ple-cache/ple-fp8.raw.json | SGLANG_QWEN4_PLE_SOURCE_REVISION=7b719225242aacd3dbd3f9407468c2ee9a9d2594 | SGLANG_QWEN4_PLE_CACHE_BYTES=268435456 | SGLANG_QWEN4_PLE_CACHE_MAX_LOOKUP_ROWS=4096' - notes: 'Serve profile = run-spark.sh serve() @ 5187a06c verbatim (docker run -d --rm --init --gpus all --ipc host --shm-size 16g --memory 116g --memory-swap 116g --cpuset-cpus 0-19 -p 127.0.0.1:8000:30000; mounts HF cache + sglang cache + ple-cache ro; local image qwen38-flash-next-gx10:73a255-ple-disk-v3). MTP-213 = SGLang official profile for this model (lmsys.org blog 2026-08-26): 2 steps / top-k 1 / 3 draft. KV BF16 (SM121 eager QSA path rejects FP8 K/V - tested). Decode EAGER by design: disk-fed PLE rows cannot be captured as fixed host pointers in a CUDA graph; prefill graph disabled. Default sampling for thinking = temp 1.0/top_p 0.95/top_k 20/min_p 0.0; non-thinking per request = temp 0.7/top_p 0.80/top_k 20/presence_penalty 1.5 (Qwen model card). Overrides: API_PORT, QWEN_NEXT_DATA_DIR, CPUSET, BIND_ADDR (default 127.0.0.1 - do not expose unauthenticated endpoint publicly). Known limits: 4 active requests share the 524K pool (two full 262K contexts max); TP=1 only for disk-backed PLE.' -performance: - claimed_by: submitter - metrics: - tok_s: 32.7 - context_len: 262144 - notes: 'Author-measured 2026-08-26 (benchmarks/asus-ascent-gx10-2026-08-26.json @ 5187a06c, structured record): single_stream 512 generated tokens, runs 28.078/32.772/32.727 tok/s, steady_state 32.7 tok/s, steady TTFT 0.2857s; four_stream_batch 4x768 tokens x2 reps wall 32.90/33.03s -> aggregate 93.372/92.996 tok/s, per-request range 23.636-23.807 tok/s, observed scheduler peak 103.36 tok/s; resident system memory 120.45 GB / 128 GB incl. OS; GPU utilization ~90%; cold load ~9.5 min. Configuration: ctx' - points: - - - context_len: 262144 - concurrency: 1 - decode_tok_s: 32.7 - ttft_ms: 286 - notes: 'bench JSON @ 5187a06c: single-stream steady-state 32.7 tok/s (runs 28.08/32.77/32.73), TTFT 0.286s; MTP-213, BF16 KV' - - - context_len: 262144 - concurrency: 4 - decode_tok_s: 93.2 - notes: 'bench JSON: four-stream aggregate 93.37/93.00 tok/s over 2 reps (scheduler peak 103.36); per-request 23.6-23.8 tok/s shared' -artifacts: - - - kind: model_weights - pointer_type: hf_model - pointer: 'https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4' - pin: 'revision 7b719225242aacd3dbd3f9407468c2ee9a9d2594 (135.24 GB NVFP4 checkpoint; 128 FP8 PLE shards -> contiguous 51.2 GB mmap via prepare_ple.py, 256-row byte verification)' - - - kind: engine_image - pointer_type: docker_digest - pointer: 'https://hub.docker.com/r/lmsysorg/sglang' - pin: 'lmsysorg/sglang @ sha256:14ed582518584c5c830206b5318a2c2769e68229c3422e48a28b952b3a888bd4 (commit d91c3682b0b429e4c70df63cd57f819588ce29b0, qwen4 source 73a255206f916366c8d26d4022f82ddfb0ab558d) + 2 source-hash-guarded patches (PLE disk cache, SM121 QSA SD' - - - kind: reference - pointer_type: git_commit - pointer: 'https://github.com/Felliks/qwen38-flash-next-one-dgx-spark' - pin: '5187a06c6c44a894ab6df2c7f07e005e1f8e66b9 - run-spark.sh (prepare/serve/smoke/status), Dockerfile, patch_qwen4_ple_disk_cache.py, patch_sm121_qsa_sdpa.py, prepare_ple.py, benchmarks/asus-ascent-gx10-2026-08-26.json, GitHub Actions validate workflow' -provenance: - source_urls: - - 'https://github.com/Felliks/qwen38-flash-next-one-dgx-spark' - - 'https://github.com/Felliks/qwen38-flash-next-one-dgx-spark/blob/main/run-spark.sh' - - 'https://github.com/Felliks/qwen38-flash-next-one-dgx-spark/blob/main/benchmarks/asus-ascent-gx10-2026-08-26.json' - - 'https://www.lmsys.org/blog/2026-08-26-qwen-flash-next/' - credit: - - - name: Felliks - url: 'https://github.com/Felliks/qwen38-flash-next-one-dgx-spark' - - - name: 'sgl-project (SGLang + official MTP-213 profile)' - url: 'https://github.com/sgl-project/sglang' - notes: "Ingested from GitHub repo Felliks/qwen38-flash-next-one-dgx-spark @ 5187a06c6c44a894ab6df2c7f07e005e1f8e66b9 (wave 3, kanban t_21a3a083). Validated on ASUS Ascent GX10; NVIDIA DGX Spark confirmations requested by author (same GB10 silicon - profile dgx-spark covers both, exact variant in tooltip/notes). Apache-2.0 code; Qwen Community License 1.0 weights. Measurements are from the included pinned checkpoint and launch path, not estimates copied from another GPU (author's statement)."