diff --git a/recipes/qwen3-8-27b-mxfp4-dflash2-1x-radeon-ai-pro-r9700-wsl2-radiance/recipe.yaml b/recipes/qwen3-8-27b-mxfp4-dflash2-1x-radeon-ai-pro-r9700-wsl2-radiance/recipe.yaml index 46eafda2c..7b9f9a8cd 100644 --- a/recipes/qwen3-8-27b-mxfp4-dflash2-1x-radeon-ai-pro-r9700-wsl2-radiance/recipe.yaml +++ b/recipes/qwen3-8-27b-mxfp4-dflash2-1x-radeon-ai-pro-r9700-wsl2-radiance/recipe.yaml @@ -2,13 +2,13 @@ slug: qwen3-8-27b-mxfp4-dflash2-1x-radeon-ai-pro-r9700-wsl2-radiance title: 'Qwen3.8-27B MXFP4 DFlash2 1x Radeon AI PRO R9700 WSL2 radiance' engine: name: vllm - version: 'radiance 0.9.3 + ggz14/radiance-vllm-mxfp4 @ 7d9a15a' + version: 'radiance 0.9.3 + ggz14/radiance-vllm-mxfp4 @ 45da506' source_type: fork git_url: 'https://codeberg.org/ggz14/radiance-vllm-mxfp4' - pinned_commit: 7d9a15a51d83733cc079fcf32ef4e3ec6b04ad7a + pinned_commit: 45da5063820c887d88c6d74af9215e5ba14bd7b3 image_digest: 'docker.io/stilldeadcode/vllm-radiance@sha256:45694209177a55a1ab3ba6702fe6e978b1b66a6e66ae3fc066f8d579f7bc4c25' install_note: "NOT installable from the stock vLLM: serving runs the digest-pinned stilldeadcode/vllm-radiance:0.9.3 image; entry.sh applies the repo's patches/radiance-wsl-r9700.patch over the pinned radiance checkout, compiles the gfx1201 W4A8 GEMM with hipcc, swaps in the pinned libr4d r4d.so, then execs vLLM through available_memory.py (VRAM-budgeted context sizing). Requires WSL2 with /dev/dxg + ROCDXG 1.2.2; gfx1201-only by design." -summary: "Qwen3.8-27B at 125-134 tok/s decode and ~216k-260k context on a single AMD Radeon AI PRO R9700 (RDNA4 gfx1201, 32 GB) under Windows 11 + WSL2: vLLM radiance 0.9.3 fork pinned by image digest with AMD's official Quark AWQ-MXFP4 checkpoint, DFlash2 FP8 k=7 speculative drafter (59-63% acceptance, 5.1-5.4 tok per verify), hand-written W4A8 GEMM + libr4d FP8 paged-attention/GDN kernels pinned b9e42ab + rx9 patch, WSL pinned-memory fix, free-VRAM context budgeting (~216k default, ~260k -Long, model limit 262144), vision tower skipped. One-command PowerShell installer (.\\qwen38.ps1 install|start), OpenAI-compatible server on :8080. Benchmarks reproduced on the author's box from a clean install (repro table + bench harness in-repo)." +summary: "Qwen3.8-27B on a single AMD Radeon AI PRO R9700 (RDNA4 gfx1201, 32 GB) under Windows 11 + WSL2: vLLM radiance 0.9.3 fork pinned by image digest with AMD's official Quark AWQ-MXFP4 checkpoint, DFlash2 FP8 k=7 speculative drafter (59-63% acceptance, 5.1-5.4 tok per verify), hand-written W4A8 GEMM + libr4d FP8 paged-attention/GDN kernels pinned b9e42ab + rx9 patch, WSL pinned-memory fix, explicit context sizing, vision tower skipped. Author-measured 125-134 tok/s decode, 2750-2970 tok/s prefill at 1.9k. The author's defaults moved to a three-sequence long-context shape (max-num-seqs 3, batched tokens 2560, max-model-len 220000) backed by a KV-profiles pin the author labels REPORTED rather than measured, plus an opt-in KV cache subsystem (off|ram|disk) that leaves the serve command byte-identical when off. One-command PowerShell installer, OpenAI-compatible server on :8080." model: name: Qwen3.8-27B-MXFP4 family: qwen-3-8-27b @@ -54,11 +54,11 @@ config: - '--gpu-memory-utilization' - '0.90' - '--max-model-len' - - '-1' + - '220000' - '--max-num-seqs' - - '1' + - '3' - '--max-num-batched-tokens' - - '2048' + - '2560' - '--kv-cache-dtype' - fp8 - '--mamba-cache-dtype' @@ -92,7 +92,7 @@ performance: metrics: tok_s: 125 context_len: 216000 - notes: 'Author-measured on the shipped profile WITH in-repo receipts: clean-install reproduction table (README, 2026-10-08) + raw tables docs/BENCHMARKS.md + shipped bench/bench.py harness (.\qwen38.ps1 bench) @ repo HEAD 3a15982. Decode greedy 125-134 tok/s (medians of 3 starts; clean repro 125.3); sampled (temp 0.7) 117-126. Prefill 2750-2970 tok/s @1.9k prompt (TTFT 0.66-0.70 s); 32k/98k/164k/258k prompts prefill in 10.9/41/83/164 s. Decode deep in context 165 @32k / 136 @98k / 145 @164k / 115 @258k. DFlash2 acceptance 59-63%, 5.1-5.4 tok per verify. Long-context recall 8/8 @ a 257,768-token prompt; coding 10/12 runs pass all 46 tests on a 1.9k Rust spec. Context ~216k default / ~260k -Long (model 262144, VRAM-sized per start). Load ~2.5-3 min. Reference: same model, tuned llama.cpp IQ4_XS build decodes 52-68 tok/s on this card (different prompt; rough per README).' + notes: 'Author-measured on the shipped profile WITH in-repo receipts: clean-install reproduction table (README, 2026-10-08) + raw tables docs/BENCHMARKS.md + shipped bench/bench.py harness (.\qwen38.ps1 bench) @ repo HEAD 3a15982. Decode greedy 125-134 tok/s (medians of 3 starts; clean repro 125.3); sampled (temp 0.7) 117-126. Prefill 2750-2970 tok/s @1.9k prompt (TTFT 0.66-0.70 s); 32k/98k/164k/258k prompts prefill in 10.9/41/83/164 s. Decode deep in context 165 @32k / 136 @98k / 145 @164k / 115 @258k. DFlash2 acceptance 59-63%, 5.1-5.4 tok per verify. Long-context recall 8/8 @ a 257,768-token prompt; coding 10/12 runs pass all 46 tests on a 1.9k Rust spec. Context ~216k default / ~260k -Long (model 262144, VRAM-sized per start). Load ~2.5-3 min. Reference: same model, tuned llama.cpp IQ4_XS build decodes 52-68 tok/s on this card (different prompt; rough per README). UPDATE 2026-10-30 (repo @ 45da5063820c887d88c6d74af9215e5ba14bd7b3, 3 commits ahead of the pinned 7d9a15a5): the single-GPU profile defaults changed - serve-mxfp4.sh now defaults MAXSEQS=3 ("3 is the long-context shape"), CHUNK=2560 (was 4096, and the author states 2560 has NOT been benchmarked here the way 4096 was), and MAXLEN=220000 (was 65536, leaning on an explicit KV pin in kv-profiles.tsv). Those are the NEW defaults; the launch below records them explicitly so the recipe does not inherit them silently. Perf points below are the author''s 2026-10-08 clean-install reproduction table measured at the PRE-CHANGE shape (4096 / 65536 / 1 seq) and were NOT re-measured at the new triple; the author labels the 220000 window as REPORTED (KV-profile pin), not measured, and CHUNK=2560 as unbenchmarked. New opt-in subsystem KVCACHE=off|ram|disk (off by default; the generated serve command is byte-identical when off) with KVCACHE_RAM_GIB / KVCACHE_DISK, 6 always-on behavioural patches for ram and a 15-patch instrumented set for disk, tooling in kv-cache/ (kvwatch.py, turnbench.py, equivbench.py, patch_*.py, kvcache-reap). New RADIANCE_GDN_LAZY=0 default, documented as corrupting multi-turn chat (repeat loops / empty replies from ~5 turns in) when enabled, and never applied at TP>=2. PAROQUANT.md gained MXFP6-PARO (W6A8) OCP MXFP6 E2M3 weights on fp8-WMMA kernels; PERFORMANCE.md gained TP=2 numbers from a second R9700 rig.' points: - context_len: 1920 @@ -180,4 +180,4 @@ provenance: - 'https://huggingface.co/amd/Qwen3.8-27B-Quark-AWQ-MXFP4' - 'https://huggingface.co/tcclaviger/Qwen3.8-27B-DFlash2-FP8' - 'https://codeberg.org/ggz14/radiance-vllm-mxfp4' - notes: 'Submitted 2026-10-09 by localai-recipes from lar-scout ingest card t_4a41485c (sweep QUEUE #2, 2026-10-09). Repo @ 3a159822ee98fe74e1478ccb835f921ccab0ab01 (HEAD via git ls-remote 2026-10-09; 2 commits, created 2026-10-08). Launch argv + docker flags + env token-verified verbatim against scripts/qwen38.sh (cmd_start/gpu_args), scripts/entry.sh, config/dflash.env and patches/radiance-wsl-r9700.patch at that sha. Pins carried: image digest sha256:45694209...fbc4c25 (tag 0.9.3), radiance commit 7d9a15a51d83733cc079fcf32ef4e3ec6b04ad7a, libr4d b9e42ab (+rx9 patch from the radiance tree), model 5233554c5fa56afda40150556b95573c2d7d29c0, drafter ee0cb26a8279b7910cc28d82a8a3e15e4728d56f, ROCDXG 1.2.2. Benchmarks are author-run WITH in-repo receipts (clean-install repro table + docs/BENCHMARKS.md + bench/bench.py shipped @ HEAD) — carried as submitter-T0 with receipts. First Radeon AI PRO R9700 recipe on the site (bucket empty, verified live 2026-10-09); first Windows/WSL2-native serving recipe; first radiance-lineage engine entry (base engine vllm, fork source). gfx1201-only kernels — other cards unsupported by design (container asserts gcnArchName before downloads). Dedupe: no other Qwen3.8-27B DFlash2-on-R9700 recipe on site; distinct from R9700 tooling-only repos (vllm-coherence, vllm-radiance-nix, r9700-stack). Hardware resolved to the pre-existing profile radeon-ai-pro-r9700 (seed-slug convention; no new profile created).' + notes: 'Submitted 2026-10-09 by localai-recipes from lar-scout ingest card t_4a41485c (sweep QUEUE #2, 2026-10-09). Repo @ 3a159822ee98fe74e1478ccb835f921ccab0ab01 (HEAD via git ls-remote 2026-10-09; 2 commits, created 2026-10-08). Launch argv + docker flags + env token-verified verbatim against scripts/qwen38.sh (cmd_start/gpu_args), scripts/entry.sh, config/dflash.env and patches/radiance-wsl-r9700.patch at that sha. Pins carried: image digest sha256:45694209...fbc4c25 (tag 0.9.3), radiance commit 7d9a15a51d83733cc079fcf32ef4e3ec6b04ad7a, libr4d b9e42ab (+rx9 patch from the radiance tree), model 5233554c5fa56afda40150556b95573c2d7d29c0, drafter ee0cb26a8279b7910cc28d82a8a3e15e4728d56f, ROCDXG 1.2.2. Benchmarks are author-run WITH in-repo receipts (clean-install repro table + docs/BENCHMARKS.md + bench/bench.py shipped @ HEAD) — carried as submitter-T0 with receipts. First Radeon AI PRO R9700 recipe on the site (bucket empty, verified live 2026-10-09); first Windows/WSL2-native serving recipe; first radiance-lineage engine entry (base engine vllm, fork source). gfx1201-only kernels — other cards unsupported by design (container asserts gcnArchName before downloads). Dedupe: no other Qwen3.8-27B DFlash2-on-R9700 recipe on site; distinct from R9700 tooling-only repos (vllm-coherence, vllm-radiance-nix, r9700-stack). Hardware resolved to the pre-existing profile radeon-ai-pro-r9700 (seed-slug convention; no new profile created). UPDATE 2026-10-30 (repo @ 45da5063820c887d88c6d74af9215e5ba14bd7b3, 3 commits ahead of the pinned 7d9a15a5): the single-GPU profile defaults changed - serve-mxfp4.sh now defaults MAXSEQS=3 ("3 is the long-context shape"), CHUNK=2560 (was 4096, and the author states 2560 has NOT been benchmarked here the way 4096 was), and MAXLEN=220000 (was 65536, leaning on an explicit KV pin in kv-profiles.tsv). Those are the NEW defaults; the launch below records them explicitly so the recipe does not inherit them silently. Perf points below are the author''s 2026-10-08 clean-install reproduction table measured at the PRE-CHANGE shape (4096 / 65536 / 1 seq) and were NOT re-measured at the new triple; the author labels the 220000 window as REPORTED (KV-profile pin), not measured, and CHUNK=2560 as unbenchmarked. New opt-in subsystem KVCACHE=off|ram|disk (off by default; the generated serve command is byte-identical when off) with KVCACHE_RAM_GIB / KVCACHE_DISK, 6 always-on behavioural patches for ram and a 15-patch instrumented set for disk, tooling in kv-cache/ (kvwatch.py, turnbench.py, equivbench.py, patch_*.py, kvcache-reap). New RADIANCE_GDN_LAZY=0 default, documented as corrupting multi-turn chat (repeat loops / empty replies from ~5 turns in) when enabled, and never applied at TP>=2. PAROQUANT.md gained MXFP6-PARO (W6A8) OCP MXFP6 E2M3 weights on fp8-WMMA kernels; PERFORMANCE.md gained TP=2 numbers from a second R9700 rig.'