diff --git a/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots/recipe.yaml b/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots/recipe.yaml index 4500d899d..a8d1afa05 100644 --- a/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots/recipe.yaml +++ b/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots/recipe.yaml @@ -1,11 +1,11 @@ slug: qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots -title: 'Qwen3.8-Flash-Next-Uncensored AWQ W4A16 on 2x CMP 170HX — 8 slots, PP2, MTP3, PLE-from-SSD (mannydelrio)' +title: 'Qwen3.8-Flash-Next-Uncensored AWQ W4A16 on 2x CMP 170HX — 8 slots, PP2, MTP5 + int4 draft head, PLE-from-SSD (mannydelrio)' engine: name: vllm source_type: release git_url: 'https://github.com/wtdcode/vllm-backport' install_note: '[base_version wtdcode/vllm-backport @ 3142ded1 (source build) + recipe patches]' -summary: 'Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE), AWQ W4A16 g32: PP2 (CMP 170HX has no P2P, so PP beats TP), MTP 3-token drafting, and the 29.8 GB PLE table served from NVMe SSD. This revision runs 8 slots with decode CUDA graphs up to 32 tokens (graph sizes round to multiples of 1 + 3 drafts = 4, so the old max of 16 covered only 4 requests). Single user 192 / 178 / 145 tok/s structured / code / prose; 8 users 986 / 734 / 620 aggregate; cold prefill ~7,000 tok/s; 262,144 context. Cards: 8 GB CMP 170HX unlocked to 64 GB (cmpunlocker v0.5), PCIe Gen2 x4, 180 W each.' +summary: 'Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE), AWQ W4A16 g32: PP2 (CMP 170HX has no P2P), MTP 5-token drafting with an int4 draft-only lm_head, 8 slots, and the 29.8 GB PLE table on NVMe. Single user 240 / 225 / 156 tok/s structured / code / prose; 8 users 1,044 / 715 / 625 aggregate; cold prefill ~6,960 tok/s; 262,144 context; HumanEval 157/164. 2x CMP 170HX unlocked to 64 GB, Gen2 x4, 180 W each.' license: CC-BY-4.0 model: name: Qwen3.8-Flash-Next-Uncensored-AWQ @@ -46,12 +46,14 @@ config: - '8' - '--max-num-batched-tokens' - '4096' + - '--block-size' + - '1632' - '--compilation-config' - - '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8,12,16,20,24,28,32],"max_cudagraph_capture_size":32}' + - '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[6,12,18,24,30,36,42,48],"max_cudagraph_capture_size":48}' - '--gpu-memory-utilization' - '0.94' - '--speculative-config' - - '{"method":"mtp","num_speculative_tokens":3}' + - '{"method":"mtp","num_speculative_tokens":5}' - '--reasoning-parser' - qwen3 - '--enable-auto-tool-choice' @@ -62,37 +64,34 @@ config: - '--port' - '18084' runtime: - env: 'VLLM_PLUGINS=tinybeast_ple_int4,tinybeast_ple_ssd | TB_PLE_SSD=1 | TB_PLE_SSD_PREFETCH=1 | TB_TORCH_MEM_CAP_GIB=61.5 | VLLM_DETERMINISTIC_MOE_ALIGN=0 | VLLM_PP_LAYER_PARTITION=25,23 | CUDA_MODULE_LOADING=EAGER | NCCL_ALGO=Ring | NCCL_PROTO=Simple' - notes: "2026-10-15 revision of the community-listed recipe /r/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3. With 4 slots (graphs to 16) 8 users gave 599 / 447 / 388 tok/s; 8 slots with graphs still to 16 was slower (396 / 332 / 286) because requests 5-8 ran without graphs; graphs to 32 cost +0.04 GiB per card, 1-4 users unchanged, KV pool unchanged at 897K tokens. Measured with MiaAI-Lab decode prompts (structured = count 1-200, code = clamp_range function, prose = hash-map explanation), temperature 0, thinking off, 400-token cap, streaming; the protocol Morrowmake publishes for CMP 170HX recipes. Decode = (completion_tokens - 1) / time after the first token; 4 slots, 10 runs gave 192.9 / 177.8 / 145.5 single-user. Cold prefill = real-text prompts of 23,945 / 34,299 / 37,905 tokens x3, unique salt first, 0 prefix-cache hits, prompt_tokens / time to first token (6,346-7,120 tok/s), measured on the 4-slot launch the same day (prefill uses no CUDA graphs). Engine is a SOURCE BUILD: wtdcode/vllm-backport @ 3142ded1 + the recipe repo's patches and the two tinybeast plugins (build steps in the reference repo). Launch argv = the repo's serve/serve.sh @ 042acb9 (2026-10-15: --max-num-seqs 8, cudagraph_capture_sizes to 32; the earlier 4-slot version is @ d82e8d1). tests/protocol_speed.py @ 042acb9 reproduces the speed numbers. If you keep 4 slots, keep the old graph sizes; if you raise slots, size graphs to slots x (1 + num_speculative_tokens). Host: 2x CMP 170HX 8 GB variant unlocked to 64 GB each (cmpunlocker v0.5), Ryzen 5 5600X, 16 GB RAM, 1 TB NVMe. Long context: a 253K-token prompt reads in 35 s and answered 8/8 planted-fact questions on this config. Optional YaRN x4 reaches 940,800 tokens on these cards (1,048,576 needs 14.91 GiB KV, 13.42 GiB available): 8/8 to 801K, 7/8 at 932K (447 s), needle found at 851K; not the default because YaRN shifts outputs slightly at every length and one full-length request fills the KV pool." + env: 'VLLM_PLUGINS=tinybeast_ple_int4,tinybeast_ple_ssd | TB_PLE_SSD=1 | TB_PLE_SSD_PREFETCH=1 | TB_TORCH_MEM_CAP_GIB=61.5 | VLLM_DETERMINISTIC_MOE_ALIGN=0 | VLLM_PP_LAYER_PARTITION=25,23 | CUDA_MODULE_LOADING=EAGER | NCCL_ALGO=Ring | NCCL_PROTO=Simple | VLLM_MTP_DRAFT_HEAD_PATH=/opt/flashnext/draft_lm_head_w4a16.safetensors' + notes: "2026-10-26 revision. Changes vs the 2026-10-15 revision (3 draft tokens): num_speculative_tokens 5, the MTP drafter uses its own int4 (W4A16, group 128, symmetric) copy of lm_head (AlexBrandes/vllm commit 14457f89e5, env VLLM_MTP_DRAFT_HEAD_PATH; the file is made from the checkpoint's lm_head with tools/quantize_lm_head.py from AlexBrandes/cmp170hx-qwen38-flash-next, RTN, 8 s on CPU; verification still uses the full bf16 head, so accepted tokens are unchanged), --block-size 1632 (5 drafts give a QSA ring of 12, which must divide the block size; the default 1616 fails at start-up) and decode graphs 6..48 = 8 slots x (1 + 5). Same-day A/B on this box: 5 drafts WITHOUT the int4 head lost prefill (5,468 tok/s) and was worse on prose and at 8 users; 3 drafts + int4 head gave 208/192/157 single-user and 1,038/760/641 at 8 users (best at 8 users, slower for one coder); 4 drafts + head sat between. Measured with MiaAI-Lab decode prompts (structured = count 1-200, code = clamp_range function, prose = hash-map explanation), temperature 0, thinking off, 400-token cap, streaming; the protocol Morrowmake publishes for CMP 170HX recipes. Single user = median of 10 runs; 2/4/8 users = median of 3 runs of completion tokens / batch wall time. Cold prefill = real-text prompts of 23,945 / 34,299 / 37,905 tokens x3, unique salt first, 0 cache hits (median of 9: 6,964). Quality: HumanEval 157/164 (unchanged from the 3-draft build), needle retrieval passes at 91K / 183K / 219K tokens, GPU1 peak 63.7 GB as before. Engine = wtdcode/vllm-backport @ 3142ded1 source build + the recipe repo's patches and plugins + the draft-head patch above; the recipe repo's serve.sh (042acb9) is the 3-draft launch and is NOT yet updated, so the launch argv and env here are authoritative. Host: 2x CMP 170HX unlocked to 64 GB each (cmpunlocker v0.5), Ryzen 5 5600X, 32 GB RAM, 1 TB NVMe. Long context: optional YaRN x4 reaches 940,800 tokens (measured 2026-10-15 on the 3-draft build, not re-run)." performance: - claimed_by: submitter - metrics: - tok_s: 192.4 - notes: 'All figures author-measured 2026-10-15 on an idle box (no other traffic in the window). MiaAI-Lab decode prompts (structured = count 1-200, code = clamp_range function, prose = hash-map explanation), temperature 0, thinking off, 400-token cap, streaming; the protocol Morrowmake publishes for CMP 170HX recipes.' - points: - - - context_len: 262144 - decode_tok_s: 192.4 - prefill_tok_s: 7006 - ttft_ms: 126 - concurrency: 1 - notes: '1 user: structured 192.4 / code 177.8 / prose 145.4 tok/s, median of 5 streaming runs after the first token; TTFT 126 ms (short prompt). Cold prefill 7,006 tok/s, median of 9 at 24K-38K tokens, no cache hits.' - - - context_len: 262144 - decode_tok_s: 324.5 - concurrency: 2 - notes: '2 users, aggregate: structured 324.5 / code 242.8 / prose 238.9 tok/s, median of 3 runs, completion tokens / batch wall time.' - - - context_len: 262144 - decode_tok_s: 598.2 - concurrency: 4 - notes: '4 users, aggregate: structured 598.2 / code 446.2 / prose 399.2 tok/s, median of 3 runs (4 slots, same day: 598.9 / 446.5 / 400.1).' - - - context_len: 262144 - decode_tok_s: 986.4 - concurrency: 8 - notes: '8 users, aggregate: structured 986.4 / code 734.2 / prose 619.6 tok/s, median of 3 runs. Before this revision (4 slots, graphs to 16): 599.3 / 446.9 / 387.8; 8 slots with graphs only to 16: 395.5 / 332.2 / 286.2.' - context_len: 262144 + tok_s: 240.4 + context_len: 262144 + notes: 'All figures author-measured 2026-10-26 on an idle box (no other traffic in the window). MiaAI-Lab decode prompts (structured = count 1-200, code = clamp_range function, prose = hash-map explanation), temperature 0, thinking off, 400-token cap, streaming; the protocol Morrowmake publishes for CMP 170HX recipes.' + points: + - + context_len: 262144 + concurrency: 1 + decode_tok_s: 240.4 + prefill_tok_s: 6964 + notes: '1 user: structured 240.4 / code 224.5 / prose 155.6 tok/s, median of 10 runs. Cold prefill 6,964 tok/s, median of 9 at 24K-38K tokens.' + - + context_len: 262144 + concurrency: 2 + decode_tok_s: 341.5 + notes: '2 users aggregate: structured 341.5 / code 278.7 / prose 251.5 tok/s, median of 3 runs (structured runs vary 206-417).' + - + context_len: 262144 + concurrency: 4 + decode_tok_s: 608.1 + notes: '4 users aggregate: structured 608.1 / code 512.5 / prose 396.0 tok/s, median of 3 runs (structured runs vary 512-700).' + - + context_len: 262144 + concurrency: 8 + decode_tok_s: 1044.0 + notes: '8 users aggregate: structured 1,044.0 / code 714.5 / prose 625.0 tok/s, median of 3 runs. Previous revision: 986.4 / 734.2 / 619.6.' artifacts: - kind: model_weights @@ -115,6 +114,9 @@ provenance: - name: mannydelrio url: 'https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx' + - + name: 'AlexBrandes (W4A16 draft lm_head patch, quantizer)' + url: 'https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next' - name: 'cmpunlocker (amoghmunikote)' url: 'https://github.com/amoghmunikote/cmpunlocker'