Qwen3.8-Flash-Next-Uncensored AWQ W4A16 on 2x CMP 170HX — 8 slots, PP2, MTP5 + int4 draft head, PLE-from-SSD (mannydelrio)

vllm serve /opt/flashnext/model-awq-int4ple --served-model-name tinybeast-flashnext-uncensored --load-format safetensors --safetensors-load-strategy lazy --pipeline-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 4096 --block-size 1632 --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[6,12,18,24,30,36,42,48],"max_cudagraph_capture_size":48}' --gpu-memory-utilization 0.94 --speculative-config '{"method":"mtp","num_speculative_tokens":5}' --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --host 127.0.0.1 --port 18084

env: 'VLLM_PLUGINS=tinybeast_ple_int4,tinybeast_ple_ssd | TB_PLE_SSD=1 | TB_PLE_SSD_PREFETCH=1 | TB_TORCH_MEM_CAP_GIB=61.5 | VLLM_DETERMINISTIC_MOE_ALIGN=0 | VLLM_PP_LAYER_PARTITION=25,23 | CUDA_MODULE_LOADING=EAGER | NCCL_ALGO=Ring | NCCL_PROTO=Simple | VLLM_MTP_DRAFT_HEAD_PATH=/opt/flashnext/draft_lm_head_w4a16.safetensors'
notes: "2026-10-26 revision. Changes vs the 2026-10-15 revision (3 draft tokens): num_speculative_tokens 5, the MTP drafter uses its own int4 (W4A16, group 128, symmetric) copy of lm_head (AlexBrandes/vllm commit 14457f89e5, env VLLM_MTP_DRAFT_HEAD_PATH; the file is made from the checkpoint's lm_head with tools/quantize_lm_head.py from AlexBrandes/cmp170hx-qwen38-flash-next, RTN, 8 s on CPU; verification still uses the full bf16 head, so accepted tokens are unchanged), --block-size 1632 (5 drafts give a QSA ring of 12, which must divide the block size; the default 1616 fails at start-up) and decode graphs 6..48 = 8 slots x (1 + 5). Same-day A/B on this box: 5 drafts WITHOUT the int4 head lost prefill (5,468 tok/s) and was worse on prose and at 8 users; 3 drafts + int4 head gave 208/192/157 single-user and 1,038/760/641 at 8 users (best at 8 users, slower for one coder); 4 drafts + head sat between. Measured with MiaAI-Lab decode prompts (structured = count 1-200, code = clamp_range function, prose = hash-map explanation), temperature 0, thinking off, 400-token cap, streaming; the protocol Morrowmake publishes for CMP 170HX recipes. Single user = median of 10 runs; 2/4/8 users = median of 3 runs of completion tokens / batch wall time. Cold prefill = real-text prompts of 23,945 / 34,299 / 37,905 tokens x3, unique salt first, 0 cache hits (median of 9: 6,964). Quality: HumanEval 157/164 (unchanged from the 3-draft build), needle retrieval passes at 91K / 183K / 219K tokens, GPU1 peak 63.7 GB as before. Engine = wtdcode/vllm-backport @ 3142ded1 source build + the recipe repo's patches and plugins + the draft-head patch above; the recipe repo's serve.sh (042acb9) is the 3-draft launch and is NOT yet updated, so the launch argv and env here are authoritative. Host: 2x CMP 170HX unlocked to 64 GB each (cmpunlocker v0.5), Ryzen 5 5600X, 32 GB RAM, 1 TB NVMe. Long context: optional YaRN x4 reaches 940,800 tokens (measured 2026-10-15 on the 3-draft build, not re-run)."

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots.json