{"slug":"qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots","title":"Qwen3.8-Flash-Next-Uncensored AWQ W4A16 on 2x CMP 170HX \u2014 8 slots, PP2, MTP5 + int4 draft head, PLE-from-SSD (mannydelrio)","summary":"Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE), AWQ W4A16 g32: PP2 (CMP 170HX has no P2P), MTP 5-token drafting with an int4 draft-only lm_head, 8 slots, and the 29.8 GB PLE table on NVMe. Single user 240 / 225 / 156 tok/s structured / code / prose; 8 users 1,044 / 715 / 625 aggregate; cold prefill ~6,960 tok/s; 262,144 context; HumanEval 157/164. 2x CMP 170HX unlocked to 64 GB, Gen2 x4, 180 W each.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":6,"commit_sha":"afee85c4c32bdc63c49516ef67d31a132fe4056a","content_hash":"a8d1afa05ba63a9ab40e0259084a2de9005062b2","created_at":"2026-10-26 15:36:10.396992-06","notes":""},"engine_source":{"type":"release","git_url":"https://github.com/wtdcode/vllm-backport","base_engine":"vllm","pip_package":null,"base_version":"wtdcode/vllm-backport @ 3142ded1 (source build) + recipe patches","image_digest":null,"install_note":null,"pinned_commit":null},"credit":[{"name":"mannydelrio","url":"https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx"},{"name":"AlexBrandes (W4A16 draft lm_head patch, quantizer)","url":"https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next"},{"name":"cmpunlocker (amoghmunikote)","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"clee/vllm-sm80","url":"https://github.com/clee/vllm-sm80"},{"name":"wtdcode (vllm-backport)","url":"https://github.com/wtdcode/vllm-backport"}],"author":{"name":"mannydelrio1","isSite":false},"uploaded_by":"mannydelrio1","config":{"launch":["vllm","serve","/opt/flashnext/model-awq-int4ple","--served-model-name","tinybeast-flashnext-uncensored","--load-format","safetensors","--safetensors-load-strategy","lazy","--pipeline-parallel-size","2","--dtype","bfloat16","--max-model-len","262144","--max-num-seqs","8","--max-num-batched-tokens","4096","--block-size","1632","--compilation-config","{\"mode\":0,\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[6,12,18,24,30,36,42,48],\"max_cudagraph_capture_size\":48}","--gpu-memory-utilization","0.94","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":5}","--reasoning-parser","qwen3","--enable-auto-tool-choice","--tool-call-parser","qwen3_coder","--host","127.0.0.1","--port","18084"],"runtime":{"env":"VLLM_PLUGINS=tinybeast_ple_int4,tinybeast_ple_ssd | TB_PLE_SSD=1 | TB_PLE_SSD_PREFETCH=1 | TB_TORCH_MEM_CAP_GIB=61.5 | VLLM_DETERMINISTIC_MOE_ALIGN=0 | VLLM_PP_LAYER_PARTITION=25,23 | CUDA_MODULE_LOADING=EAGER | NCCL_ALGO=Ring | NCCL_PROTO=Simple | VLLM_MTP_DRAFT_HEAD_PATH=/opt/flashnext/draft_lm_head_w4a16.safetensors","notes":"2026-10-26 revision. Changes vs the 2026-10-15 revision (3 draft tokens): num_speculative_tokens 5, the MTP drafter uses its own int4 (W4A16, group 128, symmetric) copy of lm_head (AlexBrandes/vllm commit 14457f89e5, env VLLM_MTP_DRAFT_HEAD_PATH; the file is made from the checkpoint's lm_head with tools/quantize_lm_head.py from AlexBrandes/cmp170hx-qwen38-flash-next, RTN, 8 s on CPU; verification still uses the full bf16 head, so accepted tokens are unchanged), --block-size 1632 (5 drafts give a QSA ring of 12, which must divide the block size; the default 1616 fails at start-up) and decode graphs 6..48 = 8 slots x (1 + 5). Same-day A/B on this box: 5 drafts WITHOUT the int4 head lost prefill (5,468 tok/s) and was worse on prose and at 8 users; 3 drafts + int4 head gave 208/192/157 single-user and 1,038/760/641 at 8 users (best at 8 users, slower for one coder); 4 drafts + head sat between. Measured with MiaAI-Lab decode prompts (structured = count 1-200, code = clamp_range function, prose = hash-map explanation), temperature 0, thinking off, 400-token cap, streaming; the protocol Morrowmake publishes for CMP 170HX recipes. Single user = median of 10 runs; 2/4/8 users = median of 3 runs of completion tokens / batch wall time. Cold prefill = real-text prompts of 23,945 / 34,299 / 37,905 tokens x3, unique salt first, 0 cache hits (median of 9: 6,964). Quality: HumanEval 157/164 (unchanged from the 3-draft build), needle retrieval passes at 91K / 183K / 219K tokens, GPU1 peak 63.7 GB as before. Engine = wtdcode/vllm-backport @ 3142ded1 source build + the recipe repo's patches and plugins + the draft-head patch above; the recipe repo's serve.sh (042acb9) is the 3-draft launch and is NOT yet updated, so the launch argv and env here are authoritative. Host: 2x CMP 170HX unlocked to 64 GB each (cmpunlocker v0.5), Ryzen 5 5600X, 32 GB RAM, 1 TB NVMe. Long context: optional YaRN x4 reaches 940,800 tokens (measured 2026-10-15 on the 3-draft build, not re-run)."},"kv_pool_tokens":897000,"max_context_len":262144},"models":[{"slug":"qwen3-8-flash-next-uncensored-awq","name":"Qwen3.8-Flash-Next-Uncensored-AWQ","hf_repo":"leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32","quant":"awq-w4a16-g32"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["abliterated","awq","long-context","mtp-speculative","pipeline-parallel","ple-ssd","uncensored"],"tasks":["chat","code","vision"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"All figures author-measured 2026-10-26 on an idle box (no other traffic in the window). MiaAI-Lab decode prompts (structured = count 1-200, code = clamp_range function, prose = hash-map explanation), temperature 0, thinking off, 400-token cap, streaming; the protocol Morrowmake publishes for CMP 170HX recipes.\", \"tok_s\": 240.400000000000005684341886080801486968994140625, \"context_len\": 262144}"}],"perf_points":[{"id":1825,"context_len":262144,"prefill_tok_s":6964,"decode_tok_s":240.400000000000005684341886080801486968994140625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"1 user: structured 240.4 / code 224.5 / prose 155.6 tok/s, median of 10 runs. Cold prefill 6,964 tok/s, median of 9 at 24K-38K tokens."},{"id":1826,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":341.5,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"2 users aggregate: structured 341.5 / code 278.7 / prose 251.5 tok/s, median of 3 runs (structured runs vary 206-417)."},{"id":1827,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":608.1000000000000227373675443232059478759765625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"4 users aggregate: structured 608.1 / code 512.5 / prose 396.0 tok/s, median of 3 runs (structured runs vary 512-700)."},{"id":1828,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":1044,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"8 users aggregate: structured 1,044.0 / code 714.5 / prose 625.0 tok/s, median of 3 runs. Previous revision: 986.4 / 734.2 / 619.6."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32","pin":"fa56146238f9fcd5ab591b7052b31e28efdca5c3 \u2014 AWQ W4A16 g32 build of orcarouter/Qwen3.8-Flash-Next-Uncensored; converted by the repo's scripts/prepare_checkpoint.py into model-awq-int4ple/ (110 GB incl. 4-bit PLE table)"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/wtdcode/vllm-backport","pin":"3142ded1671b9565503975ee946d7e19fe50ea95 \u2014 source build, no image"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx","pin":"d82e8d1de7fd591ef83faa33d8aa28c8eecbb619 \u2014 recipe repo (serve.sh, patches, plugins, scripts); this revision adds --max-num-seqs 8 and graph sizes to 32"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots.json","alternate":"/r/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots","describedby":"/openapi.json"},"updated_at":"2026-10-26 15:36:10.396992-06","created_at":"2026-10-15 22:19:28.918165-06","speeds":{"decode_tok_s":240.400000000000005684341886080801486968994140625,"prefill_tok_s":6964,"leaderboard":"/api/v1/leaderboard.json"}}