{"slug":"qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3","title":"Qwen3.8-Flash-Next-Uncensored AWQ W4A16 on 2x CMP 170HX \u2014 PP2, MTP3, PLE-from-SSD (mannydelrio)","summary":"Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE) in AWQ W4A16 (leoncca g32 build @ fa56146): PP2 with manual layer partition 25,23 (no P2P on CMP 170HX \u2014 PP beats TP, which pays a host-memory sync per layer), MTP 3-token draft, and the 29.8 GB PLE embedding table served FROM NVMe SSD via tinybeast plugins (TB_PLE_SSD=1 + prefetch so SSD reads overlap the previous GPU step). 4\u20135.7x faster than the stock TP baseline on the same cards at every prompt size, full native 262,144-token context, 897K tokens of KV. Measured by the author: 109/159 tok/s answers prose/code at 1 request (was ~37), 283/416 aggregate at 4 concurrent, prompt reading ~7,700 tok/s, TTFT 13.0 s at 100K. Quality unchanged: HumanEval 155/164, 8/8 long-doc to 222K. Engine is a SOURCE BUILD: wtdcode/vllm-backport @ 3142ded1 + the repo's patches \u2014 no published image, runtime notes carry the build steps. Cards are 8GB-variant CMP 170HX unlocked to 64 GB by cmpunlocker v0.5; 180 W/card.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"c22ad53eb75e847760d8f7a80a062e4419dc3043","content_hash":"9ce5377d01f3342eee13c43ab3526c4c4596f19c","created_at":"2026-10-20 15:34:07.132081-06","notes":"Community ingest of mannydelrio's recipe repo @ d82e8d1 (owner-named by Casey Oct 6; repo created same day, Apache-2.0). Launch argv transcribed verbatim from serve/serve.sh @ d82e8d1; runtime env verbatim from the same script; measured numbers are the author's README tables, logged claimed_by=submitter T0 (no run file). Built-on credits per the repo's own 'Built on' section: cmpunlocker (card unlock + patched driver), clee/vllm-sm80 (PP KV fix + SSD row reader ple_preadv.py), wtdcode/vllm-backport (engine fork), plus ideas from Strata and the DUFRelic recipe. Model license qwen-community-1.0 (uncensored: refusal behavior removed)."},"engine_source":{"type":"fork","git_url":"https://github.com/wtdcode/vllm-backport","base_engine":"vllm","pip_package":null,"base_version":"wtdcode/vllm-backport @ 3142ded1 (source build)","image_digest":null,"install_note":"No published image \u2014 source build required. Clone wtdcode/vllm-backport at 3142ded1671b9565503975ee946d7e19fe50ea95 into $FLASHNEXT_ROOT/vllm-backport, git apply the recipe repo's patches/vllm-backport-3142ded1.patch (pipeline-parallel KV fix from clee/vllm-sm80 + vLLM PR #57105), build into .venv per that repo's instructions (MAX_JOBS=2, CUDA 13.2). Then pip install -e plugins/tinybeast-ple-int4 -e plugins/tinybeast-ple-ssd from the recipe repo into the same venv (VLLM_PLUGINS=tinybeast_ple_int4,tinybeast_ple_ssd). Driver: nvidia-open 610.43.02 + cmpunlocker v0.5 (8GB profile, --no-iommu); older cmpunlocker v0.2 leaves a protected 141 MB region and long prompts Xid 31 on the second card.","pinned_commit":"3142ded1671b9565503975ee946d7e19fe50ea95"},"credit":[{"name":"mannydelrio","url":"https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx"},{"name":"cmpunlocker (amoghmunikote)","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"clee/vllm-sm80","url":"https://github.com/clee/vllm-sm80"},{"name":"wtdcode (vllm-backport)","url":"https://github.com/wtdcode/vllm-backport"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/opt/flashnext/model-awq-int4ple","--served-model-name","tinybeast-flashnext-uncensored","--load-format","safetensors","--safetensors-load-strategy","lazy","--pipeline-parallel-size","2","--dtype","bfloat16","--max-model-len","262144","--max-num-seqs","4","--max-num-batched-tokens","4096","--compilation-config","{\"mode\":0,\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[1,2,4,8,12,16],\"max_cudagraph_capture_size\":16}","--gpu-memory-utilization","0.94","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":3}","--reasoning-parser","qwen3","--enable-auto-tool-choice","--tool-call-parser","qwen3_coder","--host","127.0.0.1","--port","18084"],"runtime":{"env":"VLLM_PLUGINS=tinybeast_ple_int4,tinybeast_ple_ssd | TB_PLE_SSD=1 | TB_PLE_SSD_PREFETCH=1 | TB_TORCH_MEM_CAP_GIB=61.5 | VLLM_DETERMINISTIC_MOE_ALIGN=0 | VLLM_PP_LAYER_PARTITION=25,23 | CUDA_MODULE_LOADING=EAGER | NCCL_ALGO=Ring | NCCL_PROTO=Simple (verbatim from serve/serve.sh @ d82e8d1). Optional: TB_PLE_SSD_VERIFY=N cross-checks the first N CPU-computed PLE row ids against the GPU's (falls back to GPU path on mismatch); TB_PLE_SSD_TIMING=1 logs GPU-wait vs SSD-read per prompt step.","notes":"serve.sh exec's vllm serve with the launch argv above; the PLE checkpoint dir is $FLASHNEXT_ROOT/model-awq-int4ple written by scripts/prepare_checkpoint.py (resumable shard-by-shard download of leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32 @ fa56146 + bounded INT4 PLE conversion; prints READY; converted model is 110 GB on 1 TB NVMe). First start ~5.5 min. OpenAI-compatible API on 127.0.0.1:18084 as tinybeast-flashnext-uncensored; systemd user unit serve/tinybeast-vllm.service optional. Host: 2x CMP 170HX (8 GB variant, PCI 10de:20c2) unlocked to 64 GB each by cmpunlocker v0.5 (power cycle required; nvidia-smi must show 65536 MiB + ECC on), Ryzen 5 5600X + 16 GB RAM desktop. 180 W/card is the measured sweet spot (220/250 W bought only 5.6/8.4% faster long prompts for 19/36% more power). YaRN x2 to 524K works but shifts outputs (KL 0.019 vs 0.011 run-to-run floor) \u2014 repo stays native 262K."},"kv_pool_tokens":897000,"max_context_len":262144},"models":[{"slug":"qwen3-8-flash-next-uncensored-awq","name":"Qwen3.8-Flash-Next-Uncensored-AWQ","hf_repo":"leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32","quant":"awq-w4a16-g32"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["abliterated","awq","long-context","mtp-speculative","pipeline-parallel","ple-ssd","uncensored"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured on the two CMP 170HX cards (README results tables, repo @ d82e8d1): 256-token answer end-to-end 1.7 s @1K prompt / 2.9 s @8K / 6.0 s @32K / 9.7 s @64K / 14.4 s @100K (4.4x / 4.5x / 5.2x / 5.6x / 5.7x vs their stock-TP baseline ~81.4 s @100K); 4 requests at once 8K each 7.8 s (4.0x). Decode 109 tok/s prose / 159 code per answer (baseline ~37); aggregate 283 / 416 tok/s at 4 concurrent. Prompt reading ~7,700 tok/s (baseline ~1,340). TTFT 13.0 s at 100K (baseline 74.6 s). KV cache total 897K tokens (baseline 260K); max context per request 262K. HumanEval 155/164 thinking-off temp 0.6 (runs 154-158); 8/8 long-doc questions to 222K; teacher-forced comparison within run-to-run noise of reference. claimed_by=submitter T0 \u2014 no run file ingested; import via POST /api/v1/benchmarks to upgrade.\", \"tok_s\": 109, \"points\": [{\"notes\": \"author README tables @ repo d82e8d1: 109 tok/s prose answer (159 code), ~37 baseline; prompt reading ~7,700 tok/s @100K; TTFT 13.0 s @100K; 100K prompt + 256-tok answer 14.4 s total = 5.7x vs stock TP\", \"ttft_ms\": 13000, \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 109, \"prefill_tok_s\": 7700}, {\"notes\": \"author README: 4 requests at once, 8K each \u2014 aggregate 283 tok/s prose / 416 code (4.0x end-to-end vs baseline 31.0 s)\", \"concurrency\": 4, \"context_len\": 8192, \"decode_tok_s\": 283}], \"context_len\": 262144}"}],"perf_points":[{"id":1018,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":283,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"author README: 4 requests at once, 8K each \u2014 aggregate 283 tok/s prose / 416 code (4.0x end-to-end vs baseline 31.0 s)"},{"id":1017,"context_len":262144,"prefill_tok_s":7700,"decode_tok_s":109,"ttft_ms":13000,"concurrency":1,"claimed_by":"submitter","notes":"author README tables @ repo d82e8d1: 109 tok/s prose answer (159 code), ~37 baseline; prompt reading ~7,700 tok/s @100K; TTFT 13.0 s @100K; 100K prompt + 256-tok answer 14.4 s total = 5.7x vs stock TP"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32","pin":"fa56146238f9fcd5ab591b7052b31e28efdca5c3 \u2014 AWQ W4A16 g32 build of orcarouter/Qwen3.8-Flash-Next-Uncensored; converted locally by scripts/prepare_checkpoint.py into model-awq-int4ple/ (110 GB incl. 4-bit PLE table)"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/wtdcode/vllm-backport","pin":"3142ded1671b9565503975ee946d7e19fe50ea95 \u2014 source build, NO image (install_note carries the build + patches + plugin steps)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx","pin":"d82e8d1de7fd591ef83faa33d8aa28c8eecbb619 \u2014 source recipe repo @ HEAD (serve/serve.sh is the verbatim argv source; patches/, plugins/, scripts/, tests/ all at this commit)"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3.json","alternate":"/r/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:07.132081-06","created_at":"2026-10-06 15:23:04.648489-06","speeds":{"decode_tok_s":109,"prefill_tok_s":7700,"leaderboard":"/api/v1/leaderboard.json"}}