Qwen3.8-Flash-Next-Uncensored AWQ W4A16 on 2x CMP 170HX — PP2, MTP3, PLE-from-SSD (mannydelrio)

community Community submission — verify on your own hardware.

recipe by mannydelrio↗, cmpunlocker (amoghmunikote)↗, clee/vllm-sm80↗, wtdcode (vllm-backport)↗ · listed by localai-recipes

revision r2 · commit c22ad53eb7 · updated 10 d ago · license CC-BY-4.0

chat code pipeline-parallel mtp-speculative abliterated awq ple-ssd uncensored long-context

Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE) in AWQ W4A16 (leoncca g32 build @ fa56146): PP2 with manual layer partition 25,23 (no P2P on CMP 170HX — PP beats TP, which pays a host-memory sync per layer), MTP 3-token draft, and the 29.8 GB PLE embedding table served FROM NVMe SSD via tinybeast plugins (TB_PLE_SSD=1 + prefetch so SSD reads overlap the previous GPU step). 4–5.7x faster than the stock TP baseline on the same cards at every prompt size, full native 262,144-token context, 897K tokens of KV. Measured by the author: 109/159 tok/s answers prose/code at 1 request (was ~37), 283/416 aggregate at 4 concurrent, prompt reading ~7,700 tok/s, TTFT 13.0 s at 100K. Quality unchanged: HumanEval 155/164, 8/8 long-doc to 222K. Engine is a SOURCE BUILD: wtdcode/vllm-backport @ 3142ded1 + the repo's patches — no published image, runtime notes carry the build steps. Cards are 8GB-variant CMP 170HX unlocked to 64 GB by cmpunlocker v0.5; 180 W/card.

Launch command

vllm serve /opt/flashnext/model-awq-int4ple --served-model-name tinybeast-flashnext-uncensored --load-format safetensors --safetensors-load-strategy lazy --pipeline-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 4 --max-num-batched-tokens 4096 --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8,12,16],"max_cudagraph_capture_size":16}' --gpu-memory-utilization 0.94 --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --host 127.0.0.1 --port 18084

runtime

env: "VLLM_PLUGINS=tinybeast_ple_int4,tinybeast_ple_ssd | TB_PLE_SSD=1 | TB_PLE_SSD_PREFETCH=1 | TB_TORCH_MEM_CAP_GIB=61.5 | VLLM_DETERMINISTIC_MOE_ALIGN=0 | VLLM_PP_LAYER_PARTITION=25,23 | CUDA_MODULE_LOADING=EAGER | NCCL_ALGO=Ring | NCCL_PROTO=Simple (verbatim from serve/serve.sh @ d82e8d1). Optional: TB_PLE_SSD_VERIFY=N cross-checks the first N CPU-computed PLE row ids against the GPU's (falls back to GPU path on mismatch); TB_PLE_SSD_TIMING=1 logs GPU-wait vs SSD-read per prompt step."
notes: "serve.sh exec's vllm serve with the launch argv above; the PLE checkpoint dir is $FLASHNEXT_ROOT/model-awq-int4ple written by scripts/prepare_checkpoint.py (resumable shard-by-shard download of leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32 @ fa56146 + bounded INT4 PLE conversion; prints READY; converted model is 110 GB on 1 TB NVMe). First start ~5.5 min. OpenAI-compatible API on 127.0.0.1:18084 as tinybeast-flashnext-uncensored; systemd user unit serve/tinybeast-vllm.service optional. Host: 2x CMP 170HX (8 GB variant, PCI 10de:20c2) unlocked to 64 GB each by cmpunlocker v0.5 (power cycle required; nvidia-smi must show 65536 MiB + ECC on), Ryzen 5 5600X + 16 GB RAM desktop. 180 W/card is the measured sweet spot (220/250 W bought only 5.6/8.4% faster long prompts for 19/36% more power). YaRN x2 to 524K works but shifts outputs (KL 0.019 vs 0.011 run-to-run floor) — repo stays native 262K."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3.json

Engine

vllm · source↗ · 3142ded1↗

Install note: No published image — source build required. Clone wtdcode/vllm-backport at 3142ded1671b9565503975ee946d7e19fe50ea95 into $FLASHNEXT_ROOT/vllm-backport, git apply the recipe repo's patches/vllm-backport-3142ded1.patch (pipeline-parallel KV fix from clee/vllm-sm80 + vLLM PR #57105), build into .venv per that repo's instructions (MAX_JOBS=2, CUDA 13.2). Then pip install -e plugins/tinybeast-ple-int4 -e plugins/tinybeast-ple-ssd from the recipe repo into the same venv (VLLM_PLUGINS=tinybeast_ple_int4,tinybeast_ple_ssd). Driver: nvidia-open 610.43.02 + cmpunlocker v0.5 (8GB profile, --no-iommu); older cmpunlocker v0.2 leaves a protected 141 MB region and long prompts Xid 31 on the second card.

Model

Qwen3.8-Flash-Next-Uncensored-AWQ (awq-w4a16-g32) · leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32↗
AA intelligence 40 (Artificial Analysis · measured)

Hardware

NVIDIA CMP 170HX 64GB ×2 (Ampere)

Context

max context: 256K tokens

KV pool

897,000 tokens across concurrent requests

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
8,192	4	—	283 tok/s	—	reported
262,144	1	7700 tok/s	109 tok/s	13000 ms	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Author-measured on the two CMP 170HX cards (README results tables, repo @ d82e8d1): 256-token answer end-to-end 1.7 s @1K prompt / 2.9 s @8K / 6.0 s @32K / 9.7 s @64K / 14.4 s @100K (4.4x / 4.5x / 5.2x / 5.6x / 5.7x vs their stock-TP baseline ~81.4 s @100K); 4 requests at once 8K each 7.8 s (4.0x). Decode 109 tok/s prose / 159 code per answer (baseline ~37); aggregate 283 / 416 tok/s at 4 concurrent. Prompt reading ~7,700 tok/s (baseline ~1,340). TTFT 13.0 s at 100K (baseline 74.6 s). KV cache total 897K tokens (baseline 260K); max context per request 262K. HumanEval 155/164 thinking-off temp 0.6 (runs 154-158); 8/8 long-doc questions to 222K; teacher-forced comparison within run-to-run noise of reference. claimed_by=submitter T0 — no run file ingested; import via POST /api/v1/benchmarks to upgrade.

notes

Author-measured on the two CMP 170HX cards (README results tables, repo @ d82e8d1): 256-token answer end-to-end 1.7 s @1K prompt / 2.9 s @8K / 6.0 s @32K / 9.7 s @64K / 14.4 s @100K (4.4x / 4.5x / 5.2x / 5.6x / 5.7x vs their stock-TP baseline ~81.4 s @100K); 4 requests at once 8K each 7.8 s (4.0x). Decode 109 tok/s prose / 159 code per answer (baseline ~37); aggregate 283 / 416 tok/s at 4 concurrent. Prompt reading ~7,700 tok/s (baseline ~1,340). TTFT 13.0 s at 100K (baseline 74.6 s). KV cache total 897K tokens (baseline 260K); max context per request 262K. HumanEval 155/164 thinking-off temp 0.6 (runs 154-158); 8/8 long-doc questions to 222K; teacher-forced comparison within run-to-run noise of reference. claimed_by=submitter T0 — no run file ingested; import via POST /api/v1/benchmarks to upgrade.

Downloads

Weights https://huggingface.co/leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32
fa56146238f9fcd5ab591b7052b31e28efdca5c3 — AWQ W4A16 g32 build of orcarouter/Qwen3.8-Flash-Next-Uncensored; converted locally by scripts/prepare_checkpoint.py into model-awq-int4ple/ (110 GB incl. 4-bit PLE table)
Engine image https://github.com/wtdcode/vllm-backport
3142ded1671b9565503975ee946d7e19fe50ea95 — source build, NO image (install_note carries the build + patches + plugin steps)
Reference https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx
d82e8d1de7fd591ef83faa33d8aa28c8eecbb619 — source recipe repo @ HEAD (serve/serve.sh is the verbatim argv source; patches/, plugins/, scripts/, tests/ all at this commit)

Revision notes

Community ingest of mannydelrio's recipe repo @ d82e8d1 (owner-named by Casey Oct 6; repo created same day, Apache-2.0). Launch argv transcribed verbatim from serve/serve.sh @ d82e8d1; runtime env verbatim from the same script; measured numbers are the author's README tables, logged claimed_by=submitter T0 (no run file). Built-on credits per the repo's own 'Built on' section: cmpunlocker (card unlock + patched driver), clee/vllm-sm80 (PP KV fix + SSD row reader ple_preadv.py), wtdcode/vllm-backport (engine fork), plus ideas from Strata and the DUFRelic recipe. Model license qwen-community-1.0 (uncensored: refusal behavior removed).

Revision history

b9dd621513 recipe: update qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3 (r2: /max_context_len, /kv_pool_tokens) 10 d ago diff
dfdbaba624 recipe: create qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3 24 d ago

Fork

Export

recipe.yaml (raw)