{"slug":"qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n","title":"Qwen3.8-Flash-Next 3bpw MTP on 1x CMP 170HX SSD PLE AIO (iIIusi0n)","summary":"First single-CMP-170HX recipe for Qwen3.8-Flash-Next: the AutoRound 3bpw+MTP checkpoint on ONE 64 GiB SM80 GPU with the 95.37 GiB BF16 PLE table left ON the SSD \u2014 native Linux AIO async reads, 16K prompt read-ahead and a 512 MiB row cache stream it per-row while the model loads metadata-only, so the 176B MoE fits one card. Patch adds async PLE reads/read-ahead, metadata-only loading, CUDA-graph host breaks, mixed-bit expert handling and MTP layer remapping; KLD 0.1208 vs original weights. Three-round medians (128 output tokens, temp 0, thinking off, EOS ignored): 8K prefill 2,636 tok/s, single-request decode 111 tok/s, 667 tok/s aggregate at 16 concurrent. Honest state: 31 SSD/graph tests PASS, 134 requests no API errors, BUT an arithmetic-consistency issue remains open; context advertised 262K, timing tested only through 32K. Tested sparse (15 GiB RAM, 31 GiB swap) on ENTERPRISE SATA at 180 W. Documented swap: the orcarouter uncensored AutoRound 3bpw checkpoint runs the same stack.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"827bfe01549b43357fa06a8610ea5e64b55583b0","content_hash":"3a31e6a965360ba192fd941458c89f62f07e27e6","created_at":"2026-10-08 21:11:56.66944-06","notes":"Submitted 2026-10-09 from owner direct-ingest order (kanban t_49dfa0f4, Casey Oct 8: 'please add this to the single card 170hx builds'). Repo HEAD @ c7fe762c415c9f53fabccdadc25be949eb92052f verified live 2026-10-09 (21 stars/4 forks, Apache-2.0 code; model Qwen Community License 1.0). Engine pin a5a30471ff2bb7f0824f2da10e358af98d304472 is the vLLM commit the repo's own patch targets (unpatched vLLM fails \u2014 install_note carries it). Launch argv + env token-verified against scripts/serve.sh + scripts/start.sh + docs/GUIDE.md at the repo HEAD; tuning env table and benchmark invocations transcribed from GUIDE.md; measured tables transcribed from docs/RESULTS.md (three-round medians, results/ raw data referenced upstream). HF pins verified live via HF API 2026-10-09: klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP @ ce0e0b9 resolves 34 files incl. 11 main + 2 MTP shards; uncensored twin klee100/Qwen3.8-Flash-Next-Uncensored-AutoRound-3bpw-MTP @ 060ce423 (base_model:quantized orcarouter/Qwen3.8-Flash-Next-Uncensored @ e0968000 \u2014 lineage Qwen official -> OrcaRouter uncensored fine-tune -> AutoRound quant). Hardware: FIRST cmp-170hx x1 profile on site \u2014 seeded as profile cmp-170hx-x1 (family stem cmp-170hx, ampere, 64 GB) by the paired migration so ingest resolves the existing row instead of silent-creating NULL-card_family junk (0041/0047 rationale). This is the native-AIO single-card counterpart to the clee PP3 3x170HX recipe (which credits this repo's patch approach)."},"engine_source":{"type":"fork","git_url":"https://github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx","base_engine":"vllm","pip_package":null,"base_version":"patched vLLM @ a5a30471 (ple-ssd patch set)","image_digest":null,"install_note":"NOT installable from any image as written: patches/qwen38-ple-ssd.patch must be applied to vLLM pinned at commit a5a30471ff2bb7f0824f2da10e358af98d304472 and optimization/ple_ssd_io.so compiled from the patch's C source (scripts/install-runtime.sh does both; scripts/build-ple-io.sh rebuilds the .so alone). UNPATCHED vLLM FAILS on this checkpoint \u2014 the patch is not version-compatible with other trees. The C helper is pure Linux AIO syscalls (no libaio/liburing, no vLLM CUDA-kernel rebuild). Runtime env lives at ~/vllm/.venv built by scripts/install-runtime.sh.","pinned_commit":"a5a30471ff2bb7f0824f2da10e358af98d304472"},"credit":[{"name":"iIIusi0n (recipe, vLLM ple-ssd patch set, AIO helper, benchmark harness)","url":"https://github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx"},{"name":"Qwen (official Qwen3.8-Flash-Next base, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"klee100 (AutoRound 3bpw+MTP quantized checkpoints, official + uncensored)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"OrcaRouter (Qwen3.8-Flash-Next-Uncensored BF16 abliterated fine-tune)","url":"https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored"},{"name":"vLLM project (upstream vLLM @ a5a30471 the patch applies to)","url":"https://github.com/vllm-project/vllm"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["git clone https://github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx.git && cd qwen3.8-flash-next-cmp170hx","export VLLM_WORKDIR=$HOME/vllm && mkdir -p $VLLM_WORKDIR && git clone https://github.com/vllm-project/vllm.git $VLLM_WORKDIR/src && git -C $VLLM_WORKDIR/src checkout a5a30471ff2bb7f0824f2da10e358af98d304472","./scripts/install-runtime.sh","./scripts/download-model.sh","export VLLM_WORKDIR=$HOME/vllm && export CUDA_HOME=/usr/local/cuda && ./scripts/start.sh","curl -f http://127.0.0.1:8000/health"],"runtime":{"env":"VLLM_WORKDIR(~/vllm: .venv + patched src + optimization/ple_ssd_io.so)/CUDA_HOME(/usr/local/cuda)/QWEN_HOST(0.0.0.0)/QWEN_PORT(8000)/QWEN_MTP(1; 0 disables MTP)/QWEN_SEQS(16)/QWEN_BATCH_TOKENS(2048)/QWEN_GRAPH_MODE(FULL_AND_PIECEWISE)/QWEN_GPU_MEMORY(0.96)/QWEN_CONTEXT(auto)/QWEN_SSD_DEPTH(256)/QWEN_SSD_CACHE_MB(512)/QWEN_SSD_PREFETCH(16384 remaining prompt tokens; 0 disables read-ahead)/QWEN_CAPTURE_SIZES(override JSON array; keep consistent with batch tokens)/QWEN_OMP_THREADS(1)/QWEN_EAGER(0; 1 = --enforce-eager)/QWEN_PRINT_COMMAND(1 prints the resolved command and exits)/HF_HUB_OFFLINE(1)/MAX_JOBS(2) (verbatim from scripts/serve.sh @ c7fe762).","notes":"serve.sh @ c7fe762 resolves to: vllm serve $QWEN_MODEL_DIR --served-model-name Qwen3.8-Flash-Next --host 0.0.0.0 --port 8000 --dtype bfloat16 --max-model-len auto --max-num-seqs 16 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.96 --safetensors-load-strategy lazy --mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml --additional-config {\"ple_ssd_offload\":true,\"ple_ssd_workers\":16,\"ple_ssd_cache_mb\":512,\"ple_ssd_native_library\":\"$VLLM_WORKDIR/optimization/ple_ssd_io.so\",\"ple_ssd_io_depth\":256,\"ple_ssd_prefetch_tokens\":16384} plus --compilation-config {\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[1..16,18..32,64,128,256,512,1024,2048]} and --speculative-config {\"method\":\"mtp\",\"num_speculative_tokens\":1} (QWEN_MTP=1 default; QWEN_EAGER=1 swaps the compilation block for --enforce-eager; QWEN_PRINT_COMMAND=1 prints without starting). start.sh wraps serve.sh in screen session 'vllm' and refuses to replace a live one (screen -S vllm -X quit, wait for engine exit, restart); foreground alternative is ./scripts/serve.sh directly; logs at $VLLM_WORKDIR/optimization/server.log; load + graph prep took 8-10 min on the author's box. TESTED HOST ENVELOPE (deliberately sparse): 15 GiB RAM + 31 GiB swap, ENTERPRISE SATA SSD, 180 W GPU power cap \u2014 the metadata-only load and async PLE reads are what make that envelope work; a consumer SSD may not sustain the same read pattern (enterprise SATA caveat), and less swap headroom is untested. Upstream FP8 pinned-memory PLE kernel does not compile on SM80 \u2014 this deployment uses the BF16 SSD path. CUDA toolkit 13.3 at /usr/local/cuda (the older /usr/bin/nvcc could not compile the runtime sampler); tested pins: Python 3.12.13, driver 610.43.03, PyTorch 2.13.0 cu130, Transformers 5.17.0, Humming kernels 0.1.12 (requirements/tested-constraints.txt + runtime-snapshot.txt audit snapshot). Checkpoint is 153,012,762,024 bytes (~142.5 GiB), all 13 safetensors incl. MTP shards kept, PLE shard stays in the model dir (no transformed embedding file). Model download pinned to weight revision ce0e0b94083895bd836b916f29bf105c40a8162a by scripts/download-model.sh. DOCUMENTED UNCENSORED SWAP: orcarouter/Qwen3.8-Flash-Next-Uncensored (BF16 abliterated OrcaRouter fine-tune of the same Qwen base) quantized as klee100/Qwen3.8-Flash-Next-Uncensored-AutoRound-3bpw-MTP @ 060ce423a7fd31ae4b1a240b639429d7f6cb3764 runs on this same stack \u2014 point QWEN_MODEL_DIR at it. OpenAI-compatible API at http://localhost:8000/v1, model id Qwen3.8-Flash-Next, reasoning parser qwen3 + tool parser qwen3_xml; sample chat call uses chat_template_kwargs {\"enable_thinking\":false}. Reproduce timings: benchmarks/verify_server.py --long --tools, benchmarks/bench_server.py --concurrency 1 4 8 16 --rounds 3, benchmarks/bench_prefill.py --seed 729156 --repeats 3 --lengths 512 2048 8192 32768 (use a fresh prefill seed to avoid prefix-cache reuse)."}},"models":[{"slug":"qwen3-8-flash-next-3bpw-autoround-mtp","name":"Qwen3.8-Flash-Next-3bpw-AutoRound-MTP","hf_repo":"klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP","quant":"auto-round-3bpw-mtp"}],"hardware":[{"slug":"nvidia-cmp-170hx","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":1}],"generation":"ampere","tags":["autoround","enterprise-sata","metadata-load","mtp-speculative","native-aio","ple-ssd","single-card","sparse-host-ram"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured three-round medians (128 output tokens, temperature zero, thinking disabled, EOS ignored) from docs/RESULTS.md + results/ raw data @ c7fe762: single-request decode 111 tok/s, aggregate output 667 tok/s at 16 concurrent (includes prefill + HTTP overhead), fresh 8K prefill 2,636 prompt tok/s. KLD vs original weights 0.1208. The API advertises 262,144-token context but a full-length request was NOT tested \u2014 varied-token timing ran only through 32K; treat 262K as untested headroom. An arithmetic-consistency issue (an arithmetic prompt can return an incorrect answer) remains open upstream \u2014 smoke tests are not a quality benchmark.\", \"tok_s\": 111, \"points\": [{\"notes\": \"fresh 8K prefill median + single-request decode median + TTFT 0.105 s \u2014 author-measured, docs/RESULTS.md (128 output tokens, temp 0, thinking off, EOS ignored)\", \"ttft_ms\": 105, \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 111, \"prefill_tok_s\": 2636}, {\"notes\": \"16-concurrent per-request decode 47.32 tok/s median, aggregate 667.13 tok/s incl. prefill+HTTP, TTFT 0.262 s \u2014 author-measured, docs/RESULTS.md\", \"ttft_ms\": 262, \"concurrency\": 16, \"context_len\": 8192, \"decode_tok_s\": 47}, {\"notes\": \"8-concurrent per-request decode 60.63 tok/s median, aggregate 428.97 tok/s, TTFT 0.183 s \u2014 author-measured, docs/RESULTS.md\", \"ttft_ms\": 183, \"concurrency\": 8, \"context_len\": 8192, \"decode_tok_s\": 61}, {\"notes\": \"4-concurrent per-request decode 77.95 tok/s median, aggregate 277.22 tok/s, TTFT 0.195 s \u2014 author-measured, docs/RESULTS.md\", \"ttft_ms\": 195, \"concurrency\": 4, \"context_len\": 8192, \"decode_tok_s\": 78}, {\"notes\": \"fresh 2,048-token prompt, median 1.069 s \u2014 author-measured, docs/RESULTS.md prefill table (prefix-cache reuse avoided)\", \"context_len\": 2048, \"prefill_tok_s\": 1915}, {\"notes\": \"fresh 512-token prompt, median 0.317 s \u2014 author-measured, docs/RESULTS.md prefill table\", \"context_len\": 512, \"prefill_tok_s\": 1616}, {\"notes\": \"fresh 32,768-token prompt, median 15.372 s \u2014 author-measured, docs/RESULTS.md prefill table; deepest varied-token timing tested (262K advertised but untested)\", \"context_len\": 32768, \"prefill_tok_s\": 2132}], \"context_len\": 262144}"}],"perf_points":[{"id":498,"context_len":512,"prefill_tok_s":1616,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"fresh 512-token prompt, median 0.317 s \u2014 author-measured, docs/RESULTS.md prefill table"},{"id":497,"context_len":2048,"prefill_tok_s":1915,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"fresh 2,048-token prompt, median 1.069 s \u2014 author-measured, docs/RESULTS.md prefill table (prefix-cache reuse avoided)"},{"id":493,"context_len":8192,"prefill_tok_s":2636,"decode_tok_s":111,"ttft_ms":105,"concurrency":1,"claimed_by":"submitter","notes":"fresh 8K prefill median + single-request decode median + TTFT 0.105 s \u2014 author-measured, docs/RESULTS.md (128 output tokens, temp 0, thinking off, EOS ignored)"},{"id":496,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":78,"ttft_ms":195,"concurrency":4,"claimed_by":"submitter","notes":"4-concurrent per-request decode 77.95 tok/s median, aggregate 277.22 tok/s, TTFT 0.195 s \u2014 author-measured, docs/RESULTS.md"},{"id":495,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":61,"ttft_ms":183,"concurrency":8,"claimed_by":"submitter","notes":"8-concurrent per-request decode 60.63 tok/s median, aggregate 428.97 tok/s, TTFT 0.183 s \u2014 author-measured, docs/RESULTS.md"},{"id":494,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":47,"ttft_ms":262,"concurrency":16,"claimed_by":"submitter","notes":"16-concurrent per-request decode 47.32 tok/s median, aggregate 667.13 tok/s incl. prefill+HTTP, TTFT 0.262 s \u2014 author-measured, docs/RESULTS.md"},{"id":499,"context_len":32768,"prefill_tok_s":2132,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"fresh 32,768-token prompt, median 15.372 s \u2014 author-measured, docs/RESULTS.md prefill table; deepest varied-token timing tested (262K advertised but untested)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP","pin":"ce0e0b94083895bd836b916f29bf105c40a8162a"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-Uncensored-AutoRound-3bpw-MTP","pin":"060ce423a7fd31ae4b1a240b639429d7f6cb3764"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx","pin":"a5a30471ff2bb7f0824f2da10e358af98d304472"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n.json","alternate":"/r/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n","describedby":"/openapi.json"},"updated_at":"2026-10-08 21:11:56.66944-06","created_at":"2026-10-08 21:11:56.66944-06","speeds":{"decode_tok_s":111,"prefill_tok_s":2132,"leaderboard":"/api/v1/leaderboard.json"}}