Qwen3.8-Flash-Next 3bpw MTP on 1x CMP 170HX SSD PLE AIO (iIIusi0n)

'git clone https://github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx.git && cd qwen3.8-flash-next-cmp170hx' 'export VLLM_WORKDIR=$HOME/vllm && mkdir -p $VLLM_WORKDIR && git clone https://github.com/vllm-project/vllm.git $VLLM_WORKDIR/src && git -C $VLLM_WORKDIR/src checkout a5a30471ff2bb7f0824f2da10e358af98d304472' ./scripts/install-runtime.sh ./scripts/download-model.sh 'export VLLM_WORKDIR=$HOME/vllm && export CUDA_HOME=/usr/local/cuda && ./scripts/start.sh' 'curl -f http://127.0.0.1:8000/health'

env: 'VLLM_WORKDIR(~/vllm: .venv + patched src + optimization/ple_ssd_io.so)/CUDA_HOME(/usr/local/cuda)/QWEN_HOST(0.0.0.0)/QWEN_PORT(8000)/QWEN_MTP(1; 0 disables MTP)/QWEN_SEQS(16)/QWEN_BATCH_TOKENS(2048)/QWEN_GRAPH_MODE(FULL_AND_PIECEWISE)/QWEN_GPU_MEMORY(0.96)/QWEN_CONTEXT(auto)/QWEN_SSD_DEPTH(256)/QWEN_SSD_CACHE_MB(512)/QWEN_SSD_PREFETCH(16384 remaining prompt tokens; 0 disables read-ahead)/QWEN_CAPTURE_SIZES(override JSON array; keep consistent with batch tokens)/QWEN_OMP_THREADS(1)/QWEN_EAGER(0; 1 = --enforce-eager)/QWEN_PRINT_COMMAND(1 prints the resolved command and exits)/HF_HUB_OFFLINE(1)/MAX_JOBS(2) (verbatim from scripts/serve.sh @ c7fe762).'
notes: 'serve.sh @ c7fe762 resolves to: vllm serve $QWEN_MODEL_DIR --served-model-name Qwen3.8-Flash-Next --host 0.0.0.0 --port 8000 --dtype bfloat16 --max-model-len auto --max-num-seqs 16 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.96 --safetensors-load-strategy lazy --mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml --additional-config {"ple_ssd_offload":true,"ple_ssd_workers":16,"ple_ssd_cache_mb":512,"ple_ssd_native_library":"$VLLM_WORKDIR/optimization/ple_ssd_io.so","ple_ssd_io_depth":256,"ple_ssd_prefetch_tokens":16384} plus --compilation-config {"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1..16,18..32,64,128,256,512,1024,2048]} and --speculative-config {"method":"mtp","num_speculative_tokens":1} (QWEN_MTP=1 default; QWEN_EAGER=1 swaps the compilation block for --enforce-eager; QWEN_PRINT_COMMAND=1 prints without starting). start.sh wraps serve.sh in screen session ''vllm'' and refuses to replace a live one (screen -S vllm -X quit, wait for engine exit, restart); foreground alternative is ./scripts/serve.sh directly; logs at $VLLM_WORKDIR/optimization/server.log; load + graph prep took 8-10 min on the author''s box. TESTED HOST ENVELOPE (deliberately sparse): 15 GiB RAM + 31 GiB swap, ENTERPRISE SATA SSD, 180 W GPU power cap — the metadata-only load and async PLE reads are what make that envelope work; a consumer SSD may not sustain the same read pattern (enterprise SATA caveat), and less swap headroom is untested. Upstream FP8 pinned-memory PLE kernel does not compile on SM80 — this deployment uses the BF16 SSD path. CUDA toolkit 13.3 at /usr/local/cuda (the older /usr/bin/nvcc could not compile the runtime sampler); tested pins: Python 3.12.13, driver 610.43.03, PyTorch 2.13.0 cu130, Transformers 5.17.0, Humming kernels 0.1.12 (requirements/tested-constraints.txt + runtime-snapshot.txt audit snapshot). Checkpoint is 153,012,762,024 bytes (~142.5 GiB), all 13 safetensors incl. MTP shards kept, PLE shard stays in the model dir (no transformed embedding file). Model download pinned to weight revision ce0e0b94083895bd836b916f29bf105c40a8162a by scripts/download-model.sh. DOCUMENTED UNCENSORED SWAP: orcarouter/Qwen3.8-Flash-Next-Uncensored (BF16 abliterated OrcaRouter fine-tune of the same Qwen base) quantized as klee100/Qwen3.8-Flash-Next-Uncensored-AutoRound-3bpw-MTP @ 060ce423a7fd31ae4b1a240b639429d7f6cb3764 runs on this same stack — point QWEN_MODEL_DIR at it. OpenAI-compatible API at http://localhost:8000/v1, model id Qwen3.8-Flash-Next, reasoning parser qwen3 + tool parser qwen3_xml; sample chat call uses chat_template_kwargs {"enable_thinking":false}. Reproduce timings: benchmarks/verify_server.py --long --tools, benchmarks/bench_server.py --concurrency 1 4 8 16 --rounds 3, benchmarks/bench_prefill.py --seed 729156 --repeats 3 --lengths 512 2048 8192 32768 (use a fresh prefill seed to avoid prefix-cache reuse).'

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n.json