Qwen3.8-Flash-Next INT4-Mixed 170HX x2, PP2 MTP3 BF16-KV PLE-NVMe (bozocentral)

/mnt/cache/appdata/llama-swap/config/flashnext-2x64.sh '${PORT}' '# wrapper (llama-swap cmd) - llama-swap argv-splits and strips all quotes, so the JSON args are bash-quoted inside; wrapper execs:' 'docker run --runtime nvidia --rm --name flashnext-2x64' '--gpus 2' '--ipc=host --shm-size=96g --cap-add=SYS_PTRACE --ulimit memlock=-1' '-p ${PORT}:8000' '-v /mnt/cache/appdata/models/Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound-4x3090:/model:ro' '-v /mnt/cache/appdata/models/vllm/ngram_table.bin:/vllm-ngram/ngram_table.bin' '-v /mnt/cache/appdata/models/vllm/kernel-cache/vllm:/root/.cache/vllm' '-v /mnt/cache/appdata/models/vllm/kernel-cache/triton:/root/.triton' '-e HF_HUB_OFFLINE=1 -e VLLM_CACHE_ROOT=/root/.cache/vllm -e TRITON_CACHE_DIR=/root/.triton' '-e CUDA_DEVICE_ORDER=PCI_BUS_ID -e VLLM_PP_LAYER_PARTITION=24,24' '-e VLLM_PLE_MMAP_PATH=/vllm-ngram/ngram_table.bin' '-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True' '-e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 -e VLLM_USE_BREAKABLE_CUDAGRAPH=0' qwen38-flash-next:2x64 '--model /model' '--served-model-name Qwen3.8-Flash-Next' '--pipeline-parallel-size 2' '--tensor-parallel-size 1' '--speculative-config {"method":"mtp","num_speculative_tokens":3}' '--per-request-spec-decode-metrics detailed' '--engram-config {"cpu_offload":true}' '--max-model-len 262144' '--max-num-seqs 6' '--max-num-batched-tokens 1568' '--gpu-memory-utilization 0.96' '--kv-cache-dtype bfloat16' --enable-prefix-caching '--mamba-cache-mode align' '--compilation-config {"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,6,8]}' '--reasoning-parser qwen3' --enable-auto-tool-choice '--tool-call-parser qwen3_coder' --trust-remote-code '--host 0.0.0.0' '--port 8000'

env: 'HF_HUB_OFFLINE=1 | VLLM_CACHE_ROOT=/root/.cache/vllm | TRITON_CACHE_DIR=/root/.triton | CUDA_DEVICE_ORDER=PCI_BUS_ID | VLLM_PP_LAYER_PARTITION=24,24 | VLLM_PLE_MMAP_PATH=/vllm-ngram/ngram_table.bin | PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 | VLLM_USE_BREAKABLE_CUDAGRAPH=0'
host: 'Unraid 7.x, 64 GB system RAM, 2x CMP 170HX on PCIe Gen2 x4 links, one fast NVMe (the PLE table must live on NVMe); 128 GiB total VRAM carries the 65-67 GiB body + FULL BF16 KV cache with room to spare'
notes: "All flags transcribed verbatim from the bozocentral wrapper script (flashnext-2x64.sh) and llama-swap config in the LLM-instructions section, stated as of 2026-10-07. --gpus 2 (numeric) is the only form that survives llama-swap's argv splitter (device=0,1 needs quote chars that get stripped; docker also rejects 'cannot set both Count and DeviceIDs'). kernel caches on NVMe make every boot after the first ~5 min. PLE mmap: pre-size the table file sparse to EXACTLY 102400491520 bytes before first boot (0-byte or wrong-size file = boot error; sparse exact-size = refill; existing valid table = COW map, weight_loader returns early when prefilled). Shard drop (only after the table file exists and the gate logs mode=c, prefilled=True): back up the index, strip the 128 model-extra-ngram entries, rm the 33 shards; filter_duplicate_safetensors_files() hard-fails cold start if files are removed while still index-referenced. Rebuild source if the table is lost: restore index backup + scoped re-fetch of model-extra-ngram-* at snapshot 6a995be5 (keep .cache/huggingface/trees/6a995be5*.json with per-file lfs_sha256), then rm the table file (next boot rebuilds, +~10 min). MTP k=3 measured mean acceptance length ~4.0 on predictable content (197-201 tok/s), 2.3-2.9 thinking-heavy (116-122, single-stream parity at worst); k=1 mean 2.0 (122 tok/s); k=5+ impossible - QSA indexer compress-ratio hard-caps 3 drafts + 1 bonus. With k=3 the padded decode batch is seqs x 4, so cudagraph_capture_sizes covers 4-multiples when raising concurrency; 2-stream aggregate 154-212. Boot gates (docker logs): 'Resolved architecture: Qwen4ExpMTP', 'PLE table mapped from /vllm-ngram/ngram_table.bin (95.37 GiB)' + 'PLE page prefetch on' (needs cap_add SYS_PTRACE for process_madvise), Marlin WNA16 MoE kernels, 'GPU KV cache size >= 1.8M tokens (7+ requests of 262144)', 'Application startup complete'. fp8 KV is NOT used: the minamism base supports it (QSA patch), but the cache runs full BF16 - the unquantized cache still fits (~1.8M tokens) and fp8 only adds accuracy risk on a rig with this much VRAM. Do not remove PYTORCH_CUDA_ALLOC_CONF / VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS / ulimit memlock / --mamba-cache-mode align: long prompts fail or the server does not start. 'Setting attention block size to 1568' in the log = image not patched correctly."
router: 'llama-swap is the ROUTER, not the engine - it launches the docker wrapper per model and proxies; engine = vLLM. config.yaml (llama-swap): healthCheckTimeout: 2000 top-level (default 900 s is exceeded by first boots: PLE refill + shard weight load; per-model key is ignored in recent versions); models.Qwen3.8-Flash-Next: cmd /config/flashnext-2x64.sh ${PORT}, cmdStop docker stop flashnext-2x64, proxy http://${vllm_proxy_host}:${PORT}, ttl: 0, metadata.context_length 262144. Load by sending any request with model=Qwen3.8-Flash-Next through llama-swap.'
llama_swap_router_only: true

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-int4-mixed-170hx-x2-pp2-mtp3-bf16-kv-ple-nvme.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-int4-mixed-170hx-x2-pp2-mtp3-bf16-kv-ple-nvme.json