{"slug":"qwen3-8-flash-next-int4-mixed-170hx-x2-pp2-mtp3-bf16-kv-ple-nvme","title":"Qwen3.8-Flash-Next INT4-Mixed 170HX x2, PP2 MTP3 BF16-KV PLE-NVMe (bozocentral)","summary":"2x CMP 170HX (GA100 sm_80, 64 GB/card, cmpunlocker unlock) recipe for Qwen3.8-Flash-Next from bozocentral's writeup (2026-10-07; page-reference: artifact IS the URL). Minachist INT4-Mixed-AutoRound (experts INT4 gs32, dense INT8, 4x3090 branch, ~65-67 GiB): all in VRAM except the PLE: BF16 n-gram table (~95.4 GiB) mmap'd from NVMe (built once, shards deleted). vLLM 0.29.1rc1.dev47 image pinned by digest + Minachist patch set (2x24GB + MTP trio + make_mtp_int4.py). PP2/TP1 - TP dies on Gen2 x4 links. MTP-3 native (acceptance ~4.0; k=5 QSA-hard-capped). KV cache deliberately FULL BF16 (-kv-cache-dtype bfloat16 explicit, NOT fp8 - unique among the site's 170HX recipes). Served through llama-swap (router only; engine vLLM) via a bash wrapper (llama-swap argv-split strips quotes, JSON args bash-quoted), healthCheckTimeout 2000. Author-measured, warm, through llama-swap: MTP-3 197-201 tok/s predictable / 116-122 thinking-heavy, prefill ~5.8K tok/s, 240K needle HIT, 2-stream agg 154-212.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"74b4d0668cfe56e40aacf78435a980f67309b432","content_hash":"6ecc7f86abe07f1a42fe9e5007ace36e2a1c658a","created_at":"2026-10-20 16:56:40.814942-06","notes":"Page-reference ingest (owner directive, card t_4021075f): source is a blog writeup with no git HEAD - reference artifact is the article URL + as-of date 2026-10-07, page-watch pin goes to the lar-scout git-watch lane. All launch flags/config transcribed verbatim from the page's LLM-instructions section; every pinned upstream verified live at ingest time: HF snapshot 6a995be5542081bb65d9b7e8f580f0b8220ce0e7 resolves and equals branch 4x3090 (65 files: 17 body shards + 33 PLE shards BF16 [320001536,160] ~95.4 GiB + MTP overlay outputs; quant compressed-tensors mixed - routed experts INT4 gs32, linear_attn/QSA/shared + hyper-connection + lm_head/embed/PLE-proj INT8, router/vision/PLE table/MTP BF16); docker digest 43f13b4c624ab9e9e6753d0eeb5953268bff334f2a826239e6f4a2197d47bb96 resolves on hub (tags 0.29.1rc1.dev47 / nightly-5f93dd53, 22 GB); patch tree f6f90ddc39e9d212aa5b96f43999183deb4ce192 on branch main carries vllm-patch/2x24GB (flash-next-vllm.patch, decode-01..07, ttft-01, mem-01, vision-01, upstream-01) + vllm-patch/3x24GB+MTP (mtp-01-enable, mtp-02-three-cards, mtp-03-structured-output-drafts, make_mtp_int4.py); NOTICE names the patches Minachist derivative works under Apache-2.0 EXCEPT upstream-01 = vLLM PR #55506 by Karl0007 (root cause first found by zebgop-ops/qwen38-flashnext-pp); make_mtp_int4.py + Dockerfiles MIT. Patches may not apply to other nightlies (0.29.1rc1.dev47+gdc36fcce9). Perf numbers are the author's, warm, measured through llama-swap, ranges in notes - submitter T0 only, no measured/benchmark flags (no run file). MTP-3 + PP-on-2x170HX findings match the site's other 170HX recipes; BF16 KV + PLE-from-NVMe are the distinguishing features. NOTE for future watch pins: Minachist HF repo + minamism articles are upstream-of-upstream - optional future pins, not auto-added (card item h)."},"engine_source":{"type":"docker_image","git_url":"https://huggingface.co/Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound","base_engine":"vllm","pip_package":null,"base_version":"0.29.1rc1.dev47 @ sha256:43f13b4c + Minachist patches","image_digest":"docker.io/vllm/vllm-openai@sha256:43f13b4c624ab9e9e6753d0eeb5953268bff334f2a826239e6f4a2197d47bb96","install_note":"Build required: pull base vllm/vllm-openai@sha256:43f13b4c...47bb96 (22 GB), apply the 16 Minachist patch files (12 from vllm-patch/2x24GB/ + mtp-01..03 and make_mtp_int4.py from vllm-patch/3x24GB+MTP/) in the upstream vllm-patch/2x24GB/Dockerfile order (order matters), py_compile the touched files, docker build -t qwen38-flash-next:2x64. One-time MTP overlay: make_mtp_int4.py over the checkpoint, merge its config/index/model_extra_tensors into the model dir (docker cannot bind file-over-file in a ro bind). PLE table: truncate -s 102400491520 the table file BEFORE first boot; first boot refills ~92 GiB from the shards (+~10 min), then strip the 128 shard index entries and delete the 33 shards (-96 GB). HF_HUB_OFFLINE=1. GATED model: HF token with repo accepted (Qwen Community License 1.0).","pinned_commit":"f6f90ddc39e9d212aa5b96f43999183deb4ce192"},"credit":[{"name":"bozocentral (Plean)","url":"https://www.bozocentral.com/2x-cmp170hx-qwen3-flash-next"},{"name":"Minachist (minamism) - INT4-Mixed-AutoRound model + vllm-patch set","url":"https://huggingface.co/Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound"},{"name":"minamism - patch-set writeups (Article #1, Article #2)","url":"https://minamism.com/posts/flash-next-vllm-optimization/"},{"name":"amoghmunikote - cmpunlocker community 170HX unlock","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"Karl0007 - upstream-01 patch is vLLM PR #55506","url":"https://github.com/vllm-project/vllm/pull/55506"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["/mnt/cache/appdata/llama-swap/config/flashnext-2x64.sh","${PORT}","# wrapper (llama-swap cmd) - llama-swap argv-splits and strips all quotes, so the JSON args are bash-quoted inside; wrapper execs:","docker run --runtime nvidia --rm --name flashnext-2x64","--gpus 2","--ipc=host --shm-size=96g --cap-add=SYS_PTRACE --ulimit memlock=-1","-p ${PORT}:8000","-v /mnt/cache/appdata/models/Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound-4x3090:/model:ro","-v /mnt/cache/appdata/models/vllm/ngram_table.bin:/vllm-ngram/ngram_table.bin","-v /mnt/cache/appdata/models/vllm/kernel-cache/vllm:/root/.cache/vllm","-v /mnt/cache/appdata/models/vllm/kernel-cache/triton:/root/.triton","-e HF_HUB_OFFLINE=1 -e VLLM_CACHE_ROOT=/root/.cache/vllm -e TRITON_CACHE_DIR=/root/.triton","-e CUDA_DEVICE_ORDER=PCI_BUS_ID -e VLLM_PP_LAYER_PARTITION=24,24","-e VLLM_PLE_MMAP_PATH=/vllm-ngram/ngram_table.bin","-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True","-e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 -e VLLM_USE_BREAKABLE_CUDAGRAPH=0","qwen38-flash-next:2x64","--model /model","--served-model-name Qwen3.8-Flash-Next","--pipeline-parallel-size 2","--tensor-parallel-size 1","--speculative-config {\"method\":\"mtp\",\"num_speculative_tokens\":3}","--per-request-spec-decode-metrics detailed","--engram-config {\"cpu_offload\":true}","--max-model-len 262144","--max-num-seqs 6","--max-num-batched-tokens 1568","--gpu-memory-utilization 0.96","--kv-cache-dtype bfloat16","--enable-prefix-caching","--mamba-cache-mode align","--compilation-config {\"mode\":0,\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[1,2,4,6,8]}","--reasoning-parser qwen3","--enable-auto-tool-choice","--tool-call-parser qwen3_coder","--trust-remote-code","--host 0.0.0.0","--port 8000"],"runtime":{"env":"HF_HUB_OFFLINE=1 | VLLM_CACHE_ROOT=/root/.cache/vllm | TRITON_CACHE_DIR=/root/.triton | CUDA_DEVICE_ORDER=PCI_BUS_ID | VLLM_PP_LAYER_PARTITION=24,24 | VLLM_PLE_MMAP_PATH=/vllm-ngram/ngram_table.bin | PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 | VLLM_USE_BREAKABLE_CUDAGRAPH=0","host":"Unraid 7.x, 64 GB system RAM, 2x CMP 170HX on PCIe Gen2 x4 links, one fast NVMe (the PLE table must live on NVMe); 128 GiB total VRAM carries the 65-67 GiB body + FULL BF16 KV cache with room to spare","notes":"All flags transcribed verbatim from the bozocentral wrapper script (flashnext-2x64.sh) and llama-swap config in the LLM-instructions section, stated as of 2026-10-07. --gpus 2 (numeric) is the only form that survives llama-swap's argv splitter (device=0,1 needs quote chars that get stripped; docker also rejects 'cannot set both Count and DeviceIDs'). kernel caches on NVMe make every boot after the first ~5 min. PLE mmap: pre-size the table file sparse to EXACTLY 102400491520 bytes before first boot (0-byte or wrong-size file = boot error; sparse exact-size = refill; existing valid table = COW map, weight_loader returns early when prefilled). Shard drop (only after the table file exists and the gate logs mode=c, prefilled=True): back up the index, strip the 128 model-extra-ngram entries, rm the 33 shards; filter_duplicate_safetensors_files() hard-fails cold start if files are removed while still index-referenced. Rebuild source if the table is lost: restore index backup + scoped re-fetch of model-extra-ngram-* at snapshot 6a995be5 (keep .cache/huggingface/trees/6a995be5*.json with per-file lfs_sha256), then rm the table file (next boot rebuilds, +~10 min). MTP k=3 measured mean acceptance length ~4.0 on predictable content (197-201 tok/s), 2.3-2.9 thinking-heavy (116-122, single-stream parity at worst); k=1 mean 2.0 (122 tok/s); k=5+ impossible - QSA indexer compress-ratio hard-caps 3 drafts + 1 bonus. With k=3 the padded decode batch is seqs x 4, so cudagraph_capture_sizes covers 4-multiples when raising concurrency; 2-stream aggregate 154-212. Boot gates (docker logs): 'Resolved architecture: Qwen4ExpMTP', 'PLE table mapped from /vllm-ngram/ngram_table.bin (95.37 GiB)' + 'PLE page prefetch on' (needs cap_add SYS_PTRACE for process_madvise), Marlin WNA16 MoE kernels, 'GPU KV cache size >= 1.8M tokens (7+ requests of 262144)', 'Application startup complete'. fp8 KV is NOT used: the minamism base supports it (QSA patch), but the cache runs full BF16 - the unquantized cache still fits (~1.8M tokens) and fp8 only adds accuracy risk on a rig with this much VRAM. Do not remove PYTORCH_CUDA_ALLOC_CONF / VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS / ulimit memlock / --mamba-cache-mode align: long prompts fail or the server does not start. 'Setting attention block size to 1568' in the log = image not patched correctly.","router":"llama-swap is the ROUTER, not the engine - it launches the docker wrapper per model and proxies; engine = vLLM. config.yaml (llama-swap): healthCheckTimeout: 2000 top-level (default 900 s is exceeded by first boots: PLE refill + shard weight load; per-model key is ignored in recent versions); models.Qwen3.8-Flash-Next: cmd /config/flashnext-2x64.sh ${PORT}, cmdStop docker stop flashnext-2x64, proxy http://${vllm_proxy_host}:${PORT}, ttl: 0, metadata.context_length 262144. Load by sending any request with model=Qwen3.8-Flash-Next through llama-swap.","llama_swap_router_only":true},"kv_pool_tokens":1800000,"max_context_len":262144,"storage_requirements":{"ram_gb":64}},"models":[{"slug":"qwen3-8-flash-next-int4-mixed-autoround","name":"Qwen3.8-Flash-Next-INT4-Mixed-AutoRound","hf_repo":"Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound","quant":"int4-int6-int8-compressed-tensors-gs32"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["bf16-kv","llama-swap","llm-pipeline-parallel","mtp3","page-reference","ple-nvme","pp2","speculative-decoding","unraid"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"points\": [{\"notes\": \"single-stream MTP-3 (k=3), warm, measured THROUGH llama-swap; author reports 197-201 tok/s on predictable content (code/prose/low-entropy; 'Avg generation throughput' windows) and 116-122 on thinking-heavy (technical/reasoning) content - midpoint 199 shown for the predictable band, 119 for thinking-heavy in the second row; mean acceptance length 4.0 (acceptance 0.45-1.0, content-dependent); k=1 reference 122 tok/s (mean acceptance 2.0); stated as of 2026-10-07\", \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 199}, {\"notes\": \"2-stream aggregate 154-212 tok/s (varies more on hard content), warm, through llama-swap; midpoint 183 shown; stated as of 2026-10-07\", \"concurrency\": 2, \"context_len\": 262144, \"decode_tok_s\": 183}]}"}],"perf_points":[{"id":1227,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":199,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream MTP-3 (k=3), warm, measured THROUGH llama-swap; author reports 197-201 tok/s on predictable content (code/prose/low-entropy; 'Avg generation throughput' windows) and 116-122 on thinking-heavy (technical/reasoning) content - midpoint 199 shown for the predictable band, 119 for thinking-heavy in the second row; mean acceptance length 4.0 (acceptance 0.45-1.0, content-dependent); k=1 reference 122 tok/s (mean acceptance 2.0); stated as of 2026-10-07"},{"id":1228,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":183,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"2-stream aggregate 154-212 tok/s (varies more on hard content), warm, through llama-swap; midpoint 183 shown; stated as of 2026-10-07"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-int4-mixed-170hx-x2-pp2-mtp3-bf16-kv-ple-nvme.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-int4-mixed-170hx-x2-pp2-mtp3-bf16-kv-ple-nvme.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound","pin":"branch 4x3090 @ snapshot 6a995be5542081bb65d9b7e8f580f0b8220ce0e7 (= repo HEAD at access 2026-10-07); 17 body shards + 33 model-extra-ngram PLE shards + MTP-INT4 overlay merged; body ~65-67 GiB; GATED Qwen Community License 1.0"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/vllm/vllm-openai","pin":"docker.io/vllm/vllm-openai @ sha256:43f13b4c624ab9e9e6753d0eeb5953268bff334f2a826239e6f4a2197d47bb96 (tag 0.29.1rc1.dev47, hub-verified 2026-10-07); served image = local rebuild qwen38-flash-next:2x64"},{"kind":"other","pointer_type":"hf_model","pointer":"https://huggingface.co/Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound/tree/main/vllm-patch","pin":"vllm-patch tree @ branch main f6f90ddc39e9d212aa5b96f43999183deb4ce192: 16 files = 12x vllm-patch/2x24GB/ + mtp-01..03 + make_mtp_int4.py (vllm-patch/3x24GB+MTP/); upstream-01 = vLLM PR #55506 (Karl0007)"},{"kind":"reference","pointer_type":"url","pointer":"https://www.bozocentral.com/2x-cmp170hx-qwen3-flash-next","pin":"article/writeup by Plean (bozocentral, /dev/plean), stated as of 2026-10-07; NOT a git repo - provenance = URL + accessed date + cited upstreams; wrapper/llama-swap config/PLE+MTP procedures transcribed from this page"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-int4-mixed-170hx-x2-pp2-mtp3-bf16-kv-ple-nvme.json","alternate":"/r/qwen3-8-flash-next-int4-mixed-170hx-x2-pp2-mtp3-bf16-kv-ple-nvme","describedby":"/openapi.json"},"updated_at":"2026-10-20 16:56:40.814942-06","created_at":"2026-10-07 10:56:02.764472-06","speeds":{"decode_tok_s":199,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}