{"slug":"qwen3-8-flash-next-exl3-5-05bpw-2x-cmp-170hx-tp2-mtp3-morrowmake","title":"Qwen3.8-Flash-Next EXL3 5.05bpw 2x CMP 170HX TP2 MTP3 (Morrowmake 1.0.0)","summary":"First EXL3 row for Qwen3.8-Flash-Next and first Morrowmake QFN lane: turboderp/Qwen3.8-Flash-Next-exl3 (EXL3 5.05 bpw) on two CMP 170HX cards (TP2, whole routed experts per card, PCIe x16 links) with the Morrowmake sm_80 vLLM fork @ 18305948ac, container pinned by digest. Native MTP speculative decoding k=3, full 262,144-token context, BF16 KV cache, n-gram (PLE) table read from the checkpoint SSD during decode, vision + reasoning + tool calls, every weight on-GPU except the PLE table. One-command repro (./start.sh or docker compose up); checkpoint pinned to HF revision 7cef615f. Measured 2026-10-15: 152.8 tok/s single-user, 674.7 tok/s at 8 users (code), 2,651 tok/s cold prefill, 1,044,325-token KV pool; HumanEval 157/164, GSM8K 1,289/1,319.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"12a77349c7bae62691c7aca5b501917f0c67ebce","content_hash":"8bf17a2115f0ebb4099cb3d29872c9dace0521cd","created_at":"2026-10-20 21:52:36.671416-06","notes":"Recipe by Morrowmake, release v1.0.0 (2026-10-16), repo pinned at 207867e80430701f85f5e99157b03f705ff79ef0 = tag qwen38-recipe-1.0.0; ingested by localai-recipes from the Morrowmake engine-repo watch. Benchmarks upstream-measured 2026-10-15 (MiaAI-Lab protocol; 1 user = median of 10 runs over two boots, 2/4/8 users = median aggregate of 3 runs on one boot; prompt reading = median of 9 cold prefills). Serving defaults are locked by serve.sh (MAX_LEN=262144, GPU_UTIL=0.95, LAYOUT=tp2; FIRST_LIGHT=1 is a 32K/eager/no-MTP diagnostic, not serving). PLE placement ssd is the engine default; ram/device are comparison placements. EXL3 sm_80 kernel switches ship on by default with six one-variable kill switches (off: +18% single-user decode, +43-56% at 2-8 users). P2P=auto verifies peer content at every start. Recipe MIT; engine Apache-2.0; checkpoint under its own model-card licence."},"engine_source":{"type":"fork","git_url":"https://github.com/Morrowmake/qwen38-flash-next-cmp170hx-recipe","base_engine":"vllm","pip_package":null,"base_version":"18305948ac (Morrowmake/vllm-cmp170hx)","image_digest":"ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79","install_note":"Engine is Morrowmake/vllm-cmp170hx @ 18305948acba3f4c4d912cdd56f2202d2ebd94e8 (sm_80 Ampere EXL3 extension on vLLM). Runs only in the digest-pinned container ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79 (recipe release 1.0.0); not a pip install. One command from a fresh clone: ./start.sh (or docker compose up) - preflight, engine install, checkpoint download at the pinned revision, launch, /health wait. First start compiles cold; later starts reuse ./cache. Measured on PCIe x16 capacitor-mod links; x4 links not measured (start.sh reports link width and warns).","pinned_commit":"207867e80430701f85f5e99157b03f705ff79ef0"},"credit":[{"name":"Morrowmake","url":"https://github.com/Morrowmake"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/models/Qwen3.8-Flash-Next-exl3","--served-model-name","qwen3.8-flash-next","--host","0.0.0.0","--tensor-parallel-size","2","--pipeline-parallel-size","1","--enable-expert-parallel","--dtype","bfloat16","--kv-cache-dtype","bfloat16","--load-format","exl3","--mm-encoder-tp-mode","data","--attention-config","{\"indexer_kv_dtype\":\"bf16\"}","--engram-config","{\"cpu_offload\":false}","--max-model-len","262144","--gpu-memory-utilization","0.95","--max-num-seqs","8","--max-num-batched-tokens","3460","--long-prefill-token-threshold","0","--reasoning-parser","qwen3","--enable-auto-tool-choice","--tool-call-parser","qwen3_coder","--port","8000","--prefix-match-unit","64","--compilation-config","{\"cudagraph_mode\":\"PIECEWISE\"}","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":3,\"disable_eagle_block_drop\":true}"],"layouts":["tp"],"kv_pool_tokens":1044325,"max_context_len":262144,"storage_requirements":{"note":"~115 GiB checkpoint on an SSD; the 37 GiB n-gram (PLE) table is read from the SSD during decode","ssd_gb":120}},"models":[{"slug":"qwen3-8-flash-next-exl3-5-05bpw","name":"Qwen3.8-Flash-Next-EXL3-5.05bpw","hf_repo":"Qwen/Qwen3.8-Flash-Next-EXL3-5.05bpw","quant":"exl3-5-05bpw"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["bf16-kv","exl3","long-context","mtp3","mtp-speculative","ple-ssd","sm80","tensor-parallel","vision"],"tasks":["chat","code","vision"],"aa":{"aa_index":42,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/glm-5-3-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-04 14:15:00-06","aa_family_slug":"glm-5-3-flash","aa_resolved_slug":"glm-5-3-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"T0 submitter-reported (recipe release notes / README, measured by Morrowmake 2026-10-15 on 2x CMP 170HX, 180 W/card, P2P on, PCIe x16, 262,144 ctx, temperature 0, thinking off, 400-token generations): single-user structured 152.8 / code 152.6 / prose 118.3 tok/s; 2-user totals 236.5 / 265.0 / 204.6; 4-user totals 357.1 / 458.6 / 327.2; 8-user totals 588.2 / 674.7 / 489.9 (structured totals lower because replies finish early: median 346/273/279 tokens vs 400). Cold prefill 2,651 tok/s (median of 9). KV pool 1,044,325 tokens. Quality: HumanEval 157/164, GSM8K 1,289/1,319; 262,144-window boundary tests pass; needle retrieval 9/9 at 8K/32K/128K; vision 31/31.\", \"tok_s\": 152.80000000000001136868377216160297393798828125, \"points\": [{\"notes\": \"1 user, structured prompts; code 152.6, prose 118.3\", \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 152.80000000000001136868377216160297393798828125}, {\"notes\": \"cold prefill, single stream, median of 9\", \"context_len\": 262144, \"prefill_tok_s\": 2651}, {\"notes\": \"2-user aggregate, code prompts; structured 236.5, prose 204.6\", \"concurrency\": 2, \"context_len\": 262144, \"decode_tok_s\": 265}, {\"notes\": \"4-user aggregate, code prompts; structured 357.1, prose 327.2\", \"concurrency\": 4, \"context_len\": 262144, \"decode_tok_s\": 458.6000000000000227373675443232059478759765625}, {\"notes\": \"8-user aggregate, code prompts; structured 588.2, prose 489.9\", \"concurrency\": 8, \"context_len\": 262144, \"decode_tok_s\": 674.700000000000045474735088646411895751953125}], \"context_len\": 262144}"}],"perf_points":[{"id":1682,"context_len":262144,"prefill_tok_s":2651,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill, single stream, median of 9"},{"id":1678,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":152.80000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"1 user, structured prompts; code 152.6, prose 118.3"},{"id":1679,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":265,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"2-user aggregate, code prompts; structured 236.5, prose 204.6"},{"id":1680,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":458.6000000000000227373675443232059478759765625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"4-user aggregate, code prompts; structured 357.1, prose 327.2"},{"id":1681,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":674.700000000000045474735088646411895751953125,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"8-user aggregate, code prompts; structured 588.2, prose 489.9"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-exl3-5-05bpw-2x-cmp-170hx-tp2-mtp3-morrowmake.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-exl3-5-05bpw-2x-cmp-170hx-tp2-mtp3-morrowmake.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3","pin":"EXL3 5.05bpw branch @ 7cef615f7fd3681295b68848018876dfabc336c7 (~115 GiB incl. 37 GiB n-gram PLE table; MTP drafter inside)"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://github.com/Morrowmake/vllm-cmp170hx","pin":"ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/Morrowmake/qwen38-flash-next-cmp170hx-recipe","pin":"release 1.0.0 (commit 207867e80430701f85f5e99157b03f705ff79ef0)"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-exl3-5-05bpw-2x-cmp-170hx-tp2-mtp3-morrowmake.json","alternate":"/r/qwen3-8-flash-next-exl3-5-05bpw-2x-cmp-170hx-tp2-mtp3-morrowmake","describedby":"/openapi.json"},"updated_at":"2026-10-20 21:52:36.671416-06","created_at":"2026-10-20 21:49:17.344329-06","speeds":{"decode_tok_s":152.80000000000001136868377216160297393798828125,"prefill_tok_s":2651,"leaderboard":"/api/v1/leaderboard.json"}}