{"slug":"mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx","title":"MiMo-V2.6-Flash-RL 309B MoE NVFP4 4x CMP 170HX TP4 DFlash 1M ctx (ElectroHobo)","summary":"XiaomiMiMo's MiMo-V2.6-Flash-RL (309B MoE, 15B active, text/image/video/audio input) on four CMP 170HX cards (sm_80) at the full 1,048,576-token context: 322/275/170 tok/s one user (structured/code/prose), 754/738/556 across eight, 2,720 tok/s cold prefill, 1,615,953-token KV pool. NVFP4 weights (ModelOpt mixed precision via Marlin, bf16 activations), bf16 KV, MiMo's own DFlash drafter (7 drafts one user, 7/5/3 by request count), full-decode-step CUDA graph, one engine file bind-mounted (upstream vLLM #58235 vision-sink fix), every weight on the GPUs. Morrowmake's first listed community recipe on vllm-cmp170hx; measured 2026-10-18 under the Morrowmake bench protocol.","license":"MIT","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"12a7bfc3d0f95506c73c0dbaf6bb13286ef57281","content_hash":"6388d1808174e278654541bfad590d93a9202607","created_at":"2026-10-20 15:34:18.178415-06","notes":"Community ingest of ElectroHobo's recipe (MIT, (c) 2026 Applied Integrators (ElectroHobo) + Morrowmake for the start.sh/serve.sh script set), listed by Morrowmake 2026-10-19 as the fork's first community recipe; the recipe pins the exact fork commit + image digest per Morrowmake's community-recipe rules and states Morrowmake has not tested/endorsed it. All perf numbers are the author's own measurements (claimed_by=submitter T0, bench harness in-repo) - import via POST /api/v1/benchmarks to upgrade. Measured on x16-capacitor-modded cards behind one PCIe switch with P2P; x4-link performance NOT measured (start.sh warns on narrower links). Sibling recipe mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t stays (different quant+drafter+layout)."},"engine_source":{"type":"docker_image","git_url":"https://github.com/Morrowmake/vllm-cmp170hx","base_engine":"vllm","pip_package":null,"base_version":"fork @ 18305948acba (tag qwen38-recipe-1.0.0)","image_digest":"ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79","install_note":"One command from a fresh clone after one-time host setup: git clone https://github.com/ElectroHobo/mimo-v26-flash-cmp170hx-recipe && cd mimo-v26-flash-cmp170hx-recipe && ./start.sh (preflight GPU/link/disk/port checks -> engine image pull -> checkpoint download at pinned HF revision -> detached launch -> /health wait -> two-request boot check). docker compose up is the equivalent. First start compiles kernels (later starts reuse ./cache). One engine file bind-mounted: patches/mimo_v2_omni.py with upstream vLLM #58235's vision fix (VISION_PATCH=0 disables; text unchanged, images misread). Host: 4x CMP 170HX 64 GiB on PCIe x16 links (x16-capacitor-mod cards) behind one PLX switch, P2P on (P2P=auto content-checks each start), driver >=580 (measured 610.57.04), ~190 GiB disk.","pinned_commit":"18305948acba3f4c4d912cdd56f2202d2ebd94e8"},"credit":[{"name":"ElectroHobo (Applied Integrators)","url":"https://github.com/ElectroHobo/mimo-v26-flash-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx fork + engine image + start.sh script set, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"XiaomiMiMo (MiMo-V2.6-Flash-RL base model)","url":"https://huggingface.co/XiaomiMiMo"},{"name":"ProCreations (NVFP4 checkpoint)","url":"https://huggingface.co/ProCreations/MiMo-V2.6-Flash-RL-NVFP4"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["git clone https://github.com/ElectroHobo/mimo-v26-flash-cmp170hx-recipe && cd mimo-v26-flash-cmp170hx-recipe","./start.sh"],"layouts":["tp"],"runtime":{"env":"Defaults in .env (from .env.example): CUDA_VISIBLE_DEVICES=0,1,2,3 | HOST=127.0.0.1 | PORT=8000 | SERVED_MODEL_NAME=mimo-v2.6-flash | SPEC_MODE=dflash | SPEC_N=7 (dflash) / 2 (mtp) | SPEC_SCHEDULE=7/5/3 by request count (custom ranges e.g. [[1,2,7],[3,32,4]]; none = fixed SPEC_N) | SPEC_ADAPTIVE_K optional load/acceptance-adaptive depth (helps prose, costs structured) | MAX_LEN=1048576 | KV_MEM=10737418240 (10 GiB KV bytes/card ~= 1.62M tokens) | KV_DTYPE=bfloat16 (fp8_e4m3 only with mtp/none - DFlash's Triton attention has no fp8 KV on sm_80) | MAX_SEQS=32 | MAX_BATCHED=2048 (1024: -10% prefill +1.7% pool; 4096: +5% -3%) | FAIR_CHUNK=384 fair prefill slices while anyone is decoding (FAIR_PARTIAL=2) | P2P=auto (content-checked; off = ~40% slower) | EXPERT_PARALLEL=0 (measured 4-9% slower when 1) | KV_OFFLOAD_GB=0 (host-RAM KV tier; KV_OFFLOAD_DIR adds a write-through disk tier behind it) | GPU_UTIL fixed 0.94 and TP4 layout fixed (serve.sh refuses others) | FIRST_LIGHT=1 = 32K/eager/no-spec diagnostic.","notes":"API on 127.0.0.1:8000, no key by default; reasoning in message.reasoning, answer in content; thinking off per request via chat_template_kwargs.enable_thinking=false. miMoV2OmniForCausalLM selected via --hf-overrides so checkpoint files are used as downloaded (block size 128, reasoning/tool parsers mimo, prefix caching + async scheduling on). Depth-by-request schedule gave +11/+12/+39% at eight users vs fixed 7. MTP alternative is slower on every column measured (128.9/119.6/114.5 tok/s c1). Not yet measured upstream: video/audio input, sustained load, 16 users, long-context retrieval (docs/known-limits.md)."},"kv_pool_tokens":1615953,"max_context_len":1048576},"models":[{"slug":"mimo-v2-6-flash-rl","name":"MiMo-V2.6-Flash-RL","hf_repo":"XiaomiMiMo/MiMo-V2.6-Flash-RL","quant":"nvfp4"}],"hardware":[{"slug":"nvidia-cmp-170hxx4","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":4}],"generation":"ampere","tags":["community-recipe","dflash","morrowmake-fork","mtp","nvfp4","omni","speculative","tp4"],"tasks":["chat","code","vision"],"aa":{"aa_index":38,"aa_provenance":"published","aa_source_url":"https://artificialanalysis.ai/models/mimo-v2-6-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:15:00-06","aa_family_slug":"mimo-v2-6-flash-rl","aa_resolved_slug":"mimo-v2-6-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"claimed_by=submitter T0 (author's bench/cmp_recipe_bench.py, Morrowmake protocol: temp 0, thinking off, 400-token pin, per-request salt, median of 5 single-user / median of 3 aggregate; measured 2026-10-18, 74 SMs/card, P2P on, 180 W/card, PCIe x16). All decode points ran at the full 1M context; context_len on code/prose rows is the tuple marker 1048576+10240 (code) / +20480 (prose) to keep (context_len, concurrency) unique. Quality: HumanEval 161/164, GSM8K 195/200 (first 200, thinking on); 1M ctx accepted; image input correct with the #58235 patch. Decode step 15 ms c1 / 21 ms c2 / ~48 ms c8.\", \"tok_s\": 322, \"points\": [{\"notes\": \"structured, median of 5\", \"concurrency\": 1, \"context_len\": 1048576, \"decode_tok_s\": 322}, {\"notes\": \"code; context_len is the tuple marker 1048576+10240 (code row, not structured/prose)\", \"concurrency\": 1, \"context_len\": 1058816, \"decode_tok_s\": 274.8999999999999772626324556767940521240234375}, {\"notes\": \"prose; context_len is the tuple marker 1048576+20480 (prose row, not structured/code)\", \"concurrency\": 1, \"context_len\": 1069056, \"decode_tok_s\": 170.099999999999994315658113919198513031005859375}, {\"notes\": \"aggregate structured, median of 3 (c2: 427.0, c4: 572.7)\", \"concurrency\": 8, \"context_len\": 1048576, \"decode_tok_s\": 753.799999999999954525264911353588104248046875}, {\"notes\": \"aggregate code; context_len is the tuple marker 1048576+10240 (c2: 397.7, c4: 538.3)\", \"concurrency\": 8, \"context_len\": 1058816, \"decode_tok_s\": 738}, {\"notes\": \"aggregate prose; context_len is the tuple marker 1048576+20480 (c2: 255.1, c4: 412.1)\", \"concurrency\": 8, \"context_len\": 1069056, \"decode_tok_s\": 555.5}, {\"notes\": \"cold prefill median, 24K/34K/38K-token prompts, no cache hits\", \"context_len\": 1, \"prefill_tok_s\": 2722}], \"context_len\": 1048576}"}],"perf_points":[{"id":1101,"context_len":1,"prefill_tok_s":2722,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill median, 24K/34K/38K-token prompts, no cache hits"},{"id":1102,"context_len":1048576,"prefill_tok_s":null,"decode_tok_s":322,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"structured, median of 5"},{"id":1103,"context_len":1048576,"prefill_tok_s":null,"decode_tok_s":753.799999999999954525264911353588104248046875,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate structured, median of 3 (c2: 427.0, c4: 572.7)"},{"id":1104,"context_len":1058816,"prefill_tok_s":null,"decode_tok_s":274.8999999999999772626324556767940521240234375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"code; context_len is the tuple marker 1048576+10240 (code row, not structured/prose)"},{"id":1105,"context_len":1058816,"prefill_tok_s":null,"decode_tok_s":738,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate code; context_len is the tuple marker 1048576+10240 (c2: 397.7, c4: 538.3)"},{"id":1106,"context_len":1069056,"prefill_tok_s":null,"decode_tok_s":170.099999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"prose; context_len is the tuple marker 1048576+20480 (prose row, not structured/code)"},{"id":1107,"context_len":1069056,"prefill_tok_s":null,"decode_tok_s":555.5,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"aggregate prose; context_len is the tuple marker 1048576+20480 (c2: 255.1, c4: 412.1)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/ProCreations/MiMo-V2.6-Flash-RL-NVFP4","pin":"revision 34cebc5d75743a93ab76145ec609cfef9da8c417 - NVFP4 (ModelOpt mixed precision); the Hub repo's gguf/ folder is not downloaded"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://github.com/Morrowmake/vllm-cmp170hx/pkgs/container/vllm-cmp170hx","pin":"ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79 on fork 18305948acba (tag qwen38-recipe-1.0.0) + bind-mounted patches/mimo_v2_omni.py (vLLM #58235)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/ElectroHobo/mimo-v26-flash-cmp170hx-recipe","pin":"release 0.2.0 = commit c42dbc87cdfd53866d4795d414dcef41bbd9d6c1 (tag v0.2.0, HEAD at ingest) - start.sh, bench/cmp_recipe_bench.py, docs/results.md, docs/known-limits.md, patches/"}],"links":{"self":"/api/v1/recipes/mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx.json","alternate":"/r/mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:18.178415-06","created_at":"2026-10-20 08:07:43.308477-06","speeds":{"decode_tok_s":274.8999999999999772626324556767940521240234375,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}