{"slug":"mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t","title":"MiMo-V2.6-Flash-RL (omni) on 4x CMP 170HX \u2014 PP4, official fp8, three-tier KV, v3.3 (kaka86mm)","summary":"Four-card Ampere recipe for XiaomiMiMo MiMo-V2.6-Flash-RL (309B-MoE/15B active, omni: text/image/video/audio in, 1M ctx) on 4x CMP 170HX (64 GB unlocked, Gen2 x4, no P2P \u2014 PP-only). Official 161 GB fp8 checkpoint on wtdcode/vllm-backport sm80: PP4 partition 12,12,12,12, fp8_e4m3 KV (15.5 GiB, 2,061,355-token pool), fp8 o_proj, Marlin W8A16 for fp8 GEMMs, spec decoding OFF (MTP verify kernel cliff), three-tier KV stack (GPU + 100 GB RAM tier via LMCache PR#4410 engine-driven transfers + disk tier). Eight patches baked into image v3.3: exact QKV sharding (0 changed elements audited), MTP loader, fp8\u2192Marlin, ViT sink fix #58235, DiffKV split-KV + spec-3D verify + prefill q-tiling (3.0x) + QK-dot split #59673. Measured by the author (2026-10-04, v3.3): decode 64.5 short / 60 @19K / 56 @67K tok/s, aggregate 483 @16 / 752 @32 streams, 200K cold prefill ~3,650 tok/s (16x via q-tiling). NVFP4 fallback kept (92\u201394 tok/s). All four modalities + tools verified; thinking-off with mm input.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"036b9f389cc3965f671623dda781efcd7aae63bc","content_hash":"ac7293fc06185b9b9e23401a6d805f3d10a5cfe0","created_at":"2026-10-20 15:34:07.62236-06","notes":"Community ingest of kaka86mm's recipe repo @ f9f8d73 (repo owner-named by Casey Oct 6 as part of the 170HX focus wave; MIT LICENSE in repo). Launch argv + runtime env transcribed verbatim from scripts/launch-official-v33.sh @ f9f8d73 (v3.3 production; the v3.0 form launch-official.sh adds patch bind-mounts). Measured numbers are the author's README results tables (2026-09/10 measurement campaign incl. the 2026-10-04 re-measure), logged claimed_by=submitter T0 (no run file). Model is omni-modal (text/image/video/audio in, tool calls, reasoning parser) per the repo README + docs/RESULT.md verification records; the official checkpoint ships a hidden 2048-token output cap the download script rewrites to 65536. Built-on credits per the repo's own Credits section: wtdcode/vllm-backport (sm80 fork), avtc PR#106 (official-checkpoint sharding/MTP/Marlin fixes), upstream vLLM #58235 (ViT sink fix), LMCache PR#4410 (CPU/disk KV tiers), vllm#59673 (QK-dot split ported in v3.3)."},"engine_source":{"type":"fork","git_url":"https://github.com/wtdcode/vllm-backport","base_engine":"vllm","pip_package":null,"base_version":"vllm-backport sm80 (v0.13.0 lineage), kaka86mm mimo-v3-stack","image_digest":"docker.io/lazymio/vllm-backport:v0.13.0-sm80","install_note":"Production image vllm-mimo:v3.3 is LOCALLY BUILT from the recipe repo's docker/Dockerfile: FROM vllm-mimo:lmc-pr4410c (lazymio/vllm-backport:v0.13.0-sm80 + LMCache PR#4410 head build) COPY-ing the repo's eight patches/ files into the vLLM tree (docker build -f docker/Dockerfile -t vllm-mimo:v3.3 .). v3.0 form instead bind-mounts patches/ over lazymio/vllm-backport:v0.13.0-sm80 (scripts/launch-official.sh). The same eight patches are clean commits on kaka86mm/vllm-backport branch mimo-v3-stack @ 54f7737 (three-way-merged onto wtdcode master 29e66dad). The image ENTRYPOINT is already [\"vllm\",\"serve\"] \u2014 container args must NOT repeat vllm serve. Host needs driver \u2265 570 (author ran 610.57.04 with CMPUnlocker 64 GB unlock), NVIDIA container runtime, ~350 GB disk, \u226564 GB RAM (100 GB for the CPU KV tier).","pinned_commit":"54f7737ad8f3a35f3be235837b4932d8cf004402"},"credit":[{"name":"kaka86mm","url":"https://github.com/kaka86mm/vllm-mimo-170hx"},{"name":"wtdcode (vllm-backport)","url":"https://github.com/wtdcode/vllm-backport"},{"name":"avtc (vllm-backport PR#106)","url":"https://github.com/wtdcode/vllm-backport/pull/106"},{"name":"LMCache PR#4410","url":"https://github.com/LMCache/LMCache/pull/4410"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["docker","run","-d","--name","mimo26","--restart","unless-stopped","--runtime=nvidia","-e","NVIDIA_VISIBLE_DEVICES=0,1,2,3","-e","HF_HUB_OFFLINE=1","-e","VLLM_DIFFKV_FULL_ATTN_SEGMENTS=64","-e","VLLM_WORKER_MULTIPROC_METHOD=spawn","-e","VLLM_DIFFKV_PREFILL_TILE=64","-e","VLLM_MIMO_OPROJ_FP8=1","-e","LMCACHE_MP_TRANSFER_MODE=engine_driven","-e","VLLM_PP_LAYER_PARTITION=12,12,12,12","-v","$MODEL_DIR:/model:ro","--network","host","--ipc","host","vllm-mimo:v3.3","/model","--served-model-name","mimo26","--port","8099","--pipeline-parallel-size","4","--kv-cache-dtype","fp8_e4m3","--block-size","128","--max-model-len","1048576","--max-num-batched-tokens","1024","--trust-remote-code","--kv-cache-memory","17179869184","--gpu-memory-utilization","0.94","--max-num-seqs","32","--reasoning-parser","mimo","--tool-call-parser","mimo","--enable-auto-tool-choice","--enable-prefix-caching","--prefix-cache-retention-interval","1024","--async-scheduling","--no-enable-flashinfer-autotune","--media-io-kwargs","{\"video\": {\"num_frames\": 128}}","--kv-transfer-config","{\"kv_connector\":\"LMCacheMPConnector\",\"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.host\":\"tcp://127.0.0.1\",\"lmcache.mp.port\":5555,\"lmcache.mp.mp_transfer_mode\":\"engine_driven\",\"kv_buffer_size\":268435456}}"],"runtime":{"env":"VLLM_PP_LAYER_PARTITION=12,12,12,12 | VLLM_DIFFKV_FULL_ATTN_SEGMENTS=64 | VLLM_DIFFKV_PREFILL_TILE=64 | VLLM_MIMO_OPROJ_FP8=1 | LMCACHE_MP_TRANSFER_MODE=engine_driven | HF_HUB_OFFLINE=1 | VLLM_WORKER_MULTIPROC_METHOD=spawn (verbatim from scripts/launch-official-v33.sh @ f9f8d73). CPU tier runs separately: docker run --name lmcache-server ... lmc-server-cmd.sh mounted as entrypoint \u2192 'lmcache server --host 0.0.0.0 --port 5555 --chunk-size 8192 --shm-name mimo26shm --separate-object-groups --max-gpu-workers 4 --supported-transfer-mode engine_driven --l1-size-gb 100 --l1-init-size-gb 96 --no-l1-use-lazy --eviction-policy LRU --l2-adapter {type:fs,base_path:/lmcache-l2,relative_tmp_dir:tmp}'.","notes":"docker/Dockerfile bakes the eight patches (patches/mimo_v2.py exact-QKV + OPROJ_FP8, mimo_v2_mtp.py MTP drafter loader, fp8.py + online_fp8.py Marlin W8A16 routing for sm80, mimo_v2_omni_model.py + mimo_v2_omni.py ViT sink fix #58235 + processor, triton_attn_diffkv.py split-KV, triton_unified_attention_diffkv.py spec-3D + prefill q-tiling + QK-dot split #59673) into the image layer, md5-identical to the v3.0 bind-mounted files. Weights: XiaomiMiMo/MiMo-V2.6-Flash-RL official fp8 161 GB (hf download + fix the hidden 2048-token generation_config cap \u2192 65536). KV sizing: 16 GiB with the 12,12,12,12 split (rank 2 allocator warnings self-recover; 17 GiB unreachable). Speculative decoding deliberately OFF \u2014 the MTP verify pass hits a kernel cliff: 67K decode 56 (no spec) vs 16.5 (k=2); the kernel fix exists (patches + VLLM_DIFFKV_FULL_ATTN_SEGMENTS=64) taking it to 33.7 but still losing to no-spec. Prefix caching on (~33K tok/s cache-hit re-prefills). Multimodal requests should carry chat_template_kwargs enable_thinking=false (reasoning parser intermittently swallows content with mm input under thinking); audio input uses audio_url + wav (not OpenAI input_audio). Ops: always docker restart -t 60 mimo26 (a bare restart's 10 s timeout is a forced kill \u2192 stale SHM state loops the engine; after any forced kill remove /dev/shm/mimo26* segments AND recreate lmcache-server; check nvidia-smi for orphaned VLLM::Worker processes holding ~65 GB/GPU). LMCache fs-L2 known bug LMCache#5475: with PP>1 one rank's half-size group objects collide on shared L2 keys \u2014 L2 does not survive a server restart (RAM tier unaffected); no L2 size cap, watch du. The NVFP4 fallback (scripts/launch-omni.sh, ProCreations/MiMo-V2.6-Flash-RL-NVFP4 @ 34cebc5, arch switched to MiMoV2OmniForCausalLM, partition 11,13,12,12, kv-cache-memory 10 GiB, MTP k=2 probabilistic) gives 92\u201394 tok/s decode / 1,329,835-token KV pool with the same prefill \u2014 frozen at tag v1.3-prod. Power: cap cards \u2264230 W (250 W transients \u2192 Xid 43)."},"kv_pool_tokens":2061355,"max_context_len":1048576,"storage_requirements":{"ram_gb":64}},"models":[{"slug":"mimo-v2-6-flash-rl","name":"MiMo-V2.6-Flash-RL","hf_repo":"XiaomiMiMo/MiMo-V2.6-Flash-RL","quant":"official-fp8"}],"hardware":[{"slug":"nvidia-cmp-170hxx4","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":4}],"generation":"ampere","tags":["lmcache","long-context","multimodal","pipeline-parallel","three-tier-kv"],"tasks":["chat","code","stt","vision"],"aa":{"aa_index":38,"aa_provenance":"published","aa_source_url":"https://artificialanalysis.ai/models/mimo-v2-6-flash","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:15:00-06","aa_family_slug":"mimo-v2-6-flash-rl","aa_resolved_slug":"mimo-v2-6-flash"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured on the 4x170HX box, production config v3.3 (README results tables @ repo f9f8d73, 2026-10-04 re-measure): single-stream decode (greedy, ~0 ctx) 64.5 tok/s without spec (MTP k=2 gives 106 but only at trivial contexts); 60 @19K / 56 @67K ctx; aggregate 483 tok/s @16 streams / 752 @32 (short ctx, post 12,12,12,12 rebalance + 250 W cap); prefill 20K prompt 3.3 s = 5.9\u20136.2K tok/s; 200K unique prompt 56.8\u201358.4 s = 3.6\u20133.7K tok/s (3472 on v3.1 \u2192 ~3670 on v3.3, +5.7% from the QK split); KV pool 2,061,355 tokens; TTFT 0.1\u20130.5 s short prompt. claimed_by=submitter T0 \u2014 no run file ingested; import via POST /api/v1/benchmarks to upgrade.\", \"tok_s\": 64.5, \"points\": [{\"notes\": \"single-stream greedy ~0 ctx, no spec (v3.3, 2026-10-04)\", \"concurrency\": 1, \"context_len\": 1, \"decode_tok_s\": 64.5}, {\"notes\": \"single-stream decode @19K ctx\", \"concurrency\": 1, \"context_len\": 19000, \"decode_tok_s\": 60}, {\"notes\": \"single-stream decode @67K ctx, no spec\", \"concurrency\": 1, \"context_len\": 67000, \"decode_tok_s\": 56}, {\"notes\": \"aggregate short-ctx 16 streams (2026-10-04 re-measure, post rebalance + power cap)\", \"concurrency\": 16, \"context_len\": 1, \"decode_tok_s\": 483}, {\"notes\": \"aggregate short-ctx 32 streams (2026-10-04 re-measure)\", \"concurrency\": 32, \"context_len\": 1, \"decode_tok_s\": 752}, {\"notes\": \"20K-token cold unique prompt prefill 3.3 s = 5.9\u20136.2K tok/s\", \"ttft_ms\": 3300, \"concurrency\": 1, \"context_len\": 20000, \"prefill_tok_s\": 6050}, {\"notes\": \"200K-token cold unique prefill 56.8\u201358.4 s = 3.6\u20133.7K tok/s (q-tiling + QK-split v3.3); stock kernel ~170 tok/s\", \"concurrency\": 1, \"context_len\": 200000, \"prefill_tok_s\": 3650}], \"context_len\": 67000}"}],"perf_points":[{"id":1019,"context_len":1,"prefill_tok_s":null,"decode_tok_s":64.5,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream greedy ~0 ctx, no spec (v3.3, 2026-10-04)"},{"id":1022,"context_len":1,"prefill_tok_s":null,"decode_tok_s":483,"ttft_ms":null,"concurrency":16,"claimed_by":"submitter","notes":"aggregate short-ctx 16 streams (2026-10-04 re-measure, post rebalance + power cap)"},{"id":1023,"context_len":1,"prefill_tok_s":null,"decode_tok_s":752,"ttft_ms":null,"concurrency":32,"claimed_by":"submitter","notes":"aggregate short-ctx 32 streams (2026-10-04 re-measure)"},{"id":1020,"context_len":19000,"prefill_tok_s":null,"decode_tok_s":60,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode @19K ctx"},{"id":1024,"context_len":20000,"prefill_tok_s":6050,"decode_tok_s":null,"ttft_ms":3300,"concurrency":1,"claimed_by":"submitter","notes":"20K-token cold unique prompt prefill 3.3 s = 5.9\u20136.2K tok/s"},{"id":1021,"context_len":67000,"prefill_tok_s":null,"decode_tok_s":56,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"single-stream decode @67K ctx, no spec"},{"id":1025,"context_len":200000,"prefill_tok_s":3650,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"200K-token cold unique prefill 56.8\u201358.4 s = 3.6\u20133.7K tok/s (q-tiling + QK-split v3.3); stock kernel ~170 tok/s"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","pin":"5711b268169967567844e1e560e8a3966da959b1 \u2014 official fp8 ckpt (161 GB); NVFP4 fallback: ProCreations/MiMo-V2.6-Flash-RL-NVFP4 @ 34cebc5d75743a93ab76145ec609cfef9da8c417 (arch switched to MiMoV2OmniForCausalLM)"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/kaka86mm/vllm-backport/tree/mimo-v3-stack","pin":"54f7737ad8f3a35f3be235837b4932d8cf004402 \u2014 the eight patches as clean commits on wtdcode master 29e66dad (base fork wtdcode/vllm-backport); image vllm-mimo:v3.3 built from the repo docker/Dockerfile over lazymio/vllm-backport:v0.13.0-sm80 + LMCache PR#4410"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/kaka86mm/vllm-mimo-170hx","pin":"f9f8d73087c8828128975cbf8db7b32aa1398aa1 \u2014 source recipe repo @ HEAD (scripts/launch-official-v33.sh + scripts/launch-official.sh are the verbatim argv/env sources; docker/Dockerfile, patches/, scripts/bench-full.py, docs/RESULT.md all at this commit)"},{"kind":"other","pointer_type":"url","pointer":"https://github.com/LMCache/LMCache/pull/4410","pin":"head build \u2014 hybrid-model multi-group engine-driven transfers; the CPU/disk KV tiers run stock PR#4410 lmcache server code (chunk 8192, SHM transport, fs L2 with relative_tmp_dir); field data in LMCache#5440 and #5475"}],"links":{"self":"/api/v1/recipes/mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t.json","alternate":"/r/mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:07.62236-06","created_at":"2026-10-06 17:02:43.626846-06","speeds":{"decode_tok_s":60,"prefill_tok_s":3650,"leaderboard":"/api/v1/leaderboard.json"}}