MiMo-V2.6-Flash-RL (omni) on 4x CMP 170HX — PP4, official fp8, three-tier KV, v3.3 (kaka86mm)

docker run -d --name mimo26 --restart unless-stopped --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=0,1,2,3 -e HF_HUB_OFFLINE=1 -e VLLM_DIFFKV_FULL_ATTN_SEGMENTS=64 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e VLLM_DIFFKV_PREFILL_TILE=64 -e VLLM_MIMO_OPROJ_FP8=1 -e LMCACHE_MP_TRANSFER_MODE=engine_driven -e VLLM_PP_LAYER_PARTITION=12,12,12,12 -v '$MODEL_DIR:/model:ro' --network host --ipc host vllm-mimo:v3.3 /model --served-model-name mimo26 --port 8099 --pipeline-parallel-size 4 --kv-cache-dtype fp8_e4m3 --block-size 128 --max-model-len 1048576 --max-num-batched-tokens 1024 --trust-remote-code --kv-cache-memory 17179869184 --gpu-memory-utilization 0.94 --max-num-seqs 32 --reasoning-parser mimo --tool-call-parser mimo --enable-auto-tool-choice --enable-prefix-caching --prefix-cache-retention-interval 1024 --async-scheduling --no-enable-flashinfer-autotune --media-io-kwargs '{"video": {"num_frames": 128}}' --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://127.0.0.1","lmcache.mp.port":5555,"lmcache.mp.mp_transfer_mode":"engine_driven","kv_buffer_size":268435456}}'

env: "VLLM_PP_LAYER_PARTITION=12,12,12,12 | VLLM_DIFFKV_FULL_ATTN_SEGMENTS=64 | VLLM_DIFFKV_PREFILL_TILE=64 | VLLM_MIMO_OPROJ_FP8=1 | LMCACHE_MP_TRANSFER_MODE=engine_driven | HF_HUB_OFFLINE=1 | VLLM_WORKER_MULTIPROC_METHOD=spawn (verbatim from scripts/launch-official-v33.sh @ f9f8d73). CPU tier runs separately: docker run --name lmcache-server ... lmc-server-cmd.sh mounted as entrypoint → 'lmcache server --host 0.0.0.0 --port 5555 --chunk-size 8192 --shm-name mimo26shm --separate-object-groups --max-gpu-workers 4 --supported-transfer-mode engine_driven --l1-size-gb 100 --l1-init-size-gb 96 --no-l1-use-lazy --eviction-policy LRU --l2-adapter {type:fs,base_path:/lmcache-l2,relative_tmp_dir:tmp}'."
notes: "docker/Dockerfile bakes the eight patches (patches/mimo_v2.py exact-QKV + OPROJ_FP8, mimo_v2_mtp.py MTP drafter loader, fp8.py + online_fp8.py Marlin W8A16 routing for sm80, mimo_v2_omni_model.py + mimo_v2_omni.py ViT sink fix #58235 + processor, triton_attn_diffkv.py split-KV, triton_unified_attention_diffkv.py spec-3D + prefill q-tiling + QK-dot split #59673) into the image layer, md5-identical to the v3.0 bind-mounted files. Weights: XiaomiMiMo/MiMo-V2.6-Flash-RL official fp8 161 GB (hf download + fix the hidden 2048-token generation_config cap → 65536). KV sizing: 16 GiB with the 12,12,12,12 split (rank 2 allocator warnings self-recover; 17 GiB unreachable). Speculative decoding deliberately OFF — the MTP verify pass hits a kernel cliff: 67K decode 56 (no spec) vs 16.5 (k=2); the kernel fix exists (patches + VLLM_DIFFKV_FULL_ATTN_SEGMENTS=64) taking it to 33.7 but still losing to no-spec. Prefix caching on (~33K tok/s cache-hit re-prefills). Multimodal requests should carry chat_template_kwargs enable_thinking=false (reasoning parser intermittently swallows content with mm input under thinking); audio input uses audio_url + wav (not OpenAI input_audio). Ops: always docker restart -t 60 mimo26 (a bare restart's 10 s timeout is a forced kill → stale SHM state loops the engine; after any forced kill remove /dev/shm/mimo26* segments AND recreate lmcache-server; check nvidia-smi for orphaned VLLM::Worker processes holding ~65 GB/GPU). LMCache fs-L2 known bug LMCache#5475: with PP>1 one rank's half-size group objects collide on shared L2 keys — L2 does not survive a server restart (RAM tier unaffected); no L2 size cap, watch du. The NVFP4 fallback (scripts/launch-omni.sh, ProCreations/MiMo-V2.6-Flash-RL-NVFP4 @ 34cebc5, arch switched to MiMoV2OmniForCausalLM, partition 11,13,12,12, kv-cache-memory 10 GiB, MTP k=2 probabilistic) gives 92–94 tok/s decode / 1,329,835-token KV pool with the same prefill — frozen at tag v1.3-prod. Power: cap cards ≤230 W (250 W transients → Xid 43)."

Fetch https://localairecipes.com/api/v1/recipes/mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/mimo-v2-6-flash-rl-omni-on-4x-cmp-170hx-pp4-official-fp8-three-t.json