DeepSeek-V4.1-Flash EXL3 3bpw TC-MoE on 4x CMP 170HX TP4 (original checkpoint)

'docker run -d --name dsv41-tp4 --cap-add SYS_NICE --gpus all --network host --ipc host --ulimit memlock=-1 --ulimit stack=67108864' '-v <model_dir>:/model-view:ro -v <exl3_py>:/usr/local/lib/python3.12/dist-packages/vllm_exl3/exl3.py:ro -v <sm80_dir>:/usr/local/lib/python3.12/dist-packages/vllm/sm80:ro' '-v <envs_py>:/usr/local/lib/python3.12/dist-packages/vllm/envs.py:ro -v <cuda_py>:/usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py:ro' '-v <tc_hook>:/opt/dsv41-tc-moe/exl3_tc_hook.py:ro -v <tc_so>:/opt/dsv41-tc-moe/_dsv41_tc_moe_C.abi3.so:ro' '-e VLLM_EXL3_MODEL_DIR=/model-view -e VLLM_PLUGINS=vllm_exl3 -e VLLM_USE_V2_MODEL_RUNNER=0 -e DSV41_DSPARK_V1=1' '-e VLLM_EXL3_COOP=1 -e VLLM_HOST_ALLREDUCE=1 -e NCCL_P2P_DISABLE=1 -e NCCL_IGNORE_DISABLED_P2P=1 -e NCCL_CUMEM_ENABLE=0' '-e VLLM_EXL3_TC_MOE_V2=1 -e VLLM_EXL3_TC_MOE_PREFILL=1 -e VLLM_EXL3_TC_PREFILL_ROWS=2048' '-e VLLM_EXL3_TC_MOE_SO=/opt/dsv41-tc-moe/_dsv41_tc_moe_C.abi3.so -e VLLM_EXL3_MOE_TP_ALIGN=128 -e VLLM_EXL3_MOE_TP_ROTATE=1 -e TORCH_CUDA_ARCH_LIST=8.0' '--entrypoint vllm <dsv41_image>' 'serve /model-view' '--quantization exl3 --load-format safetensors --safetensors-load-strategy lazy' '--pipeline-parallel-size 1 --tensor-parallel-size 4' '--served-model-name deepseek-v4.1-flash --host 0.0.0.0 --port 8095' '--max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.97' '--kv-cache-dtype fp8_ds_mla --engram-config {"cpu_offload": true} --enable-prefix-caching' '--compilation-config {"cudagraph_mode": "FULL_AND_PIECEWISE", "cudagraph_capture_sizes": [1, 2, 3, 4, 6, 8, 12, 18, 24, 30, 36, 42, 48]}' --async-scheduling '--speculative-config {"method": "dspark", "num_speculative_tokens": 5}' '--tokenizer-mode deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41'

band: 'batches of 2-8 users; batch 7 pads onto the 48-row batch-8 graph; draft length stays 5 (6 rows per request). Draft length 7 and the Mia weights are NOT part of this selection.'
lane: '2-8 user decode lane = experiments/ds1017-r2-b2-b8 (separate source, DEFAULT OFF). The published C1 source stays the one-user bundle in experiments/ds1017-native-mixed-and-split (native mixed experts, split attention, captured K5 drafter).'
image: "UNAVAILABLE (never published): the author states 'There is still no pullable image' (SERVING-2026-10-26.md). The only image identifier in the stack, sha256:55d2bd9a6664aea2540e3f8e373d784b88420d35c3cedefc9f03923cd687b6f5, is a LOCAL docker image id recorded for the measured serve (1017 repo README.md:93), not a registry reference; the site's image_digest is null. Cloning the repo does not turn the lane on: no image boots the 2-8 user graphs or the sampled drafter yet."
hardware: '4x CMP 170HX (sm_80) host as measured; the October 26 run used the 1017 weights on this host class.'
measurements: 'https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX/blob/main/benchmarks/2026-10-26/README.md'
lane_defaults: 'DSV41_R2_BATCHES=2-8; DSV41_R2_COMPONENTS=attention,experts,collective,sampler,sampled; DSV41_R2_PAD7=1; DSV41_R2_SPLITS=per batch nearest the one-user 96-CTA shape (empty = default); DSV41_R2_EXPERTS=coop, or tc when tensor-core v2 covers the row count; DSV41_R2_COLLECTIVE=pair at 48 rows, otherwise the host all-reduce size rule; DSV41_R2_CAP_MIB=768 (retained-graph cap checked at startup).'
measured_plan: 'With DSV41_R2=1, DSV41_R2_PAD7=1 and VLLM_EXL3_TC_MOE_V2=1 and no per-batch overrides the defaults are the measured plan: cooperative experts for batches 2-6, tensor-core experts + pair collective at batch 8, batch 7 served by the batch-8 graph. New attention for every batch size; a temperature-0 request stays greedy at every batch size.'
published_from: "CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE @ e559217ac4f90dd8c2e4820f97b259e584efcf1f (experiments/ds1017-native-mixed-and-split/SERVING-2026-10-26.md + experiments/ds1017-r2-b2-b8/README.md; commits 'Publish the 2-8 user lane and its default-off flags' and 'Note the October 26 serving selection and the missing image')"
default_off_flags:
  -
    flag: DSV41_R2
    effect: 'Master switch for the 2-8 lane; 1 is the only armed value, malformed flag raises at boot. Being present in the repository does NOT turn the lane on.'
    default: '0 (off)'
  -
    flag: VLLM_EXL3_TC_MOE_V2
    effect: 'Required for the tensor-core expert path on the 48-row graph; the launch command of this recipe arms it explicitly (-e VLLM_EXL3_TC_MOE_V2=1).'
    default: '0 (off)'
  -
    flag: dsv41_draft_tq
    effect: 'Per-request sampled-drafter flag; NOT turned on by DSV41_R2. The October 26 selection keeps the drafter greedy at batches 2-8 and uses it only for a single temperature-1 request (outside this lane).'
    default: 'off'
  -
    flag: DSV41_ARENA_CACHE_DIR
    effect: 'Optional gate in scripts/run-tp4-tcmoe.sh: unset/empty = zero argv change; when set it adds -v DIR:DIR, -e VLLM_EXL3_ARENA_CACHE=1 and -e VLLM_EXL3_ARENA_CACHE_DIR=DIR.'
    default: 'unset (off)'
launch_consistency: "The launch command above arms VLLM_EXL3_TC_MOE_V2=1; DSV41_R2 is NOT set, so the published serve keeps the 2-8 lane default-off until armed. The optional arena-cache group (DSV41_ARENA_CACHE_DIR gate) is NOT in this recipe's launch command — it stays default-off; add the three flags above to enable it."

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4.json