Qwen3.8-Flash-Next on RTX PRO 6000 Blackwell — Karmic Kraken TP1 MTP3

Launch command

serve local-inference-lab/Qwen3.8-Flash-Next-NVFP4 --async-scheduling --block-size 64 --compilation-config ''\''{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'\''' --decode-context-parallel-size 1 --dtype bfloat16 --enable-auto-tool-choice --enable-chunked-prefill --no-enable-flashinfer-autotune --enable-prefix-caching --enable-prompt-tokens-details --gdn-decode-kernel b12x --gpu-memory-utilization 0.96 --host 0.0.0.0 --kv-cache-dtype fp8 --no-language-model-only --linear-backend b12x --load-format instanttensor --mamba-cache-mode align --mamba-ssm-cache-dtype auto --max-cudagraph-capture-size 64 --max-model-len 262144 --max-num-batched-tokens 6019 --max-num-seqs 16 --mm-encoder-tp-mode data --mm-processor-cache-gb 0.0 --moe-backend b12x --pipeline-parallel-size 1 --port 8000 --quantization modelopt_mixed --reasoning-parser qwen3 --served-model-name Qwen3.8-Flash-Next --speculative-config ''\''{"method":"mtp","moe_backend":"b12x","num_speculative_tokens":3}'\''' --tensor-parallel-size 1 --tool-call-parser qwen3_xml

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-1x6000-kk-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-1x6000-kk-mtp3.json

Install requirements

• Docker (or a compatible OCI container runtime)
• NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

• Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
• GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
• Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
• Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Revision notes

Benchmark-qualified configuration from local-inference-lab/rtx6kpro (Karmic Kraken serving report). Five warmed 30-second windows per cell on RTX PRO 6000 Blackwell Max-Q; decode ctx-0 temp 1. Launch argv transcribed verbatim from the repo's unified Docker guide (same composition). Measured: C1 173.2 tok/s, C8 698.4 tok/s, 32K prefill 12073 tok/s.

Revision history

• 8fb728a8f9 recipe: update qwen3-8-flash-next-1x6000-kk-mtp3 (r2: /max_context_len) 10 d ago diff
• bfdf9daa3e history: re-squash corpus to a single root commit after the slug reconciliation (prior tip kept at refs/lar-archive/pre-single-slug-resquash) 25 d ago

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)