Qwen3.8-27B PARO-MXFP6 2x Radeon AI PRO R9700 — vLLM radiance TP2

podman compose up -d '# compose.yaml service '\''vllm'\'' @ 83b778f — image docker.io/stilldeadcode/vllm-radiance:0.9.3, entrypoint bash /opt/paro-prelude.sh, command: (verbatim):' /models/Swift-Qwen3.8-27B-PARO-MXFP6 --served-model-name qwen3.8-27b swift-qwen3.8-27b-paro-mxfp6 --chat-template /patches/qwen-fixed-v22.3.jinja --limit-mm-per-prompt '{"image": 99, "audio": 0, "video": 0}' --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --kv-cache-dtype fp8 --max-model-len 262144 --max-num-batched-tokens 8192 --max-num-seqs 8 --enable-prefix-caching --mamba-cache-mode align --gpu-memory-utilization 0.92 --kv-cache-memory-bytes 13500000000 -tp 2 --attention-backend R4D --speculative-config '{"method": "dflash", "model": "/models/Qwen3.8-27B-DFlash2-FP8", "num_speculative_tokens": 7, "attention_backend": "TRITON_ATTN", "disable_padded_drafter_batch": true, "draft_sample_method": "greedy"}' --no-async-scheduling --compilation-config '{"pass_config": {"fuse_norm_quant": true, "fuse_act_quant": true}, "compile_sizes": [1, 2, 4, 8], "inductor_compile_config": {"enable_auto_functionalized_v2": false, "size_asserts": false, "alignment_asserts": false, "scalar_asserts": false, "combo_kernels": true, "benchmark_combo_kernel": true, "triton.cooperative_reductions": true}}' --host 0.0.0.0 --port 8180

env: 'configs/env/paro.env (verbatim @ 83b778f): ROCR_VISIBLE_DEVICES=0,1 HIP_VISIBLE_DEVICES=0,1 HF_HUB_OFFLINE=1 PARO_MODEL_DIR=Swift-Qwen3.8-27B-PARO-MXFP6 PARO_DRAFTER_DIR=Qwen3.8-27B-DFlash2-FP8 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 (all other VLLM_ROCM_USE_AITER_* =0) NCCL_PROTO=Simple RADIANCE_USE_R4D=1 RADIANCE_USE_R4D_AR=1 RADIANCE_USE_R4D_AR_QUANT=1 RADIANCE_AR_MAX_KB=86016 RADIANCE_AR_QNB=96 RADIANCE_AR_QNT=1024 RADIANCE_AR_OVERLAP=0 RADIANCE_PRESHUFFLE=1 RADIANCE_FUSE_RMS_QUANT=1 R4D_ATTN_FP8=3 RADIANCE_SKINNY_GEMM=1 RADIANCE_TOPK_TRITON_MIN_ROWS=1 RADIANCE_KV_GROUP_OPT=1 RADIANCE_GDN_FUSED_UPDATE=1 RADIANCE_GDN_MERGE_INPROJ=0 RADIANCE_GDN_FUSED_MAX_ITEMS=32 RADIANCE_GDN_PATHS=both RADIANCE_DYNAMIC_WIDTH=1 RADIANCE_DYNW_ALPHA=0.35 RADIANCE_DYNW_MARGIN=2 RADIANCE_DYNW_MIN=2 RADIANCE_DYNW_MIN_BATCH=3 RADIANCE_FAST_DRAFT=1 RADIANCE_DRAFT_TAU=0.20 RADIANCE_DRAFT_RERANK=80 RADIANCE_VERIFY_HEAD=1 RADIANCE_VERIFY_HEAD_MAX_M=32 RADIANCE_MXFP4_W4A8=1 RADIANCE_MXFP4_WPERM=1 RADIANCE_MXFP4_DECODE_NT=1 RADIANCE_MXFP4_DECODE_MAX_M=64 RADIANCE_MXFP4_EPIFAST=1 RADIANCE_MXFP4_TN4_MIN_M=2048 RADIANCE_MXFP4_A_TILED_MIN_M=513 RADIANCE_PAROQUANT=1 RADIANCE_PQ_I8=0 RADIANCE_PQ_PG=0 RADIANCE_PQ_ZPE=0 RADIANCE_PQ_PG_PRODUCER=3 RADIANCE_PQ_CHECKALL= RADIANCE_PQ_DECODE_MAX_M=64 RADIANCE_PQ_WPERM=1 RADIANCE_PQ_DECODE_NT=1 RADIANCE_PQ_ATILED=1 RADIANCE_PQ_AT_LBK=128 RADIANCE_PQ_AT_HOIST=1 RADIANCE_PQ_PTOK=1 RADIANCE_PQ_FUSED_TOKQ=1 RADIANCE_PQ_ROT_STREAM=1 RADIANCE_PQ_ROT_STREAM2=1 RADIANCE_PQ_ROT_STREAM3=0 RADIANCE_PQ_AR_CHECK=0 RADIANCE_PQ_AR_FALLBACK=0 RADIANCE_PQ_ROT_V2=1 VLLM_CACHE_ROOT=/cache/vllm TORCHINDUCTOR_CACHE_DIR=/cache/inductor TRITON_CACHE_DIR=/cache/triton AITER_ROOT_DIR=/cache/aiter PARO_PDF_DPI=144 PARO_PDF_MAX_PAGES=50 | plus x-vllm-base env: HF_HOME=/models VLLM_NO_USAGE_STATS=1.'
notes: "One-time setup BEFORE first serve: git submodule update --init build/radiance-vllm-mxfp4 (pinned submodule 44e3ee35, hugypufy fork until ggz14 merges MXFP6 PR #46) then ./setup-paro (fetches the ~25 GiB checkpoint + ~2 GiB drafter into ./models via the image's huggingface_hub, builds patched libr4d into ./radiance-cache/libr4d/b9e42ab-rx6). Start: podman compose up -d — first start hipcc-compiles kernels + JIT-compiles Triton/inductor (several minutes, looks idle; cached under ./radiance-cache/paro-mxfp6, warm starts ~90 s). The prelude dies loudly if the checkpoint, drafter, or patched libr4d is missing, and validates config.json quantization_config (paroquant_mxfp6, bits 6, group 128, mx_block 32). Alternate models replace (not join) the default service: ./model use <profile> swaps the 'vllm' network alias; a 'flash' profile (Qwen3.8-Flash-Next 125B MoE on stilldeadcode/radiance:1.3.0, expert-tiered, 12 GiB pinned host pool per card) ships in the same compose. Stack layout: Caddy terminates internal-CA TLS on the only published ports (443 path-routed /v1/* to vllm, /hermes/* to the gateway; 9119 for Hermes Desktop); the Hermes agent runs in a sandboxed container (gateway + SSH-dispatched workbench sidecar), egressing through Squid which denies all private/LAN ranges; Honcho (pgvector Postgres + CPU TEI embeddings, Qwen3-Embedding-0.6B) is the agent's self-hosted memory with all LLM calls going to the local vLLM. Gotchas from the README: add every stack-network hostname to NO_PROXY or Squid refuses it; never add user:/userns_mode: to the vLLM service (non-root AITER crash-loop); the protocol.py overlay is version-locked to the image's vLLM v0.27.1 — re-diff when bumping the image tag. Bench harness (bench/run-bench.sh: decode1/decode4/prefill32k/mixed4 random + chat1/chat4/chatmixed4 ShareGPT + docs32k) ships in-repo; no results are committed and the compose does not mount the bench runner."

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-paro-mxfp6-2x-radeon-ai-pro-r9700-vllm-radiance-tp2.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-paro-mxfp6-2x-radeon-ai-pro-r9700-vllm-radiance-tp2.json