{"slug":"qwen3-8-27b-paro-mxfp6-2x-radeon-ai-pro-r9700-vllm-radiance-tp2","title":"Qwen3.8-27B PARO-MXFP6 2x Radeon AI PRO R9700 \u2014 vLLM radiance TP2","summary":"27B Qwen served from a single podman-compose stack on two Radeon AI PRO R9700 (gfx1201) cards: vLLM runs the prebuilt stilldeadcode/vllm-radiance:0.9.3 image (RDNA4 patches + tuned kernels baked in) with a PARO prelude that applies the radiance source patches, hipcc-builds the ParoQuant kernels and swaps in a patched libr4d at every start. ParoQuant MXFP6 (E2M3, 6.25 bpw, W6A8) of the Swift fine-tune at tensor parallel 2, R4D attention, fp8 KV, 262144-token context, external DFlash2-FP8 drafter with 7 draft tokens. Caddy fronts the only published port with an internal-CA TLS OpenAI+Anthropic API; a sandboxed Hermes agent container egresses through Squid and never touches the host or LAN. Model ids qwen3.8-27b and swift-qwen3.8-27b-paro-mxfp6.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"eeb81045e6312bd245e460da2277b6b8557f1059","content_hash":"7e8ac2b04a18ee509166698bd6ef1546224f7c51","created_at":"2026-10-20 21:52:36.618781-06","notes":"Submitted 2026-10-10 from lar-scout ingest card t_a39242ac. Repo @ 83b778f472fffe019f1af3b55fd58c18a5ea979b (HEAD verified 2026-10-10 via git ls-remote; tree enumerated at that sha; 0 stars, no LICENSE file). Serve argv transcribed VERBATIM from compose.yaml vllm service command: @ 83b778f, token-verified against configs/paro/prelude.sh (prelude execs /opt/radiance_entrypoint.sh with /models/$PARO_MODEL_DIR prepended, so the compose command IS the vllm argv) and configs/env/paro.env (every RADIANCE_*/AITER/PQ knob transcribed verbatim). Engine pin docker.io/stilldeadcode/vllm-radiance:0.9.3 verified against Docker Hub API (digest sha256:4569\u2026c25, pushed 2026-08-25). Checkpoint HF sha 580065fc + drafter repo existence verified via HF API from the ingest host; drafter pin is current-head at ingest (setup-paro does not revision-pin it), checkpointer's quality+perf tables published on the same card. Perf rows are TRANSCRIBED from the hugypufy HF model card (quant author's 2x R9700 TP2 rig), NOT measured by the recipe author \u2014 the recipe repo publishes no numbers and ships bench/run-bench.sh with zero committed results; the card states rows are single runs on a newer radiance build than the 0.9.3 image pinned here. Model family qwen-3-8-27b explicit; model/quant/hardware rows: model Qwen3.8-27B-PARO-MXFP6 NEW, quant mxfp6-paro NEW (slug mxfp6-paro), hardware radeon-ai-pro-r9700-x2 seeded by migration 0065 pre-POST (mirrors 0064 dgx-spark-x4, t_ad3aebd3, same run). No LICENSE in the recipe repo \u2014 license field omitted."},"engine_source":{"type":"docker_image","git_url":null,"base_engine":"vllm","pip_package":null,"base_version":"vllm-radiance 0.9.3 (vLLM v0.27.1 + RDNA4 patches)","image_digest":"docker.io/stilldeadcode/vllm-radiance@sha256:45694209177a55a1ab3ba6702fe6e978b1b66a6e66ae3fc066f8d579f7bc4c25","install_note":"Serving image is docker.io/stilldeadcode/vllm-radiance:0.9.3 (gfx1201-only; Docker Hub digest verified at ingest). Not stock vLLM: compose runs it as container root because AITER JIT-writes kernels into site-packages at startup, the PARO prelude (configs/paro/prelude.sh) applies the radiance source patches and hipcc-builds the MXFP6 + ParoQuant HIP kernels into site-packages on every container start (idempotent), and ./setup-paro builds a patched libr4d (codeberg StillDeadcode/libr4d @ b9e42ab + r4d_radiance_extras.patch, key b9e42ab-rx6) because the image's own libr4d NaNs this model's gated-delta-net layers. Podman rootless on Fedora; needs /dev/kfd + /dev/dri, render-group gid, and the port-443 sysctl from the README.","pinned_commit":null},"credit":[{"name":"vinniefranco (podman compose stack, TLS/egress design, bench harness)","url":"https://github.com/vinniefranco/hermes-vllm-2x-r9700"},{"name":"hugypufy (PARO MXFP6 checkpoint + radiance-vllm-mxfp4 fork + published perf table)","url":"https://huggingface.co/hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6"},{"name":"StillDeadcode (vllm-radiance image, radiance engine, libr4d)","url":"https://codeberg.org/StillDeadcode/vllm-radiance"},{"name":"tcclaviger (DFlash2-FP8 drafter)","url":"https://huggingface.co/tcclaviger/Qwen3.8-27B-DFlash2-FP8"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["podman","compose","up","-d","# compose.yaml service 'vllm' @ 83b778f \u2014 image docker.io/stilldeadcode/vllm-radiance:0.9.3, entrypoint bash /opt/paro-prelude.sh, command: (verbatim):","/models/Swift-Qwen3.8-27B-PARO-MXFP6","--served-model-name","qwen3.8-27b","swift-qwen3.8-27b-paro-mxfp6","--chat-template","/patches/qwen-fixed-v22.3.jinja","--limit-mm-per-prompt","{\"image\": 99, \"audio\": 0, \"video\": 0}","--enable-auto-tool-choice","--tool-call-parser","qwen3_xml","--reasoning-parser","qwen3","--kv-cache-dtype","fp8","--max-model-len","262144","--max-num-batched-tokens","8192","--max-num-seqs","8","--enable-prefix-caching","--mamba-cache-mode","align","--gpu-memory-utilization","0.92","--kv-cache-memory-bytes","13500000000","-tp","2","--attention-backend","R4D","--speculative-config","{\"method\": \"dflash\", \"model\": \"/models/Qwen3.8-27B-DFlash2-FP8\", \"num_speculative_tokens\": 7, \"attention_backend\": \"TRITON_ATTN\", \"disable_padded_drafter_batch\": true, \"draft_sample_method\": \"greedy\"}","--no-async-scheduling","--compilation-config","{\"pass_config\": {\"fuse_norm_quant\": true, \"fuse_act_quant\": true}, \"compile_sizes\": [1, 2, 4, 8], \"inductor_compile_config\": {\"enable_auto_functionalized_v2\": false, \"size_asserts\": false, \"alignment_asserts\": false, \"scalar_asserts\": false, \"combo_kernels\": true, \"benchmark_combo_kernel\": true, \"triton.cooperative_reductions\": true}}","--host","0.0.0.0","--port","8180"],"runtime":{"env":"configs/env/paro.env (verbatim @ 83b778f): ROCR_VISIBLE_DEVICES=0,1 HIP_VISIBLE_DEVICES=0,1 HF_HUB_OFFLINE=1 PARO_MODEL_DIR=Swift-Qwen3.8-27B-PARO-MXFP6 PARO_DRAFTER_DIR=Qwen3.8-27B-DFlash2-FP8 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 (all other VLLM_ROCM_USE_AITER_* =0) NCCL_PROTO=Simple RADIANCE_USE_R4D=1 RADIANCE_USE_R4D_AR=1 RADIANCE_USE_R4D_AR_QUANT=1 RADIANCE_AR_MAX_KB=86016 RADIANCE_AR_QNB=96 RADIANCE_AR_QNT=1024 RADIANCE_AR_OVERLAP=0 RADIANCE_PRESHUFFLE=1 RADIANCE_FUSE_RMS_QUANT=1 R4D_ATTN_FP8=3 RADIANCE_SKINNY_GEMM=1 RADIANCE_TOPK_TRITON_MIN_ROWS=1 RADIANCE_KV_GROUP_OPT=1 RADIANCE_GDN_FUSED_UPDATE=1 RADIANCE_GDN_MERGE_INPROJ=0 RADIANCE_GDN_FUSED_MAX_ITEMS=32 RADIANCE_GDN_PATHS=both RADIANCE_DYNAMIC_WIDTH=1 RADIANCE_DYNW_ALPHA=0.35 RADIANCE_DYNW_MARGIN=2 RADIANCE_DYNW_MIN=2 RADIANCE_DYNW_MIN_BATCH=3 RADIANCE_FAST_DRAFT=1 RADIANCE_DRAFT_TAU=0.20 RADIANCE_DRAFT_RERANK=80 RADIANCE_VERIFY_HEAD=1 RADIANCE_VERIFY_HEAD_MAX_M=32 RADIANCE_MXFP4_W4A8=1 RADIANCE_MXFP4_WPERM=1 RADIANCE_MXFP4_DECODE_NT=1 RADIANCE_MXFP4_DECODE_MAX_M=64 RADIANCE_MXFP4_EPIFAST=1 RADIANCE_MXFP4_TN4_MIN_M=2048 RADIANCE_MXFP4_A_TILED_MIN_M=513 RADIANCE_PAROQUANT=1 RADIANCE_PQ_I8=0 RADIANCE_PQ_PG=0 RADIANCE_PQ_ZPE=0 RADIANCE_PQ_PG_PRODUCER=3 RADIANCE_PQ_CHECKALL= RADIANCE_PQ_DECODE_MAX_M=64 RADIANCE_PQ_WPERM=1 RADIANCE_PQ_DECODE_NT=1 RADIANCE_PQ_ATILED=1 RADIANCE_PQ_AT_LBK=128 RADIANCE_PQ_AT_HOIST=1 RADIANCE_PQ_PTOK=1 RADIANCE_PQ_FUSED_TOKQ=1 RADIANCE_PQ_ROT_STREAM=1 RADIANCE_PQ_ROT_STREAM2=1 RADIANCE_PQ_ROT_STREAM3=0 RADIANCE_PQ_AR_CHECK=0 RADIANCE_PQ_AR_FALLBACK=0 RADIANCE_PQ_ROT_V2=1 VLLM_CACHE_ROOT=/cache/vllm TORCHINDUCTOR_CACHE_DIR=/cache/inductor TRITON_CACHE_DIR=/cache/triton AITER_ROOT_DIR=/cache/aiter PARO_PDF_DPI=144 PARO_PDF_MAX_PAGES=50 | plus x-vllm-base env: HF_HOME=/models VLLM_NO_USAGE_STATS=1.","notes":"One-time setup BEFORE first serve: git submodule update --init build/radiance-vllm-mxfp4 (pinned submodule 44e3ee35, hugypufy fork until ggz14 merges MXFP6 PR #46) then ./setup-paro (fetches the ~25 GiB checkpoint + ~2 GiB drafter into ./models via the image's huggingface_hub, builds patched libr4d into ./radiance-cache/libr4d/b9e42ab-rx6). Start: podman compose up -d \u2014 first start hipcc-compiles kernels + JIT-compiles Triton/inductor (several minutes, looks idle; cached under ./radiance-cache/paro-mxfp6, warm starts ~90 s). The prelude dies loudly if the checkpoint, drafter, or patched libr4d is missing, and validates config.json quantization_config (paroquant_mxfp6, bits 6, group 128, mx_block 32). Alternate models replace (not join) the default service: ./model use <profile> swaps the 'vllm' network alias; a 'flash' profile (Qwen3.8-Flash-Next 125B MoE on stilldeadcode/radiance:1.3.0, expert-tiered, 12 GiB pinned host pool per card) ships in the same compose. Stack layout: Caddy terminates internal-CA TLS on the only published ports (443 path-routed /v1/* to vllm, /hermes/* to the gateway; 9119 for Hermes Desktop); the Hermes agent runs in a sandboxed container (gateway + SSH-dispatched workbench sidecar), egressing through Squid which denies all private/LAN ranges; Honcho (pgvector Postgres + CPU TEI embeddings, Qwen3-Embedding-0.6B) is the agent's self-hosted memory with all LLM calls going to the local vLLM. Gotchas from the README: add every stack-network hostname to NO_PROXY or Squid refuses it; never add user:/userns_mode: to the vLLM service (non-root AITER crash-loop); the protocol.py overlay is version-locked to the image's vLLM v0.27.1 \u2014 re-diff when bumping the image tag. Bench harness (bench/run-bench.sh: decode1/decode4/prefill32k/mixed4 random + chat1/chat4/chatmixed4 ShareGPT + docs32k) ships in-repo; no results are committed and the compose does not mount the bench runner."},"max_context_len":262144},"models":[{"slug":"swift-qwen3-8-27b-paro-mxfp6","name":"Swift-Qwen3.8-27B-PARO-MXFP6","hf_repo":"hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6","quant":"mxfp6-paro"}],"hardware":[{"slug":"radeon-ai-pro-r9700-x2","vendor":"amd","card":"Radeon AI Pro R9700","card_family":"radeon-ai-pro-r9700","vram_gb":32,"device_count":2}],"generation":"amd","tags":["agent-serving","amd","dflash2","fp8-kv","gfx1201","mxfp6","paroquant","podman-compose","rdna4","speculative","tensor-parallel","tls-caddy"],"tasks":["chat","code"],"aa":{"aa_index":34,"aa_provenance":"published","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-07 18:00:00-06","aa_family_slug":"qwen-3-8-27b","aa_resolved_slug":"qwen-3-8-27b"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"TRANSCRIBED from the PARO-MXFP6 checkpointer's HF model card (hugypufy, 2026-09-17 revision 580065fc) \u2014 its published 2x Radeon AI PRO R9700 TP=2 table for this checkpoint with the DFlash2-FP8 drafter at 7 draft tokens and the KV cache pinned at 13.5 GB per GPU, the same config this compose serves. The recipe repo (vinniefranco) publishes NO benchmark numbers; these are the quant author's measured rows, not the recipe author's. Source states each row is a single measured run (speed rows at a 12.5 GB pin) on a NEWER radiance build than the 0.9.3 image pinned here, so figures may move \u2014 treat as indicative, not lab-verified.\", \"tok_s\": 138, \"points\": [{\"notes\": \"Single-stream decode, transcribed from hugypufy HF model card perf table (2x R9700 TP2, DFlash2-FP8 k=7, 13.5 GB KV pin); single run, newer radiance build than the 0.9.3 pin\", \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 138}, {\"notes\": \"Concurrent-decode table c1 row (13.5 GB KV pin), same source\", \"concurrency\": 1, \"context_len\": 8192, \"decode_tok_s\": 124.7000000000000028421709430404007434844970703125}, {\"notes\": \"Concurrent decode c2 aggregate (13.5 GB KV pin), same source\", \"concurrency\": 2, \"context_len\": 8192, \"decode_tok_s\": 209.599999999999994315658113919198513031005859375}, {\"notes\": \"Concurrent decode c4 aggregate (13.5 GB KV pin), same source\", \"concurrency\": 4, \"context_len\": 8192, \"decode_tok_s\": 321.6000000000000227373675443232059478759765625}, {\"notes\": \"Concurrent decode c8 aggregate (13.5 GB KV pin), same source\", \"concurrency\": 8, \"context_len\": 8192, \"decode_tok_s\": 394}, {\"notes\": \"Prefill 2k, transcribed from hugypufy HF model card (no decode row published for this arm)\", \"concurrency\": 1, \"context_len\": 2048, \"prefill_tok_s\": 3327}, {\"notes\": \"Prefill 8k (peak of the 2k/8k/16k/32k/60k sweep: 3327/3628/3570/3479/3348)\", \"concurrency\": 1, \"context_len\": 8192, \"prefill_tok_s\": 3628}, {\"notes\": \"Prefill 16k, same sweep\", \"concurrency\": 1, \"context_len\": 16000, \"prefill_tok_s\": 3570}, {\"notes\": \"Prefill 32k, same sweep\", \"concurrency\": 1, \"context_len\": 32000, \"prefill_tok_s\": 3479}, {\"notes\": \"Prefill 60k, same sweep; KV cache tokens 685,554 and 12.85 GiB weights per GPU also published\", \"concurrency\": 1, \"context_len\": 60000, \"prefill_tok_s\": 3348}], \"context_len\": 262144}"}],"perf_points":[{"id":1667,"context_len":2048,"prefill_tok_s":3327,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Prefill 2k, transcribed from hugypufy HF model card (no decode row published for this arm)"},{"id":1668,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":138,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Single-stream decode, transcribed from hugypufy HF model card perf table (2x R9700 TP2, DFlash2-FP8 k=7, 13.5 GB KV pin); single run, newer radiance build than the 0.9.3 pin"},{"id":1669,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":209.599999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":2,"claimed_by":"submitter","notes":"Concurrent decode c2 aggregate (13.5 GB KV pin), same source"},{"id":1670,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":321.6000000000000227373675443232059478759765625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"Concurrent decode c4 aggregate (13.5 GB KV pin), same source"},{"id":1671,"context_len":8192,"prefill_tok_s":null,"decode_tok_s":394,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Concurrent decode c8 aggregate (13.5 GB KV pin), same source"},{"id":1672,"context_len":16000,"prefill_tok_s":3570,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Prefill 16k, same sweep"},{"id":1673,"context_len":32000,"prefill_tok_s":3479,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Prefill 32k, same sweep"},{"id":1674,"context_len":60000,"prefill_tok_s":3348,"decode_tok_s":null,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Prefill 60k, same sweep; KV cache tokens 685,554 and 12.85 GiB weights per GPU also published"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-paro-mxfp6-2x-radeon-ai-pro-r9700-vllm-radiance-tp2.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-paro-mxfp6-2x-radeon-ai-pro-r9700-vllm-radiance-tp2.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6","pin":"580065fca8f831f8ad70e3dfc0b47b205bba3985 \u2014 PARO MXFP6 (E2M3) of the Swift fine-tune (ukisai/Swift-Qwen3.8-27b), HF API sha verified at ingest (setup-paro snapshot_download, ~25 GiB; 12.85 GiB weights per GPU)"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/tcclaviger/Qwen3.8-27B-DFlash2-FP8","pin":"f02593d0ea2e3b7eedbb831fb26f465b1e6238bb current at ingest \u2014 DFlash2 FP8 drafter k=7 (setup-paro DRAFT_REPO, fetched to ./models/Qwen3.8-27B-DFlash2-FP8; --speculative-config model path)"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/stilldeadcode/vllm-radiance","pin":"sha256:45694209177a55a1ab3ba6702fe6e978b1b66a6e66ae3fc066f8d579f7bc4c25 (tag 0.9.3, Docker Hub API verified at ingest) \u2014 compose x-vllm-base image + setup-paro IMAGE"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/vinniefranco/hermes-vllm-2x-r9700","pin":"83b778f472fffe019f1af3b55fd58c18a5ea979b @ HEAD 'Pin hermes-agent to v0.21.6' \u2014 compose.yaml vllm service command: is the verbatim serve argv; configs/env/paro.env + configs/paro/prelude.sh the env/prelude; setup-paro the one-time setup"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://codeberg.org/hugypufy/radiance-vllm-mxfp4","pin":"submodule build/radiance-vllm-mxfp4 @ 44e3ee355d6fe56fe7554176184deab7d93273a0 \u2014 upstream ParoQuant/MXFP6 sources (patches + kernels) applied by prelude.sh; tracks the fork until ggz14 merges MXFP6 PR #46"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://codeberg.org/StillDeadcode/libr4d","pin":"b9e42ab (short-sha pin per setup-paro R4D_PIN; built with the submodule tree's r4d_radiance_extras.patch as key b9e42ab-rx6 \u2014 stock libr4d NaNs this model's gated-delta-net layers)"}],"links":{"self":"/api/v1/recipes/qwen3-8-27b-paro-mxfp6-2x-radeon-ai-pro-r9700-vllm-radiance-tp2.json","alternate":"/r/qwen3-8-27b-paro-mxfp6-2x-radeon-ai-pro-r9700-vllm-radiance-tp2","describedby":"/openapi.json"},"updated_at":"2026-10-20 21:52:36.618781-06","created_at":"2026-10-20 21:49:16.786127-06","speeds":{"decode_tok_s":138,"prefill_tok_s":3348,"leaderboard":"/api/v1/leaderboard.json"}}