{"meta":{"count":4,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["170hx","radeon-ai-pro-r9700-x2"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T18:36:29+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","title":"Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM","summary":"Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":197.80000000000001136868377216160297393798828125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.955646-06","commit_sha":"af7d7dde12376551476c8b9e36cd72c68dc32408","content_hash":"c82e80ffa9955d905b61bda381cf7d89db1350ba","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/Mia-AiLab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json","alternate":"/r/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm"}},{"slug":"qwen3-8-27b-paro-mxfp6-2x-radeon-ai-pro-r9700-vllm-radiance-tp2","title":"Qwen3.8-27B PARO-MXFP6 2x Radeon AI PRO R9700 \u2014 vLLM radiance TP2","summary":"27B Qwen served from a single podman-compose stack on two Radeon AI PRO R9700 (gfx1201) cards: vLLM runs the prebuilt stilldeadcode/vllm-radiance:0.9.3 image (RDNA4 patches + tuned kernels baked in) with a PARO prelude that applies the radiance source patches, hipcc-builds the ParoQuant kernels and swaps in a patched libr4d at every start. ParoQuant MXFP6 (E2M3, 6.25 bpw, W6A8) of the Swift fine-tune at tensor parallel 2, R4D attention, fp8 KV, 262144-token context, external DFlash2-FP8 drafter with 7 draft tokens. Caddy fronts the only published port with an internal-CA TLS OpenAI+Anthropic API; a sandboxed Hermes agent container egresses through Squid and never touches the host or LAN. Model ids qwen3.8-27b and swift-qwen3.8-27b-paro-mxfp6.","tier":"community","beta":false,"generation":"amd","decode_tok_s":138,"prefill_tok_s":3348,"updated_at":"2026-10-20 21:52:36.618781-06","commit_sha":"eeb81045e6312bd245e460da2277b6b8557f1059","content_hash":"7e8ac2b04a18ee509166698bd6ef1546224f7c51","votes":0,"credit":[{"name":"vinniefranco (podman compose stack, TLS/egress design, bench harness)","url":"https://github.com/vinniefranco/hermes-vllm-2x-r9700"},{"name":"hugypufy (PARO MXFP6 checkpoint + radiance-vllm-mxfp4 fork + published perf table)","url":"https://huggingface.co/hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6"},{"name":"StillDeadcode (vllm-radiance image, radiance engine, libr4d)","url":"https://codeberg.org/StillDeadcode/vllm-radiance"},{"name":"tcclaviger (DFlash2-FP8 drafter)","url":"https://huggingface.co/tcclaviger/Qwen3.8-27B-DFlash2-FP8"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-paro-mxfp6-2x-radeon-ai-pro-r9700-vllm-radiance-tp2.json","alternate":"/r/qwen3-8-27b-paro-mxfp6-2x-radeon-ai-pro-r9700-vllm-radiance-tp2"}},{"slug":"qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix","title":"Qwen3.8-27B INT8 MTP3 prefix-cache on 1x CMP 170HX Hynix","summary":"Qwen3.8-27B at 65-67 tok/s decode (74 structured) with MTP k=3 speculative decoding AND prefix caching simultaneously on a single NVIDIA CMP 170HX 64GB (Hynix HBM2e, GA100/sm80), 262,144-token context on fp8_e4m3 KV, 24/7 production on pinned upstream vLLM nightly 0.26.1rc1.dev1024+g1fe3a1571 \u2014 the first build where prefix caching and MTP spec decode coexist (GDN mamba-align). Cached TTFT on a 7K shared prefix drops ~4.0 s -> 1.44 s (33.5% hit rate) while decode gains 7-9% over the 0.27.1 fork fallback \u2014 the 'MTP vs prefix cache, pick one' tradeoff is gone. Load-bearing trio --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 (default chunked-prefill batch = 0.0% hit rate while looking healthy); --prefix-match-unit 400 halves cached TTFT further (must divide the 1600-token GDN block); batch budget 32000 OOMs and hard-wedges the card. Samsung-40GB side recipe (int8 + DFlash2, 97.8 tok/s) with an sm80 int8-repack CPU-fallback patch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":68.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:22.585629-06","commit_sha":"ad3f55dab02cedfb2b187f7dcfc3f1647e0aaa0c","content_hash":"68f95048b88c132d1ddbffc8ceceaf5c38be5140","votes":0,"credit":[{"name":"ahnguyen17 (recipe, patches, probes, A/B benchmarks)","url":"https://github.com/ahnguyen17/cmp-170hx-vllm"},{"name":"lued (Qwen3.8-27B-INT8-W8A16-MTP checkpoint)","url":"https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP"},{"name":"vllm-project (vLLM upstream; #50021 GDN spec bounds, #45238 align-mode cache retention)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json","alternate":"/r/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix"}},{"slug":"qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n","title":"Qwen3.8-Flash-Next 3bpw MTP on 1x CMP 170HX SSD PLE AIO (iIIusi0n)","summary":"First single-CMP-170HX recipe for Qwen3.8-Flash-Next: the AutoRound 3bpw+MTP checkpoint on ONE 64 GiB SM80 GPU with the 95.37 GiB BF16 PLE table left ON the SSD \u2014 native Linux AIO async reads, 16K prompt read-ahead and a 512 MiB row cache stream it per-row while the model loads metadata-only, so the 176B MoE fits one card. Patch adds async PLE reads/read-ahead, metadata-only loading, CUDA-graph host breaks, mixed-bit expert handling and MTP layer remapping; KLD 0.1208 vs original weights. Three-round medians (128 output tokens, temp 0, thinking off, EOS ignored): 8K prefill 2,636 tok/s, single-request decode 111 tok/s, 667 tok/s aggregate at 16 concurrent. Honest state: 31 SSD/graph tests PASS, 134 requests no API errors, BUT an arithmetic-consistency issue remains open; context advertised 262K, timing tested only through 32K. Tested sparse (15 GiB RAM, 31 GiB swap) on ENTERPRISE SATA at 180 W. Documented swap: the orcarouter uncensored AutoRound 3bpw checkpoint runs the same stack.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":111,"prefill_tok_s":2132,"updated_at":"2026-10-08 21:11:56.66944-06","commit_sha":"827bfe01549b43357fa06a8610ea5e64b55583b0","content_hash":"3a31e6a965360ba192fd941458c89f62f07e27e6","votes":0,"credit":[{"name":"iIIusi0n (recipe, vLLM ple-ssd patch set, AIO helper, benchmark harness)","url":"https://github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx"},{"name":"Qwen (official Qwen3.8-Flash-Next base, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"klee100 (AutoRound 3bpw+MTP quantized checkpoints, official + uncensored)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"OrcaRouter (Qwen3.8-Flash-Next-Uncensored BF16 abliterated fine-tune)","url":"https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored"},{"name":"vLLM project (upstream vLLM @ a5a30471 the patch applies to)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n.json","alternate":"/r/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n"}}]}