{"meta":{"count":4,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["rtx-pro-6000-blackwell-x3","170hx"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T17:27:12+00:00"},"recipes":[{"slug":"deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell","title":"DeepSeek-V4.1-Flash-1017 Kraken TP3 3x RTX PRO 6000 Blackwell","summary":"DeepSeek-V4.1-Flash-1017 at TP3 on three 96 GB RTX PRO 6000 Blackwell cards with the full 1M-token context, serving DeepSeek's original weights (FP8 dense + MXFP4 routed experts, nothing requantized). Expert layers 20-33 park in host RAM against a 1.5 GiB promotion reserve; once healthy their hottest experts promote back to GPU (hot-first order from 1017 route counts, lossless FlashInfer CSF scale compression). Spreading that fixed budget across 14 layers instead of 6 shallows each layer's cold tail - the author reports 68.0 -> 72.3 engine steps/s at 32K decode (+6.3%), no quality, KV or prefill cost. Engram layer 1 pinned in host RAM, layer 14 streamed from NVMe. DSpark speculative decoding, 5 drafts verified, NVFP4 drafter head. 1,048,576-token window over a 2,185,581-token KV pool, up to 4 sequences. Measured on image 20261019.3: prefill 13,486/12,245 tok/s (32K/128K), 7,193 at 983K; decode 186.7/340.7 c1/c4 at 32K, 179.2/307.1 at 128K, 474 aggregate on 4 shorts, 271.8 coding peak.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":228.5,"prefill_tok_s":11445.5,"updated_at":"2026-10-29 23:29:37.139691-06","commit_sha":"fe33370ee6a18643174519c8b9e956f05e807711","content_hash":"252307b64eed922ec9291880d6efb1938f670372","votes":0,"credit":[{"name":"BoxTops4Coochie (recipe, TP3 port, measurements)","url":"https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes"},{"name":"infernix - ds41f-tp3 expert-parking design on karmic-kraken","url":"https://github.com/infernix"},{"name":"Local Inference Lab - vLLM karmic-kraken fork + FlashInfer b12x kernels","url":"https://github.com/local-inference-lab"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell.json","alternate":"/r/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell"}},{"slug":"qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","title":"Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM","summary":"Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":197.80000000000001136868377216160297393798828125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.955646-06","commit_sha":"af7d7dde12376551476c8b9e36cd72c68dc32408","content_hash":"c82e80ffa9955d905b61bda381cf7d89db1350ba","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/Mia-AiLab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json","alternate":"/r/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm"}},{"slug":"qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix","title":"Qwen3.8-27B INT8 MTP3 prefix-cache on 1x CMP 170HX Hynix","summary":"Qwen3.8-27B at 65-67 tok/s decode (74 structured) with MTP k=3 speculative decoding AND prefix caching simultaneously on a single NVIDIA CMP 170HX 64GB (Hynix HBM2e, GA100/sm80), 262,144-token context on fp8_e4m3 KV, 24/7 production on pinned upstream vLLM nightly 0.26.1rc1.dev1024+g1fe3a1571 \u2014 the first build where prefix caching and MTP spec decode coexist (GDN mamba-align). Cached TTFT on a 7K shared prefix drops ~4.0 s -> 1.44 s (33.5% hit rate) while decode gains 7-9% over the 0.27.1 fork fallback \u2014 the 'MTP vs prefix cache, pick one' tradeoff is gone. Load-bearing trio --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 (default chunked-prefill batch = 0.0% hit rate while looking healthy); --prefix-match-unit 400 halves cached TTFT further (must divide the 1600-token GDN block); batch budget 32000 OOMs and hard-wedges the card. Samsung-40GB side recipe (int8 + DFlash2, 97.8 tok/s) with an sm80 int8-repack CPU-fallback patch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":68.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:22.585629-06","commit_sha":"ad3f55dab02cedfb2b187f7dcfc3f1647e0aaa0c","content_hash":"68f95048b88c132d1ddbffc8ceceaf5c38be5140","votes":0,"credit":[{"name":"ahnguyen17 (recipe, patches, probes, A/B benchmarks)","url":"https://github.com/ahnguyen17/cmp-170hx-vllm"},{"name":"lued (Qwen3.8-27B-INT8-W8A16-MTP checkpoint)","url":"https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP"},{"name":"vllm-project (vLLM upstream; #50021 GDN spec bounds, #45238 align-mode cache retention)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json","alternate":"/r/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix"}},{"slug":"qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n","title":"Qwen3.8-Flash-Next 3bpw MTP on 1x CMP 170HX SSD PLE AIO (iIIusi0n)","summary":"First single-CMP-170HX recipe for Qwen3.8-Flash-Next: the AutoRound 3bpw+MTP checkpoint on ONE 64 GiB SM80 GPU with the 95.37 GiB BF16 PLE table left ON the SSD \u2014 native Linux AIO async reads, 16K prompt read-ahead and a 512 MiB row cache stream it per-row while the model loads metadata-only, so the 176B MoE fits one card. Patch adds async PLE reads/read-ahead, metadata-only loading, CUDA-graph host breaks, mixed-bit expert handling and MTP layer remapping; KLD 0.1208 vs original weights. Three-round medians (128 output tokens, temp 0, thinking off, EOS ignored): 8K prefill 2,636 tok/s, single-request decode 111 tok/s, 667 tok/s aggregate at 16 concurrent. Honest state: 31 SSD/graph tests PASS, 134 requests no API errors, BUT an arithmetic-consistency issue remains open; context advertised 262K, timing tested only through 32K. Tested sparse (15 GiB RAM, 31 GiB swap) on ENTERPRISE SATA at 180 W. Documented swap: the orcarouter uncensored AutoRound 3bpw checkpoint runs the same stack.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":111,"prefill_tok_s":2132,"updated_at":"2026-10-08 21:11:56.66944-06","commit_sha":"827bfe01549b43357fa06a8610ea5e64b55583b0","content_hash":"3a31e6a965360ba192fd941458c89f62f07e27e6","votes":0,"credit":[{"name":"iIIusi0n (recipe, vLLM ple-ssd patch set, AIO helper, benchmark harness)","url":"https://github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx"},{"name":"Qwen (official Qwen3.8-Flash-Next base, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"klee100 (AutoRound 3bpw+MTP quantized checkpoints, official + uncensored)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"OrcaRouter (Qwen3.8-Flash-Next-Uncensored BF16 abliterated fine-tune)","url":"https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored"},{"name":"vLLM project (upstream vLLM @ a5a30471 the patch applies to)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n.json","alternate":"/r/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n"}}]}