{"meta":{"count":6,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["rtx-pro-6000-blackwell-x2","170hx"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T12:38:54+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","title":"Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM","summary":"Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":197.80000000000001136868377216160297393798828125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.955646-06","commit_sha":"af7d7dde12376551476c8b9e36cd72c68dc32408","content_hash":"c82e80ffa9955d905b61bda381cf7d89db1350ba","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/Mia-AiLab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json","alternate":"/r/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm"}},{"slug":"ds4-vision-2x6000-jj-dspark-k3","title":"DeepSeek V4 Flash Vision on 2x RTX PRO 6000 Blackwell \\x{2014} TP2 DSpark K3","summary":"Community image TP2 DSpark K3 Vision serving recipe: B12X sparse MoE, FP8 KV, three draft layers, 1M context. Saved numbers from the KK comparison.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":169.19999999999998863131622783839702606201171875,"prefill_tok_s":8993,"updated_at":"2026-10-20 21:54:52.636866-06","commit_sha":"17fd5669c2c9eaeb55ffb611f348686755997213","content_hash":"a67645b428d443142e2632eb2d7b7ff2eb8c27c4","votes":0,"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/ds4-vision-2x6000-jj-dspark-k3.json","alternate":"/r/ds4-vision-2x6000-jj-dspark-k3"}},{"slug":"ds4-flash-2x6000-jj-dspark-k5","title":"DeepSeek V4 Flash-0731 on 2x RTX PRO 6000 Blackwell \\x{2014} TP2 DSpark K5","summary":"Community image TP2 DSpark K5 serving recipe: B12X sparse MoE, FP8 KV, InstantTensor loading, 1M context. Saved numbers from the KK comparison (Max-Q +6000).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":214.900000000000005684341886080801486968994140625,"prefill_tok_s":11167,"updated_at":"2026-10-20 21:54:52.587962-06","commit_sha":"aa77cf295a865cd2380aa3150da706889728140b","content_hash":"655e2c0c651bd72eb6a493c12824a232b005a10c","votes":0,"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/ds4-flash-2x6000-jj-dspark-k5.json","alternate":"/r/ds4-flash-2x6000-jj-dspark-k5"}},{"slug":"qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix","title":"Qwen3.8-27B INT8 MTP3 prefix-cache on 1x CMP 170HX Hynix","summary":"Qwen3.8-27B at 65-67 tok/s decode (74 structured) with MTP k=3 speculative decoding AND prefix caching simultaneously on a single NVIDIA CMP 170HX 64GB (Hynix HBM2e, GA100/sm80), 262,144-token context on fp8_e4m3 KV, 24/7 production on pinned upstream vLLM nightly 0.26.1rc1.dev1024+g1fe3a1571 \u2014 the first build where prefix caching and MTP spec decode coexist (GDN mamba-align). Cached TTFT on a 7K shared prefix drops ~4.0 s -> 1.44 s (33.5% hit rate) while decode gains 7-9% over the 0.27.1 fork fallback \u2014 the 'MTP vs prefix cache, pick one' tradeoff is gone. Load-bearing trio --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 (default chunked-prefill batch = 0.0% hit rate while looking healthy); --prefix-match-unit 400 halves cached TTFT further (must divide the 1600-token GDN block); batch budget 32000 OOMs and hard-wedges the card. Samsung-40GB side recipe (int8 + DFlash2, 97.8 tok/s) with an sm80 int8-repack CPU-fallback patch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":68.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:22.585629-06","commit_sha":"ad3f55dab02cedfb2b187f7dcfc3f1647e0aaa0c","content_hash":"68f95048b88c132d1ddbffc8ceceaf5c38be5140","votes":0,"credit":[{"name":"ahnguyen17 (recipe, patches, probes, A/B benchmarks)","url":"https://github.com/ahnguyen17/cmp-170hx-vllm"},{"name":"lued (Qwen3.8-27B-INT8-W8A16-MTP checkpoint)","url":"https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP"},{"name":"vllm-project (vLLM upstream; #50021 GDN spec bounds, #45238 align-mode cache retention)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json","alternate":"/r/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix"}},{"slug":"glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3","title":"GLM-5.3-Flash QAD (CSF) on 2x RTX PRO 6000 Blackwell \u2014 TP2/DCP2 MTP3","summary":"Local Inference Lab production two-card serving config: TP2/DCP2, 8 concurrent seqs, MTP3 on B12X, lossless-CSF NVFP4 MoE (MXFP8-CSF-QAD weights, 188.87 GB), online MXFP8 attention and shared experts, NVFP4 MTP experts, FP8 KV, host embeddings, MXFP8 vision, 1,048,576 context at 0.985 GPU memory. Reported KV pool 2,053,877 tokens at 0.984 memory with the 4096-token batch budget; guild-benchmarked at 207/488/672 tok/s at 1/4/8 users, about 68% of the 4-card TP4 reference on the same checkpoint.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":207,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:06.136201-06","commit_sha":"379e6111fef42f62bf472d1b559f4514dd06c1b6","content_hash":"fc89c71f949991e16656968e86cdcababa686f4a","votes":0,"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/blackwell-llm-docker"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3.json","alternate":"/r/glm-5-3-flash-qad-csf-on-2x-rtx-pro-6000-blackwell-tp2-dcp2-mtp3"}},{"slug":"qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n","title":"Qwen3.8-Flash-Next 3bpw MTP on 1x CMP 170HX SSD PLE AIO (iIIusi0n)","summary":"First single-CMP-170HX recipe for Qwen3.8-Flash-Next: the AutoRound 3bpw+MTP checkpoint on ONE 64 GiB SM80 GPU with the 95.37 GiB BF16 PLE table left ON the SSD \u2014 native Linux AIO async reads, 16K prompt read-ahead and a 512 MiB row cache stream it per-row while the model loads metadata-only, so the 176B MoE fits one card. Patch adds async PLE reads/read-ahead, metadata-only loading, CUDA-graph host breaks, mixed-bit expert handling and MTP layer remapping; KLD 0.1208 vs original weights. Three-round medians (128 output tokens, temp 0, thinking off, EOS ignored): 8K prefill 2,636 tok/s, single-request decode 111 tok/s, 667 tok/s aggregate at 16 concurrent. Honest state: 31 SSD/graph tests PASS, 134 requests no API errors, BUT an arithmetic-consistency issue remains open; context advertised 262K, timing tested only through 32K. Tested sparse (15 GiB RAM, 31 GiB swap) on ENTERPRISE SATA at 180 W. Documented swap: the orcarouter uncensored AutoRound 3bpw checkpoint runs the same stack.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":111,"prefill_tok_s":2132,"updated_at":"2026-10-08 21:11:56.66944-06","commit_sha":"827bfe01549b43357fa06a8610ea5e64b55583b0","content_hash":"3a31e6a965360ba192fd941458c89f62f07e27e6","votes":0,"credit":[{"name":"iIIusi0n (recipe, vLLM ple-ssd patch set, AIO helper, benchmark harness)","url":"https://github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx"},{"name":"Qwen (official Qwen3.8-Flash-Next base, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"klee100 (AutoRound 3bpw+MTP quantized checkpoints, official + uncensored)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"OrcaRouter (Qwen3.8-Flash-Next-Uncensored BF16 abliterated fine-tune)","url":"https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored"},{"name":"vLLM project (upstream vLLM @ a5a30471 the patch applies to)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n.json","alternate":"/r/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n"}}]}