{"meta":{"count":9,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["170hx","rtx-pro-6000-blackwell"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T11:04:04+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3","title":"Qwen3.8-Flash-Next NVFP4 on RTX PRO 6000 Blackwell \u2014 TP1 MTP3","summary":"Single-card Blackwell recipe for Qwen3.8-Flash-Next (hybrid GDN/QSA MoE, 48 layers, 512 experts) on one 96 GB RTX PRO 6000: the Community image community image R34 (localinferencelab/vllm, v0.26.1rc0+glm53.r34), TP1 with MTP3 speculative decoding (b12x MoE backend), 256k context. Benchmark-qualified on RTX PRO 6000 Blackwell Max-Q: decode 148 tok/s at ctx-0 (temp 1), prefill 3860 tok/s at 2048 batched tokens. Launch argv transcribed verbatim from the r35 community deployment. Weights: JMNI-Labs/Qwen3.8-Flash-Next-NVFP4-QAD5500-Hybrid \u2014 step-5500 QAD trained tensors recombined with main 7c4f1bc1 MXFP8 attention + NVFP4 MTP experts; same-stack measured +5% MTP-normalized decode, +25% with MTP vs the main checkpoint (evals within noise, MMLU-Pro 66.9 vs 67.2).","tier":"curated","beta":false,"generation":"blackwell","decode_tok_s":137.19999999999998863131622783839702606201171875,"prefill_tok_s":6627.3000000000001818989403545856475830078125,"updated_at":"2026-10-29 23:29:38.177885-06","commit_sha":"28709ecba026aff18371426bc26025e5a4185209","content_hash":"1e37b0f978392ba766c01cfdf9ae9ee3e002ee84","votes":1,"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3"}},{"slug":"qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","title":"Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM","summary":"Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":197.80000000000001136868377216160297393798828125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.955646-06","commit_sha":"af7d7dde12376551476c8b9e36cd72c68dc32408","content_hash":"c82e80ffa9955d905b61bda381cf7d89db1350ba","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/Mia-AiLab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json","alternate":"/r/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm"}},{"slug":"qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode","title":"Qwen3.8-27B NVFP4A4-AWQ 1x RTX PRO 6000 SGLang Fast Decode","summary":"Recipe by AppNetDeveloper. ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ on 1x RTX PRO 6000 Blackwell (SM120) with stock SGLang 0.5.22 plus engine overlays, no fork: FR-Spec pruned draft lm_head for a quantized FP8 head (slices weight AND per-row scale along vocab, bit-identical kept rows); KDA NVFP4 SM120 GEMM allowlist extension (KN=5120x34816,17408x5120) 85-89% DRAM bandwidth at M<=16 vs 50-70% default; tuned triton_scaled_mm fp8_w8a8_channelwise JSONs. On 0.5.22 the overlays land as ONE patch, patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1). NEXTN/MTP, FP8_e4m3 KV cache, HiCache, 40k hot-token draft list; hybrid GDN+MTP needs --enable-linear-replayssm-spec. Author bench 2026-10-14 (400 output tokens, unique prefixes): 131.7 tok/s 1req 45k ctx vs 98.5 (+34%); 4req@45k +13.5%; 8req@27k +3%; 1req@141k +24%; same quality, no extra memory. MTP5 + FR-Spec is the author production default; measured on real agent text: 149 vs 140 @1req, 117 vs 103 @2req, 77 vs 69 @4req over MTP4.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":131.69999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-28 16:25:30.123672-06","commit_sha":"607c8d8babf1cd4237a56128e0614aceb2176332","content_hash":"ba21205503ad8f52405ab4becff180d4e348b45f","votes":0,"credit":[{"name":"AppNetDeveloper","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode"},{"name":"SGLang 0.5.22 port + MTP5/FR-Spec real-text results @ c00cfec525 (published upstream 2026-10-21)","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode/commit/c00cfec525efb20e9c4185ca6c86e61a3235dbc1"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode.json","alternate":"/r/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode"}},{"slug":"qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang","title":"Qwen3.8-27B Bit-Exact Spec-Decode 1x RTX PRO 6000 SGLang","summary":"Recipe by bill-h-lin. Qwen3.8-27B BF16 (BF16 KV, temperature 0) on one RTX PRO 6000 Blackwell Max-Q (SM120) with SGLang made bit-exact under speculative decoding: verify GDN recurrence routed through the decode kernel (old verify kernel tiled state differently, kept beta fp32), verify attention run as decode rows, a DeepGEMM sm120 bf16 small-M GEMM fence for repeatability (upstream from 0.2.3), and upstream sgl-project/sglang#42145 routing small-M bf16 GEMMs to a bit-identical Triton tile to escape the DeepGEMM sm120 bandwidth cliff. Base: SGLang main 542a043 (2026-09-23), DeepGEMM 0.2.0. DFlash drafter (incoai/Qwen3.8-27B-DFlash2) at 8 draft tokens: 106.6 tok/s exact vs 109.8 non-exact vLLM short, 73.9 vs 67.4 at 2.2k, 42.6 vs 56.1 at 128k. Receipts: 24/24 token-id+logprob parity vs no-speculation (forced-reject, 8 in-flight, 128k); 0/1200 run-to-run divergences with the fence; divergences cluster where top-1/top-2 logits are within 0.25 (0.125 for backend/--max-num-seqs).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":73.900000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-27 20:30:16.519777-06","commit_sha":"d47db89f9ef31a0a96530e29d246e7f562509ee6","content_hash":"5fd4ef10f7e92f70e8b5831e5c33c58cfa0d635c","votes":0,"credit":[{"name":"bill-h-lin","url":"https://github.com/bill-h-lin/sglang-exact-spec-decode"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang.json","alternate":"/r/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang"}},{"slug":"qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix","title":"Qwen3.8-27B INT8 MTP3 prefix-cache on 1x CMP 170HX Hynix","summary":"Qwen3.8-27B at 65-67 tok/s decode (74 structured) with MTP k=3 speculative decoding AND prefix caching simultaneously on a single NVIDIA CMP 170HX 64GB (Hynix HBM2e, GA100/sm80), 262,144-token context on fp8_e4m3 KV, 24/7 production on pinned upstream vLLM nightly 0.26.1rc1.dev1024+g1fe3a1571 \u2014 the first build where prefix caching and MTP spec decode coexist (GDN mamba-align). Cached TTFT on a 7K shared prefix drops ~4.0 s -> 1.44 s (33.5% hit rate) while decode gains 7-9% over the 0.27.1 fork fallback \u2014 the 'MTP vs prefix cache, pick one' tradeoff is gone. Load-bearing trio --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 (default chunked-prefill batch = 0.0% hit rate while looking healthy); --prefix-match-unit 400 halves cached TTFT further (must divide the 1600-token GDN block); batch budget 32000 OOMs and hard-wedges the card. Samsung-40GB side recipe (int8 + DFlash2, 97.8 tok/s) with an sm80 int8-repack CPU-fallback patch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":68.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:22.585629-06","commit_sha":"ad3f55dab02cedfb2b187f7dcfc3f1647e0aaa0c","content_hash":"68f95048b88c132d1ddbffc8ceceaf5c38be5140","votes":0,"credit":[{"name":"ahnguyen17 (recipe, patches, probes, A/B benchmarks)","url":"https://github.com/ahnguyen17/cmp-170hx-vllm"},{"name":"lued (Qwen3.8-27B-INT8-W8A16-MTP checkpoint)","url":"https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP"},{"name":"vllm-project (vLLM upstream; #50021 GDN spec bounds, #45238 align-mode cache retention)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json","alternate":"/r/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h","title":"GLM-5.3-Flash EXL3 3.05bpw on 1x RTX PRO 6000 - GPU expert cache, host-RAM streaming","summary":"GLM-5.3-Flash (320B, 42x288-expert MoE) served from turboderp's EXL3 3.05bpw checkpoint (116.7 GiB) on ONE 96 GB RTX PRO 6000 Blackwell: a GPU expert cache keeps the 40 most-routed experts static in VRAM plus 148 LFU cache slots on-card, and streams the remaining 100 experts per layer from 91.5 GiB of pinned host RAM by rewriting ExLlamaV3's expert pointer tables in place - stock ExLlamaV3 1.6.0 kernels and unmodified TabbyAPI (pinned commit) behind a front shim. 61.5 tok/s decode short-context, 68 tok/s at 32K-127K in context, 1.9K-2.9K tok/s prefill, 524,288-token context, native vision, MTP drafting; mean KLD 0.0680 vs the original model and 59/60 on a 60-task graded tool-use bench - roughly half the llama.cpp UD-IQ3_XXS fidelity error (0.1258) while beating it on every axis. Replaces the author's own llama.cpp recipe in the same repo.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":64.7999999999999971578290569595992565155029296875,"prefill_tok_s":2736.09999999999990905052982270717620849609375,"updated_at":"2026-10-20 15:34:15.674077-06","commit_sha":"b273a7fb8e437bb6978e08e600c43acc1fbb6fda","content_hash":"9a80e9b5da56789795695cd17f70e97159a46050","votes":0,"credit":[{"name":"SSHdotCodes","url":"https://github.com/SSHdotCodes/glm-5.3-flash-pro6000"},{"name":"turboderp","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h"}},{"slug":"qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang","title":"Qwen3.8-Flash-Next NVFP4 RTX PRO 6000 Blackwell SM120 SGLang","summary":"Experimental SGLang fork for Qwen3.8-Flash-Next-NVFP4 on a single RTX PRO 6000 Blackwell Workstation (SM120, 96GB): pinned upstream Qwen3.8 (PR #36497 @ 73a255206f), NVMe-backed PLE (PR #36567 @ e14d1c3cb6/405e494199/2179ad9bcb/d4477bd298), SM120 QSA sparse decode (PR #36556, snapshot 93fd20af6f - extends the sparse GQA decode path from SM121 DOWN to SM120 so no incompatible fallback fires) + repo additions: (1) breakable-CUDA-graph dataclass outputs (recursive alloc/copy/slice so a 1-request speculative target-verify graph safely holds 4 rows of verify logits + hidden states; CPU unit coverage), (2) hot-PLE row gathering (all-hit LRU fast path avoiding page plans + duplicate-row reuse for repetitive batches; commit 7df4aea02b), (3) GDN projection + Conv1D decode fusion (commit b2f0b02763, SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV=1 default), (4) QSA indexer query-tile tuning to 8 rows for SM120 w/ 4 index heads dim 128 + text RoPE bound from known CPU seq lens (commit 2b56a39e70). Meas","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":224.099999999999994315658113919198513031005859375,"prefill_tok_s":13460.70000000000072759576141834259033203125,"updated_at":"2026-10-20 15:34:12.156597-06","commit_sha":"3b38bdadad8817372ec43bc98243ffb2e37523d6","content_hash":"372a4539d8514d0fbc2308f988545bff470cf6ae","votes":0,"credit":[{"name":"yhfgyyf","url":"https://github.com/yhfgyyf/sglang-qwen38-flash-next-sm120"},{"name":"sgl-project (SGLang upstream + PRs 36497/36567/36556)","url":"https://github.com/sgl-project/sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang.json","alternate":"/r/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang"}},{"slug":"qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple","title":"Qwen3.8-Flash-Next NVFP4 RTX PRO 6000 Blackwell SGLang NVMe PLE","summary":"Single-card RTX PRO 6000 Blackwell (SM120, 96GB) recipe for Qwen3.8-Flash-Next on SGLang: the 51B N-gram PLE table is read straight off NVMe with io_uring as tokens need it (measured gather 0.7-2.7 ms, under 2% of decode budget) instead of landing in the 95GB host-RAM PLE worker the official recipes assume - that is what leaves the GPU free for 78.2GB weights plus a real 233,856-token FP8 KV cache. Build: lmsysorg/sglang:qwen38flashnext day-0 image + source overlay of jzinno/sglang @ d4477bd298 (tree head of PR #36567 NVMe PLE streaming stacked on #36497 model support) + cherry-picked PRs 36556 (SM120/SM121 sparse decode - without it the QSA kernel MLIRErrors at startup), 36749 (BCG buffers sized by token rows - the ONLY way breakable CUDA graphs compose with NEXTN speculation; took single-stream decode 57.6 -> 167 tok/s), 36750 (max_thinking_tokens on the OpenAI endpoint) + local patches/0001-qsa-fp8-kv-dequant-on-read.patch (chunked-prompt FP8 KV crash fix; upstream #36644 preferred","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":180.900000000000005684341886080801486968994140625,"prefill_tok_s":9692.5,"updated_at":"2026-10-20 15:34:11.662562-06","commit_sha":"783bd7b94154dd45b3903efa0785c087fe7b4fb7","content_hash":"58c9d00a88399c3296fc8da5d7397a97ae63a3ed","votes":0,"credit":[{"name":"yepapa-nest","url":"https://github.com/yepapa-nest/qwen38-flashnext-rtx6000"},{"name":"jzinno (NVMe PLE streaming PR #36567 source tree)","url":"https://github.com/jzinno/sglang"},{"name":"sgl-project (SGLang upstream + PRs 36556/36749/36750)","url":"https://github.com/sgl-project/sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple.json","alternate":"/r/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple"}},{"slug":"qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n","title":"Qwen3.8-Flash-Next 3bpw MTP on 1x CMP 170HX SSD PLE AIO (iIIusi0n)","summary":"First single-CMP-170HX recipe for Qwen3.8-Flash-Next: the AutoRound 3bpw+MTP checkpoint on ONE 64 GiB SM80 GPU with the 95.37 GiB BF16 PLE table left ON the SSD \u2014 native Linux AIO async reads, 16K prompt read-ahead and a 512 MiB row cache stream it per-row while the model loads metadata-only, so the 176B MoE fits one card. Patch adds async PLE reads/read-ahead, metadata-only loading, CUDA-graph host breaks, mixed-bit expert handling and MTP layer remapping; KLD 0.1208 vs original weights. Three-round medians (128 output tokens, temp 0, thinking off, EOS ignored): 8K prefill 2,636 tok/s, single-request decode 111 tok/s, 667 tok/s aggregate at 16 concurrent. Honest state: 31 SSD/graph tests PASS, 134 requests no API errors, BUT an arithmetic-consistency issue remains open; context advertised 262K, timing tested only through 32K. Tested sparse (15 GiB RAM, 31 GiB swap) on ENTERPRISE SATA at 180 W. Documented swap: the orcarouter uncensored AutoRound 3bpw checkpoint runs the same stack.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":111,"prefill_tok_s":2132,"updated_at":"2026-10-08 21:11:56.66944-06","commit_sha":"827bfe01549b43357fa06a8610ea5e64b55583b0","content_hash":"3a31e6a965360ba192fd941458c89f62f07e27e6","votes":0,"credit":[{"name":"iIIusi0n (recipe, vLLM ple-ssd patch set, AIO helper, benchmark harness)","url":"https://github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx"},{"name":"Qwen (official Qwen3.8-Flash-Next base, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"klee100 (AutoRound 3bpw+MTP quantized checkpoints, official + uncensored)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"OrcaRouter (Qwen3.8-Flash-Next-Uncensored BF16 abliterated fine-tune)","url":"https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored"},{"name":"vLLM project (upstream vLLM @ a5a30471 the patch applies to)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n.json","alternate":"/r/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n"}}]}