{"meta":{"count":7,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["170hx-x6","rtx-pro-6000-blackwell"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T14:28:32+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3","title":"Qwen3.8-Flash-Next NVFP4 on RTX PRO 6000 Blackwell \u2014 TP1 MTP3","summary":"Single-card Blackwell recipe for Qwen3.8-Flash-Next (hybrid GDN/QSA MoE, 48 layers, 512 experts) on one 96 GB RTX PRO 6000: the Community image community image R34 (localinferencelab/vllm, v0.26.1rc0+glm53.r34), TP1 with MTP3 speculative decoding (b12x MoE backend), 256k context. Benchmark-qualified on RTX PRO 6000 Blackwell Max-Q: decode 148 tok/s at ctx-0 (temp 1), prefill 3860 tok/s at 2048 batched tokens. Launch argv transcribed verbatim from the r35 community deployment. Weights: JMNI-Labs/Qwen3.8-Flash-Next-NVFP4-QAD5500-Hybrid \u2014 step-5500 QAD trained tensors recombined with main 7c4f1bc1 MXFP8 attention + NVFP4 MTP experts; same-stack measured +5% MTP-normalized decode, +25% with MTP vs the main checkpoint (evals within noise, MMLU-Pro 66.9 vs 67.2).","tier":"curated","beta":false,"generation":"blackwell","decode_tok_s":137.19999999999998863131622783839702606201171875,"prefill_tok_s":6627.3000000000001818989403545856475830078125,"updated_at":"2026-10-29 23:29:38.177885-06","commit_sha":"28709ecba026aff18371426bc26025e5a4185209","content_hash":"1e37b0f978392ba766c01cfdf9ae9ee3e002ee84","votes":1,"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3"}},{"slug":"qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode","title":"Qwen3.8-27B NVFP4A4-AWQ 1x RTX PRO 6000 SGLang Fast Decode","summary":"Recipe by AppNetDeveloper. ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ on 1x RTX PRO 6000 Blackwell (SM120) with stock SGLang 0.5.22 plus engine overlays, no fork: FR-Spec pruned draft lm_head for a quantized FP8 head (slices weight AND per-row scale along vocab, bit-identical kept rows); KDA NVFP4 SM120 GEMM allowlist extension (KN=5120x34816,17408x5120) 85-89% DRAM bandwidth at M<=16 vs 50-70% default; tuned triton_scaled_mm fp8_w8a8_channelwise JSONs. On 0.5.22 the overlays land as ONE patch, patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1). NEXTN/MTP, FP8_e4m3 KV cache, HiCache, 40k hot-token draft list; hybrid GDN+MTP needs --enable-linear-replayssm-spec. Author bench 2026-10-14 (400 output tokens, unique prefixes): 131.7 tok/s 1req 45k ctx vs 98.5 (+34%); 4req@45k +13.5%; 8req@27k +3%; 1req@141k +24%; same quality, no extra memory. MTP5 + FR-Spec is the author production default; measured on real agent text: 149 vs 140 @1req, 117 vs 103 @2req, 77 vs 69 @4req over MTP4.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":131.69999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-28 16:25:30.123672-06","commit_sha":"607c8d8babf1cd4237a56128e0614aceb2176332","content_hash":"ba21205503ad8f52405ab4becff180d4e348b45f","votes":0,"credit":[{"name":"AppNetDeveloper","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode"},{"name":"SGLang 0.5.22 port + MTP5/FR-Spec real-text results @ c00cfec525 (published upstream 2026-10-21)","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode/commit/c00cfec525efb20e9c4185ca6c86e61a3235dbc1"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode.json","alternate":"/r/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode"}},{"slug":"deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5","title":"DeepSeek-V4.1-Flash SM80 DeepInfer kernel port 6x CMP 170HX DSpark K5","summary":"DeepSeek-V4.1-Flash served on six 64 GB SM80 GPUs (GA100 die, owner-classified 170HX) with HalfVulpes' Deepinfer-SM80 patch set: a 30-file vLLM V1 overlay porting DeepSeek's DeepInfer kernel architecture (SM100 tcgen05/TMEM) to Ampere mma.sync tensor cores - unified 512-dim SM80 sparse-MLA decode/prefill kernels with bitshift UE8M0 scale decoding, a bit-exact single-CTA Triton TopK+SWA combiner (22.17x), direct-store KV-cache gather (7.64x @48K), SM80-aligned TileLang mHC routing (2.89x), and YOCO L20-L39 ragged indexer-metadata reuse (19 redundant kernel chains eliminated per chunk). TP2 x PP3 (14/14/12) + DSpark K=5 + 1M context, 4.65M-token GPU-resident KV. Author-measured: 64K cold prefill 4,507 tok/s (+24.2% vs baseline vLLM SM80), 64K TTFT 10.7 s (-18.9%), c1 DSpark decode 160.8 tok/s code / 188.8 tok/s 16K+ long-gen, c4 170.5 aggregate. Microbenchmarks bit-exact (0 diff) or within BF16 noise. Topology guide: PP6 = 2.1x slower single-stream; TP2+TP4 asymmetric hurts Marlin tiles.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":160.80000000000001136868377216160297393798828125,"prefill_tok_s":4506.8000000000001818989403545856475830078125,"updated_at":"2026-10-28 15:48:17.404025-06","commit_sha":"ced28259b90a43ec103ecd450b68455a56d2808d","content_hash":"e00265eca8b84734c2c7de2b67d558b0d702d25e","votes":0,"credit":[{"name":"HalfVulpes (Deepinfer-SM80 port + measurements)","url":"https://github.com/HalfVulpes/Deepinfer-SM80"},{"name":"DeepSeek-AI (DeepInfer engine architecture + DeepSeek-V4.1-Flash checkpoint)","url":"https://github.com/deepseek-ai/DeepInfer"},{"name":"vLLM project (V1 engine + TPxPP runtime)","url":"https://github.com/vllm-project/vllm"},{"name":"TileLang + OpenAI Triton (SM80 kernel synthesis)","url":"https://github.com/tile-ai/tilelang"},{"name":"FlashInfer + IST-DASLab Marlin (sparse-MLA + MXFP4/W4A16 Ampere GEMM primitives)","url":"https://github.com/flashinfer-ai/flashinfer"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5.json","alternate":"/r/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5"}},{"slug":"qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang","title":"Qwen3.8-27B Bit-Exact Spec-Decode 1x RTX PRO 6000 SGLang","summary":"Recipe by bill-h-lin. Qwen3.8-27B BF16 (BF16 KV, temperature 0) on one RTX PRO 6000 Blackwell Max-Q (SM120) with SGLang made bit-exact under speculative decoding: verify GDN recurrence routed through the decode kernel (old verify kernel tiled state differently, kept beta fp32), verify attention run as decode rows, a DeepGEMM sm120 bf16 small-M GEMM fence for repeatability (upstream from 0.2.3), and upstream sgl-project/sglang#42145 routing small-M bf16 GEMMs to a bit-identical Triton tile to escape the DeepGEMM sm120 bandwidth cliff. Base: SGLang main 542a043 (2026-09-23), DeepGEMM 0.2.0. DFlash drafter (incoai/Qwen3.8-27B-DFlash2) at 8 draft tokens: 106.6 tok/s exact vs 109.8 non-exact vLLM short, 73.9 vs 67.4 at 2.2k, 42.6 vs 56.1 at 128k. Receipts: 24/24 token-id+logprob parity vs no-speculation (forced-reject, 8 in-flight, 128k); 0/1200 run-to-run divergences with the fence; divergences cluster where top-1/top-2 logits are within 0.25 (0.125 for backend/--max-num-seqs).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":73.900000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-27 20:30:16.519777-06","commit_sha":"d47db89f9ef31a0a96530e29d246e7f562509ee6","content_hash":"5fd4ef10f7e92f70e8b5831e5c33c58cfa0d635c","votes":0,"credit":[{"name":"bill-h-lin","url":"https://github.com/bill-h-lin/sglang-exact-spec-decode"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang.json","alternate":"/r/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h","title":"GLM-5.3-Flash EXL3 3.05bpw on 1x RTX PRO 6000 - GPU expert cache, host-RAM streaming","summary":"GLM-5.3-Flash (320B, 42x288-expert MoE) served from turboderp's EXL3 3.05bpw checkpoint (116.7 GiB) on ONE 96 GB RTX PRO 6000 Blackwell: a GPU expert cache keeps the 40 most-routed experts static in VRAM plus 148 LFU cache slots on-card, and streams the remaining 100 experts per layer from 91.5 GiB of pinned host RAM by rewriting ExLlamaV3's expert pointer tables in place - stock ExLlamaV3 1.6.0 kernels and unmodified TabbyAPI (pinned commit) behind a front shim. 61.5 tok/s decode short-context, 68 tok/s at 32K-127K in context, 1.9K-2.9K tok/s prefill, 524,288-token context, native vision, MTP drafting; mean KLD 0.0680 vs the original model and 59/60 on a 60-task graded tool-use bench - roughly half the llama.cpp UD-IQ3_XXS fidelity error (0.1258) while beating it on every axis. Replaces the author's own llama.cpp recipe in the same repo.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":64.7999999999999971578290569595992565155029296875,"prefill_tok_s":2736.09999999999990905052982270717620849609375,"updated_at":"2026-10-20 15:34:15.674077-06","commit_sha":"b273a7fb8e437bb6978e08e600c43acc1fbb6fda","content_hash":"9a80e9b5da56789795695cd17f70e97159a46050","votes":0,"credit":[{"name":"SSHdotCodes","url":"https://github.com/SSHdotCodes/glm-5.3-flash-pro6000"},{"name":"turboderp","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h"}},{"slug":"qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang","title":"Qwen3.8-Flash-Next NVFP4 RTX PRO 6000 Blackwell SM120 SGLang","summary":"Experimental SGLang fork for Qwen3.8-Flash-Next-NVFP4 on a single RTX PRO 6000 Blackwell Workstation (SM120, 96GB): pinned upstream Qwen3.8 (PR #36497 @ 73a255206f), NVMe-backed PLE (PR #36567 @ e14d1c3cb6/405e494199/2179ad9bcb/d4477bd298), SM120 QSA sparse decode (PR #36556, snapshot 93fd20af6f - extends the sparse GQA decode path from SM121 DOWN to SM120 so no incompatible fallback fires) + repo additions: (1) breakable-CUDA-graph dataclass outputs (recursive alloc/copy/slice so a 1-request speculative target-verify graph safely holds 4 rows of verify logits + hidden states; CPU unit coverage), (2) hot-PLE row gathering (all-hit LRU fast path avoiding page plans + duplicate-row reuse for repetitive batches; commit 7df4aea02b), (3) GDN projection + Conv1D decode fusion (commit b2f0b02763, SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV=1 default), (4) QSA indexer query-tile tuning to 8 rows for SM120 w/ 4 index heads dim 128 + text RoPE bound from known CPU seq lens (commit 2b56a39e70). Meas","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":224.099999999999994315658113919198513031005859375,"prefill_tok_s":13460.70000000000072759576141834259033203125,"updated_at":"2026-10-20 15:34:12.156597-06","commit_sha":"3b38bdadad8817372ec43bc98243ffb2e37523d6","content_hash":"372a4539d8514d0fbc2308f988545bff470cf6ae","votes":0,"credit":[{"name":"yhfgyyf","url":"https://github.com/yhfgyyf/sglang-qwen38-flash-next-sm120"},{"name":"sgl-project (SGLang upstream + PRs 36497/36567/36556)","url":"https://github.com/sgl-project/sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang.json","alternate":"/r/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang"}},{"slug":"qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple","title":"Qwen3.8-Flash-Next NVFP4 RTX PRO 6000 Blackwell SGLang NVMe PLE","summary":"Single-card RTX PRO 6000 Blackwell (SM120, 96GB) recipe for Qwen3.8-Flash-Next on SGLang: the 51B N-gram PLE table is read straight off NVMe with io_uring as tokens need it (measured gather 0.7-2.7 ms, under 2% of decode budget) instead of landing in the 95GB host-RAM PLE worker the official recipes assume - that is what leaves the GPU free for 78.2GB weights plus a real 233,856-token FP8 KV cache. Build: lmsysorg/sglang:qwen38flashnext day-0 image + source overlay of jzinno/sglang @ d4477bd298 (tree head of PR #36567 NVMe PLE streaming stacked on #36497 model support) + cherry-picked PRs 36556 (SM120/SM121 sparse decode - without it the QSA kernel MLIRErrors at startup), 36749 (BCG buffers sized by token rows - the ONLY way breakable CUDA graphs compose with NEXTN speculation; took single-stream decode 57.6 -> 167 tok/s), 36750 (max_thinking_tokens on the OpenAI endpoint) + local patches/0001-qsa-fp8-kv-dequant-on-read.patch (chunked-prompt FP8 KV crash fix; upstream #36644 preferred","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":180.900000000000005684341886080801486968994140625,"prefill_tok_s":9692.5,"updated_at":"2026-10-20 15:34:11.662562-06","commit_sha":"783bd7b94154dd45b3903efa0785c087fe7b4fb7","content_hash":"58c9d00a88399c3296fc8da5d7397a97ae63a3ed","votes":0,"credit":[{"name":"yepapa-nest","url":"https://github.com/yepapa-nest/qwen38-flashnext-rtx6000"},{"name":"jzinno (NVMe PLE streaming PR #36567 source tree)","url":"https://github.com/jzinno/sglang"},{"name":"sgl-project (SGLang upstream + PRs 36556/36749/36750)","url":"https://github.com/sgl-project/sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple.json","alternate":"/r/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple"}}]}