{"meta":{"count":9,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["rtx-pro-6000-blackwell","rtx-3090"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T13:37:34+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3","title":"Qwen3.8-Flash-Next NVFP4 on RTX PRO 6000 Blackwell \u2014 TP1 MTP3","summary":"Single-card Blackwell recipe for Qwen3.8-Flash-Next (hybrid GDN/QSA MoE, 48 layers, 512 experts) on one 96 GB RTX PRO 6000: the Community image community image R34 (localinferencelab/vllm, v0.26.1rc0+glm53.r34), TP1 with MTP3 speculative decoding (b12x MoE backend), 256k context. Benchmark-qualified on RTX PRO 6000 Blackwell Max-Q: decode 148 tok/s at ctx-0 (temp 1), prefill 3860 tok/s at 2048 batched tokens. Launch argv transcribed verbatim from the r35 community deployment. Weights: JMNI-Labs/Qwen3.8-Flash-Next-NVFP4-QAD5500-Hybrid \u2014 step-5500 QAD trained tensors recombined with main 7c4f1bc1 MXFP8 attention + NVFP4 MTP experts; same-stack measured +5% MTP-normalized decode, +25% with MTP vs the main checkpoint (evals within noise, MMLU-Pro 66.9 vs 67.2).","tier":"curated","beta":false,"generation":"blackwell","decode_tok_s":137.19999999999998863131622783839702606201171875,"prefill_tok_s":6627.3000000000001818989403545856475830078125,"updated_at":"2026-10-29 23:29:38.177885-06","commit_sha":"28709ecba026aff18371426bc26025e5a4185209","content_hash":"1e37b0f978392ba766c01cfdf9ae9ee3e002ee84","votes":1,"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin","title":"GLM-5.3-Flash EXL3 3.05bpw on 1x RTX 3090 \u2014 RAM/NVMe expert tiering (near-exact)","summary":"320B MoE served on ONE 24 GB RTX 3090 with stock exllamav3 1.5.1 plus process-local monkeypatches: the 12,096 routed experts (42 MoE layers x 288, 8 picked per token per layer) tier across VRAM, pinned host RAM and a packed NVMe store. An elastic VRAM cache holds the ~1,300 hottest experts (12.4 GB); evicted experts move into a pinned no-duplicates RAM tier sized from the container cap; a 117 GB O_DIRECT record store covers the rest, with layer-ahead prefetch, a victim write-back ring and an AVX2 CPU lane computing RAM-resident decode picks in place. Modes: fast (all experts in RAM, ~218 GiB free, 28.2 tok/s, KL 0.005), nvme (55 GiB cap, 19.5 tok/s, KL 0.008; 16 GiB cap 14.7 lab / 14.38 sweep), nvme-exact (bit-exact, 8.3 tok/s), and an experimental Intel Arc Pro B70 peer-offload tier (GLM53_B70=1, 26.5 tok/s, KL 0.003 - not lab-run). 131,072-token KV, OpenAI API on port 30000, model id glm-5.3-flash, no output caps. Registry lab six gates pass at 55 GiB (PR #199); 16 GiB reported.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":24.10000000000000142108547152020037174224853515625,"prefill_tok_s":1015,"updated_at":"2026-10-29 21:32:24.549403-06","commit_sha":"53520bbf5e06f65db6670e8693efb018f75026c8","content_hash":"65ab07a0787590e7c21331b006d4a13553080d02","votes":0,"credit":[{"name":"sybil-solutions (image, tiering runtime, omarchy plugin)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"turboderp (ExLlamaV3 + EXL3 3.05bpw checkpoint)","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin"}},{"slug":"qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode","title":"Qwen3.8-27B NVFP4A4-AWQ 1x RTX PRO 6000 SGLang Fast Decode","summary":"Recipe by AppNetDeveloper. ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ on 1x RTX PRO 6000 Blackwell (SM120) with stock SGLang 0.5.22 plus engine overlays, no fork: FR-Spec pruned draft lm_head for a quantized FP8 head (slices weight AND per-row scale along vocab, bit-identical kept rows); KDA NVFP4 SM120 GEMM allowlist extension (KN=5120x34816,17408x5120) 85-89% DRAM bandwidth at M<=16 vs 50-70% default; tuned triton_scaled_mm fp8_w8a8_channelwise JSONs. On 0.5.22 the overlays land as ONE patch, patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1). NEXTN/MTP, FP8_e4m3 KV cache, HiCache, 40k hot-token draft list; hybrid GDN+MTP needs --enable-linear-replayssm-spec. Author bench 2026-10-14 (400 output tokens, unique prefixes): 131.7 tok/s 1req 45k ctx vs 98.5 (+34%); 4req@45k +13.5%; 8req@27k +3%; 1req@141k +24%; same quality, no extra memory. MTP5 + FR-Spec is the author production default; measured on real agent text: 149 vs 140 @1req, 117 vs 103 @2req, 77 vs 69 @4req over MTP4.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":131.69999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-28 16:25:30.123672-06","commit_sha":"607c8d8babf1cd4237a56128e0614aceb2176332","content_hash":"ba21205503ad8f52405ab4becff180d4e348b45f","votes":0,"credit":[{"name":"AppNetDeveloper","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode"},{"name":"SGLang 0.5.22 port + MTP5/FR-Spec real-text results @ c00cfec525 (published upstream 2026-10-21)","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode/commit/c00cfec525efb20e9c4185ca6c86e61a3235dbc1"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode.json","alternate":"/r/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode"}},{"slug":"qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang","title":"Qwen3.8-27B Bit-Exact Spec-Decode 1x RTX PRO 6000 SGLang","summary":"Recipe by bill-h-lin. Qwen3.8-27B BF16 (BF16 KV, temperature 0) on one RTX PRO 6000 Blackwell Max-Q (SM120) with SGLang made bit-exact under speculative decoding: verify GDN recurrence routed through the decode kernel (old verify kernel tiled state differently, kept beta fp32), verify attention run as decode rows, a DeepGEMM sm120 bf16 small-M GEMM fence for repeatability (upstream from 0.2.3), and upstream sgl-project/sglang#42145 routing small-M bf16 GEMMs to a bit-identical Triton tile to escape the DeepGEMM sm120 bandwidth cliff. Base: SGLang main 542a043 (2026-09-23), DeepGEMM 0.2.0. DFlash drafter (incoai/Qwen3.8-27B-DFlash2) at 8 draft tokens: 106.6 tok/s exact vs 109.8 non-exact vLLM short, 73.9 vs 67.4 at 2.2k, 42.6 vs 56.1 at 128k. Receipts: 24/24 token-id+logprob parity vs no-speculation (forced-reject, 8 in-flight, 128k); 0/1200 run-to-run divergences with the fence; divergences cluster where top-1/top-2 logits are within 0.25 (0.125 for backend/--max-num-seqs).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":73.900000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-27 20:30:16.519777-06","commit_sha":"d47db89f9ef31a0a96530e29d246e7f562509ee6","content_hash":"5fd4ef10f7e92f70e8b5831e5c33c58cfa0d635c","votes":0,"credit":[{"name":"bill-h-lin","url":"https://github.com/bill-h-lin/sglang-exact-spec-decode"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang.json","alternate":"/r/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang"}},{"slug":"qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4","title":"Qwen3.8-27B AWQ-INT4 + DSpark on RTX 3090s \u2014 SGLang TP1/TP2/TP4","summary":"Validated SGLang serving for the dense Qwen3.8-27B hybrid GDN vision-language model (48 Gated-DeltaNet + 16 gated-attention layers) on RTX 3090 24 GB: one digest-pinned container whose TP env selects tuned 1x/2x/4x presets. AWQ-INT4 W4A16 (Marlin) beats FP8 on Ampere; DSpark speculative decoding (RadixArk 1.36B draft) on 2x/4x with auto-fallback on TP1. Image input everywhere, video on 2x/4x. Author-measured: 136.8 tok/s decode at 8k on 4x (169.8 aggregate at C8), 112.9 on 2x, 43.2 on 1x; up to 128k context on 4x.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":127.099999999999994315658113919198513031005859375,"prefill_tok_s":1372.700000000000045474735088646411895751953125,"updated_at":"2026-10-20 21:52:36.534482-06","commit_sha":"26a1363966d9065a310e9ec7caf124206a25f3f2","content_hash":"3fa45739100d87065553d5fb598424a66d4742fb","votes":0,"credit":[{"name":"sybil-solutions / 0xSero (SGLang presets, image build, benchmarks)","url":"https://github.com/sybil-solutions/qwen-serve"},{"name":"cyankiwi (AWQ-INT4 W4A16 checkpoint)","url":"https://huggingface.co/cyankiwi/Qwen3.8-27B-AWQ-INT4"},{"name":"RadixArk (DSpark speculative drafter)","url":"https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4.json","alternate":"/r/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h","title":"GLM-5.3-Flash EXL3 3.05bpw on 1x RTX PRO 6000 - GPU expert cache, host-RAM streaming","summary":"GLM-5.3-Flash (320B, 42x288-expert MoE) served from turboderp's EXL3 3.05bpw checkpoint (116.7 GiB) on ONE 96 GB RTX PRO 6000 Blackwell: a GPU expert cache keeps the 40 most-routed experts static in VRAM plus 148 LFU cache slots on-card, and streams the remaining 100 experts per layer from 91.5 GiB of pinned host RAM by rewriting ExLlamaV3's expert pointer tables in place - stock ExLlamaV3 1.6.0 kernels and unmodified TabbyAPI (pinned commit) behind a front shim. 61.5 tok/s decode short-context, 68 tok/s at 32K-127K in context, 1.9K-2.9K tok/s prefill, 524,288-token context, native vision, MTP drafting; mean KLD 0.0680 vs the original model and 59/60 on a 60-task graded tool-use bench - roughly half the llama.cpp UD-IQ3_XXS fidelity error (0.1258) while beating it on every axis. Replaces the author's own llama.cpp recipe in the same repo.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":64.7999999999999971578290569595992565155029296875,"prefill_tok_s":2736.09999999999990905052982270717620849609375,"updated_at":"2026-10-20 15:34:15.674077-06","commit_sha":"b273a7fb8e437bb6978e08e600c43acc1fbb6fda","content_hash":"9a80e9b5da56789795695cd17f70e97159a46050","votes":0,"credit":[{"name":"SSHdotCodes","url":"https://github.com/SSHdotCodes/glm-5.3-flash-pro6000"},{"name":"turboderp","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h"}},{"slug":"qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang","title":"Qwen3.8-Flash-Next NVFP4 RTX PRO 6000 Blackwell SM120 SGLang","summary":"Experimental SGLang fork for Qwen3.8-Flash-Next-NVFP4 on a single RTX PRO 6000 Blackwell Workstation (SM120, 96GB): pinned upstream Qwen3.8 (PR #36497 @ 73a255206f), NVMe-backed PLE (PR #36567 @ e14d1c3cb6/405e494199/2179ad9bcb/d4477bd298), SM120 QSA sparse decode (PR #36556, snapshot 93fd20af6f - extends the sparse GQA decode path from SM121 DOWN to SM120 so no incompatible fallback fires) + repo additions: (1) breakable-CUDA-graph dataclass outputs (recursive alloc/copy/slice so a 1-request speculative target-verify graph safely holds 4 rows of verify logits + hidden states; CPU unit coverage), (2) hot-PLE row gathering (all-hit LRU fast path avoiding page plans + duplicate-row reuse for repetitive batches; commit 7df4aea02b), (3) GDN projection + Conv1D decode fusion (commit b2f0b02763, SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV=1 default), (4) QSA indexer query-tile tuning to 8 rows for SM120 w/ 4 index heads dim 128 + text RoPE bound from known CPU seq lens (commit 2b56a39e70). Meas","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":224.099999999999994315658113919198513031005859375,"prefill_tok_s":13460.70000000000072759576141834259033203125,"updated_at":"2026-10-20 15:34:12.156597-06","commit_sha":"3b38bdadad8817372ec43bc98243ffb2e37523d6","content_hash":"372a4539d8514d0fbc2308f988545bff470cf6ae","votes":0,"credit":[{"name":"yhfgyyf","url":"https://github.com/yhfgyyf/sglang-qwen38-flash-next-sm120"},{"name":"sgl-project (SGLang upstream + PRs 36497/36567/36556)","url":"https://github.com/sgl-project/sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang.json","alternate":"/r/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sm120-sglang"}},{"slug":"qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple","title":"Qwen3.8-Flash-Next NVFP4 RTX PRO 6000 Blackwell SGLang NVMe PLE","summary":"Single-card RTX PRO 6000 Blackwell (SM120, 96GB) recipe for Qwen3.8-Flash-Next on SGLang: the 51B N-gram PLE table is read straight off NVMe with io_uring as tokens need it (measured gather 0.7-2.7 ms, under 2% of decode budget) instead of landing in the 95GB host-RAM PLE worker the official recipes assume - that is what leaves the GPU free for 78.2GB weights plus a real 233,856-token FP8 KV cache. Build: lmsysorg/sglang:qwen38flashnext day-0 image + source overlay of jzinno/sglang @ d4477bd298 (tree head of PR #36567 NVMe PLE streaming stacked on #36497 model support) + cherry-picked PRs 36556 (SM120/SM121 sparse decode - without it the QSA kernel MLIRErrors at startup), 36749 (BCG buffers sized by token rows - the ONLY way breakable CUDA graphs compose with NEXTN speculation; took single-stream decode 57.6 -> 167 tok/s), 36750 (max_thinking_tokens on the OpenAI endpoint) + local patches/0001-qsa-fp8-kv-dequant-on-read.patch (chunked-prompt FP8 KV crash fix; upstream #36644 preferred","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":180.900000000000005684341886080801486968994140625,"prefill_tok_s":9692.5,"updated_at":"2026-10-20 15:34:11.662562-06","commit_sha":"783bd7b94154dd45b3903efa0785c087fe7b4fb7","content_hash":"58c9d00a88399c3296fc8da5d7397a97ae63a3ed","votes":0,"credit":[{"name":"yepapa-nest","url":"https://github.com/yepapa-nest/qwen38-flashnext-rtx6000"},{"name":"jzinno (NVMe PLE streaming PR #36567 source tree)","url":"https://github.com/jzinno/sglang"},{"name":"sgl-project (SGLang upstream + PRs 36556/36749/36750)","url":"https://github.com/sgl-project/sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple.json","alternate":"/r/qwen3-8-flash-next-nvfp4-rtx-pro-6000-blackwell-sglang-nvme-ple"}},{"slug":"qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts","title":"Qwen3.8-Flash-Next W4A16-FP8PLE on 1x RTX 3090 \u2014 CPU cold experts, MTP3, 128K (DominikBucko)","summary":"Single-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in the albucino W4A16-FP8PLE build (@ ef55414, same checkpoint as the 2x3090 runtime): the GPU holds attention + dense weights + the 32 busiest experts per layer, a pinned AVX2 CPU thread pool computes the other 480 experts straight from RAM during decode (GPU takes ~22% over PCIe), and prefill streams all experts through the GPU in 8,192-token chunks. INT8 KV cache fits the full 135,168-token context in 3.05 GB; MTP 3-token draft; PLE table read in place from NVMe. The most common local-AI machine \u2014 one 24 GB card + 64 GB RAM. Author-measured (submitter T0, release image): 131,099-token prompt reads at 2,106 tok/s (62 s to first token) then decodes 47.4 tok/s; 43-51 tok/s decode range on 32 cores; 32K prompt at 2,141-2,219 tok/s prefill. Same vLLM runtime as the 2x3090 recipe, repackaged for half the hardware, nothing pruned. Argv transcribed verbatim from scripts/serve-container.sh @ 10b24c2.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":48.89999999999999857891452847979962825775146484375,"prefill_tok_s":2123.5,"updated_at":"2026-10-20 15:34:09.15642-06","commit_sha":"088542fd7b452ae34d7f6b0253b939a3ef6b350d","content_hash":"a354418ea2f8a6dbcef09d963e5901758b7dd549","votes":0,"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/qwen38-flash-next-3090"},{"name":"albucino (checkpoint)","url":"https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts.json","alternate":"/r/qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts"}}]}