{"meta":{"count":3,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["rtx-3090-x4","170hx-x6"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T18:55:31+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f","title":"Qwen3.8-Flash-Next W4A16-Merlin on 4x RTX 3090 \u2014 TP2xPP2+EP, MTP3, FP8 KV, 925K pool (halt95)","summary":"Four-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in halt95's W4A16-Merlin build (Intel AutoRound INT4 + FP8 PLE + MTP draft, @ 17284bf): TP2 x PP2 with expert parallel, manual layer partition 25,23, FP8 KV with a 924,993-token pool (three full 262K sessions resident; block-size 4096, prefix-match-unit 64), MTP 3-token probabilistic draft with RecoverSSM, host-mapped PLE pull transport, host-resident embeddings, side-cache layout + Model Runner V2, 42 images/request. Agent-focused: 92.5% of follow-up prompt tokens from cache across 8 sessions, median follow-up first token 1.61 s (0.96 s solo). Maintainer-measured (submitter T0, v2.5.1 bench card 2026-10-06): 5,219-5,533 tok/s cold prefill 10K-261K, 158-174 tok/s single-stream decode (thinking on), 534 tok/s aggregate at 8 concurrent, 262K needle exact, GSM8K 198/200. Engine is a SOURCE BUILD: vLLM v0.30.0 + 97 commits (release bundle verified by commit AND tree hash); argv verbatim from scripts/serve-v2.5.sh @ 28cbb72.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":168.099999999999994315658113919198513031005859375,"prefill_tok_s":5360,"updated_at":"2026-10-31 11:51:35.845609-06","commit_sha":"423a58839f188df961890110971ddb250e6fe6b5","content_hash":"b1b20a73286839a8b304e2a3bcb8ee2b35fd43e2","votes":0,"credit":[{"name":"halt95","url":"https://github.com/halt95/merlin-engine"},{"name":"Qwen (Qwen3.8-Flash-Next)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"Intel AutoRound (INT4 experts)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"RadixArk (FP8 PLE table)","url":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4"},{"name":"DominikBucko (MTP INT4 packing recipe)","url":"https://github.com/DominikBucko/qwen38-flash-next-2x3090"},{"name":"peakcrosser7 (Flash-Next vLLM branch)","url":"https://github.com/peakcrosser7/vllm/commits/release/qwen38next_offload"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f.json","alternate":"/r/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f"}},{"slug":"deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5","title":"DeepSeek-V4.1-Flash SM80 DeepInfer kernel port 6x CMP 170HX DSpark K5","summary":"DeepSeek-V4.1-Flash served on six 64 GB SM80 GPUs (GA100 die, owner-classified 170HX) with HalfVulpes' Deepinfer-SM80 patch set: a 30-file vLLM V1 overlay porting DeepSeek's DeepInfer kernel architecture (SM100 tcgen05/TMEM) to Ampere mma.sync tensor cores - unified 512-dim SM80 sparse-MLA decode/prefill kernels with bitshift UE8M0 scale decoding, a bit-exact single-CTA Triton TopK+SWA combiner (22.17x), direct-store KV-cache gather (7.64x @48K), SM80-aligned TileLang mHC routing (2.89x), and YOCO L20-L39 ragged indexer-metadata reuse (19 redundant kernel chains eliminated per chunk). TP2 x PP3 (14/14/12) + DSpark K=5 + 1M context, 4.65M-token GPU-resident KV. Author-measured: 64K cold prefill 4,507 tok/s (+24.2% vs baseline vLLM SM80), 64K TTFT 10.7 s (-18.9%), c1 DSpark decode 160.8 tok/s code / 188.8 tok/s 16K+ long-gen, c4 170.5 aggregate. Microbenchmarks bit-exact (0 diff) or within BF16 noise. Topology guide: PP6 = 2.1x slower single-stream; TP2+TP4 asymmetric hurts Marlin tiles.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":160.80000000000001136868377216160297393798828125,"prefill_tok_s":4506.8000000000001818989403545856475830078125,"updated_at":"2026-10-28 15:48:17.404025-06","commit_sha":"ced28259b90a43ec103ecd450b68455a56d2808d","content_hash":"e00265eca8b84734c2c7de2b67d558b0d702d25e","votes":0,"credit":[{"name":"HalfVulpes (Deepinfer-SM80 port + measurements)","url":"https://github.com/HalfVulpes/Deepinfer-SM80"},{"name":"DeepSeek-AI (DeepInfer engine architecture + DeepSeek-V4.1-Flash checkpoint)","url":"https://github.com/deepseek-ai/DeepInfer"},{"name":"vLLM project (V1 engine + TPxPP runtime)","url":"https://github.com/vllm-project/vllm"},{"name":"TileLang + OpenAI Triton (SM80 kernel synthesis)","url":"https://github.com/tile-ai/tilelang"},{"name":"FlashInfer + IST-DASLab Marlin (sparse-MLA + MXFP4/W4A16 Ampere GEMM primitives)","url":"https://github.com/flashinfer-ai/flashinfer"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5.json","alternate":"/r/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5"}},{"slug":"qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4","title":"Qwen3.8-27B AWQ-INT4 + DSpark on RTX 3090s \u2014 SGLang TP1/TP2/TP4","summary":"Validated SGLang serving for the dense Qwen3.8-27B hybrid GDN vision-language model (48 Gated-DeltaNet + 16 gated-attention layers) on RTX 3090 24 GB: one digest-pinned container whose TP env selects tuned 1x/2x/4x presets. AWQ-INT4 W4A16 (Marlin) beats FP8 on Ampere; DSpark speculative decoding (RadixArk 1.36B draft) on 2x/4x with auto-fallback on TP1. Image input everywhere, video on 2x/4x. Author-measured: 136.8 tok/s decode at 8k on 4x (169.8 aggregate at C8), 112.9 on 2x, 43.2 on 1x; up to 128k context on 4x.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":127.099999999999994315658113919198513031005859375,"prefill_tok_s":1372.700000000000045474735088646411895751953125,"updated_at":"2026-10-20 21:52:36.534482-06","commit_sha":"26a1363966d9065a310e9ec7caf124206a25f3f2","content_hash":"3fa45739100d87065553d5fb598424a66d4742fb","votes":0,"credit":[{"name":"sybil-solutions / 0xSero (SGLang presets, image build, benchmarks)","url":"https://github.com/sybil-solutions/qwen-serve"},{"name":"cyankiwi (AWQ-INT4 W4A16 checkpoint)","url":"https://huggingface.co/cyankiwi/Qwen3.8-27B-AWQ-INT4"},{"name":"RadixArk (DSpark speculative drafter)","url":"https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4.json","alternate":"/r/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4"}}]}