{"meta":{"count":3,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["rtx-3090-x4","rtx-4090"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-11-03T03:35:45+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-iq3-s-mtp-on-1x-rtx-4090-llama-cpp-tandem-cpp","title":"Qwen3.8-Flash-Next IQ3_S + MTP on 1x RTX 4090 \u2014 llama.cpp + tandem.cpp MoE expert cache (ewhacc)","summary":"MoE expert-cache offload for llama.cpp via the tandem.cpp patch set: hot experts stay in a GPU expert cache, missed experts are split between CPU compute and an upload into the cache, sized from measured CPU and PCIe speeds; single-user decode is the target. Author's GSM8K first-200 decode numbers on the consumer PC (i5-12400F, 64 GB DDR4, RTX 4090 24 GB): Qwen3.8-Flash-Next IQ3_S + MTP 97.3 tok/s greedy and 96.1 sampled vs llama.cpp f5222f25c 29.8/29.4 (3.3x); gpt-oss-120b 51.5/48.0. REPRO CAVEAT: all numbers are author-self-reported on a repo <24h old with one commit day of history (initial commit 2026-11-02, no releases/tags); nothing reproduced on-box or by a third party - evidence is claimed, tier community. Requires CUDA 13.0-13.4 on Linux (sm_89/sm_120) and 64 GB host RAM for the 78 GiB IQ3_S weights; 1.4-2.1x gains on other MoE models that don't fit VRAM.","tier":"community","beta":false,"generation":"ada","decode_tok_s":73.7999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-11-02 09:53:20.10132-07","commit_sha":"543df76dcb783c7c365b8ea88a2d81ed9d36de76","content_hash":"fd8e878ab3aa782deec1708f8b0913a0ee5ea7a7","votes":0,"credit":[{"name":"ewhacc - recipe, patch set and measurements","url":"https://github.com/ewhacc/tandem.cpp"},{"name":"llama.cpp / ggml (MIT)","url":"https://github.com/ggml-org/llama.cpp"},{"name":"MoE expert-split rule adapted from Strata (Niko1221)","url":"https://github.com/Niko1221/Strata"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-iq3-s-mtp-on-1x-rtx-4090-llama-cpp-tandem-cpp.json","alternate":"/r/qwen3-8-flash-next-iq3-s-mtp-on-1x-rtx-4090-llama-cpp-tandem-cpp"}},{"slug":"qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f","title":"Qwen3.8-Flash-Next W4A16-Merlin on 4x RTX 3090 \u2014 TP2xPP2+EP, MTP3, FP8 KV, 925K pool (halt95)","summary":"Four-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in halt95's W4A16-Merlin build (Intel AutoRound INT4 + FP8 PLE + MTP draft, @ 17284bf): TP2 x PP2 with expert parallel, manual layer partition 25,23, FP8 KV with a 924,993-token pool (three full 262K sessions resident; block-size 4096, prefix-match-unit 64), MTP 3-token probabilistic draft with RecoverSSM, host-mapped PLE pull transport, host-resident embeddings, side-cache layout + Model Runner V2, 42 images/request. Agent-focused: 92.5% of follow-up prompt tokens from cache across 8 sessions, median follow-up first token 1.61 s (0.96 s solo). Maintainer-measured (submitter T0, v2.5.1 bench card 2026-10-06): 5,219-5,533 tok/s cold prefill 10K-261K, 158-174 tok/s single-stream decode (thinking on), 534 tok/s aggregate at 8 concurrent, 262K needle exact, GSM8K 198/200. Engine is a SOURCE BUILD: vLLM v0.30.0 + 97 commits (release bundle verified by commit AND tree hash); argv verbatim from scripts/serve-v2.5.sh @ 28cbb72.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":168.099999999999994315658113919198513031005859375,"prefill_tok_s":5360,"updated_at":"2026-10-31 11:51:35.845609-06","commit_sha":"423a58839f188df961890110971ddb250e6fe6b5","content_hash":"b1b20a73286839a8b304e2a3bcb8ee2b35fd43e2","votes":0,"credit":[{"name":"halt95","url":"https://github.com/halt95/merlin-engine"},{"name":"Qwen (Qwen3.8-Flash-Next)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"Intel AutoRound (INT4 experts)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"RadixArk (FP8 PLE table)","url":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4"},{"name":"DominikBucko (MTP INT4 packing recipe)","url":"https://github.com/DominikBucko/qwen38-flash-next-2x3090"},{"name":"peakcrosser7 (Flash-Next vLLM branch)","url":"https://github.com/peakcrosser7/vllm/commits/release/qwen38next_offload"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f.json","alternate":"/r/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f"}},{"slug":"qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4","title":"Qwen3.8-27B AWQ-INT4 + DSpark on RTX 3090s \u2014 SGLang TP1/TP2/TP4","summary":"Validated SGLang serving for the dense Qwen3.8-27B hybrid GDN vision-language model (48 Gated-DeltaNet + 16 gated-attention layers) on RTX 3090 24 GB: one digest-pinned container whose TP env selects tuned 1x/2x/4x presets. AWQ-INT4 W4A16 (Marlin) beats FP8 on Ampere; DSpark speculative decoding (RadixArk 1.36B draft) on 2x/4x with auto-fallback on TP1. Image input everywhere, video on 2x/4x. Author-measured: 136.8 tok/s decode at 8k on 4x (169.8 aggregate at C8), 112.9 on 2x, 43.2 on 1x; up to 128k context on 4x.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":127.099999999999994315658113919198513031005859375,"prefill_tok_s":1372.700000000000045474735088646411895751953125,"updated_at":"2026-10-20 21:52:36.534482-06","commit_sha":"26a1363966d9065a310e9ec7caf124206a25f3f2","content_hash":"3fa45739100d87065553d5fb598424a66d4742fb","votes":0,"credit":[{"name":"sybil-solutions / 0xSero (SGLang presets, image build, benchmarks)","url":"https://github.com/sybil-solutions/qwen-serve"},{"name":"cyankiwi (AWQ-INT4 W4A16 checkpoint)","url":"https://huggingface.co/cyankiwi/Qwen3.8-27B-AWQ-INT4"},{"name":"RadixArk (DSpark speculative drafter)","url":"https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4.json","alternate":"/r/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4"}}]}