localairecipes launch configs for local inference

Qwen3.8-Flash-Next IQ3_S + MTP on 1x RTX 4090 — llama.cpp + tandem.cpp MoE expert cache (ewhacc)

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by ewhacc - recipe, patch set and measurements↗, llama.cpp / ggml (MIT)↗, MoE expert-split rule adapted from Strata (Niko1221)↗ · listed by localai-recipes

revision r2 · commit 543df76dcb · updated 1 h ago · license MIT

chat single-gpu long-context speculative-decoding mtp expert-cache moe moe-offload gsm8k patch-set cuda

MoE expert-cache offload for llama.cpp via the tandem.cpp patch set: hot experts stay in a GPU expert cache, missed experts are split between CPU compute and an upload into the cache, sized from measured CPU and PCIe speeds; single-user decode is the target. Author's GSM8K first-200 decode numbers on the consumer PC (i5-12400F, 64 GB DDR4, RTX 4090 24 GB): Qwen3.8-Flash-Next IQ3_S + MTP 97.3 tok/s greedy and 96.1 sampled vs llama.cpp f5222f25c 29.8/29.4 (3.3x); gpt-oss-120b 51.5/48.0. REPRO CAVEAT: all numbers are author-self-reported on a repo <24h old with one commit day of history (initial commit 2026-11-02, no releases/tags); nothing reproduced on-box or by a third party - evidence is claimed, tier community. Requires CUDA 13.0-13.4 on Linux (sm_89/sm_120) and 64 GB host RAM for the 78 GiB IQ3_S weights; 1.4-2.1x gains on other MoE models that don't fit VRAM.

Launch command

'LLAMA_MTP_DRAFT_VOCAB=100000 llama-server -m models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf -ngl 99 -cmoe --moe-cache-mib auto -lm none -np 1 -c 16384 --spec-type draft-mtp -md models/qwen38-mtp-outq4_K-expsq4_K.gguf --spec-draft-n-max 3 --spec-draft-sampling probabilistic -bs -ot '\''^token_embd\.weight=CUDA0'\''' '# Build: bash build.sh (clones llama.cpp @ f5222f25c, applies tandem.patch, cmake -DGGML_CUDA=ON, target llama-server)' '# Weights: hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF --include '\''IQ3_S/*'\'' --local-dir models; hf download ewhacc/qwen38-mtp-outq4_K-expsq4_K --include '\''*.gguf'\'' --local-dir models' '# -cmoe keeps every expert in host RAM; --moe-cache-mib auto gives leftover GPU memory (after model, KV, compute buffers) minus --moe-cache-margin-mib (default 1024, draft size added) to the expert cache; -lm none reads weights into RAM instead of mmap; -np 1 -c 16384 pins ONE conversation at 16k context (default 4 slots share one small ctx)' '# Less RAM than the model (78 GiB IQ3_S on 64 GB): drop -lm none (mapped file, experts pinned when RAM holds model + 10 GiB). Much less RAM (16 GB): also --cache-ram 0 (prompt cache otherwise evicts experts from page cache; 7x slower in the author'\''s 12 GB test)'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-iq3-s-mtp-on-1x-rtx-4090-llama-cpp-tandem-cpp.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-iq3-s-mtp-on-1x-rtx-4090-llama-cpp-tandem-cpp.json

Engine

Install note: Patch set, not a fork repo: bash build.sh clones llama.cpp, checks out base f5222f25c (pinned in BASE_COMMIT), applies tandem.patch (6634 insertions over 35 files, incl. new llama-moe-cache.cpp), builds llama-server with CUDA. By hand: git clone https://github.com/ggml-org/llama.cpp tandem && cd tandem && git checkout f5222f25c && git apply ../tandem.patch && cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j --target llama-server. Tested CUDA 13.0-13.4, Ubuntu 22.04/24.04, sm_89 and sm_120. Weights are NOT bundled: hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF --include 'IQ3_S/*' --local-dir models; hf download ewhacc/qwen38-mtp-outq4_K-expsq4_K --include '*.gguf' --local-dir models. -cmoe/--moe-cache-mib auto are patch-extended llama.cpp flags (auto sizing + --moe-cache-margin-mib are patch additions); --spec-type draft-mtp is upstream llama.cpp.

Model

Hardware

Context

  • max context: 16K tokens

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
16,3841—97.3 tok/s—reported
26,6241—96.1 tok/s—reported
36,8641—51.5 tok/s—reported
47,1041—48 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAll figures claimed by the author (ewhacc) at b03b001467: consumer PC i5-12400F (6 cores), 64 GB DDR4 43 GB/s, RTX 4090 24 GB, PCIe 4.0 x16; first 200 GSM8K test questions, zero-shot chat, max 2048 tokens, decode tok/s; llama.cpp f5222f25c baseline with default -fit on (same GPU memory, same draft model). context_len is a TUPLE MARKER for the (context_len, concurrency) perf index, not a real context length: 16384 = Qwen IQ3_S+MTP greedy (real launch -c 16384); 26624 = 16384+10240 Qwen IQ3_S+MTP default sampling; 36864 = 16384+20480 gpt-oss-120b greedy; 47104 = 16384+30720 gpt-oss-120b recommended sampling. Repo <24h old, single commit day, no third-party reproduction. Workstation/server tables (RTX 6000 Ada, EPYC + RTX PRO 6000 Blackwell) measured on other cards - excluded here.

Downloads

Revision notes

Ingested 2026-11-02 from repo b03b0014674576b685d9641f01da1bbccf2d5afc (HEAD verified via git ls-remote; initial commit 60b7b0b dated 2026-11-02, repo <24h old at sweep time, 6 stars, no tags/releases). Patch set over llama.cpp f5222f25c (2026-10-28) recorded in BASE_COMMIT; base commit verified live on GitHub ('cuda: fuse MoE (MUL_MAT_ID) gate/up and GLU in MMQ (#31188)'). Launch argv and flag semantics transcribed verbatim from README at the pinned sha; --moe-cache-mib 'auto' and --moe-cache-margin-mib confirmed in tandem.patch common/arg.cpp; -cmoe, -lm, -np, --spec-type are upstream llama.cpp base flags (patch context). Also surfaced on Reddit r/LocalLLaMA post 1xldent (same author, same numbers, folded into this card). EVIDENCE TIER: claimed - author-self-measured on one consumer box; the Strata comparison row is the author's own; nothing reproduced on-box or by a third party. HF revisions ed59f920/848c5ad8 captured by the scout sweep 2026-11-02. corrections_to_card: thin card t_fac98175 said IQ3_S+MTP vs llama.cpp 29.8 - matches README; twin card claimed no rtx-4090 hardware profile (0 rows) - FALSE: live hardware_profiles id 300 slug rtx-4090 (nvidia, 24GB, device_count 1, generation ada, card_family rtx-4090) + id 301 rtx-4090-x2 exist on preprod DB; the audit checked recipe usage, not the profile table. Twin's scout-format submission JSON (engine.source_type 'git', base_version, hardware object, performance array) re-shaped to the live submission envelope: source_type fork, base pin moved to version+install_note, perf re-keyed to points.

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)