{"slug":"qwen3-8-flash-next-iq3-s-mtp-on-1x-rtx-4090-llama-cpp-tandem-cpp","title":"Qwen3.8-Flash-Next IQ3_S + MTP on 1x RTX 4090 \u2014 llama.cpp + tandem.cpp MoE expert cache (ewhacc)","summary":"MoE expert-cache offload for llama.cpp via the tandem.cpp patch set: hot experts stay in a GPU expert cache, missed experts are split between CPU compute and an upload into the cache, sized from measured CPU and PCIe speeds; single-user decode is the target. Author's GSM8K first-200 decode numbers on the consumer PC (i5-12400F, 64 GB DDR4, RTX 4090 24 GB): Qwen3.8-Flash-Next IQ3_S + MTP 97.3 tok/s greedy and 96.1 sampled vs llama.cpp f5222f25c 29.8/29.4 (3.3x); gpt-oss-120b 51.5/48.0. REPRO CAVEAT: all numbers are author-self-reported on a repo <24h old with one commit day of history (initial commit 2026-11-02, no releases/tags); nothing reproduced on-box or by a third party - evidence is claimed, tier community. Requires CUDA 13.0-13.4 on Linux (sm_89/sm_120) and 64 GB host RAM for the 78 GiB IQ3_S weights; 1.4-2.1x gains on other MoE models that don't fit VRAM.","license":"MIT","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"543df76dcb783c7c365b8ea88a2d81ed9d36de76","content_hash":"fd8e878ab3aa782deec1708f8b0913a0ee5ea7a7","created_at":"2026-11-02 09:53:20.10132-07","notes":"Ingested 2026-11-02 from repo b03b0014674576b685d9641f01da1bbccf2d5afc (HEAD verified via git ls-remote; initial commit 60b7b0b dated 2026-11-02, repo <24h old at sweep time, 6 stars, no tags/releases). Patch set over llama.cpp f5222f25c (2026-10-28) recorded in BASE_COMMIT; base commit verified live on GitHub ('cuda: fuse MoE (MUL_MAT_ID) gate/up and GLU in MMQ (#31188)'). Launch argv and flag semantics transcribed verbatim from README at the pinned sha; --moe-cache-mib 'auto' and --moe-cache-margin-mib confirmed in tandem.patch common/arg.cpp; -cmoe, -lm, -np, --spec-type are upstream llama.cpp base flags (patch context). Also surfaced on Reddit r/LocalLLaMA post 1xldent (same author, same numbers, folded into this card). EVIDENCE TIER: claimed - author-self-measured on one consumer box; the Strata comparison row is the author's own; nothing reproduced on-box or by a third party. HF revisions ed59f920/848c5ad8 captured by the scout sweep 2026-11-02. corrections_to_card: thin card t_fac98175 said IQ3_S+MTP vs llama.cpp 29.8 - matches README; twin card claimed no rtx-4090 hardware profile (0 rows) - FALSE: live hardware_profiles id 300 slug rtx-4090 (nvidia, 24GB, device_count 1, generation ada, card_family rtx-4090) + id 301 rtx-4090-x2 exist on preprod DB; the audit checked recipe usage, not the profile table. Twin's scout-format submission JSON (engine.source_type 'git', base_version, hardware object, performance array) re-shaped to the live submission envelope: source_type fork, base pin moved to version+install_note, perf re-keyed to points."},"engine_source":{"type":"fork","git_url":"https://github.com/ewhacc/tandem.cpp","base_engine":"llama.cpp","pip_package":null,"base_version":"llama.cpp f5222f25c (2026-10-28) + tandem.patch","image_digest":null,"install_note":"Patch set, not a fork repo: bash build.sh clones llama.cpp, checks out base f5222f25c (pinned in BASE_COMMIT), applies tandem.patch (6634 insertions over 35 files, incl. new llama-moe-cache.cpp), builds llama-server with CUDA. By hand: git clone https://github.com/ggml-org/llama.cpp tandem && cd tandem && git checkout f5222f25c && git apply ../tandem.patch && cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j --target llama-server. Tested CUDA 13.0-13.4, Ubuntu 22.04/24.04, sm_89 and sm_120. Weights are NOT bundled: hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF --include 'IQ3_S/*' --local-dir models; hf download ewhacc/qwen38-mtp-outq4_K-expsq4_K --include '*.gguf' --local-dir models. -cmoe/--moe-cache-mib auto are patch-extended llama.cpp flags (auto sizing + --moe-cache-margin-mib are patch additions); --spec-type draft-mtp is upstream llama.cpp.","pinned_commit":"b03b0014674576b685d9641f01da1bbccf2d5afc"},"credit":[{"name":"ewhacc - recipe, patch set and measurements","url":"https://github.com/ewhacc/tandem.cpp"},{"name":"llama.cpp / ggml (MIT)","url":"https://github.com/ggml-org/llama.cpp"},{"name":"MoE expert-split rule adapted from Strata (Niko1221)","url":"https://github.com/Niko1221/Strata"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["LLAMA_MTP_DRAFT_VOCAB=100000 llama-server -m models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf -ngl 99 -cmoe --moe-cache-mib auto -lm none -np 1 -c 16384 --spec-type draft-mtp -md models/qwen38-mtp-outq4_K-expsq4_K.gguf --spec-draft-n-max 3 --spec-draft-sampling probabilistic -bs -ot '^token_embd\\.weight=CUDA0'","# Build: bash build.sh (clones llama.cpp @ f5222f25c, applies tandem.patch, cmake -DGGML_CUDA=ON, target llama-server)","# Weights: hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF --include 'IQ3_S/*' --local-dir models; hf download ewhacc/qwen38-mtp-outq4_K-expsq4_K --include '*.gguf' --local-dir models","# -cmoe keeps every expert in host RAM; --moe-cache-mib auto gives leftover GPU memory (after model, KV, compute buffers) minus --moe-cache-margin-mib (default 1024, draft size added) to the expert cache; -lm none reads weights into RAM instead of mmap; -np 1 -c 16384 pins ONE conversation at 16k context (default 4 slots share one small ctx)","# Less RAM than the model (78 GiB IQ3_S on 64 GB): drop -lm none (mapped file, experts pinned when RAM holds model + 10 GiB). Much less RAM (16 GB): also --cache-ram 0 (prompt cache otherwise evicts experts from page cache; 7x slower in the author's 12 GB test)"],"max_context_len":16384,"storage_requirements":{"note":"78 GiB IQ3_S GGUF (2 files) + 1.8 GiB Q4_K MTP head; model exceeds the 64 GB DDR4 reference box - both engines read from the mapped SSD file","ram_gb":64,"ssd_gb":80}},"models":[{"slug":"qwen3-8-flash-next","name":"Qwen3.8-Flash-Next","hf_repo":"local-inference-lab/Qwen3.8-Flash-Next-NVFP4","quant":"gguf-iq3-s-mtp"}],"hardware":[{"slug":"rtx-4090","vendor":"nvidia","card":"RTX 4090","card_family":"rtx-4090","vram_gb":24,"device_count":1}],"generation":"ada","tags":["cuda","expert-cache","gsm8k","long-context","moe","moe-offload","mtp","patch-set","single-gpu","speculative-decoding"],"tasks":["chat"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":"Index v4.3.2","aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"All figures claimed by the author (ewhacc) at b03b001467: consumer PC i5-12400F (6 cores), 64 GB DDR4 43 GB/s, RTX 4090 24 GB, PCIe 4.0 x16; first 200 GSM8K test questions, zero-shot chat, max 2048 tokens, decode tok/s; llama.cpp f5222f25c baseline with default -fit on (same GPU memory, same draft model). context_len is a TUPLE MARKER for the (context_len, concurrency) perf index, not a real context length: 16384 = Qwen IQ3_S+MTP greedy (real launch -c 16384); 26624 = 16384+10240 Qwen IQ3_S+MTP default sampling; 36864 = 16384+20480 gpt-oss-120b greedy; 47104 = 16384+30720 gpt-oss-120b recommended sampling. Repo <24h old, single commit day, no third-party reproduction. Workstation/server tables (RTX 6000 Ada, EPYC + RTX PRO 6000 Blackwell) measured on other cards - excluded here.\", \"tok_s\": 97.2999999999999971578290569595992565155029296875, \"context_len\": 16384}"}],"perf_points":[{"id":2356,"context_len":16384,"prefill_tok_s":null,"decode_tok_s":97.2999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Qwen3.8-Flash-Next IQ3_S + MTP, greedy - real launch -c 16384; llama.cpp f5222f25c baseline 29.8"},{"id":2357,"context_len":26624,"prefill_tok_s":null,"decode_tok_s":96.099999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Qwen IQ3_S + MTP, model default sampling (T 1.0, top-k 20, top-p 0.95) - tuple marker 16384+10240; llama.cpp baseline 29.4"},{"id":2358,"context_len":36864,"prefill_tok_s":null,"decode_tok_s":51.5,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"gpt-oss-120b MXFP4, greedy, no draft - tuple marker 16384+20480; llama.cpp baseline 27.8 (same 4090 box)"},{"id":2359,"context_len":47104,"prefill_tok_s":null,"decode_tok_s":48,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"gpt-oss-120b MXFP4, recommended sampling (T 1.0, top-p 1.0) - tuple marker 16384+30720; llama.cpp baseline 27.7"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-iq3-s-mtp-on-1x-rtx-4090-llama-cpp-tandem-cpp.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-iq3-s-mtp-on-1x-rtx-4090-llama-cpp-tandem-cpp.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF","pin":"IQ3_S 78 GiB two-file GGUF @ HF revision ed59f92082b1e93c0e96d60a8b11aab089b52f09 (measured weights)"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/ewhacc/qwen38-mtp-outq4_K-expsq4_K","pin":"Q4_K MTP head @ HF revision 848c5ad85766f4a212587df40f73ebcbee58de89; Qwen Community License 1.0"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/ewhacc/tandem.cpp","pin":"b03b0014674576b685d9641f01da1bbccf2d5afc (tandem.patch, build.sh, BASE_COMMIT, bench scripts)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/ggml-org/llama.cpp","pin":"f5222f25c (2026-10-28, BASE_COMMIT) - patched llama.cpp base"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-iq3-s-mtp-on-1x-rtx-4090-llama-cpp-tandem-cpp.json","alternate":"/r/qwen3-8-flash-next-iq3-s-mtp-on-1x-rtx-4090-llama-cpp-tandem-cpp","describedby":"/openapi.json"},"updated_at":"2026-11-02 09:53:20.10132-07","created_at":"2026-11-02 09:35:44.423668-07","speeds":{"decode_tok_s":73.7999999999999971578290569595992565155029296875,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}