{"slug":"qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa","title":"Qwen3.8-Flash-Next W4A16-FP8PLE on 2x RTX 3090 \u2014 TP2, expert-offload, MTP3 (DominikBucko)","summary":"Two-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in the albucino W4A16-FP8PLE build (Intel AutoRound INT4 backbone + FP8 PLE table + INT4 MTP draft, checkpoint @ ef55414): TP2 + expert parallel on the humming MoE path, UVA CPU offload (30 GB of cold experts in RAM), dynamic LRU expert cache (hot-84), BF16 KV at full 262,144-token context, MTP 3-token speculative draft, prefix caching. Author-measured (submitter T0, 128 GB RAM host): 131K prompt first token in 31.3 s and up to 4,191 tok/s prefill on the 8K-chunk prefill profile; agent 128K profile decodes 104.8-111.5 tok/s; fast 256K profile reads 260,096 tokens in 90.8 s (2,865 tok/s). Also ships 64 GB RAM and opt-in uncensored (abliteration) profiles. Engine is the pinned vLLM runtime image 0.1.dev20073+g8e685d198, republished as a release image pinned by digest; argv transcribed verbatim from scripts/serve-container.sh @ 939905b.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"bb7322bf4f7bdf564d617b1ab9f9d2985bf71877","content_hash":"5e0fd674c07293431c87af75dcd971af50a99d37","created_at":"2026-10-20 15:34:08.646082-06","notes":"Community ingest of DominikBucko/qwen38-flash-next-2x3090 @ 939905b0dab8c6eb3852ff35cbbd42d098af2451 (owner-named by Casey Oct 6; 108 stars; last push 2026-10-02, Apache-2.0). Launch argv transcribed verbatim from scripts/serve-container.sh @ 939905b with configs/2x3090-128gb.env defaults inlined (128 GB headline profile); env exports verbatim from the same script. Token-level check done against the raw script before POST. Perf numbers are the author's README tables, logged claimed_by=submitter T0. Credits per repo repro.lock/THIRD_PARTY_NOTICES: checkpoint = Intel AutoRound target + RadixArk FP8 PLE + compact INT4 MTP, published by albucino."},"engine_source":{"type":"docker_image","git_url":null,"base_engine":"vllm","pip_package":null,"base_version":"0.1.dev20073+g8e685d198 (qwen38-flash-next runtime)","image_digest":"ghcr.io/dominikbucko/qwen38-flash-next-2x3090@sha256:75adfc64ba1576b8a677f455d54891c152444e4bf43fbc37cb8f1885a8db3387","install_note":null,"pinned_commit":"8e685d198"},"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/qwen38-flash-next-2x3090"},{"name":"albucino (checkpoint)","url":"https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE"},{"name":"Intel AutoRound (W4A16 target)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"RadixArk (FP8 PLE table)","url":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/model","--served-model-name","Qwen3.8-Flash-Next","--host","0.0.0.0","--port","8000","--tensor-parallel-size","2","--enable-expert-parallel","--all2all-backend","allgather_reducescatter","--moe-backend","humming","--dtype","bfloat16","--language-model-only","--load-format","safetensors","--safetensors-load-strategy","lazy","--max-parallel-loading-workers","1","--offload-backend","uva","--cpu-offload-gb","30","--cpu-offload-params","experts","--max-model-len","262144","--max-num-seqs","1","--max-num-batched-tokens","4096","--kv-cache-dtype","auto","--kv-cache-memory-bytes","4429185024","--enable-chunked-prefill","--enable-prefix-caching","--mamba-cache-mode","align","--no-async-scheduling","--disable-custom-all-reduce","--compilation-config","{\"mode\":0,\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}","--trust-remote-code","--enable-auto-tool-choice","--tool-call-parser","qwen3_coder","--reasoning-parser","qwen3","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":3,\"use_local_argmax_reduction\":true,\"model\":\"/model/runtime/mtp-int4-g32\"}"],"runtime":{"env":"CUDA_VISIBLE_DEVICES=0,1 | PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | VLLM_PLE_CPU_OFFLOAD=1 | VLLM_WNA16_STATIC_HOT_CACHE_FILE=/workspace/static_hot_cache_rankings.json | VLLM_FORCE_DYNAMIC_SPEC_SCHEDULING=1 | VLLM_WNA16_DYNAMIC_LRU=1 | VLLM_WNA16_MIXED_VMM_HOT_CACHE=1 | VLLM_WNA16_STATIC_HOT_CACHE_SIZE=84 | VLLM_WNA16_STATIC_HOT_CACHE_MAX_TOKENS=16 | VLLM_PREFIX_CACHE_RETENTION_INTERVAL=1600 | VLLM_QSA_EXACT_TOPK=0 | MTP_DEPTH=3 | ENABLE_VISION=0 | DISABLE_CUSTOM_ALL_REDUCE=1 | VLLM_PLE_OFFLOAD_READY_TIMEOUT=1200 (verbatim from scripts/serve-container.sh @ 939905b with the configs/2x3090-128gb.env profile defaults inlined)","notes":"Run path: git clone the repo @ 939905b, hf download albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE --revision ef554143369a706525336f6b42a09094835dc077 --local-dir <MODEL_DIR>, cp .env.example .env, set MODEL_DIR, then IMAGE=ghcr.io/dominikbucko/qwen38-flash-next-2x3090@sha256:75adfc64ba1576b8a677f455d54891c152444e4bf43fbc37cb8f1885a8db3387 make serve (or make build-image first). The container mounts MODEL_DIR at /model and execs scripts/serve-container.sh, which is the verbatim argv source above; /model/runtime/mtp-int4-g32 is the compact INT4 MTP draft inside the checkpoint. The release image is built FROM the pinned runtime image vllm/vllm-openai:qwen38-flash-next@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8 (repro.lock). Profiles: append configs/fast-256k.env (needs CUDA P2P + free RAM for the PLE table), configs/agent-128k.env or configs/agent-128k-prefill.env for 135,168-token agent work, configs/2x3090-64gb.env for 64 GB hosts (QWEN38_HOT_ONLY path, MAX_NUM_SEQS=1 enforced); uncensored mode = QWEN38_ABLITERATION=orcarouter. Host needs 128 GB RAM (64 GB profile excepted), Docker + NVIDIA Container Toolkit, 32-64 GiB swap. OpenAI-compatible endpoint http://127.0.0.1:8000/v1. DISABLE_CUSTOM_ALL_REDUCE=0 is incompatible with expandable_segments:True in this runtime (script exits 2)."},"max_context_len":262144,"storage_requirements":{"ram_gb":128}},"models":[{"slug":"qwen3-8-flash-next-w4a16-fp8ple","name":"Qwen3.8-Flash-Next-W4A16-FP8PLE","hf_repo":"albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE","quant":"w4a16-fp8-ple"}],"hardware":[{"slug":"rtx-3090-x2","vendor":"nvidia","card":"RTX 3090","card_family":"rtx-3090","vram_gb":24,"device_count":2}],"generation":"ampere","tags":["expert-offload","long-context","mtp-speculative"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author README tables (repo @ 939905b, 128 GB host, best-of-N single request, submitter T0 \u2014 no run file ingested; import via POST /api/v1/benchmarks to upgrade). Prefill profile (8K chunks, hot-88): 131,072+2,048 first token 31.3-36.0 s, 3,643-4,191 input tok/s. Agent 128K profile (4K chunks, hot-100): 43.3 s TTFT best of 2, 3,029 input tok/s, decode 104.8-111.5 tok/s (Sept 29 best-of-2 run: 109.8). Fast 256K profile: 260,096+2,048 first token 90.8 s, 2,864-2,865 input tok/s, decode 103.1 tok/s (Sept 25 best of 3). 64 GB RAM profile: 131,099+512 at 3,401-3,413 tok/s prefill, 84.2-88.8 tok/s decode. Decode depends on free RAM (~60 GB experts + 51.2 GB PLE table resident).\", \"tok_s\": 111.5, \"points\": [{\"notes\": \"agent-128k profile, best of 2 (README Sept 29): 43.3 s TTFT, 3,029 input tok/s, 109.8 decode tok/s\", \"ttft_ms\": 43300, \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 109.7999999999999971578290569595992565155029296875, \"prefill_tok_s\": 3029}, {\"notes\": \"agent-128k-prefill profile (8K chunks, hot-88), best run (README Sept 30): 31.3 s TTFT, 4,191 input tok/s (range 3,643-4,191); decode 91.9-101.8 tok/s\", \"ttft_ms\": 31300, \"concurrency\": 1, \"context_len\": 131072, \"prefill_tok_s\": 4191}, {\"notes\": \"fast-256k profile: 260,096+2,048, first token 90.8 s (Sept 29 runtime), 2,864-2,865 input tok/s; decode 103.1 tok/s (Sept 25 best of 3)\", \"ttft_ms\": 90800, \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 103.099999999999994315658113919198513031005859375, \"prefill_tok_s\": 2865}], \"context_len\": 131072}"}],"perf_points":[{"id":1032,"context_len":131072,"prefill_tok_s":3029,"decode_tok_s":109.7999999999999971578290569595992565155029296875,"ttft_ms":43300,"concurrency":1,"claimed_by":"submitter","notes":"agent-128k profile, best of 2 (README Sept 29): 43.3 s TTFT, 3,029 input tok/s, 109.8 decode tok/s"},{"id":1033,"context_len":262144,"prefill_tok_s":2865,"decode_tok_s":103.099999999999994315658113919198513031005859375,"ttft_ms":90800,"concurrency":1,"claimed_by":"submitter","notes":"fast-256k profile: 260,096+2,048, first token 90.8 s (Sept 29 runtime), 2,864-2,865 input tok/s; decode 103.1 tok/s (Sept 25 best of 3)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE","pin":"ef554143369a706525336f6b42a09094835dc077 \u2014 canonical checkpoint commit (repro.lock published_model; later documentation-only commits do not change model tensors). Intel AutoRound W4A16 target + FP8 E4M3FN PLE table + compact INT4-g32 MTP draft"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://ghcr.io/dominikbucko/qwen38-flash-next-2x3090","pin":"sha256:75adfc64ba1576b8a677f455d54891c152444e4bf43fbc37cb8f1885a8db3387 \u2014 v0.5.0 release image (tags v0.5.0, sha-e7279ab); base runtime vllm/vllm-openai:qwen38-flash-next@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/DominikBucko/qwen38-flash-next-2x3090","pin":"939905b0dab8c6eb3852ff35cbbd42d098af2451 \u2014 source recipe repo @ HEAD (scripts/serve-container.sh is the verbatim argv source; configs/, docker/, repro.lock.json all at this commit)"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa.json","alternate":"/r/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:08.646082-06","created_at":"2026-10-06 17:12:38.000919-06","speeds":{"decode_tok_s":106.5,"prefill_tok_s":2947,"leaderboard":"/api/v1/leaderboard.json"}}