Qwen3.8-Flash-Next W4A16-FP8PLE on 2x RTX 3090 — TP2, expert-offload, MTP3 (DominikBucko)

community Community submission — verify on your own hardware.

recipe by DominikBucko↗, albucino (checkpoint)↗, Intel AutoRound (W4A16 target)↗, RadixArk (FP8 PLE table)↗ · listed by localai-recipes

revision r2 · commit bb7322bf4f · updated 10 d ago · license CC-BY-4.0

chat code mtp-speculative long-context expert-offload

Two-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in the albucino W4A16-FP8PLE build (Intel AutoRound INT4 backbone + FP8 PLE table + INT4 MTP draft, checkpoint @ ef55414): TP2 + expert parallel on the humming MoE path, UVA CPU offload (30 GB of cold experts in RAM), dynamic LRU expert cache (hot-84), BF16 KV at full 262,144-token context, MTP 3-token speculative draft, prefix caching. Author-measured (submitter T0, 128 GB RAM host): 131K prompt first token in 31.3 s and up to 4,191 tok/s prefill on the 8K-chunk prefill profile; agent 128K profile decodes 104.8-111.5 tok/s; fast 256K profile reads 260,096 tokens in 90.8 s (2,865 tok/s). Also ships 64 GB RAM and opt-in uncensored (abliteration) profiles. Engine is the pinned vLLM runtime image 0.1.dev20073+g8e685d198, republished as a release image pinned by digest; argv transcribed verbatim from scripts/serve-container.sh @ 939905b.

Launch command

vllm serve /model --served-model-name Qwen3.8-Flash-Next --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 --enable-expert-parallel --all2all-backend allgather_reducescatter --moe-backend humming --dtype bfloat16 --language-model-only --load-format safetensors --safetensors-load-strategy lazy --max-parallel-loading-workers 1 --offload-backend uva --cpu-offload-gb 30 --cpu-offload-params experts --max-model-len 262144 --max-num-seqs 1 --max-num-batched-tokens 4096 --kv-cache-dtype auto --kv-cache-memory-bytes 4429185024 --enable-chunked-prefill --enable-prefix-caching --mamba-cache-mode align --no-async-scheduling --disable-custom-all-reduce --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}' --trust-remote-code --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --speculative-config '{"method":"mtp","num_speculative_tokens":3,"use_local_argmax_reduction":true,"model":"/model/runtime/mtp-int4-g32"}'

runtime

env: 'CUDA_VISIBLE_DEVICES=0,1 | PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | VLLM_PLE_CPU_OFFLOAD=1 | VLLM_WNA16_STATIC_HOT_CACHE_FILE=/workspace/static_hot_cache_rankings.json | VLLM_FORCE_DYNAMIC_SPEC_SCHEDULING=1 | VLLM_WNA16_DYNAMIC_LRU=1 | VLLM_WNA16_MIXED_VMM_HOT_CACHE=1 | VLLM_WNA16_STATIC_HOT_CACHE_SIZE=84 | VLLM_WNA16_STATIC_HOT_CACHE_MAX_TOKENS=16 | VLLM_PREFIX_CACHE_RETENTION_INTERVAL=1600 | VLLM_QSA_EXACT_TOPK=0 | MTP_DEPTH=3 | ENABLE_VISION=0 | DISABLE_CUSTOM_ALL_REDUCE=1 | VLLM_PLE_OFFLOAD_READY_TIMEOUT=1200 (verbatim from scripts/serve-container.sh @ 939905b with the configs/2x3090-128gb.env profile defaults inlined)'
notes: 'Run path: git clone the repo @ 939905b, hf download albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE --revision ef554143369a706525336f6b42a09094835dc077 --local-dir <MODEL_DIR>, cp .env.example .env, set MODEL_DIR, then IMAGE=ghcr.io/dominikbucko/qwen38-flash-next-2x3090@sha256:75adfc64ba1576b8a677f455d54891c152444e4bf43fbc37cb8f1885a8db3387 make serve (or make build-image first). The container mounts MODEL_DIR at /model and execs scripts/serve-container.sh, which is the verbatim argv source above; /model/runtime/mtp-int4-g32 is the compact INT4 MTP draft inside the checkpoint. The release image is built FROM the pinned runtime image vllm/vllm-openai:qwen38-flash-next@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8 (repro.lock). Profiles: append configs/fast-256k.env (needs CUDA P2P + free RAM for the PLE table), configs/agent-128k.env or configs/agent-128k-prefill.env for 135,168-token agent work, configs/2x3090-64gb.env for 64 GB hosts (QWEN38_HOT_ONLY path, MAX_NUM_SEQS=1 enforced); uncensored mode = QWEN38_ABLITERATION=orcarouter. Host needs 128 GB RAM (64 GB profile excepted), Docker + NVIDIA Container Toolkit, 32-64 GiB swap. OpenAI-compatible endpoint http://127.0.0.1:8000/v1. DISABLE_CUSTOM_ALL_REDUCE=0 is incompatible with expandable_segments:True in this runtime (script exits 2).'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa.json

Engine

vllm · 8e685d19↗ · ghcr.io/dominikbucko/qwen38-flash-next-2x3090@sha256:75adfc64ba1576b8a677f455d54891c152444e4bf43fbc37cb8f1885a8db3387↗

Model

Qwen3.8-Flash-Next-W4A16-FP8PLE (w4a16-fp8-ple) · albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE↗
AA intelligence 40 (Artificial Analysis · measured)

Hardware

NVIDIA RTX 3090 24GB ×2 (Ampere)

Context

max context: 256K tokens

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
131,072	1	3029 tok/s	109.8 tok/s	43300 ms	reported
262,144	1	2865 tok/s	103.1 tok/s	90800 ms	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Author README tables (repo @ 939905b, 128 GB host, best-of-N single request, submitter T0 — no run file ingested; import via POST /api/v1/benchmarks to upgrade). Prefill profile (8K chunks, hot-88): 131,072+2,048 first token 31.3-36.0 s, 3,643-4,191 input tok/s. Agent 128K profile (4K chunks, hot-100): 43.3 s TTFT best of 2, 3,029 input tok/s, decode 104.8-111.5 tok/s (Sept 29 best-of-2 run: 109.8). Fast 256K profile: 260,096+2,048 first token 90.8 s, 2,864-2,865 input tok/s, decode 103.1 tok/s (Sept 25 best of 3). 64 GB RAM profile: 131,099+512 at 3,401-3,413 tok/s prefill, 84.2-88.8 tok/s decode. Decode depends on free RAM (~60 GB experts + 51.2 GB PLE table resident).

notes

Author README tables (repo @ 939905b, 128 GB host, best-of-N single request, submitter T0 — no run file ingested; import via POST /api/v1/benchmarks to upgrade). Prefill profile (8K chunks, hot-88): 131,072+2,048 first token 31.3-36.0 s, 3,643-4,191 input tok/s. Agent 128K profile (4K chunks, hot-100): 43.3 s TTFT best of 2, 3,029 input tok/s, decode 104.8-111.5 tok/s (Sept 29 best-of-2 run: 109.8). Fast 256K profile: 260,096+2,048 first token 90.8 s, 2,864-2,865 input tok/s, decode 103.1 tok/s (Sept 25 best of 3). 64 GB RAM profile: 131,099+512 at 3,401-3,413 tok/s prefill, 84.2-88.8 tok/s decode. Decode depends on free RAM (~60 GB experts + 51.2 GB PLE table resident).

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE
ef554143369a706525336f6b42a09094835dc077 — canonical checkpoint commit (repro.lock published_model; later documentation-only commits do not change model tensors). Intel AutoRound W4A16 target + FP8 E4M3FN PLE table + compact INT4-g32 MTP draft
Engine image https://ghcr.io/dominikbucko/qwen38-flash-next-2x3090
sha256:75adfc64ba1576b8a677f455d54891c152444e4bf43fbc37cb8f1885a8db3387 — v0.5.0 release image (tags v0.5.0, sha-e7279ab); base runtime vllm/vllm-openai:qwen38-flash-next@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8
Reference https://github.com/DominikBucko/qwen38-flash-next-2x3090
939905b0dab8c6eb3852ff35cbbd42d098af2451 — source recipe repo @ HEAD (scripts/serve-container.sh is the verbatim argv source; configs/, docker/, repro.lock.json all at this commit)

Revision notes

Community ingest of DominikBucko/qwen38-flash-next-2x3090 @ 939905b0dab8c6eb3852ff35cbbd42d098af2451 (owner-named by Casey Oct 6; 108 stars; last push 2026-10-02, Apache-2.0). Launch argv transcribed verbatim from scripts/serve-container.sh @ 939905b with configs/2x3090-128gb.env defaults inlined (128 GB headline profile); env exports verbatim from the same script. Token-level check done against the raw script before POST. Perf numbers are the author's README tables, logged claimed_by=submitter T0. Credits per repo repro.lock/THIRD_PARTY_NOTICES: checkpoint = Intel AutoRound target + RadixArk FP8 PLE + compact INT4 MTP, published by albucino.

Revision history

e1ffa1d76c recipe: update qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa (r2: /max_context_len, /storage_requirements) 10 d ago diff
c110eaf8c4 recipe: create qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa 24 d ago

Fork

Export

recipe.yaml (raw)