DeepSeek-V4.1-Flash-1017 dsx 2.6-bit GPTQ 2x RTX 3090 (Engram)

community Community submission — verify on your own hardware.

recipe by lxe (dsx engine + 2bit reference quant)↗, icecubetr (2.6-bit GPTQ quant + measurements)↗, deepseek-ai (DeepSeek-V4.1-Flash-1017 checkpoint)↗, Lucebox (1017 eval set)↗ · listed by localai-recipes

revision r2 · commit 942f2d4b4c · updated 2 d ago · license MIT

chat code engram custom-engine dsx exact-routing gptq hybrid-experts cpu-experts 3090

DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) quantized to 187.6 GB single GGUF (2.6127 bpw routed-expert average: 5717 PTQ1_0 / 2545 Q2_0 / 7098 Q3_K per-expert GPTQ with allocator-spent bit budget; dense Q8_0 attention/shared expert/head, F16 embedding/compressor/indexer/Engram-wkv, F32 norms/router/mHC), served by dsx, a Rust+CUDA engine built for this checkpoint, on two RTX 3090s with 192 GB RAM. ~26 tok/s decode / ~170 tok/s prefill at --ctx 262144 with all 15,360 routed experts resident. Exact routing: no expert pruning, no router bias - the rank file orders placement only. Engram n-gram tables (~203 GB) are NOT quantized: read at runtime from the official safetensor shards by a prefetch thread. Hottest ~3,300 experts in VRAM across both GPUs (GPU0's set adapts to the conversation, DSX_ADAPT=8 swaps/token), remaining experts computed on 14 CPU threads from a hugepage RAM arena. Quality vs FP8/FP4 teacher (dsx eval, Lucebox 1017 set): KL 0.1972, top-1 82.59%, PPL 9.3626.

Launch command

'hf download icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit --local-dir dsx-2.6bit' 'hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 --local-dir v41-1017 --include model.safetensors.index.json model-00047-of-00048.safetensors model-00048-of-00048.safetensors tokenizer.json' 'CUDA_ARCHS=86 cargo build --release -p dsx --features cuda' './target/release/dsx-server --model dsx-2.6bit/DeepSeek-V4.1-Flash-1017-dsx-2.6bit.gguf --engram v41-1017 --tokenizer v41-1017/tokenizer.json --host 0.0.0.0 --port 8090 --ctx 262144 --chunk 4096 --gpu0-gb=-1.0 --gpu1-gb=-1.0 --arena-gb 120 --rank dsx-2.6bit/expert-rank.json --vision dsx-2.6bit/vision.safetensors'

runtime

env: "Placement flags: --gpu0-gb/--gpu1-gb = GB of experts per GPU (negative = all free VRAM minus that; --gpu1-gb 0 for one GPU); --arena-gb = RAM for CPU-computed experts; --rank = expert hotness order (expert-rank.json from the quant repo); --vision / --no-vision; DSX_ADAPT=N = GPU0 expert swaps per token (default 8, 0 = static). 192 GB box uses --gpu0-gb=-1.0 --gpu1-gb=-1.0 --arena-gb 120 (the 2.6-bit card's sizing); 128 GB: --gpu0-gb=-1.5 --gpu1-gb=-1.0 --arena-gb 110. Thinking on by default; reasoning_effort:none or thinking:{type:disabled} turns it off. The launch line above is the quant card's own run command (--ctx 262144); --ctx 1048576 is the model's full 1M but lower it unless you need it."

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram.json

Engine

dsx · source↗ · 98f12597↗

Install note: Source build: clone github.com/lxe/dsx, CUDA_ARCHS=86 cargo build --release -p dsx --features cuda (no release tags, no published image; MIT). Weights: hf download icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit (187.6 GB single GGUF + vision.safetensors + expert-rank.json). Engram+tokenizer from deepseek-ai/DeepSeek-V4.1-Flash-1017: --include model.safetensors.index.json model-00047-of-00048.safetensors model-00048-of-00048.safetensors tokenizer.json. Needs 2x 24 GB (sm_86; sm_89/sm_120 build too), ~140 GB VRAM+RAM, x86 AVX2 (VNNI preferred), ~350 GB disk. OpenAI /v1 + Anthropic /v1/messages + web UI on :8090.

Model

DeepSeek-V4.1-Flash-1017 (dsx-mixed-2-6bit-per-expert-ptq1-0-q2-0-q3-k-gptq-dense-q8-0) · deepseek-ai/DeepSeek-V4.1-Flash-1017↗
AA intelligence 45 (Artificial Analysis · measured)

Hardware

NVIDIA RTX 3090 24GB ×2 (Ampere)

Context

max context: 256K tokens

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
262,144	—	170 tok/s	—	—	reported
262,144	1	—	26 tok/s	—	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	claimed_by=submitter T0 (quant author's own box: 2x RTX 3090, 192 GB RAM, --ctx 262144, all 15,360 experts resident): ~26 tok/s decode, ~170 tok/s prompt processing. Quality via dsx eval on Lucebox/DeepSeek-V4.1-Flash-1017-ROCMFP-GGUF eval set (8x1024 held-out tokens; KL/top-1 vs FP8/FP4 teacher top-256 log-probs): KL 0.1972, top-1 82.59%, PPL 9.3626, NLL 2.236722 over 8,184 tokens. Comparators, same set: FP8/FP4 teacher KL 0 / 100% / 8.56; Lucebox ROCMFP 2.5 bpw KL 0.253 / 78.7% / 9.78; engine author's 2.03 bpw (lxe/dsx-2bit, 128 GB box, 30-37 tok/s decode) KL 0.462 / 71.5% / 11.66.

notes

claimed_by=submitter T0 (quant author's own box: 2x RTX 3090, 192 GB RAM, --ctx 262144, all 15,360 experts resident): ~26 tok/s decode, ~170 tok/s prompt processing. Quality via dsx eval on Lucebox/DeepSeek-V4.1-Flash-1017-ROCMFP-GGUF eval set (8x1024 held-out tokens; KL/top-1 vs FP8/FP4 teacher top-256 log-probs): KL 0.1972, top-1 82.59%, PPL 9.3626, NLL 2.236722 over 8,184 tokens. Comparators, same set: FP8/FP4 teacher KL 0 / 100% / 8.56; Lucebox ROCMFP 2.5 bpw KL 0.253 / 78.7% / 9.78; engine author's 2.03 bpw (lxe/dsx-2bit, 128 GB box, 30-37 tok/s decode) KL 0.462 / 71.5% / 11.66.

Downloads

Weights https://huggingface.co/icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit
70794f816419731761a8ff07af8ee457020faff1 - 187.6 GB single GGUF; Engram NOT included (read from official shards at runtime)
Weights https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
official checkpoint - only index + shards 47-48 + tokenizer needed (Engram + tokenizer source)
Engine image https://github.com/lxe/dsx
98f1259732f8c3a5f445b25e080daf70fc300967 (HEAD at revision 2026-10-28, audited; no release tags; MIT). NVIDIA runner in nvidia/; AMD MI325X runner (amd/) is a separate future card.

Revision notes

Community ingest surfaced via r/LocalLLaMA 2026-10-19 (quant author's release thread; not re-verified from .55 - Reddit hard-403). Ingest corrections vs scout card: launch argv re-transcribed at dsx HEAD 971c072 (README quickstart + icecubetr model card Run section); quant ships a SINGLE GGUF named DeepSeek-V4.1-Flash-1017-dsx-2.6bit.gguf (card's 00001-of-00004 shard name belongs to the 2-bit sibling), --chunk 4096 and --vision added per the quant card, placement flags follow the quant card's 192 GB sizing. Card said 'speculative' tag: DSpark exists in dsx core but is wired only into the offline runner (nvidia/src/main.rs), FINDINGS.md section 9 lists speculative decoding as a negative result (off in serving) - tag dropped. Engine source_type=release (built from the git repo, but engine_source.type mirrors the submission enum, not 'git'). All numbers claimed_by=submitter T0; import via POST /api/v1/benchmarks to upgrade. amd/ MI325X runner (110-114 tok/s @256K, MXFP4, KL 0.049) is a deliberate SEPARATE candidate - not folded into this card. PIN REVISION (kanban t_7ae73b9f, owner go Casey 2026-10-28, parent t_fa31dc3c): engine pin 971c0725bfd3..98f1259732f8 audited via GitHub compare (ahead_by=338, 338 commits). Published argv re-verified flag-by-flag at 98f1259732 (nvidia/src/bin/server.rs: --model --engram --tokenizer --host --port --ctx --gpu0-gb and --gpu1-gb kept as aliases of --main-gb/--aux-gb --arena-gb --rank --chunk 4096 --vision — all supported); GGUF load of the published PTQ1_0/Q2_0/Q3_K/Q8_0/F16 mix still supported (core/src/gguf.rs GType + core/src/formats.rs dequant; additions only). NVIDIA toolchain unchanged (README: Rust stable + CUDA 13, x86 AVX2, AVX-VNNI/AVX-512 VNNI preferred; CUDA_ARCHS=86 build identical; amd/ stays a non-default workspace member). Quant + measured facts untouched (third-party icecubetr quant; perf points unchanged). Post-2026-10-27 01:16Z commits deferred to the next sweep.

Revision history

942f2d4b4c recipe: update deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram (r2: /engine/pinned_commit, /engine/version, engine_source mirror, artifacts engine_image pin — engine pin bump 971c0725bf 45 h ago diff
1b69fcc513 recipe: create deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram 11 d ago

Fork

Export

recipe.yaml (raw)