{"slug":"deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram","title":"DeepSeek-V4.1-Flash-1017 dsx 2.6-bit GPTQ 2x RTX 3090 (Engram)","summary":"DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) quantized to 187.6 GB single GGUF (2.6127 bpw routed-expert average: 5717 PTQ1_0 / 2545 Q2_0 / 7098 Q3_K per-expert GPTQ with allocator-spent bit budget; dense Q8_0 attention/shared expert/head, F16 embedding/compressor/indexer/Engram-wkv, F32 norms/router/mHC), served by dsx, a Rust+CUDA engine built for this checkpoint, on two RTX 3090s with 192 GB RAM. ~26 tok/s decode / ~170 tok/s prefill at --ctx 262144 with all 15,360 routed experts resident. Exact routing: no expert pruning, no router bias - the rank file orders placement only. Engram n-gram tables (~203 GB) are NOT quantized: read at runtime from the official safetensor shards by a prefetch thread. Hottest ~3,300 experts in VRAM across both GPUs (GPU0's set adapts to the conversation, DSX_ADAPT=8 swaps/token), remaining experts computed on 14 CPU threads from a hugepage RAM arena. Quality vs FP8/FP4 teacher (dsx eval, Lucebox 1017 set): KL 0.1972, top-1 82.59%, PPL 9.3626.","license":"MIT","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"942f2d4b4ca3b3ca6697019daf09d9615a47b0af","content_hash":"57641e58a944d825b6d88744c42b9451d8ee916a","created_at":"2026-10-28 15:52:50.29183-06","notes":"Community ingest surfaced via r/LocalLLaMA 2026-10-19 (quant author's release thread; not re-verified from .55 - Reddit hard-403). Ingest corrections vs scout card: launch argv re-transcribed at dsx HEAD 971c072 (README quickstart + icecubetr model card Run section); quant ships a SINGLE GGUF named DeepSeek-V4.1-Flash-1017-dsx-2.6bit.gguf (card's 00001-of-00004 shard name belongs to the 2-bit sibling), --chunk 4096 and --vision added per the quant card, placement flags follow the quant card's 192 GB sizing. Card said 'speculative' tag: DSpark exists in dsx core but is wired only into the offline runner (nvidia/src/main.rs), FINDINGS.md section 9 lists speculative decoding as a negative result (off in serving) - tag dropped. Engine source_type=release (built from the git repo, but engine_source.type mirrors the submission enum, not 'git'). All numbers claimed_by=submitter T0; import via POST /api/v1/benchmarks to upgrade. amd/ MI325X runner (110-114 tok/s @256K, MXFP4, KL 0.049) is a deliberate SEPARATE candidate - not folded into this card. PIN REVISION (kanban t_7ae73b9f, owner go Casey 2026-10-28, parent t_fa31dc3c): engine pin 971c0725bfd3..98f1259732f8 audited via GitHub compare (ahead_by=338, 338 commits). Published argv re-verified flag-by-flag at 98f1259732 (nvidia/src/bin/server.rs: --model --engram --tokenizer --host --port --ctx --gpu0-gb and --gpu1-gb kept as aliases of --main-gb/--aux-gb --arena-gb --rank --chunk 4096 --vision \u2014 all supported); GGUF load of the published PTQ1_0/Q2_0/Q3_K/Q8_0/F16 mix still supported (core/src/gguf.rs GType + core/src/formats.rs dequant; additions only). NVIDIA toolchain unchanged (README: Rust stable + CUDA 13, x86 AVX2, AVX-VNNI/AVX-512 VNNI preferred; CUDA_ARCHS=86 build identical; amd/ stays a non-default workspace member). Quant + measured facts untouched (third-party icecubetr quant; perf points unchanged). Post-2026-10-27 01:16Z commits deferred to the next sweep."},"engine_source":{"type":"release","git_url":"https://github.com/lxe/dsx","base_engine":"dsx","pip_package":null,"base_version":"dsx @ 98f1259732","image_digest":null,"install_note":"Source build: clone github.com/lxe/dsx, CUDA_ARCHS=86 cargo build --release -p dsx --features cuda (no release tags, no published image; MIT). Weights: hf download icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit (187.6 GB single GGUF + vision.safetensors + expert-rank.json). Engram+tokenizer from deepseek-ai/DeepSeek-V4.1-Flash-1017: --include model.safetensors.index.json model-00047-of-00048.safetensors model-00048-of-00048.safetensors tokenizer.json. Needs 2x 24 GB (sm_86; sm_89/sm_120 build too), ~140 GB VRAM+RAM, x86 AVX2 (VNNI preferred), ~350 GB disk. OpenAI /v1 + Anthropic /v1/messages + web UI on :8090.","pinned_commit":"98f1259732f8c3a5f445b25e080daf70fc300967"},"credit":[{"name":"lxe (dsx engine + 2bit reference quant)","url":"https://github.com/lxe/dsx"},{"name":"icecubetr (2.6-bit GPTQ quant + measurements)","url":"https://huggingface.co/icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit"},{"name":"deepseek-ai (DeepSeek-V4.1-Flash-1017 checkpoint)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"Lucebox (1017 eval set)","url":"https://huggingface.co/Lucebox/DeepSeek-V4.1-Flash-1017-ROCMFP-GGUF"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["hf download icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit --local-dir dsx-2.6bit","hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 --local-dir v41-1017 --include model.safetensors.index.json model-00047-of-00048.safetensors model-00048-of-00048.safetensors tokenizer.json","CUDA_ARCHS=86 cargo build --release -p dsx --features cuda","./target/release/dsx-server --model dsx-2.6bit/DeepSeek-V4.1-Flash-1017-dsx-2.6bit.gguf --engram v41-1017 --tokenizer v41-1017/tokenizer.json --host 0.0.0.0 --port 8090 --ctx 262144 --chunk 4096 --gpu0-gb=-1.0 --gpu1-gb=-1.0 --arena-gb 120 --rank dsx-2.6bit/expert-rank.json --vision dsx-2.6bit/vision.safetensors"],"runtime":{"env":"Placement flags: --gpu0-gb/--gpu1-gb = GB of experts per GPU (negative = all free VRAM minus that; --gpu1-gb 0 for one GPU); --arena-gb = RAM for CPU-computed experts; --rank = expert hotness order (expert-rank.json from the quant repo); --vision / --no-vision; DSX_ADAPT=N = GPU0 expert swaps per token (default 8, 0 = static). 192 GB box uses --gpu0-gb=-1.0 --gpu1-gb=-1.0 --arena-gb 120 (the 2.6-bit card's sizing); 128 GB: --gpu0-gb=-1.5 --gpu1-gb=-1.0 --arena-gb 110. Thinking on by default; reasoning_effort:none or thinking:{type:disabled} turns it off. The launch line above is the quant card's own run command (--ctx 262144); --ctx 1048576 is the model's full 1M but lower it unless you need it."},"max_context_len":262144,"storage_requirements":{"note":"~187.6 GB quant GGUF + ~203 GB Engram shards 47-48 + tokenizer (engine README says ~350-400 GB disk; model card says ~400 GB for GGUF+Engram+tokenizer).","ram_gb":192,"nvme_gb":400}},"models":[{"slug":"deepseek-v4-1-flash-1017","name":"DeepSeek-V4.1-Flash-1017","hf_repo":"deepseek-ai/DeepSeek-V4.1-Flash-1017","quant":"dsx-mixed-2-6bit-per-expert-ptq1-0-q2-0-q3-k-gptq-dense-q8-0"}],"hardware":[{"slug":"rtx-3090-x2","vendor":"nvidia","card":"RTX 3090","card_family":"rtx-3090","vram_gb":24,"device_count":2}],"generation":"ampere","tags":["3090","cpu-experts","custom-engine","dsx","engram","exact-routing","gptq","hybrid-experts"],"tasks":["chat","code"],"aa":{"aa_index":45,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/deepseek-v4-1-flash-1017","aa_index_version":"Index v4.3.2 (2026-10-18)","aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-19 12:00:00-06","aa_family_slug":"deepseek-v4-1-flash-1017","aa_resolved_slug":"deepseek-v4-1-flash-1017"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"claimed_by=submitter T0 (quant author's own box: 2x RTX 3090, 192 GB RAM, --ctx 262144, all 15,360 experts resident): ~26 tok/s decode, ~170 tok/s prompt processing. Quality via dsx eval on Lucebox/DeepSeek-V4.1-Flash-1017-ROCMFP-GGUF eval set (8x1024 held-out tokens; KL/top-1 vs FP8/FP4 teacher top-256 log-probs): KL 0.1972, top-1 82.59%, PPL 9.3626, NLL 2.236722 over 8,184 tokens. Comparators, same set: FP8/FP4 teacher KL 0 / 100% / 8.56; Lucebox ROCMFP 2.5 bpw KL 0.253 / 78.7% / 9.78; engine author's 2.03 bpw (lxe/dsx-2bit, 128 GB box, 30-37 tok/s decode) KL 0.462 / 71.5% / 11.66.\", \"tok_s\": 26, \"points\": [{\"notes\": \"all experts resident, 192 GB RAM box; 2x RTX 3090\", \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 26}, {\"notes\": \"prompt processing, 2x RTX 3090; concurrency omitted (NULL) so the row never collides with the c=1 decode tuple\", \"context_len\": 262144, \"prefill_tok_s\": 170}], \"context_len\": 262144}"}],"perf_points":[{"id":1886,"context_len":262144,"prefill_tok_s":170,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"prompt processing, 2x RTX 3090; concurrency omitted (NULL) so the row never collides with the c=1 decode tuple"},{"id":1885,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":26,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"all experts resident, 192 GB RAM box; 2x RTX 3090"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit","pin":"70794f816419731761a8ff07af8ee457020faff1 - 187.6 GB single GGUF; Engram NOT included (read from official shards at runtime)"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017","pin":"official checkpoint - only index + shards 47-48 + tokenizer needed (Engram + tokenizer source)"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/lxe/dsx","pin":"98f1259732f8c3a5f445b25e080daf70fc300967 (HEAD at revision 2026-10-28, audited; no release tags; MIT). NVIDIA runner in nvidia/; AMD MI325X runner (amd/) is a separate future card."}],"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram.json","alternate":"/r/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram","describedby":"/openapi.json"},"updated_at":"2026-10-28 15:52:50.29183-06","created_at":"2026-10-20 21:49:11.030299-06","speeds":{"decode_tok_s":26,"prefill_tok_s":170,"leaderboard":"/api/v1/leaderboard.json"}}