Qwen3.8-27B AWQ-INT4 + DSpark on RTX 3090s — SGLang TP1/TP2/TP4

Launch command

docker run -d --name qwen38-3090 --gpus all --shm-size 32g -p 8000:8000 -v '$PWD/models:/models' -e TP=4 ghcr.io/0xsero/qwen38-3090-sglang@sha256:cc2bfc369c709c53d239ef2af41d4ae2530904975816bb2ca484188008586ac9

runtime

env: 'TP=1|2|4 (tensor parallel preset, default 1) | DSPARK=1|0 (default 1; entrypoint auto-disables on TP1 — draft+target=21.7 GB leaves <1 GB KV on one 24 GB card) | PORT=8000 | MODELS_DIR=/models | weights download pinned revisions cyankiwi/Qwen3.8-27B-AWQ-INT4@63768c1 + RadixArk/Qwen3.8-27B-DSpark@85ef153 into /models, trusted only after the entrypoint records the exact downloaded revision'
notes: 'docker/entrypoint.sh @ 13f8f42 resolves the server argv internally: python3 -m sglang.launch_server --model-path /models/Qwen3.8-27B-AWQ-INT4 --tp {TP} --chunked-prefill-size 4096 --mamba-radix-cache-strategy extra_buffer plus per-preset flags — TP1+DSpark: --mem-fraction-static 0.92 --max-mamba-cache-size 4 --disable-prefill-cuda-graph --cuda-graph-max-bs 4 | TP1 plain: 0.92 / 8 / bs 8 | TP2+DSpark: 0.85 / 16 / bs 16 | TP2 plain: 0.85 / 32 / bs 16 | TP4+DSpark: --mem-fraction-static 0.72 (full graphs) | TP4 plain: 0.80; DSpark adds --speculative-algorithm DSPARK --speculative-draft-model-path /models/Qwen3.8-27B-DSpark --speculative-dspark-block-size 7 --speculative-draft-model-quantization unquant. --mamba-radix-cache-strategy extra_buffer is REQUIRED for the hybrid GDN arch (wrong strategies deadlock CUDA graph capture); the mamba state pool sizes concurrency (TP4 max 9 concurrent despite the 275,018-token KV pool; TP2 48). Max context: 8k (TP1) / 64k (TP2) / 128k (TP4). TP1 DSpark possible but impractical (KV ~1 GB). Entry-point preset logic is CI-tested (test/entrypoint-test.sh).'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4.json

Install requirements

• Docker (or a compatible OCI container runtime)
• NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

• Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
• GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
• Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
• Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

• Weights https://huggingface.co/cyankiwi/Qwen3.8-27B-AWQ-INT4
  63768c10df38c0395e12ef49edac1bd539eaeeea — AWQ_REVISION pinned by docker/entrypoint.sh @ 13f8f42; W4A16 group-32 pack-quantized (compressed-tensors), GDN/visual/lm_head layers unquantized
• Drafter https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark
  85ef153be924f17ce4bf62726954eeaa4a73e854 — DRAFT_REVISION pinned by docker/entrypoint.sh @ 13f8f42; 1.36B DSpark draft (DSparkDraftModel), served unquant
• Engine image https://ghcr.io/0xsero/qwen38-3090-sglang
  sha256:cc2bfc369c709c53d239ef2af41d4ae2530904975816bb2ca484188008586ac9 — built from source commit 55ab5b924c3d2176c65bf1132d6c2a5d51ab394e (README pin note)
• Reference https://github.com/sybil-solutions/qwen-serve
  13f8f426e6c7e7de7b141a7e7b060828a660087d @ HEAD 'fix(qwen3.8-b70): publish the image under the repository's own package' — qwen3.8/3090-sglang/ is the recipe source: entrypoint.sh argv, Dockerfile stack pins, benchmarks/tp{1,2,4}-results.json raw rows

Revision notes

Ingested by localai-recipes from sybil-solutions/qwen-serve @ 13f8f426e6c7e7de7b141a7e7b060828a660087d (2026-10-12): launch argv transcribed verbatim from qwen3.8/3090-sglang/entrypoint.sh, benchmark rows transcribed from benchmarks/tp{1,2,4}-results.json, image digest + model/drafter revisions from the README + entrypoint pins; W4A16 group-32 quant config verified from the checkpoint config.json at the pinned revision. The submission draft guessed Qwen/Qwen3.8-27B-Instruct-AWQ — corrected at ingest to the recipe's actual checkpoint cyankiwi/Qwen3.8-27B-AWQ-INT4. The ghcr image is published under the 0xSero account (repo org sybil-solutions). Repo also carries qwen3.6/b70 + qwen3.8/b70 (Arc B70) lanes — future candidates, out of scope here.

Revision history

• cbbf61b9b8 recipe: create qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4 18 d ago diff
• 833bfca827 recipe: create qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4 18 d ago

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)