diff --git a/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin/recipe.yaml b/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin/recipe.yaml index 70268a202..65ab07a07 100644 --- a/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin/recipe.yaml +++ b/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin/recipe.yaml @@ -64,7 +64,7 @@ config: - 'ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69' runtime: env: 'GLM53_MODE=nvme (fast | exact | nvme | nvme-exact) | GLM53_EC_MAX_SLOTS=1376 (VRAM expert-cache cap) | --memory 55g --memory-swap 55g sizes the pinned RAM tier from the cgroup cap (16 GiB cap variant: --memory 16g --memory-swap 16g; all-RAM fast: --ulimit memlock=-1 --shm-size 16g, no cap) | GLM53_B70=1 adds the experimental Intel Arc Pro B70 peer-offload expert tier (needs GLM53_MODE=nvme and the b70tier/b70srv.py companion server up first with the shared ring dir mounted; 120 GiB VRAM pair total) | GLM53_MODEL_DIR=/models GLM53_MODEL_REVISION=332ab457b709b7ba30dd9a448be5de03b80a7ac9 PORT=30000 (defaults, verbatim from docker/entrypoint.sh @ 6769b27).' - notes: "docker/entrypoint.sh @ 6769b27 resolves the server argv internally: python3 glm53/serve.py -m /models -cs 131072 --max-batch-size 8 -chunk_size 8192 -ambs 4 --host 0.0.0.0 --port 30000 --served-name glm-5.3-flash (mode nvme adds the shipped S3b2 tuning: GLM53_NV=2 GLM53_NV_CPU=1, 24-slot VRAM victim ring, layer-ahead prefetch, N135 CPU-lane forward, N136 decode switches; the entrypoint die-gates on VRAM < 23,000 MiB and on AVX2+FMA+F16C for the CPU tier, and warns when memlock is capped). One-time store build BEFORE the server: docker run --rm -v /data/glm53:/models:ro -v /mnt/nvme/glm53:/nvx IMG pack-store (117 GB expert store, O_DIRECT sha256 verify of every record + byte-compare of a 300-record sample; GLM53_MODE=nvme* refuses to start without /nvx/glm53_flash_exl3_3.05bpw_experts.bin + .json). Weights: hf download turboderp/GLM-5.3-Flash-exl3 --revision 332ab457 (3.05bpw branch tip, 125.3 GB) or let the entrypoint snapshot_download. Hardware: 1x RTX 3090 PCIe 4.0 x16 (24 GB), EPYC 7443P 24c/48t 8-channel DDR4 (NVMe modes ask for 40+ logical CPUs and AVX2+FMA+F16C), 4x Samsung 9100 PRO 1 TB md RAID0 xfs on PCIe 4.0 x16 (242.6 GB: 125.3 weights + 117.3 store; 16 GiB cap needs the same store, 969-expert tier). Ready in ~80 s at http://127.0.0.1:30000/v1. Alternative: Omarchy Local AI one-click — omarchy plugin add https://github.com/sybil-solutions/omarchy-local-ai --enable, then pick 'GLM-5.3-Flash, EXL3 3.05 bpw (55 GB RAM, experts on NVMe)'." + notes: "docker/entrypoint.sh @ 6769b27 resolves the server argv internally: python3 glm53/serve.py -m /models -cs 131072 --max-batch-size 8 -chunk_size 8192 -ambs 4 --host 0.0.0.0 --port 30000 --served-name glm-5.3-flash (mode nvme adds the shipped S3b2 tuning: GLM53_NV=2 GLM53_NV_CPU=1, 24-slot VRAM victim ring, layer-ahead prefetch, N135 CPU-lane forward, N136 decode switches; the entrypoint die-gates on VRAM < 23,000 MiB and on AVX2+FMA+F16C for the CPU tier, and warns when memlock is capped). One-time store build BEFORE the server: docker run --rm -v /data/glm53:/models:ro -v /mnt/nvme/glm53:/nvx IMG pack-store (117 GB expert store, O_DIRECT sha256 verify of every record + byte-compare of a 300-record sample; GLM53_MODE=nvme* refuses to start without /nvx/glm53_flash_exl3_3.05bpw_experts.bin + .json). Weights: hf download turboderp/GLM-5.3-Flash-exl3 --revision 332ab457 (3.05bpw branch tip, 125.3 GB) or let the entrypoint snapshot_download. Hardware: 1x RTX 3090 PCIe 4.0 x16 (24 GB), EPYC 7443P 24c/48t 8-channel DDR4 (NVMe modes ask for 40+ logical CPUs and AVX2+FMA+F16C), 4x Samsung 9100 PRO 1 TB md RAID0 xfs on PCIe 4.0 x16 (242.6 GB: 125.3 weights + 117.3 store; 16 GiB cap needs the same store, 969-expert tier). Ready in ~80 s at http://127.0.0.1:30000/v1. Alternative: Omarchy Local AI one-click — omarchy plugin add https://github.com/0xSero/omarchy-local-ai --enable, then pick 'GLM-5.3-Flash, EXL3 3.05 bpw (55 GB RAM, experts on NVMe)'." performance: claimed_by: submitter metrics: