DiffusionGemma 26B-A4B-it
A W8A8 INT8 quantization of Google's DiffusionGemma 26B-A4B-it block-diffusion MoE model, cutting it from ~49GB to ~26GB so it runs on Ampere-class GPUs like the RTX 3090 instead of requiring Hopper/Ada for FP8.
- DiffusionGemma
- vLLM
- single-GPU
- Ampere
- INT8
- quantization
- MoE
- block diffusion
- tool calling
Hardware1–2× RTX 3090 (≥32GB)
PrecisionINT8 W8A8 dynamic
RuntimevLLM (DiffusionGemma branch)
Footprint~26GB (from ~49GB bf16)
- A block-diffusion (not purely autoregressive) 26B MoE model, INT8-quantized so it runs on Ampere cards like the RTX 3090 instead of needing Hopper/Ada for FP8.
- Updated on the model card: serving now enables tool calling and "thinking"/reasoning mode (gemma4 parser) — reasoning comes back in message.reasoning.
- High fidelity: <1.2% reconstruction error, ~0.99996 cosine similarity vs. the bf16 original.
- The default max_new_tokens (256) is too low for "thinking" outputs — raise it to ~8192 to avoid truncated generations.
- GPU: Ampere or newer. Reference/tested config is 2× RTX 3090; single-GPU deployment needs ≥32GB VRAM.
- Install vLLM from a branch with DiffusionGemma support — mainline vLLM (through at least the 0.14 nightly) doesn't include it.
- Serve with --trust-remote-code, set --max-num-seqs for your concurrency, and pass diffusion sampler overrides via --hf-overrides.
- Enable tool calling and thinking mode with --enable-auto-tool-choice --reasoning-parser gemma4 --tool-call-parser gemma4 and --default-chat-template-kwargs '{"enable_thinking": true}' — reasoning output comes back in message.reasoning.
- Override the generation config to raise max_new_tokens well past the 256 default (8192 is a reasonable ceiling — thinking mode alone needs ≥1024 tokens of budget).
- Tune --gpu-memory-utilization down (e.g. ~0.75 on 24GB-class cards) to leave headroom for per-decode sampler-buffer overhead.
- Optional: trade some quality for ~3× faster decoding with --diffusion-config '{"canvas_length": 256, "max_denoising_steps": 16}'.
vllm serve aidendle94/diffusiongemma-26B-A4B-it-INT8-dynamic \
--trust-remote-code \
--max-num-seqs 4 \
--hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}' \
--override-generation-config '{"max_new_tokens": 8192}' \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--default-chat-template-kwargs '{"enable_thinking": true}'No community results reported yet — run it and be the first to share what you get.