DiffusionGemma 26B-A4B-it INT8 Dynamic
A W8A8 INT8 quantization of Google's DiffusionGemma 26B-A4B-it block-diffusion MoE, cut from ~49GB to ~26GB to run on Ampere-class GPUs like the RTX 3090 instead of Hopper/Ada for FP8.
2× RTX 3090 (or 1× GPU ≥32GB VRAM) · INT8 W8A8 dynamic · vLLM · source Hugging Face
- DiffusionGemma
- vLLM
- single-GPU
- Ampere
- INT8
- quantization
- MoE
- block diffusion
01Overview
- A block-diffusion (not purely autoregressive) 26B MoE model, INT8-quantized so it runs on Ampere cards like the RTX 3090 instead of needing Hopper/Ada for FP8.
- Serving enables tool calling and thinking/reasoning mode (gemma4 parser); reasoning comes back in message.reasoning.
- High fidelity: <1.2% reconstruction error, ~0.99996 cosine similarity vs. the bf16 original.
- The default max_new_tokens (256) is too low for thinking outputs — raise it to ~8192 to avoid truncated generations.
The condensed setup
- GPU: Ampere or newer. Reference/tested config is 2× RTX 3090; single-GPU deployment needs ≥32GB VRAM.
- Install vLLM from a branch with DiffusionGemma support — mainline vLLM (through at least the 0.14 nightly) doesn't include it.
- Serve with --trust-remote-code, set --max-num-seqs for your concurrency, and pass diffusion sampler overrides via --hf-overrides.
- Enable tool calling and thinking mode with --enable-auto-tool-choice --reasoning-parser gemma4 --tool-call-parser gemma4 and --default-chat-template-kwargs '{"enable_thinking": true}' — reasoning output comes back in message.reasoning.
- Override the generation config to raise max_new_tokens past the 256 default (8192 is a reasonable ceiling — thinking mode alone needs ≥1024 tokens).
- Tune --gpu-memory-utilization down (e.g. ~0.75 on 24GB-class cards) to leave headroom for per-decode sampler-buffer overhead.
- Optional: trade some quality for ~3× faster decoding with --diffusion-config '{"canvas_length": 256, "max_denoising_steps": 16}'.
02The recipe
vllm serve aidendle94/diffusiongemma-26B-A4B-it-INT8-dynamic \
--trust-remote-code \
--max-num-seqs 4 \
--hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}' \
--override-generation-config '{"max_new_tokens": 8192}' \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--default-chat-template-kwargs '{"enable_thinking": true}'03Technical details
DiffusionGemma 26B-A4B-it is a block-diffusion language model built on a Gemma MoE backbone: 26B total parameters with roughly 4B active per token, instruction-tuned. This repository is a downstream INT8 quantization of the bf16 original (google/diffusiongemma-26B-A4B-it), released for deployability: the original is ~49GB, too large for most consumer GPUs, and this build cuts that to ~26GB.
Why INT8, not FP8
FP8 kernels need Hopper- or Ada-class hardware (H100, L40S, RTX 40-series and up). This build uses INT8 through vLLM's Triton INT8 MoE backend, which supports Ampere-generation cards — specifically the RTX 3090.
What is quantized
- Weights: int8, symmetric, per-output-channel, via RTN (round-to-nearest) with "memoryless_minmax" calibration — a static one-shot method, not a learned scheme like GPTQ/AWQ.
- Activations: int8, symmetric, per-token, dynamic — computed at runtime on every forward pass.
- Coverage: ~11,725 quantized tensors across all 128 experts and 48 layers; embeddings, the LM head, MoE routers, the vision tower, and self-conditioning components stay in bf16.
- Reported fidelity vs. the bf16 original: relative reconstruction error < 1.2%, cosine similarity ≈ 0.99996.
Tool calling & reasoning
The model card documents --enable-auto-tool-choice, --reasoning-parser gemma4, --tool-call-parser gemma4, and --default-chat-template-kwargs '{"enable_thinking": true}'. With thinking enabled, the reasoning trace comes back in message.reasoning, separate from the final answer. Use the bundled chat_template.jinja as-is — the model self-emits its own channel delimiters, so a custom template is likely to break parsing.
Operational considerations
- Default max_new_tokens is 256; raise it to ~8192 or thinking outputs get truncated (thinking alone needs at least 1024 tokens).
- Each concurrent decode uses roughly 1.9GB of full-vocab fp32 sampler buffers beyond what vLLM's memory profiler accounts for. Budget for this explicitly.
- Tensor-parallel size > 1 on consumer cards needs workarounds until upstream fixes land; a single GPU with ≥32GB VRAM avoids this.
- Trivial imperative prompts (e.g. "Say OK.") can deterministically produce an empty output — a known behavior of the base bf16 model, not the quantization.
Optional: faster decoding
A diffusion-config override with a lower confidence threshold trades some quality for roughly 3x faster decoding:
--diffusion-config '{"canvas_length": 256, "max_denoising_steps": 16}' \
--hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1, "diffusion_confidence_threshold": 0.0}'04Reported results
No community results reported yet — run it and be the first to share what you get.