Google DeepMind has introduced DiffusionGemma, a text diffusion model developed by retrofitting the existing Gemma 4 model. The model processes blocks of 256 tokens in parallel, similar to image diffusion models, and achieves a throughput of about 1,500 tokens per second on an Nvidia H100 accelerator. This approach allows for significantly faster output generation compared to standard language models, which generate text one token at a time. The retrofitting process used less than ten percent of the original training token budget, according to the technical report. DiffusionGemma maintains comparable accuracy to the base model while delivering several times the output speed of previous diffusion models.
The model underwent two training stages to balance quality and speed. The first stage focused on reconstructing noisy text blocks from example data, while the second combined reinforcement learning with sampler distillation, a process Google calls SD·RL. This approach improved reasoning benchmark scores by an average of ten points and nearly quadrupled the number of tokens processed per compute step. As a result, DiffusionGemma's answers are about 50 percent shorter, further boosting speed. The model can also correct mistakes during later denoising steps, unlike autoregressive models that commit to answers early in the process.
Google acknowledges that DiffusionGemma trails the autoregressive Gemma 4 on quality benchmarks but leads in output speed. The model's performance is limited by its retrofitting process, which was not designed from the start as a diffusion model. The training phase was relatively short, and the SD·RL step prioritized speed over peak quality. Additionally, the model occasionally gets stuck in repetition loops and can forget to close reasoning sections on multimodal tasks, which affects benchmark scores. The speed advantage holds mainly for single-user scenarios, as standard language models catch up with around 32 concurrent requests.
Source: thedecoder
