Language Diffusion Survey

I gave this talk at the Latent Space (opens in a new tab) Paper Club in May 2025, shortly after Google announced Gemini Diffusion. Inception Labs had introduced Mercury a few months earlier with a similar pitch: accuracy comparable to autoregressive models at much higher output speed, on the order of a thousand tokens per second. That seemed worth understanding, so instead of covering one paper I surveyed the lineage that leads from denoising autoencoders to language diffusion models that compete with GPT-2 and Llama 2.

You can also download the full slide deck (PDF, 9MB). The talk was optimized for exploration rather than exploitation: a dozen papers, none in much depth. The full reference list is at the bottom.

What a generative model is trying to do

A generative model aims to learn the underlying probability distribution of a dataset. The true distribution is unknown, so we train a parameterized model to approximate it. Training can be framed as minimizing the KL divergence between the two distributions, or equivalently as maximizing the likelihood of the observed data. The catch is that the distributions we care about, like the distribution of natural images or English sentences, have shapes far too complex for something like a Gaussian to capture. Neural networks can represent them, and the question is how you structure the learning problem so the network actually does.

Different families of generative models answer that question differently. I made this taxonomy as an Obsidian canvas, loosely following Ian Goodfellow’s 2017 GAN tutorial:

Autoregressive models and normalizing flows have tractable densities you can compute and optimize directly. VAEs and diffusion models define a density but optimize a lower bound on it. GANs and energy-based models never define the density explicitly; they just give you a way to sample. All of them are learning some compressed representation of the data and a way to generate from it, and ideas cross-pollinate between the branches constantly.

Where diffusion came from

The earliest thread I could trace is a 2008 paper on denoising autoencoders by Vincent, Larochelle, Bengio, and Manzagol. The idea is simple: randomly zero out parts of the input and train the model to fill in the blanks. In 2013 Bengio and collaborators pushed it further and proved that training a model to denoise implicitly estimates the data-generating process, and that alternating between sampling from the denoising model and the corruption process converges to that estimate. In other words, any denoising autoencoder can be turned into a generative model.

Sohl-Dickstein et al. made the connection to physics in 2015 (Sohl-Dickstein et al., 2015). Their framing borrowed from nonequilibrium thermodynamics: gradually add noise to data through a forward diffusion process, then learn to reverse it. The analogy I like is a drop of ink in water. The forward process is watching the ink spread out until it is uniformly dispersed. The learned reverse process is inferring, from the dispersed state, where the drop went in.

Not much happened with diffusion for the next few years. GANs dominated image generation, though they leaned toward realism over diversity and suffered from mode collapse. I hit this firsthand on a course project: I trained a GAN on Fashion-MNIST and about half of my generated samples came out as boots. VQ-VAE 2 and VQGAN established the other major pattern of the era: compress images into discrete sequences, then predict them autoregressively.

Diffusion came back around 2020 from two directions at once. Song and Ermon’s score-matching line of work trained a network to estimate the gradient of the log-density at each noise level, and their 2020 paper worked out how to choose noise scales in a principled way. The math there goes deeper than I fully understand or want to get into here. Then Ho et al. published DDPM, the breakout paper. Two things made it work: a simplified objective where the network just predicts the noise that was added at a randomly chosen timestep, and a large U-Net borrowed from image segmentation to do the predicting. That combination hit state-of-the-art FID on image benchmarks without any adversarial training. A year later Song et al. unified the diffusion and score-based views under stochastic differential equations, which is the framework most later work builds on.

Getting diffusion to work on text

Everything above operates on continuous values, where adding Gaussian noise to a pixel makes sense. Text is a sequence of discrete tokens, and there is no obvious way to add a little bit of noise to a token. This is the core obstacle for language diffusion, and the paper that cracked it is Structured Denoising Diffusion Models in Discrete State-Spaces (D3PM) (Austin et al., 2021).

Their move was to define corruption through transition matrices over the vocabulary: at each timestep, each token either stays put or transitions according to some structured process. The variant that stuck is the absorbing-state process, where tokens randomly turn into a special [MASK] token. Run it long enough and the whole sequence is masks; the reverse process learns to unmask.

The paper also drew a parallel I found clarifying: BERT is a one-step diffusion model (corrupt by masking, denoise in one shot), and autoregressive models are discrete diffusion models with a deterministic corruption process that masks exactly the next token. This came up in the Q&A, and I’ll get to it below.

Scaling up

Between 2021 and 2024 there was a lot of work I skipped over, including approaches like Diffusion-LM and DiffuSeq that run diffusion in a continuous embedding space and then round to tokens. That direction hasn’t produced today’s state of the art, but the papers are in the references.

The line that did pan out is masked diffusion at scale:

Scaling up Masked Diffusion Models on Text (Nie et al., 2024), from October 2024, showed that masked diffusion models follow scaling laws similar to autoregressive models, with a constant multiplier. Their 1.1B parameter model was competitive with GPT-2 1.5B and, on some benchmarks, Llama 2 7B. The multiplier is real, though: reaching the same validation loss took about 16x the compute of the autoregressive baseline.

LLaDA (Nie et al., 2025), from February 2025, scaled the same recipe to 8B parameters and 2.3 trillion training tokens, matching or beating Llama 2 7B and Llama 3 8B on a number of benchmarks. Because there is no causal mask, the model attends to the whole sequence in both directions. That shows up concretely on reversal tasks: given the last lines of a poem, predict what came before. LLaDA beat much larger autoregressive models, including GPT-4o, on that task. They also introduced a remasking trick at sampling time: after predicting all masked tokens, re-mask the low-confidence predictions and try them again.

Block Diffusion (Arriola et al., 2025), from March 2025, is a hybrid: generate fixed-size blocks autoregressively, but use diffusion within each block. Pure diffusion models regenerate everything from scratch and can’t reuse a KV cache the way autoregressive decoding can; generating block by block restores that, while keeping parallel sampling inside each block.

d1 (Zhao et al., 2025), from April 2025, tackles post-training, which LLaDA largely skipped. They fine-tune LLaDA on the s1k reasoning dataset, then apply diffu-GRPO, a critic-free RL algorithm adapted for the non-sequential nature of diffusion generation. The gains on math and reasoning benchmarks over the base model are substantial.

Why bother?

Two properties make language diffusion interesting despite the compute multiplier.

The first is speed. Diffusion predicts many tokens per step instead of one, which is where the Mercury and Gemini Diffusion throughput claims come from. When we asked Gemini during the talk, it reported Gemini Diffusion runs around 1,479 tokens per second, with a caveat of roughly 0.8 seconds of initial overhead.

The second is bidirectional context. An autoregressive model editing the middle of a file has to take the code after the edit point as prior context and attend to it as if it came earlier. A diffusion model has no causal mask, so it attends to both sides of the edit directly. Code infilling is the obvious application, and it is probably not a coincidence that the early diffusion demos lean on code editing.

Questions from the discussion

The paper club format means the best parts are often interruptions. A few worth keeping, paraphrased:

If BERT is one-step diffusion, how are autoregressive models discrete diffusion? I gave a rough answer and the group sharpened it. Training an autoregressive model amounts to a corruption process that masks exactly one position, the next token, and denoises it, with a deterministic answer instead of a stochastic one. One attendee put it well: autoregressive training is the special case where everything ahead of you is masked and you unmask one position at a time, while BERT masks 15% of positions anywhere in the sequence. Masked diffusion generalizes both: mask an arbitrary fraction, anywhere, and learn to reverse it in steps.

Does diffusion inference scale quadratically with sequence length? Depends entirely on the backbone. The high-profile models (Stable Diffusion, DALL-E, DiT-style video models) use transformers or attention blocks somewhere, so quadratic attention cost shows up. Some earlier DDPM-style models are mostly convolutional. The diffusion process itself is orthogonal to the transformer-vs-not question; diffusion is an alternative to autoregressive generation, not to attention.

Where does attention live in image diffusion models? This one turned into a real argument. The conditioning view: attention (often cross-attention over text embeddings) mostly grounds the generation in the prompt, and the diffusion backbone is separate. The counterpoint, which I’d also argue: even the original 2020 DDPM U-Net has self-attention operators at its 16x16 resolutions, and later architectures add more throughout the backbone. We agreed to argue about it offline.

Since the talk (July 2026)

A year on, most of the threads above kept going. Inception Labs published a Mercury technical report (opens in a new tab) in June 2025, reporting over 1,000 tokens per second on an H100. ByteDance and Tsinghua released Seed Diffusion Preview (opens in a new tab) in August 2025, a code model reporting 2,146 tokens per second, roughly twice Mercury Coder’s throughput and 1.5x Gemini Diffusion’s by their measurements. The LLaDA line kept scaling: LLaDA 1.5 (opens in a new tab) added variance-reduced preference optimization, LLaDA-MoE (opens in a new tab) (September 2025) was the first mixture-of-experts diffusion LM pre-trained from scratch, and LLaDA 2.0 (opens in a new tab) (early 2026) pushed the architecture to a 100B-parameter MoE. The framing has shifted from “can masked diffusion match GPT-2” to shipping code models where latency is the selling point.

I gave a later talk at the same paper club:

Hierarchical Reasoning Models

A written version of my Latent Space talk on HRM: how the architecture works, and what the ARC Prize analysis says actually drives its performance.

Further reading

The wider reading around the five surveyed papers.

Background on diffusion:

• Extracting and Composing Robust Features with Denoising Autoencoders (opens in a new tab), Vincent, Larochelle, Bengio, Manzagol, 2008
• Generalized Denoising Auto-Encoders as Generative Models (opens in a new tab), Bengio et al., 2013
• Improved Techniques for Training Score-Based Generative Models (opens in a new tab), Song & Ermon, 2020
• Denoising Diffusion Probabilistic Models (opens in a new tab), Ho et al., 2020
• Score-Based Generative Modeling Through Stochastic Differential Equations (opens in a new tab), Song et al., 2020

Language diffusion (beyond the surveyed papers):

• Diffusion-LM Improves Controllable Text Generation (opens in a new tab), Li et al., 2022
• DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models (opens in a new tab), Gong et al., 2022

Misc:

• Gemini Diffusion (opens in a new tab), Google DeepMind
• Introducing Mercury (opens in a new tab), Inception Labs
• Diffusion and Score-Based Generative Models (opens in a new tab) (lecture), Yang Song, 2022
• What are Diffusion Models? (opens in a new tab), Lilian Weng, 2021
• Discrete Generative Modeling with Masked Diffusions (opens in a new tab) (talk), Jiaxin Shi, 2024

References

These are notes on the works marked ★ — cite the original rather than this page.

1. Arriola, M., Gokaslan, A., & others. (2025). Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models. arXiv Preprint arXiv:2503.09573. https://arxiv.org/abs/2503.09573 ★
2. Austin, J., Johnson, D. D., Ho, J., Tarlow, D., & van den Berg, R. (2021). Structured Denoising Diffusion Models in Discrete State-Spaces. arXiv Preprint arXiv:2107.03006. https://arxiv.org/abs/2107.03006 ★
3. Nie, S., Zhu, F., & others. (2024). Scaling up Masked Diffusion Models on Text. arXiv Preprint arXiv:2410.18514. https://arxiv.org/abs/2410.18514 ★
4. Nie, S., Zhu, F., & others. (2025). Large Language Diffusion Models. arXiv Preprint arXiv:2502.09992. https://arxiv.org/abs/2502.09992 ★
5. Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. arXiv Preprint arXiv:1503.03585. https://arxiv.org/abs/1503.03585
6. Zhao, S., Gupta, D., Zheng, Q., & Grover, A. (2025). d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning. arXiv Preprint arXiv:2504.12216. https://arxiv.org/abs/2504.12216 ★

Figures: Forward/reverse diffusion figure — from [5] (reproduced in my slides); D3PM Figure 2 — from [2] (reproduced in my slides); LLaDA Figure 2 — from [4] (reproduced in my slides); Block Diffusion Figure 1 — from [1] (reproduced in my slides).

@article{crosse2025language,
  title   = {Language Diffusion Survey},
  author  = {Crosse, Tyler},
  journal = {tylercrosse.com},
  year    = {2025},
  month   = {May},
  url     = {https://tylercrosse.com/ideas/2025/language-diffusion-survey/}
}