AI & ML: Generative Models

AI & Machine Learning › Generative Models

A generative model learns an approximation $p_\theta(\mathbf{x})$ of the distribution that produced its training data, so it can draw new samples from it and, in some cases, evaluate how likely a given sample is. This page covers the families that matter in practice: diffusion and flow-matching models (the dominant approach for images, video, and audio), GANs, VAEs, autoregressive models (the basis of large language models), and the newer discrete diffusion language models. It focuses on the objectives and sampling procedures; the loss functions themselves are derived in more depth on Loss Functions & Objectives, and a hands-on pipeline walkthrough lives in Stable Diffusion Fundamentals.

The Families at a Glance

Every family answers the same question — how to turn simple randomness into structured data — with a different trade-off between sample quality, diversity (mode coverage), sampling speed, training stability, and whether the model gives you a likelihood.

Diffusion Models

Diffusion models, introduced by Sohl-Dickstein et al. (2015) and made practical by DDPM (Ho, Jain & Abbeel, 2020), define a fixed forward process that gradually corrupts data into Gaussian noise and train a network to run that process in reverse. Generating an image then means starting from pure noise and repeatedly applying the learned denoiser.

flowchart LR
    x0["x_0<br/>data"] -->|"q: add noise"| x1["x_t<br/>partly noisy"]
    x1 -->|"q: add noise"| xT["x_T<br/>pure Gaussian noise"]
    xT -.->|"p_theta: denoise"| x1b["x_t"]
    x1b -.->|"p_theta: denoise"| x0b["x_0<br/>new sample"]

Solid arrows: the fixed forward (noising) process used to build training pairs. Dashed arrows: the learned reverse process used for generation.

DDPM: The Discrete-Time Formulation

With a variance schedule $\beta_1, \dots, \beta_T$ and $\bar\alpha_t = \prod_{s=1}^{t}(1-\beta_s)$, the forward process has a closed form at any step, so a noisy sample can be produced in one shot:

The reverse transitions $p_\theta(\mathbf{x}{t-1} \mid \mathbf{x}_t)$ are Gaussians whose mean is computed from a network $\varepsilon\theta$ that predicts the added noise. The variational bound on the likelihood simplifies, after reweighting, to a plain regression loss:

Training is therefore: sample an image, a timestep, and a noise vector; build $\mathbf{x}_t$; regress the noise. There is no adversary and no intractable integral, which is why diffusion training is far more stable than GAN training.

Parameterizations. Predicting the noise $\varepsilon$ is one of three equivalent targets. The network can instead predict the clean image $\mathbf{x}_0$, or the velocity $\mathbf{v} = \sqrt{\bar\alpha_t}\,\varepsilon - \sqrt{1-\bar\alpha_t}\,\mathbf{x}_0$ (Salimans & Ho, 2022). They differ in how the loss is weighted across noise levels; $\mathbf{v}$-prediction behaves better at very high noise and is used by several production models.

Schedules. The original linear $\beta_t$ schedule destroys information too quickly at low resolution; the cosine schedule (Nichol & Dhariwal, 2021) and later noise-level-aware formulations fixed this. Karras et al.’s EDM framework (2022) separated the design choices — noise schedule, network preconditioning, loss weighting, and sampler — and showed each can be tuned independently.

from diffusion_models import DDPM

ddpm = DDPM(noise_predictor, T=1000, beta_start=1e-4, beta_end=0.02)  # noise_predictor: a U-Net or DiT

loss = ddpm.loss(batch_images)                                  # noise-prediction MSE
samples = ddpm.sample(shape=(16, 3, 256, 256))                  # 1000 ancestral steps
fast = ddpm.ddim_sample(shape=(16, 3, 256, 256), ddim_timesteps=50)  # deterministic DDIM

The Score-Based (SDE) View

Song et al. (2021) unified diffusion models with score matching by taking the number of steps to infinity. The forward process becomes a stochastic differential equation

and Anderson’s time-reversal result gives the reverse-time SDE, which depends on the data only through the score $\nabla_{\mathbf{x}} \log p_t(\mathbf{x})$:

A network $s_\theta(\mathbf{x}, t)$ is trained to approximate the score by denoising score matching. Noise prediction and score estimation are the same thing up to scale: for the DDPM forward process, $s_\theta(\mathbf{x}t, t) = -\varepsilon\theta(\mathbf{x}_t, t)/\sqrt{1-\bar\alpha_t}$.

The same marginals $p_t$ are also produced by a deterministic probability-flow ODE,

which is what makes fast deterministic samplers (DDIM is a discretization of it), exact likelihood computation, and latent-space interpolation possible. Common choices of $f$ and $g$ are the variance-preserving SDE (the continuous limit of DDPM) and the variance-exploding SDE with geometric noise levels $\sigma(t) = \sigma_{\min}\left(\sigma_{\max}/\sigma_{\min}\right)^{t}$.

Samplers

Because the reverse process is an SDE or ODE, sampling is numerical integration, and better integrators mean fewer network evaluations (NFEs) for the same quality.

Flow Matching and Rectified Flow

Flow matching (Lipman et al., 2023) and the closely related rectified flow (Liu et al., 2023) skip the SDE machinery and learn a velocity field that transports noise to data directly. The simplest version draws a straight line between a data point and a noise sample,

and trains a network $\mathbf{v}_\theta$ to predict the constant velocity along it:

Sampling integrates $d\mathbf{x}/dt = \mathbf{v}_\theta$ from $t=1$ (noise) to $t=0$ (data). Gaussian diffusion is a special case with curved paths; straighter paths are easier to integrate in few steps. Stable Diffusion 3 (Esser et al., 2024) showed that rectified flow with a timestep distribution biased toward the middle noise levels outperformed the standard diffusion formulations at scale, and FLUX.1 and most image and video models released since 2024 are trained with flow-matching objectives. See SD3 Guide and FLUX Guide for the practical consequences.

Guidance and Conditioning

Conditional models (text-to-image, image-to-video) receive the condition $c$ — usually text-encoder embeddings — through cross-attention (U-Net era) or by concatenating text and image tokens into one joint attention sequence (MM-DiT in SD3 and FLUX).

Classifier-free guidance (Ho & Salimans, 2022) trains one network both with and without the condition (the condition is randomly dropped during training) and extrapolates between the two predictions at sampling time:

A guidance scale $w > 1$ trades diversity for prompt adherence and apparent quality; too high a value oversaturates images. CFG doubles the cost per step, so some models (FLUX.1-dev, for example) are guidance-distilled to take $w$ as an input and run a single pass. Structural conditioning such as edges, depth, or pose is added with auxiliary networks like ControlNet.

Latent Diffusion and the Backbone

Running diffusion on raw pixels is expensive at high resolution. Latent diffusion (Rombach et al., 2022) first trains an autoencoder that compresses an image by about 8x per side into a latent tensor, runs diffusion in that latent space, and decodes the result. This is the design behind every Stable Diffusion release and most commercial image and video generators.

flowchart LR
    P["Text prompt"] --> TE["Text encoder(s)<br/>CLIP / T5 / LLM"]
    N["Gaussian noise<br/>in latent space"] --> D
    TE -->|"conditioning"| D["Denoiser<br/>U-Net or DiT<br/>run for N steps"]
    S["Sampler / scheduler"] --> D
    D --> Z["Clean latent z_0"]
    Z --> V["VAE decoder"]
    V --> I["Image / video frames"]

The denoiser architecture has shifted from the convolutional U-Net (SD 1.x, SDXL) to the Diffusion Transformer (DiT; Peebles & Xie, 2023), which treats the latent as a sequence of patches and scales predictably with parameters and compute, just like language transformers. SD3’s MM-DiT, FLUX, and current video models are transformer-based. For video, the latent is a spatio-temporal volume of patches, and the same machinery extends across time.

Fast Sampling: Distillation

Twenty to fifty network calls per image is the main practical cost of diffusion. Several distillation methods reduce it to one to eight:

• Progressive distillation (Salimans & Ho, 2022) repeatedly trains a student to match two teacher steps with one.
• Consistency models (Song et al., 2023) learn a function that maps any point on an ODE trajectory directly to its endpoint, enabling one- or few-step generation; latent consistency models apply this to Stable Diffusion.
• Adversarial diffusion distillation (SDXL Turbo, 2023) and its latent successors combine a distillation loss with a GAN discriminator, producing sharp images in 1–4 steps.
• Distribution-matching and rectified-flow reflow approaches straighten trajectories so a few Euler steps suffice.

The cost of distillation is some loss of diversity and controllability (for example, reduced sensitivity to negative prompts and CFG).

Applications Beyond Still Images

• Video: diffusion transformers over spatio-temporal latents. OpenAI’s Sora (previewed February 2024) made the approach famous; Google’s Veo 3 (2025) added natively generated synchronized audio, and open-weight video models (e.g. HunyuanVideo, Wan) followed.
• Audio and music: latent diffusion over spectrograms or codec latents (Stable Audio), alongside autoregressive codec-token models.
• Science: protein backbone design (RFdiffusion), biomolecular structure prediction (AlphaFold 3 uses a diffusion module to produce atomic coordinates), materials and small-molecule generation.
• Robotics: diffusion policies generate action trajectories conditioned on observations.
• Editing: inpainting, outpainting, and instruction-based image editing reuse the same denoiser with partial noising or extra conditioning; see Inpainting & Editing.

Strengths and Limitations

Generative Adversarial Networks (GANs)

A GAN (Goodfellow et al., 2014) trains a generator $G$ that maps noise $\mathbf{z}$ to samples and a discriminator $D$ that classifies samples as real or generated. They play a minimax game:

At the optimal discriminator the generator minimizes the Jensen–Shannon divergence between the real and generated distributions. In practice the generator is trained with the non-saturating loss $-\log D(G(\mathbf{z}))$, and variants such as the Wasserstein GAN with gradient penalty and spectral normalization were developed to stabilize training (see Loss Functions).

• Strengths: a single forward pass per sample; very sharp outputs; the StyleGAN family produced photorealistic faces years before diffusion caught up.
• Weaknesses: unstable, hyperparameter-sensitive training; mode collapse (the generator covers only part of the distribution); no likelihood; hard to scale to diverse, text-conditioned data.
• Where GANs live now: as a component rather than the whole model — the perceptual/adversarial loss that makes VAE decoders sharp, adversarial distillation of diffusion models, super-resolution (Real-ESRGAN), and neural vocoders for speech.

Variational Autoencoders (VAEs)

A VAE (Kingma & Welling, 2014) pairs an encoder $q_\phi(\mathbf{z} \mid \mathbf{x})$ that maps an input to a distribution over latent codes with a decoder $p_\theta(\mathbf{x} \mid \mathbf{z})$ that reconstructs it. The marginal likelihood is intractable, so training maximizes the evidence lower bound (ELBO):

The first term rewards accurate reconstruction; the second keeps the approximate posterior close to a simple prior $p(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$ so that sampling from the prior yields plausible outputs. The reparameterization trick $\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon}$ makes the expectation differentiable. The approximation machinery is covered under variational inference in Machine Learning Foundations.

• Strengths: stable training, a smooth latent space useful for interpolation and representation learning, a principled probabilistic objective.
• Weaknesses: samples from a pure VAE are blurry, because a factorized Gaussian decoder averages over plausible outputs.
• Modern role: the autoencoder in latent diffusion is a VAE trained with a small KL weight plus perceptual and adversarial losses, making it a high-quality compressor rather than a generator. Its latent channel count (4 in SD 1.x/SDXL, 16 in SD3 and FLUX) is a major factor in fine-detail quality. VQ-VAE replaces the continuous latent with a learned codebook of discrete tokens, which is what allows images and audio to be modeled by autoregressive transformers.

Autoregressive Models

Autoregressive models factor the joint distribution with the chain rule and generate one element at a time, each conditioned on everything before it:

Training is cross-entropy on next-token prediction, parallelized across positions with a causal attention mask. At inference the model produces a distribution over the next token, one token is chosen, appended, and the process repeats:

flowchart LR
    C["Context tokens"] --> T["Transformer<br/>(causal attention, KV cache)"]
    T --> L["Logits over vocabulary"]
    L --> S["Decoding rule<br/>greedy, temperature,<br/>top-k, top-p, min-p"]
    S --> N["Next token"]
    N -->|"append and repeat"| C

This is how every large language model generates text. Decoding strategy matters: greedy decoding is repetitive, temperature scales the logits, top-$k$ and nucleus (top-$p$) sampling truncate the low-probability tail, and speculative decoding uses a small draft model to propose several tokens that the large model verifies in one pass, cutting latency without changing the output distribution. Transformer internals, inference, and serving are covered in the AI Deep Dive.

• Strengths: exact likelihoods; a single, extremely scalable objective; native fit for discrete, sequential data.
• Weaknesses: generation is inherently sequential (one forward pass per token), and errors cannot be revised once emitted.
• Beyond text: WaveNet and PixelCNN were early autoregressive audio and image models. Modern systems tokenize images or audio with a VQ-style codec and model the tokens with a transformer — the approach behind natively multimodal models, and behind GPT-4o’s built-in image generation (March 2025). Visual autoregressive modeling (VAR, 2024) predicts coarse-to-fine token maps rather than raster order, and hybrid designs (e.g. MAR) replace the discrete codebook with a small per-token diffusion head.

Normalizing Flows

A normalizing flow builds $p_\theta(\mathbf{x})$ from an invertible, differentiable map $\mathbf{x} = f_\theta(\mathbf{z})$ of a simple base density, and computes the exact likelihood with the change-of-variables formula:

Architectures (RealNVP, Glow) are designed so the Jacobian determinant is cheap, which restricts expressiveness; flows never matched diffusion on image quality, though transformer-based autoregressive flows have narrowed the gap. Their lasting influence is conceptual: continuous normalizing flows are the direct ancestors of flow matching.

Discrete Diffusion for Language

Diffusion ideas have been carried back to discrete tokens. Masked diffusion language models start from a fully masked sequence and, over several steps, predict and commit tokens in parallel, remasking low-confidence positions — generalizing BERT-style masked modeling into a generative process. Research models such as LLaDA (2025) showed competitive quality with similarly sized autoregressive models, and commercial previews (Inception Labs’ Mercury, Google’s Gemini Diffusion) emphasized very high tokens-per-second for code and text. The appeal is parallel generation and the ability to revise earlier tokens; the open questions are quality at the largest scales, variable-length output, and efficient caching. As of 2026 autoregressive decoding still dominates production LLMs.

Timeline

timeline
    title Milestones in deep generative modeling
    2013-2014 : VAE (Kingma and Welling) : GAN (Goodfellow et al.)
    2015-2016 : Diffusion probabilistic models (Sohl-Dickstein) : PixelCNN and WaveNet
    2017-2019 : VQ-VAE : StyleGAN : GPT-2
    2020-2021 : DDPM : Score-based SDEs : DALL-E and CLIP
    2022 : Latent diffusion / Stable Diffusion : Classifier-free guidance : EDM
    2023 : DiT : Consistency models : Flow matching and rectified flow
    2024 : SD3 (rectified-flow MM-DiT) : FLUX.1 : Sora preview
    2025-2026 : Native autoregressive image generation in LLMs : Video with audio (Veo 3) : Discrete diffusion LMs

Evaluation

Generative models are hard to evaluate because there is no single correct output.

Current practice combines automated metrics with prompt suites that test specific skills (object counts, spatial relations, text rendering) and large-scale human preference rankings.

Continue Reading

See Also

• Loss Functions & Objectives — the ELBO, adversarial, diffusion, and flow-matching objectives in detail
• Stable Diffusion Fundamentals — the latent-diffusion pipeline, hands-on
• SD3 Guide and FLUX Guide — rectified-flow transformers in practice
• Deep Learning Architectures — the transformers and convolutional networks these models are built from
• AI Deep Dive — how LLMs are trained and served
• Frontier Research & Ethics — scaling, safety, and the governance of generative AI