Stable Diffusion Fundamentals

AI/ML Documentation » Stable Diffusion Fundamentals

Stable Diffusion is a family of open-weight latent diffusion models that generate images from text. A diffusion model is trained to remove noise from corrupted images. To generate, it starts from random noise and removes noise step by step, with a text prompt guiding the result. This page explains that mechanism and the three networks that implement it. It then covers what each generation setting does: sampler, scheduler, steps, CFG scale, seed, resolution, and denoising strength. The concepts apply with small changes to every model in the family, from SD 1.5 through SDXL, and to flow-matching successors such as SD3 and FLUX.

Latent Diffusion

Early diffusion models denoised directly in pixel space, which made each step very expensive at useful resolutions. Stable Diffusion, released in August 2022 and based on the latent diffusion paper by Rombach et al., moves the iterative work into a compressed latent space. A variational autoencoder (VAE) encodes an image into a latent roughly 48× smaller. All denoising steps run on that latent, and the result is decoded back to pixels once, at the end.

flowchart LR
    Px["Pixel image<br/>512×512×3<br/>(786,432 values)"] -- "VAE encode" --> Lat["Latent<br/>64×64×4<br/>(16,384 values)"]
    Lat -- "iterative denoising<br/>(all the expensive work)" --> Lat2["Denoised latent"]
    Lat2 -- "VAE decode" --> Px2["Pixel image"]

Text-to-image generation never needs the encoder. It starts from latent noise and uses only the decoder. The encoder is needed only when an input image is involved: img2img, inpainting, and training.

Model Lineage

The table lists Stability AI’s releases, the successors that replaced them, and the newer open models they are commonly compared against.

The broad trend runs from U-Net to transformer (DiT/MMDiT) denoisers, from noise prediction to flow matching, and from CLIP to large language-model text encoders. The mechanism described on this page is the same across all of them. Model-specific details are covered in the SDXL, SD3, and FLUX guides.

The Diffusion Process

Forward Process (Adding Noise)

The forward process gradually corrupts a clean latent $x_0$ with Gaussian noise over $T$ timesteps (typically $T = 1000$). Because each step adds independent Gaussian noise, any timestep can be sampled directly in closed form:

The noise schedule $\bar{\alpha}_t$ decreases from close to 1 (almost clean) at $t = 0$ toward 0 (almost pure noise) at $t = T$. Samplers often describe the same thing with the noise level $\sigma_t = \sqrt{(1 - \bar{\alpha}_t)/\bar{\alpha}_t}$, which is why sampler settings talk about “sigmas”.

Training Objective

The denoiser $\epsilon_\theta(x_t, t, c)$ receives the noisy latent, the timestep, and the text conditioning $c$, and predicts the noise that was added. Training minimizes the mean squared error:

flowchart LR
    Img["Training image"] --> Enc["VAE encode"] --> X0["x0"]
    X0 --> Mix["Mix with noise<br/>at random t"]
    Eps["Sampled noise ε"] --> Mix
    Mix --> XT["xt"]
    XT --> Net["Denoiser ε_θ(xt, t, c)"]
    Cap["Caption"] --> TE["Text encoder"] --> Net
    Net --> Pred["Predicted noise"]
    Pred --> Loss["MSE loss"]
    Eps --> Loss

The network never learns to “draw” directly. It learns one function, given a noisy input, what does the noise look like?, at every noise level. Sampling, guidance, img2img, and inpainting are all built on repeated calls to that function.

Prediction Targets

Predicting the noise is only one way to parameterize the output. The alternatives are mathematically interchangeable but behave differently in practice, and a checkpoint must be run with the parameterization it was trained with.

Loading a v-prediction checkpoint as if it were ε-prediction, or the reverse, gives gray, washed-out, or noisy output. This is a common source of “broken model” reports.

Reverse Process (Generation)

Generation runs the process backwards. It starts from pure noise $x_T$. At each step the denoiser estimates the noise, the sampler uses that estimate to compute a slightly less noisy latent, and the loop repeats until $t = 0$. Early steps, at high noise, determine composition and color layout. Late steps, at low noise, add texture and fine detail. This split is why the SDXL refiner and hires-fix passes work on the low-noise end only.

flowchart LR
    N["Pure noise<br/>t = T"] --> C["Composition<br/>and color blocks"] --> S["Shapes<br/>and structure"] --> D["Texture<br/>and fine detail"] --> F["Clean latent<br/>t = 0"]
    P["Prompt conditioning"] -.-> C
    P -.-> S
    P -.-> D

The Three Networks

A Stable Diffusion checkpoint bundles three separately trained networks.

flowchart LR
    Prompt["Prompt"] --> TE["Text encoder<br/>(CLIP / T5 / LLM)"]
    Seed["Seed"] --> Noise["Initial latent noise"]
    TE -->|"token embeddings<br/>(cross-attention)"| Den
    Noise --> Den["Denoiser<br/>(U-Net or DiT)"]
    Sch["Sampler + scheduler"] <-->|"one call per step"| Den
    Den --> Lat["Clean latent"]
    Lat --> VAE["VAE decoder"] --> Img["Image"]

VAE

The VAE maps between pixels and latents. SD 1.x and SDXL use a KL-regularized autoencoder with 8× spatial downsampling and 4 latent channels. SD3 and FLUX.1 moved to 16-channel latents, and FLUX.2 introduced a new autoencoder, which preserve much more fine detail (small text, faces at a distance) at the cost of a harder denoising task. The VAE sets an upper bound on detail and color fidelity. It is also the usual cause of washed-out colors, color casts, or black images (fp16 overflow in the original SDXL VAE). VAEs are architecture-specific: an SD 1.5 VAE cannot decode SDXL or FLUX latents.

Denoiser (U-Net or DiT)

The denoiser is the large network that predicts noise or velocity at every step, and it holds most of the parameters.

• U-Net (SD 1.x, 2.x, SDXL): a convolutional encoder-decoder with skip connections. Transformer blocks inside it apply self-attention (image regions attend to each other) and cross-attention (image regions attend to prompt tokens).
• Diffusion Transformer, DiT / MMDiT (SD3, FLUX, and newer models): the latent is cut into patches and processed as a token sequence. MMDiT processes text and image tokens jointly, with information flowing both ways, instead of injecting text only through cross-attention. Transformers scale more predictably with parameters and data, which is why every new flagship model uses one.

LoRAs, and most other fine-tuning, modify the denoiser’s attention and projection weights. Some LoRAs also modify the text encoder.

Text Encoder

The text encoder turns the prompt into a sequence of embeddings that condition the denoiser. Its capacity largely determines how well the model follows prompts.

CLIP was trained to match images with short captions, so it encodes a loose set of concepts. T5 and LLM encoders encode syntax, which is why “a red cube on top of a blue sphere” works reliably only on the newer models.

Samplers and Schedulers

Two settings control how the reverse process is integrated. They are chosen independently.

• Sampler (solver): the numerical method that turns each noise prediction into the next latent, such as Euler, DPM++ 2M, or UniPC.
• Scheduler (sigma schedule): which noise levels the N steps visit, and how they are spaced.

flowchart LR
    Steps["Step count N"] --> Sched["Scheduler<br/>chooses N+1 sigma levels"]
    Sched --> Loop
    subgraph Loop["Sampling loop (per step)"]
        direction LR
        Den["Denoiser predicts<br/>noise at sigma_i"] --> Solv["Sampler computes<br/>latent at sigma_(i+1)"]
    end
    Loop --> Out["Clean latent"]

Samplers

Two properties matter more than the name of the sampler:

• Deterministic vs stochastic. With a fixed seed, every sampler is reproducible on the same hardware and software. Only deterministic samplers converge, though: adding steps refines the same image. Ancestral and SDE samplers inject new noise, so 30 steps and 50 steps give noticeably different images.
• Order. Higher-order and multistep solvers (DPM++ 2M, UniPC) reach a given quality in fewer steps than Euler, but can overshoot at very low step counts or very high CFG.

Schedulers

Common options in ComfyUI and A1111/Forge:

The scheduler used in training (for example linear or cosine $\bar\alpha_t$) is fixed by the checkpoint. The sampling scheduler only chooses which points on that curve to visit.

Classifier-Free Guidance (CFG)

A conditional model’s prediction alone follows the prompt only weakly. Classifier-free guidance runs the denoiser twice per step, once with the prompt ($\epsilon_{\text{cond}}$) and once with an empty or negative prompt ($\epsilon_{\text{uncond}}$), and extrapolates away from the unconditional prediction:

At $w = 1$ this is simply the conditional prediction, with no extrapolation. Larger $w$ pushes the sample further toward what distinguishes the prompt from “anything”. Prompt adherence and contrast increase until colors oversaturate and details burn.

sequenceDiagram
    participant S as Sampler
    participant D as Denoiser
    S->>D: latent + prompt embedding
    D-->>S: eps_cond
    S->>D: latent + negative (or empty) embedding
    D-->>S: eps_uncond
    Note over S: eps = eps_uncond + w * (eps_cond - eps_uncond)
    S->>S: step to next sigma

Practical consequences:

• The negative prompt replaces the unconditional branch. This is how negative prompts work, and why they have no effect at CFG 1.
• CFG roughly doubles compute per step, because it runs two forward passes. This is why distilled models that run at CFG 1 are twice as fast per step.
• Guidance-distilled models (FLUX.1-dev, FLUX.2-dev) learned the guided output directly. They run at CFG 1 with a separate guidance value (about 2.5-4) passed as model input. Raising real CFG on them mostly degrades output.

Several techniques reduce the side effects of high CFG: CFG rescale (introduced for zero-terminal-SNR models), dynamic thresholding, and APG (adaptive projected guidance). Perturbed-attention guidance (PAG) takes a different approach. It adds structure by contrasting against a deliberately degraded prediction instead of relying only on a high $w$. See Advanced Techniques.

Generation Parameters

Resolution

Generate at or near the pixel count the model was trained on. The model learned composition at that scale. Much larger canvases produce duplicated subjects and repeated limbs, and much smaller ones look soft.

For larger output, generate at native size and then upscale in a second pass: latent or pixel upscale, then img2img at about 0.3-0.5 denoise, optionally tiled with a tile ControlNet.

Steps

Seed

The seed initializes the random number generator that produces the starting noise. The same seed, prompt, settings, model, and software stack reproduce the same image. Results can still differ across GPU vendors, precisions, or attention backends, and between CPU- and GPU-generated noise (A1111 generates on the GPU by default, ComfyUI on the CPU). Keeping the seed fixed while changing one parameter is the standard way to see what that parameter does.

Denoising Strength (img2img and Inpainting)

Image-to-image starts from an existing image instead of pure noise. The image is encoded, noised to an intermediate timestep, and the loop runs from there. Denoising strength (0-1) chooses how far into the schedule to start, which is the same as how much of the original the model may change.

flowchart LR
    In["Input image"] --> E["VAE encode"] --> Add["Add noise to level s<br/>(denoising strength)"]
    Add --> Loop["Run the remaining<br/>fraction s of steps"] --> Dec["VAE decode"] --> Out["Output"]

With 30 steps and strength 0.4, only the last 12 steps run. Inpainting applies the same process inside a mask. Dedicated inpainting checkpoints, or models with native editing such as FLUX.1 Kontext, blend the edges better. For structural control that img2img cannot give, see ControlNet and Inpainting & Editing.

Prompting Mechanics

The prompt is split into tokens, the text encoder produces embeddings, and the denoiser attends to them through attention. Some consequences follow from that:

• Earlier tokens tend to carry more weight in CLIP-based models. CLIP’s causal attention and training on short captions favor the start of the prompt. Put the subject first, then details, style, and lighting. This matters less with T5 or LLM encoders.
• Cross-attention is local. Each image region attends to the tokens most relevant to it, which is why concepts “bleed” (a “red hat” also turns the shirt red). Regional prompting and attention masking address this. See Advanced Techniques.
• Negative prompts take the place of the unconditional branch in CFG (see above). Short, specific negatives work better than long boilerplate lists, especially on SDXL and newer models.
• Prompt style depends on the training captions. SD 1.5 anime fine-tunes and Pony, Illustrious, and NoobAI expect danbooru tags. Base SDXL accepts short sentences. SD3, FLUX, and LLM-encoder models expect full descriptive prose and can render quoted text.

Weighting Syntax

A1111, Forge, and ComfyUI scale a token’s embedding with the following syntax:

Weights above about 1.5 usually cause artifacts. On T5 and LLM-encoded models, weighting is less effective and often unsupported. Rephrase instead.

Memory and Performance

Approximate VRAM needed to generate at native resolution with a single model loaded:

In order of cost, the main levers are:

1. Memory-efficient attention. PyTorch SDPA (the default in current front-ends), xFormers, or FlashAttention.
2. Lower-precision weights. fp16/bf16, then fp8, then GGUF or NF4 4-bit quantization. Quality loss is small down to about 8 bits and noticeable below 5.
3. Offloading. Keep only the active network on the GPU. ComfyUI and Forge do this automatically, and diffusers exposes enable_model_cpu_offload().
4. Tiled VAE decode for large images, where the decode alone can exceed VRAM.

See the Optimization Guide for details.

Flow Matching

SD3, FLUX, and most models since 2024 are trained with rectified flow, a form of flow matching, instead of the DDPM noise objective. The noisy sample is a straight-line interpolation between data $x_0$ and noise $\epsilon$:

The network $v_\theta$ is trained to predict the constant velocity along that line:

Generation integrates the ordinary differential equation $\mathrm{d}x/\mathrm{d}t = v_\theta$ from $t = 1$ (noise) to $t = 0$ (data). Because the training paths are straight, even a plain Euler solver with 20-30 steps follows them accurately. Distillation can reduce this to 1-8 steps.

Flow models also use timestep shift. More sampling steps are spent at high noise for larger images, because a high-resolution image keeps its coarse structure visible at noise levels that would erase a small one. ComfyUI exposes this as ModelSamplingSD3 or ModelSamplingFlux (shift).

Few-Step Generation

Distillation trains a student model to match what the full model produces over many steps, but in only a few steps:

Distilled models run at CFG ≈ 1, ignore or weaken negative prompts, and usually need a specific sampler and scheduler, which the model card gives. See the Optimization Guide for the speed trade-offs.

Troubleshooting

See Also

• SDXL Guide - Dual encoders, size conditioning, and the refiner
• SD3 Guide and FLUX Guide - Transformer and flow-matching models
• Base Models Comparison - Choosing a base model
• Model Types - Checkpoints, LoRAs, VAEs, and embeddings
• ComfyUI Guide - Building generation workflows
• ControlNet - Structural control
• Inpainting & Editing - Masked and instruction-based editing
• LoRA Training - Training custom styles and subjects
• Optimization Guide - Quantization, attention backends, and distillation
• Advanced Techniques - Regional prompting, guidance variants, and upscaling
• AI/ML Documentation Hub - Complete AI/ML documentation index