Stable Diffusion Fundamentals

Table of contents

1. Why Learn How Stable Diffusion Works?
2. What Makes Stable Diffusion Special
3. How Diffusion Models Work
   1. Training: Learning to Denoise
   2. Generation: Reversing the Process
   3. Why This Matters Practically
4. The Three Main Components
   1. VAE: The Compressor
   2. U-Net: The Denoiser
   3. Text Encoder: The Translator
5. The Generation Pipeline
6. Sampling Methods: Choosing a Sampler
   1. When to Use Which Sampler
   2. Practical Recommendations
7. CFG Scale: Balancing Creativity and Control
   1. Choosing CFG Values
8. Key Parameters Explained
   1. Resolution
   2. Steps
   3. Seed
9. Writing Effective Prompts
   1. Prompt Structure
   2. Negative Prompts
   3. Emphasis and Weighting
10. Advanced Concepts
    1. Attention: How Text Connects to Image
    2. Noise Schedules
11. Memory and Performance
    1. VRAM Requirements by Model
    2. Reducing Memory Usage
12. Common Issues and Solutions
    1. Quick Quality Improvements
13. The Technology Continues Evolving
    1. Speed Improvements
    2. Better Architectures
14. Putting It Into Practice
15. Conclusion
16. See Also

Why Learn How Stable Diffusion Works?

You can generate images without understanding the underlying technology, but knowing how diffusion models work helps you in practical ways:

• Better prompts - Understanding what the model “sees” helps you write prompts it interprets correctly
• Smarter troubleshooting - When results disappoint, you will know which settings actually matter
• Effective LoRA use - Knowing how models learn helps you combine LoRAs effectively
• Informed model choices - Different architectures have real trade-offs you can evaluate

Consider the following as you read:

The core idea is surprisingly simple. Diffusion models learn to reverse a process of gradually adding noise to images. Once trained, they can start with pure noise and progressively refine it into a coherent image, guided by your text description.

What Makes Stable Diffusion Special

Stable Diffusion, released in 2022, made AI image generation accessible by solving a key problem: earlier diffusion models required enormous computational resources because they worked directly with pixels.

Stable Diffusion instead works in “latent space” - a compressed mathematical representation where a 512x512 image becomes a much smaller 64x64 representation. This compression reduces computation by roughly 50x while preserving the information needed for high-quality images.

The underlying principle remains the same across generations, but each advance improves quality, speed, or both.

How Diffusion Models Work

The process has two phases: training (learning from images) and generation (creating new images).

Training: Learning to Denoise

During training, the model learns by observing what happens when you gradually destroy images with noise:

1. Take a clear training image
2. Add a small amount of random noise
3. Ask the model: “What noise was added?”
4. Compare its answer to the actual noise and improve

Repeat this millions of times with varying amounts of noise, and the model learns to recognize and predict noise at any level. It never learns to “create” images directly - it learns to clean them up.

Generation: Reversing the Process

When you generate an image, the model runs in reverse:

1. Start with pure random noise
2. Ask: “What noise is in this image?”
3. Subtract the predicted noise
4. Repeat until the image is clear

Your text prompt guides which “clean” image the model steers toward. Each step removes a bit of noise while nudging the result toward matching your description.

Pure noise → Shapes emerge → Details form → Final image
  Step 1         Step 10          Step 25        Step 30

Why This Matters Practically

Understanding this process explains several things you will encounter:

• More steps = more refinement - Each step removes noise and adds detail, but returns diminish after 30-50 steps
• CFG scale = prompt strength - Higher values force the model to match your prompt more aggressively
• Seeds control randomness - The same seed produces the same starting noise, enabling reproducible results

The Three Main Components

Stable Diffusion combines three specialized neural networks, each handling a different part of the process.

VAE: The Compressor

The VAE (Variational Autoencoder) translates between pixel images and the compressed latent space where diffusion happens.

Why it matters: Different VAEs produce different color characteristics. If your images have washed-out colors or strange tints, trying a different VAE often helps.

U-Net: The Denoiser

The U-Net (or DiT in newer models) is the core network that predicts noise. It takes three inputs:

• The current noisy image
• How far along in the denoising process we are
• Your text prompt (as numbers)

Why it matters: This is where LoRAs make their modifications. When you train or apply a LoRA, you are adjusting how this network interprets prompts and generates features.

Text Encoder: The Translator

CLIP (or T5 in newer models) converts your text prompt into numerical representations the U-Net can understand.

Why it matters: The text encoder determines how well the model understands your prompt. SDXL uses two text encoders for better comprehension. FLUX uses T5, which handles longer, more natural descriptions better than CLIP.

The Generation Pipeline

Here is what happens when you click “Generate”:

1. Your prompt gets encoded - The text encoder converts your words into numerical vectors
2. Random noise is created - Based on your seed, initial noise fills the latent space
3. Denoising loop runs - For each step, the U-Net predicts noise and removes it
4. VAE decodes the result - The final latent gets converted to a viewable image

In code form, the core loop looks like this:

for step in range(num_steps):
    noise_prediction = unet(current_latent, step, text_embedding)
    current_latent = remove_noise(current_latent, noise_prediction)
final_image = vae.decode(current_latent)

The entire process typically takes 5-30 seconds depending on your settings and hardware.

Sampling Methods: Choosing a Sampler

The “sampler” determines exactly how noise gets removed at each step. Different samplers produce different results and have different speed characteristics.

When to Use Which Sampler

Practical Recommendations

Start with DPM++ 2M - It works well for most purposes and is a good default.

Use Euler for speed - When iterating quickly on prompts, Euler at 20 steps shows you the general direction fast.

Try DPM++ SDE for final renders - When quality matters more than speed, this sampler often produces the best detail.

Euler a for creative exploration - The added randomness can produce unexpected and interesting variations.

CFG Scale: Balancing Creativity and Control

CFG (Classifier-Free Guidance) scale controls how strongly the model follows your prompt versus generating more “natural” images.

The model actually runs your prompt twice internally - once with your text and once without. CFG scale determines how much to amplify the difference between these two predictions.

Choosing CFG Values

Note: FLUX models use a different guidance system and typically use CFG=1 with a separate guidance parameter.

Key Parameters Explained

Every generation involves several settings. Here is what each one controls and how to choose values.

Resolution

Generate at the resolution your model was trained on for best results:

Tip: Generating larger than the training resolution often causes repetition artifacts. Instead, generate at native resolution and upscale afterward.

Steps

More steps mean more refinement, but with diminishing returns:

Seed

The seed determines the random starting noise:

• Random (-1) - Different result each generation
• Fixed number - Same prompt + seed = same image (mostly)
• Seed variation - Change seed slightly to explore similar results

Writing Effective Prompts

Your prompt is the primary way you communicate with the model. Understanding how the model interprets prompts helps you get better results.

Prompt Structure

The model pays more attention to words that appear earlier. Structure your prompts with the most important elements first:

Subject → Details → Style → Quality modifiers
"A red dragon, scales shimmering, perched on a mountain, fantasy digital art"

Negative Prompts

Negative prompts tell the model what to avoid. Common negative prompts address known model weaknesses:

"blurry, low quality, bad anatomy, extra limbs, watermark"

These work by steering the generation away from patterns associated with those words.

Emphasis and Weighting

Most interfaces support adjusting word importance:

Use weighting sparingly. Heavy weighting can cause artifacts or oversaturation of the emphasized concept.

Advanced Concepts

This section covers topics that help advanced users optimize results and understand model behavior more deeply.

Attention: How Text Connects to Image

The “cross-attention” mechanism is how your prompt influences specific parts of the image. The model learns which words should affect which regions.

This is why prompt order matters and why LoRAs can change how specific words are interpreted. Tools like attention visualization can show you which words are affecting which parts of your image.

Noise Schedules

The “scheduler” in your settings controls how aggressively noise gets removed at each step. Different schedules work better for different situations:

Most users can leave this at the default, but experimenting can improve results for specific use cases.

Memory and Performance

Understanding VRAM requirements helps you choose appropriate models and settings for your hardware.

VRAM Requirements by Model

Reducing Memory Usage

If you encounter out-of-memory errors, try these solutions in order:

1. Use fp16 models - Half precision uses half the memory with minimal quality loss
2. Enable low VRAM mode - Your workflow tool likely has this setting
3. Reduce resolution - Generate smaller and upscale afterward
4. Use quantized models - fp8 or GGUF formats use even less memory
5. Enable CPU offloading - Slower but works with limited GPU memory

Common Issues and Solutions

When results are not as expected, here are the most common problems and their fixes:

Quick Quality Improvements

These additions often improve results without other changes:

• Lighting terms: “soft lighting”, “dramatic shadows”, “golden hour”
• Camera terms: “85mm portrait”, “wide angle”, “close-up”
• Quality modifiers: “highly detailed”, “sharp focus” (less effective on newer models)

The Technology Continues Evolving

The field moves quickly. Here are the major developments that change how generation works:

Speed Improvements

Better Architectures

Newer models like FLUX and SD3 use “flow matching” instead of traditional diffusion. This produces straighter paths from noise to image, allowing faster generation with better quality.

Putting It Into Practice

The concepts covered here translate directly to better generation:

1. Start simple - Use default settings, focus on prompt quality first
2. Iterate systematically - Change one parameter at a time to understand its effect
3. Match model to task - Photorealism needs different models than anime art
4. Save what works - Record seeds and settings for successful generations
5. Learn from failures - Artifacts tell you which parameters to adjust

Conclusion

Stable Diffusion makes high-quality image generation accessible on consumer hardware. The core concept - learning to reverse noise - is simple, but the details of prompts, parameters, and model selection determine your results.

With this foundation, you are ready to explore:

• ComfyUI Guide for building practical workflows
• LoRA Training for creating custom styles
• Model Types for understanding all the components

See Also

• ComfyUI Guide - Visual workflow creation for Stable Diffusion
• LoRA Training - Train custom models and styles
• ControlNet - Precise control over generation
• Model Types - Understanding LoRAs, VAEs, and embeddings
• Base Models Comparison - SD 1.5, SDXL, FLUX compared
• Advanced Techniques - Expert generation techniques
• AI Fundamentals - Core AI/ML concepts
• AI/ML Documentation Hub - Complete AI/ML documentation index