Model Types Explained

AI/ML Documentation » Model Types Explained

A diffusion image generator is not one file. It is a pipeline of separately trained networks: a text encoder that turns the prompt into embeddings, a denoiser (U-Net or diffusion transformer) that generates in a compressed latent space, and a VAE that decodes latents into pixels. Around that core sit add-ons: LoRAs and other adapters that change the denoiser’s behavior, and control models that steer it with images. This page explains what each component is, how the model families differ, which file formats and precisions you will run into, and what is compatible with what.

Quick Reference

A checkpoint in the SD 1.5/SDXL sense is a single .safetensors file that bundles the denoiser, text encoder(s), and VAE. Newer and larger models are usually distributed as separate component files instead, because the text encoder alone can be larger than an entire SDXL checkpoint and is often swapped for a quantized copy.

How the Components Fit Together

flowchart LR
    Prompt["Prompt"] --> TE["Text encoder<br/>CLIP / T5 / LLM"]
    Emb["Embeddings"] -.-> TE
    TE -->|conditioning| Den["Denoiser<br/>U-Net or DiT"]
    Noise["Random latent noise"] --> Den
    Den -->|"repeat N steps"| Den
    LoRA["LoRA / LyCORIS"] -.->|"patches weights"| Den
    Ctrl["ControlNet"] -.->|"adds residuals"| Den
    IPA["Image adapter /<br/>reference image"] -.->|"extra attention input"| Den
    Den --> Lat["Clean latent"]
    Lat --> VAE["VAE decoder"]
    VAE --> Img["Image"]

1. The text encoder runs once per prompt and produces a sequence of embedding vectors (plus, for some models, a pooled summary vector).
2. The denoiser starts from random noise in latent space and removes noise over a series of steps (roughly 20-50 for standard models, 1-8 for distilled ones). At each step it attends to the text embeddings. Classifier-free guidance (CFG) usually runs it twice per step, once with and once without the prompt.
3. The VAE decoder turns the final latent into pixels. For all current families the latent is 8x smaller than the image in each spatial dimension, so a 1024x1024 image is denoised as a 128x128 latent.

The dashed add-ons never replace the base model. A LoRA patches its weights, a ControlNet injects extra features, and an image adapter adds another input for attention. That is why add-ons are tied to one specific base architecture. See Stable Diffusion Fundamentals for the underlying diffusion process.

Base Models and Families

The denoiser defines the family, and the family decides which text encoder, VAE, LoRAs, and control models are compatible. The field moved from convolutional U-Nets (SD 1.5, SDXL) to diffusion transformers (DiT/MMDiT) trained with flow matching (SD3, FLUX, Qwen-Image, Z-Image), and from CLIP text encoders to T5 and then to full LLM or vision-language encoders.

Parameter counts are for the denoiser only. Check each model card for exact license terms, which vary between variants. Pony Diffusion V6, Illustrious-XL, and NoobAI-XL are SDXL-architecture fine-tunes: they load anywhere SDXL loads, but their LoRAs work best within their own lineage. See Pony and Fine-Tunes, SDXL, SD3, FLUX, and Base Models Comparison for per-family guidance.

Choosing a base model by content

File Formats and Precision Variants

Container formats

Single-file vs. split components

SD 1.5 and SDXL are usually shared as one all-in-one checkpoint. FLUX, SD 3.5, Qwen-Image, Z-Image, and video models such as Wan are usually shared as separate files: a diffusion-model file, one or more text-encoder files, and a VAE file. In ComfyUI these go in different folders and use different loader nodes, such as Load Diffusion Model, Load CLIP / DualCLIPLoader, and Load VAE.

Precision variants

The same model is often published at several precisions. Smaller variants need less VRAM. Quality loss is negligible at 8 bits and noticeable in fine detail at 4 bits.

For the mechanics of these formats, see Model Compression and Optimization & Performance.

Text Encoders

The text encoder decides how well a model understands the prompt. Each generation of models has used a more capable encoder:

• The 77-token limit belongs to CLIP. UIs such as ComfyUI and A1111 work around it by splitting long prompts into 77-token chunks and concatenating the embeddings, which works but weakens attention to later chunks.
• CLIP skip takes the embedding from an earlier CLIP layer. Many anime fine-tunes of SD 1.5 were trained with CLIP skip 2 and look worse without it. SDXL fine-tunes vary, so follow the model card. CLIP skip has no meaning for T5 or LLM encoders.
• Prompt weighting syntax such as (word:1.3) was designed around CLIP embeddings. Its effect on T5 and LLM encoders is weaker and less predictable.
• Encoder size matters for VRAM. FLUX.1’s T5-XXL is ~9.5 GB in bf16, and FLUX.2’s 24B text encoder is larger than its entire denoiser was a generation ago. Loading the encoder in fp8 or GGUF, or offloading it after encoding, is often the cheapest VRAM win. The encoder runs only once per prompt, so offloading it costs almost nothing.

VAE (Variational Autoencoder)

The VAE compresses images into the latent space the denoiser works in, and decodes the result back to pixels. Every family has its own VAE, and a latent from one family’s denoiser cannot be decoded by another family’s VAE.

More latent channels means less information is lost in compression. That is a large part of why the 16-channel models render small text and textures so much better than SD 1.5/SDXL.

When to change the VAE. Consider it for faded or desaturated colors (common with SD 1.5 fine-tunes that shipped without a baked VAE), for black or NaN images from SDXL in fp16, or for a VAE mismatch after merging models. Otherwise, use the VAE that ships with the model.

Tiled VAE. Decoding a high-resolution latent is a memory spike of its own. Tiled decoding (ComfyUI VAE Decode (Tiled), diffusers enable_vae_tiling()) processes overlapping tiles and uses far less peak memory, at a small speed cost and occasionally faint seams.

LoRA and LyCORIS

A LoRA (Low-Rank Adaptation) stores a learned difference to some of the denoiser’s weight matrices (and optionally the text encoder’s) as the product of two thin matrices. At load time the difference is added to the base weights, scaled by a user-chosen strength. This makes LoRAs small and stackable, and it also means they only fit the architecture they were trained on. How LoRAs are trained is covered in LoRA Training.

Using LoRAs

• Trigger words. Many LoRAs were trained with a specific token or phrase in their captions. Include it in the prompt. It is listed on the model page or in the file’s metadata (ss_tag_frequency).
• Stacking. Effects add, so two strong LoRAs can over-saturate or distort the image. Lower each strength as you add more, and watch for LoRAs that fight over the same features, such as two faces or two styles.
• Model vs. CLIP strength. ComfyUI’s LoraLoader has separate strengths for the denoiser and the text encoder. Most modern LoRAs, including nearly all FLUX LoRAs, train only the denoiser.

LyCORIS and other variants

LyCORIS is a family of LoRA-like parameterizations with different trade-offs. Current tools (ComfyUI, Forge, diffusers via PEFT) load the common ones directly.

Compatibility

A LoRA works only with the architecture it was trained for:

Loading a LoRA on the wrong architecture fails outright (“keys not found” or shape mismatch) or does nothing. Loading an SDXL LoRA on a distant SDXL fine-tune loads fine but can give weak or odd results.

ControlNet and Structural Control

A ControlNet is a trainable copy of the denoiser’s encoder that takes a control image (pose skeleton, depth map, edges, and so on) and adds its features into the base model, so the output follows that structure. Full coverage, including preprocessors, is in ControlNet.

• Union / Pro models. A single ControlNet that accepts many control types, such as Xinsir’s ControlNet Union for SDXL and InstantX/Shakker Union for FLUX.1. One file replaces a folder of single-purpose models.
• Lighter alternatives. T2I-Adapters are small and fast. For FLUX.1, Black Forest Labs’ official FLUX.1 Canny/Depth came as both full models and control LoRAs.
• Strength and timing. Strength 0.5-0.8 guides the image while leaving the model room to work. 1.0 is strict. Applying control only over the first 30-60% of steps (start/end percent) fixes composition and leaves details free.

Image Prompting and Editing

Several kinds of model let a reference image steer generation. They differ in what they take from the image:

Adapters vs. ControlNet. An image adapter says what the output should look like (style, identity, content). A ControlNet says where things should be (structure). They combine well: IP-Adapter for style plus OpenPose for pose gives a specific character in a specific pose.

Edit models have taken over much of what inpainting and adapter stacks used to do. See Inpainting & Editing.

Embeddings (Textual Inversion)

A textual inversion embedding is a handful of learned vectors that act as a new “word” in a CLIP text encoder’s vocabulary. The model’s weights are not changed, which makes the files tiny (kilobytes). The trade-off is that an embedding can only express what the frozen model can already draw.

• Main use today: negative embeddings for SD 1.5 and SDXL, such as EasyNegative and badhandv4 for SD 1.5 and assorted SDXL equivalents. You reference them by filename in the negative prompt, for example embedding:EasyNegative in ComfyUI.
• Compatibility: an embedding is tied to its text encoder. SD 1.5 embeddings don’t work on SDXL (which needs vectors for both of its CLIP encoders). Embeddings have essentially no role with T5 or LLM encoders, so there is no FLUX, SD 3.5, or Qwen-Image equivalent in common use.

Legacy: Hypernetworks

Hypernetworks were small networks that modified the cross-attention layers of SD 1.x. LoRAs replaced them in 2023 because LoRAs are smaller, faster, easier to train, and give better results. You will only meet hypernetworks in old SD 1.5 archives.

Model Merging

Merging combines the weights of models of the same architecture into a new checkpoint. Many popular community checkpoints are merges.

Merging is empirical. Results are hard to predict, and a merged model’s license is bounded by the most restrictive ingredient.

Speed-Optimized Variants

Step-distilled models generate in 1-8 denoising steps instead of 20-50, trading some diversity and fine detail for speed. They usually need CFG near 1 and a matching sampler and scheduler.

Guidance-distilled models such as FLUX.1 [dev] are a separate case. They take guidance as an input and run one pass per step instead of two, but they still need about 20-30 steps. See Optimization & Performance.

Memory Requirements

Approximate weight sizes. Peak VRAM is higher because activations add to it, and it depends on resolution and batch size.

The weights don’t all need to be on the GPU at once. The text encoder, denoiser, and VAE run one after another, and ComfyUI and diffusers offload whichever component is idle. That is how a 12 GB card runs FLUX.1 [dev] in fp8. See Optimization & Performance for offloading and other VRAM tactics.

Organizing Model Files

ComfyUI’s default layout, which other tools largely mirror:

models/
├── checkpoints/       # All-in-one SD 1.5 / SDXL checkpoints
├── diffusion_models/  # Standalone denoisers: FLUX, SD 3.5, Qwen-Image, Z-Image, Wan
│                      #   (older name: unet/)
├── text_encoders/     # CLIP-L, CLIP-G, T5-XXL, Qwen, Mistral encoders (older name: clip/)
├── vae/               # Standalone VAEs
├── loras/             # LoRA and LyCORIS files
├── controlnet/        # ControlNets, T2I-Adapters
├── clip_vision/       # Image encoders used by IP-Adapter / Redux
├── embeddings/        # Textual inversions
└── upscale_models/    # ESRGAN-style upscalers

Put the family and precision in filenames (for example detailer_lora_sdxl.safetensors or flux1-dev_fp8_e4m3fn.safetensors). Architecture mismatch is the most common cause of silent failures.

Choosing Components

Start with a base model that fits your content and hardware, and add components only when a specific need comes up:

flowchart TD
    Start["Pick a base model<br/>(content + VRAM)"] --> Q1{"Specific style or<br/>character needed?"}
    Q1 -->|Yes| LoRA["Add LoRA(s)<br/>matching the family"]
    Q1 -->|No| Q2
    LoRA --> Q2{"Exact pose or<br/>layout needed?"}
    Q2 -->|Yes| CN["Add ControlNet"]
    Q2 -->|No| Q3
    CN --> Q3{"Have a reference image<br/>for style / identity?"}
    Q3 -->|Yes| IP["Image adapter<br/>or edit model"]
    Q3 -->|No| Q4
    IP --> Q4{"Too slow or<br/>out of memory?"}
    Q4 -->|Yes| Opt["Quantized variant,<br/>distilled / turbo model,<br/>offloading"]
    Q4 -->|No| Gen["Generate"]
    Opt --> Gen

See Also

• Stable Diffusion Fundamentals - How latent diffusion works
• Base Models Comparison - Family-by-family comparison
• FLUX Guide - FLUX.1 and FLUX.2 in detail
• LoRA Training - Training your own LoRAs
• ControlNet - Structural control in depth
• ComfyUI Guide - Wiring these components together in node workflows
• Model Compression - How quantized variants are produced
• AI/ML Documentation Hub - Complete AI/ML documentation index