AI/ML Documentation

This section is a reference for open-weight image generation: how diffusion and flow-matching models work, how the major model families differ, how to steer and customise them with ComfyUI, LoRAs, ControlNet and editing models, and how to run them efficiently and reliably at scale. It also includes a separate page on game AI. General machine-learning theory lives in AI Fundamentals.

How the Section Is Organised

flowchart LR
    subgraph F["Foundations"]
        F1["Stable Diffusion<br/>Fundamentals"]
        F2["Model Types"]
        F3["Base Models<br/>Comparison"]
    end
    subgraph M["Model families"]
        M1["SDXL"]
        M2["SD3"]
        M3["FLUX"]
        M4["Pony &<br/>fine-tunes"]
    end
    subgraph T["Tools and control"]
        T1["ComfyUI"]
        T2["LoRA Training"]
        T3["ControlNet"]
        T4["Inpainting &<br/>Editing"]
    end
    subgraph P["Production"]
        P1["Optimization"]
        P2["Model<br/>Compression"]
        P3["Production<br/>Pipelines"]
        P4["MLOps"]
    end
    F --> M --> T --> P

Start with the foundations if the vocabulary (latent, sampler, CFG, text encoder) is new; otherwise jump straight to the page for your task.

How Image Generation Works

A modern text-to-image model is a pipeline of separately trained components. The prompt is encoded once; the denoiser then runs for a number of steps, gradually turning random noise in a compressed latent space into a latent image, which the VAE decodes to pixels. Add-ons attach to specific points in this pipeline.

flowchart LR
    Prompt["Prompt"] --> TE["Text encoder(s)<br/>CLIP, T5, or an LLM"]
    Noise["Random latent<br/>(seed)"] --> Den["Denoiser<br/>U-Net or transformer<br/>runs N steps"]
    TE --> Den
    Den --> VAE["VAE decoder"]
    VAE --> Img["Image"]
    LoRA["LoRA<br/>(weight patch)"] -.-> Den
    CN["ControlNet<br/>(pose, depth, edges)"] -.-> Den
    IPA["IP-Adapter / reference<br/>images"] -.-> Den
    Samp["Sampler + scheduler<br/>steps, guidance"] -.-> Den

• Denoiser. Older families (SD 1.5, SDXL) use a convolutional U-Net trained to predict noise. Newer ones (SD3, FLUX, Qwen-Image, Z-Image) use a diffusion transformer trained with flow matching, which predicts a velocity from noise toward the image.
• Text encoder. CLIP for SD 1.5/SDXL; CLIP plus T5 for SD3 and FLUX.1; full language or vision-language models (Qwen, Mistral) in the newest families, which is why they follow long prompts and render text well.
• Guidance. Classifier-free guidance (CFG) runs the denoiser with and without the prompt and extrapolates; guidance-distilled models such as FLUX.1 [dev] take a guidance value directly and keep CFG at 1.
• Steps. Standard models need roughly 20-40 steps; distilled “turbo”, “lightning”, LCM and schnell-style models need 1-8.

Stable Diffusion Fundamentals explains each stage in detail, and Model Types covers the file formats and add-ons.

The Model Landscape (2026)

The trend since 2024 is larger transformer denoisers, language-model text encoders, and single models that both generate and edit. SDXL-based fine-tunes remain popular because they are cheap to run and train. See the Base Models Comparison for a detailed comparison.

Getting Started

Choosing an Interface

These pages use ComfyUI because its graph makes every component explicit and because it can be driven headlessly for production pipelines.

First Image

ComfyUI can be installed as the desktop application, the Windows portable build, or from source:

git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI
python -m venv .venv && source .venv/bin/activate
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128   # match your CUDA version
pip install -r requirements.txt
python main.py            # then open http://127.0.0.1:8188

Put a checkpoint in models/checkpoints/, load the default workflow, type a prompt and queue it. The ComfyUI Guide covers the containerised setup used elsewhere on this site, model folders, custom nodes and the HTTP API.

Hardware

VRAM is the constraint that matters most. Figures are for comfortable 1024 px generation with common fp16/fp8 weights; quantised (GGUF, 4-bit) builds and CPU offload lower them at a speed cost.

NVIDIA GPUs have the broadest support. Apple Silicon works through PyTorch’s MPS backend and AMD through ROCm (Linux) or DirectML/ZLUDA-style layers, with fewer optimised kernels and occasional node incompatibilities. Plan for 100-500 GB of disk once you collect several model families. The Optimization & Performance guide covers quantisation and offloading.

All Pages

Foundations

• Stable Diffusion Fundamentals — diffusion, latents, samplers, CFG and the parameters that control results.
• Model Types — checkpoints, LoRAs, VAEs, text encoders, ControlNets and IP-Adapters, and how they combine.
• Base Models Comparison — choosing among SD 1.5, SDXL, SD3, FLUX and fine-tunes.

Model families

• SDXL Guide — the U-Net workhorse, its refiner, and settings.
• Stable Diffusion 3 Guide — the MM-DiT flow-matching family from Stability AI.
• FLUX Guide — FLUX.1 and FLUX.2: architecture, guidance distillation, variants and licenses.
• Pony & Community Fine-Tunes — Pony, Illustrious and other SDXL-derived models.

Tools and control

• ComfyUI Guide — the node-based workflow builder.
• LoRA Training — datasets, captions, hyperparameters and trainers.
• ControlNet — pose, edge, depth and other spatial conditioning.
• Inpainting & Editing — masking, inpainting, outpainting and instruction editing.
• Advanced Techniques — regional prompting, latent tricks, flow matching and distillation.
• Output Formats — diffusion for video, audio and 3D, and exporting each.

Production

• Optimization & Performance — quantisation, VRAM reduction and inference speed-ups.
• Model Compression — pruning, distillation, quantisation and low-rank methods.
• Production Pipelines — batch generation, the ComfyUI API, queues and asset pipelines.
• MLOps & Production — experiment tracking, registries, rollouts and monitoring.

Other

• Game AI Systems — pathfinding, behavior trees, utility AI, planners and ML-driven NPCs.

Troubleshooting

Prompting differs by family: SD 1.5 and SDXL fine-tunes respond to comma-separated tags and weighting, while T5- and LLM-encoded models (SD3, FLUX, Qwen-Image) respond best to plain descriptive sentences. Put the subject first and describe lighting, composition and style explicitly.

Resources

Models and code

• Hugging Face — official releases and research models
• Civitai — community checkpoints and LoRAs
• ComfyUI and the ComfyUI documentation
• Hugging Face diffusers — the reference Python library

Papers

• High-Resolution Image Synthesis with Latent Diffusion Models (Rombach et al., 2022) — Stable Diffusion
• Flow Matching for Generative Modeling (Lipman et al., 2022) and Rectified Flow (Liu et al., 2022)
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (Esser et al., 2024) — SD3 and MM-DiT
• Adding Conditional Control to Text-to-Image Diffusion Models (Zhang et al., 2023) — ControlNet
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)

Community

• r/StableDiffusion — news and discussion across all open image models

Related Documentation

• AI Fundamentals - Simplified — conceptual introduction without heavy math
• AI Fundamentals - Complete — machine learning and deep learning in depth
• AI Documentation Hub — all AI-related documentation on this site
• Game Development — the wider context for the game AI page