AI/ML Documentation

This section takes you from your first generated image to training your own artistic styles and shipping them to production. It is organized in three arcs:

• Foundations — how diffusion turns noise into images, the model stack, and how to choose a base model for your task and hardware.
• Tools — build node workflows in ComfyUI, train your own LoRAs, edit existing images, and steer composition precisely with ControlNet.
• Production — optimize for your hardware, compress models, automate pipelines, and run reliable, observable services with MLOps.

Suggested path: start with the Foundations, pick up the Tools that match your goal, then move into Production workflows. Use the “Choose Your Path” table below to jump straight to your starting point.

Why Learn AI Image Generation?

AI image generation has transformed from a research curiosity into a practical creative tool. Artists use it to explore new styles, designers prototype concepts in minutes instead of hours, and developers build automated content pipelines. The technology is accessible enough to run on consumer hardware, yet powerful enough for professional applications.

Consider the following before diving in:

• What do you want to create? Photorealistic images, artistic illustrations, anime characters, or product mockups each benefit from different approaches
• How much control do you need? Quick generation versus precise artistic direction require different tools and workflows
• Will you need custom styles or subjects? Training your own models unlocks personalized results that generic models cannot achieve

This documentation covers the practical skills you need, from understanding how the technology works to building production-ready workflows.

Quick Start: Your First Image

The fastest way to generate an image is through ComfyUI’s web interface:

# Start ComfyUI and open http://localhost:8188
docker compose up -d comfyui-server

Once the interface loads, you can use the default workflow immediately. Type your prompt, click “Queue Prompt,” and watch your image generate.

For programmatic access or automation, the MCP API accepts JSON requests:

curl -X POST http://localhost:8189/mcp/tool \
  -H "Content-Type: application/json" \
  -d '{"tool": "generate-image", "arguments": {"prompt": "mountain landscape at sunset"}}'

Hardware at a glance — requirements scale with the models you run:

Most modern NVIDIA GPUs work well. AMD and Apple Silicon have growing support but may require additional configuration. See the ComfyUI Guide for detailed setup, and the Optimization & Performance guide if you need to fit larger models on smaller cards.

Choose Your Path

Different goals require different starting points. Find your path below:

Key Concepts

Understanding a few core ideas will help you make better decisions about models, settings, and workflows.

How Diffusion Models Create Images

Diffusion models learn by studying how images gradually dissolve into random noise, then learning to reverse that process. When you generate an image, the model starts with pure noise and progressively refines it into a coherent picture, guided by your text prompt.

This happens in “latent space” (a compressed mathematical representation) rather than pixel-by-pixel, which is why modern models can run on consumer hardware. Each generation step removes a bit of noise while steering toward your described content.

The Model Stack

AI image generation uses several specialized components working together:

• Base Model - The foundation that understands image-text relationships (SD 1.5, SDXL, FLUX)
• VAE - Compresses images for efficient processing, then decompresses the result
• Text Encoder - Translates your prompt into numbers the model understands
• LoRA - Small add-ons that teach the base model new styles or subjects
• ControlNet - Guides composition using reference images, poses, or edges

Think of the base model as a skilled artist, LoRAs as specialized training, and ControlNet as a reference sketch the artist follows.

Choosing a Workflow Tool

Several interfaces exist for working with these models:

This documentation focuses on ComfyUI because its node-based approach teaches you how the components connect and enables the most advanced workflows.

Model Generations at a Glance

The field evolves quickly. Here is how the major model families compare:

Start with SDXL for the best balance of quality, speed, and ecosystem; use SD 1.5 on limited hardware or for legacy LoRAs; choose FLUX when photorealism matters most; pick SD3 for text-heavy or precision work. See Base Models Comparison for detailed technical differences.

Documentation Overview

Explore the full library, grouped by purpose:

Understanding the Foundations

• Stable Diffusion Fundamentals — how diffusion models turn noise into images, and the parameters that control your results.
• Model Types — the building blocks (checkpoints, LoRAs, VAEs, CLIP/T5, ControlNet, IP-Adapter) and how they fit together.
• Base Models Comparison — SD 1.5 vs SDXL vs SD3 vs FLUX vs Pony: choosing the right foundation for your task and hardware.

Practical Tools

• ComfyUI Guide — the node-based workflow builder for complex, automatable generation pipelines.
• LoRA Training — train custom models for your own styles, characters, or concepts on consumer hardware.
• ControlNet — guide composition with poses, edges, depth maps, and segmentation for precise control.
• Inpainting & Editing — mask, regenerate, extend, and blend regions to edit existing images instead of rerolling them.

Going Further

• Output Formats — diffusion across every medium (image, video, audio, 3D) and how to export each.
• Advanced Techniques — latent interpolation, regional prompting, flow matching, distillation, and expert optimization.
• Game AI Systems — pathfinding, behavior trees, steering, and ML-driven NPCs for real-time interactive AI.

Optimization & Production

• Optimization & Performance — quantization, VRAM-reduction tactics, inference speedups, and batching for diffusion models and LLMs.
• Model Compression — pruning, distillation, quantization, low-rank factorization, and edge deployment, with the accuracy you trade.
• Production Pipelines — headless generation at scale: batch jobs, parameter sweeps, the ComfyUI API, queues, and asset pipelines.
• MLOps & Production — reproducible training, experiment tracking, model registries, rollouts, and drift monitoring for reliable services.

Troubleshooting

When something goes wrong, these are the most common causes and fixes.

Out of Memory Errors

Your GPU ran out of VRAM. Try these solutions in order:

1. Use a smaller model version (fp16 instead of fp32, fp8 for FLUX)
2. Reduce image resolution
3. Enable “low VRAM” or “CPU offloading” in your workflow tool
4. Close other applications using the GPU

For systematic VRAM reduction and quantization, see Optimization & Performance.

Slow Generation

Generation taking too long usually means inefficient settings:

1. Reduce sampling steps (20-30 is often sufficient)
2. Switch to a faster sampler (DPM++ 2M, Euler)
3. Verify GPU is being used (check nvidia-smi)
4. Ensure models are loaded once, not reloaded per image

Poor Quality Results

When images do not match your expectations:

Prompt tips: put the subject first (“a knight in armor” beats “detailed, 4k, masterpiece, knight”), be specific, include lighting/setting/style, and use negative prompts to exclude what you do not want.

Resources and Community

Where to Find Models

• CivitAI - Largest collection of LoRAs, checkpoints, and community models
• Hugging Face - Official model releases and research models

Learning and Help

• Reddit r/StableDiffusion - Active community discussions
• ComfyUI GitHub - Official documentation and issues
• Stable Diffusion Discord - Real-time community help

Research Papers

• Stable Diffusion Paper - Original architecture
• Stable Diffusion 3 Paper - Latest architecture advances

Related Documentation

Broader AI and machine learning concepts beyond image generation:

• AI Fundamentals - Simplified - Conceptual introduction without heavy math
• AI Fundamentals - Complete - Technical deep-dive into AI concepts
• AI Documentation Hub - All AI-related documentation
• Game AI Systems - Real-time AI for NPCs and interactive behaviors

Hardware Note: This documentation assumes NVIDIA GPU access. AMD and Apple Silicon support is improving but may require additional configuration and have limited feature availability.