AI: Loss Functions & Objectives

AI & Machine Learning › Loss Functions & Objectives

A loss function (or objective) is the scalar a model is trained to minimize. After the choice of architecture and data it is the most consequential design decision in a learning system: it defines what “good” means, determines the gradients that flow during backpropagation, and implicitly encodes assumptions about noise, label distribution, and the geometry of the output space. The same network trained with mean-squared error and with cross-entropy learns different things. This page covers the major families — regression, classification, metric and contrastive learning, ranking, knowledge distillation, language-model and preference objectives, and generative objectives — with the probabilistic reasoning behind each and guidance on choosing among them.

Two Organizing Ideas

Losses as Negative Log-Likelihoods

Most standard losses are not arbitrary distances: they are negative log-likelihoods under an assumed conditional distribution $p_\theta(y \mid x)$. Maximum-likelihood estimation minimizes

and each noise model yields a familiar loss:

Choosing a loss is choosing a noise model. MSE assumes symmetric, light-tailed errors; MAE assumes heavy tails and is therefore robust to outliers. The lens tells you when a loss is appropriate, not just how to compute it.

Surrogate Loss vs. Task Metric

The quantity you care about — classification accuracy, intersection-over-union, NDCG, BLEU — is usually piecewise-constant in the parameters and has zero gradient almost everywhere. Training therefore minimizes a smooth surrogate (cross-entropy, hinge, Dice, pairwise logistic) that upper-bounds or correlates with the metric and provides usable gradients. A good surrogate is calibrated for the metric: minimizing it drives the metric toward its optimum. Keep the distinction in mind when training loss improves but the metric does not.

flowchart LR
    X["Input x"] --> M["Model f_theta"]
    M --> Yh["Prediction"]
    Yh --> L["Surrogate loss<br/>(differentiable)"]
    Y["Target y"] --> L
    L -->|"gradient"| M
    Yh --> E["Task metric<br/>(accuracy, IoU, NDCG)<br/>used for evaluation only"]
    Y --> E

Regression Losses

Regression predicts a continuous target; the loss determines how residuals — especially large ones — are penalized.

Mean Squared Error (L2)

MSE is the negative log-likelihood of Gaussian noise with constant variance, and its minimizer is the conditional mean $\mathbb{E}[y \mid x]$. The gradient with respect to a prediction, $\partial \mathcal{L}/\partial \hat{y}_i = -\tfrac{2}{N}(y_i - \hat{y}_i)$, grows linearly with the residual, so the largest errors dominate the update. That makes MSE smooth, convex in the prediction, and well-behaved near the optimum — and highly sensitive to outliers, since a single mislabeled point with a large residual can drag the fit toward it.

Mean Absolute Error (L1)

MAE is the negative log-likelihood of Laplace noise, and its minimizer is the conditional median. Its gradient has constant magnitude (the sign of the residual), so a distant outlier pulls no harder than a near miss — the source of its robustness. The costs are non-differentiability at zero (a subgradient is used) and a gradient that carries no information about how close the prediction is, which can slow late-stage convergence.

Huber (Smooth L1)

Huber loss is quadratic for small residuals and linear for large ones. With threshold $\delta$ and residual $r = y - \hat{y}$:

Small $\delta$ behaves like MAE, large $\delta$ like MSE. Smooth L1 is the same shape rescaled by $1/\delta$; with $\delta = 1$ the two coincide, and it is the standard bounding-box regression loss in detectors such as Faster R-CNN. Huber loss is also the usual choice for the temporal-difference error in deep Q-learning.

MSE grows quadratically and is dominated by large residuals; MAE grows linearly with a kink at zero; Huber is quadratic inside $\lvert r\rvert \le \delta$ and linear outside.

Log-Cosh and Quantile Loss

• Log-cosh, $\sum_i \log\cosh(\hat{y}_i - y_i)$, approximates $r^2/2$ for small residuals and $\lvert r\rvert - \log 2$ for large ones, and is twice differentiable everywhere — a thresholdless alternative to Huber, useful for second-order methods such as gradient boosting.
• Quantile (pinball) loss predicts a chosen quantile $\tau \in (0,1)$ and is the basis of prediction intervals and probabilistic forecasting:

With $\tau = 0.5$ it is half the MAE; with $\tau = 0.9$ under-prediction costs nine times as much as over-prediction, so the model learns an upper bound that the target falls below about 90% of the time. Training several quantiles jointly (e.g. 0.1, 0.5, 0.9) gives an interval forecast.

Classification Losses

Cross-Entropy

For $K$ classes the network outputs logits $\mathbf{z}$, converted by the softmax into probabilities $\hat{p}_k = e^{z_k}/\sum_j e^{z_j}$. With a one-hot target and true class $c$, the categorical cross-entropy is

The binary case, with sigmoid output $\hat{p}$ and label $y \in {0,1}$, is

Multi-label problems (several classes can be present at once) apply an independent BCE per class rather than a softmax.

Cross-entropy dominates classification because of its gradient. For softmax with cross-entropy, the gradient with respect to the logits is simply

“predicted minus true.” It does not saturate when the model is confidently wrong, whereas softmax followed by MSE produces vanishing gradients in exactly that regime. Minimizing cross-entropy is equivalent to minimizing $D_{\mathrm{KL}}(p_{\text{data}} \,|\, \hat{p})$, and it is a proper scoring rule: it is minimized only by reporting the true probabilities, which is why cross-entropy-trained models can be calibrated.

Label Smoothing

One-hot targets push logits toward infinity and produce overconfident models. Label smoothing (Szegedy et al., 2016) mixes the target with a uniform distribution:

This bounds the logit gap the model tries to reach, acts as a regularizer, and typically improves top-1 accuracy; $\epsilon \approx 0.1$ is standard in image classifiers and was used in the original Transformer. Its effect on calibration is mixed — it reduces overconfidence but can harm post-hoc temperature scaling — and it degrades the teacher signal when the model is later used for knowledge distillation (Müller et al., 2019).

Focal Loss

In dense object detection the overwhelming majority of candidate boxes are easy background; even with small individual losses their number swamps the gradient. Focal loss (Lin et al., 2017) down-weights well-classified examples. With $p_t$ the predicted probability of the true class:

With $\gamma = 2$, an easy example at $p_t = 0.99$ has its loss scaled by $10^{-4}$, while a hard example at $p_t = 0.3$ keeps about half of it. $\gamma = 0$ recovers ($\alpha$-weighted) cross-entropy. Focal loss is what allowed the single-stage RetinaNet to match two-stage detectors, and it is widely used for imbalanced dense prediction.

Focal loss for several values of $\gamma$. Larger $\gamma$ suppresses the loss of confidently correct examples (right side) while leaving hard examples (left side) nearly untouched.

Hinge Loss

The hinge loss is the objective of support vector machines. For $y \in {-1, +1}$ and raw score $s$:

It is zero once a point is on the correct side with margin at least 1 and grows linearly inside the margin, so it ignores points that are already confidently correct. That yields a sparse set of support vectors and a max-margin boundary, but no calibrated probabilities. Its squared and multi-class variants appear occasionally in deep networks, and the hinge form survives in GAN discriminator losses.

Segmentation Overlap Losses

Per-pixel cross-entropy on a segmentation mask is dominated by the majority (background) class. The Dice loss optimizes a soft version of the Dice coefficient, the overlap between predicted probabilities $p_i$ and ground truth $g_i$:

with a small smoothing constant $s$. Because it is a ratio over the whole region it is insensitive to class imbalance; the soft Jaccard (IoU) and Lovász-softmax losses are related surrogates. In medical imaging the common default is Dice plus cross-entropy, which combines Dice’s imbalance robustness with cross-entropy’s smooth per-pixel gradients.

Metric Learning: Contrastive, Triplet, and InfoNCE

Metric-learning losses train an embedding space in which similar items are close and dissimilar items far apart. They act on relationships between embeddings rather than absolute predictions, and they underpin face recognition, retrieval, and self-supervised pretraining.

Contrastive (Pairwise) Loss

For a pair of embeddings with label $Y$ (1 = similar, 0 = dissimilar) and distance $D = \lVert z_i - z_j \rVert_2$:

Similar pairs are pulled together; dissimilar pairs are pushed apart only until they exceed the margin $m$, after which they contribute no gradient.

Triplet Loss

For an anchor $a$, positive $p$, and negative $n$, triplet loss requires the anchor to be closer to the positive than to the negative by a margin $m$:

Because it constrains relative rather than absolute distances it is more flexible than the pairwise loss, and it was central to FaceNet (2015). Its practical difficulty is mining: most random triplets already satisfy the margin and yield zero gradient, so training must select hard or semi-hard negatives.

InfoNCE

InfoNCE (van den Oord et al., 2018) generalizes the triplet idea to many negatives and frames metric learning as classification: identify the one positive among a set of candidates. With cosine similarity and temperature $\tau$:

It is a cross-entropy over similarities, and minimizing it maximizes a lower bound on the mutual information between the two views; the bound is at most $\log(1 + K)$ for $K$ negatives, which is one reason more negatives help (MoCo’s momentum queue, CLIP’s batches of tens of thousands). A small $\tau$ sharpens the distribution and concentrates the gradient on the hardest negatives. SimCLR, MoCo, and CLIP all use this loss; CLIP applies it symmetrically in both image-to-text and text-to-image directions.

Sigmoid contrastive loss. SigLIP (Zhai et al., 2023) replaces the batch-wide softmax with an independent binary classification for every image–text pair — matching pairs are positives, all others negatives — using a learned temperature and bias. Removing the global normalization makes the loss cheaper to distribute across devices and performs well at smaller batch sizes; SigLIP-family encoders are widely used as the vision tower in open multimodal models.

Ranking Losses

Search, recommendation, and retrieval care about order, not isolated scores. Ranking losses act on pairs or whole lists.

• Pairwise (RankNet). For items $i$ ranked above $j$ with scores $s_i, s_j$, model $P(i \succ j) = \sigma(s_i - s_j)$ and minimize its cross-entropy, $\log\left(1 + \exp(-(s_i - s_j))\right)$. The margin ranking loss $\max(0,\ m - (s_i - s_j))$ enforces a fixed gap instead.
• Metric-aware (LambdaRank, LambdaMART). Pairwise losses treat all misordered pairs equally, but NDCG and MAP weight the top of the list most. LambdaRank scales each pair’s gradient by the change in NDCG that swapping the pair would cause, optimizing the non-differentiable metric through a weighted pairwise surrogate. LambdaMART — the same idea with gradient-boosted trees — remains a strong learning-to-rank baseline.
• Listwise (ListNet, ListMLE, softmax cross-entropy). Define a distribution over items or permutations and minimize cross-entropy against the ideal ordering. The sampled softmax over one positive and many negatives — structurally identical to InfoNCE — is the standard objective for two-tower retrieval models and dense passage retrieval.

Knowledge Distillation

Distillation (Hinton, Vinyals & Dean, 2015) trains a small student to match a large teacher’s output distribution rather than only the hard labels. Both logits are softened with a temperature $T$, and the student minimizes a mix of the ordinary loss and a KL term:

where $\sigma$ is the softmax. The softened teacher distribution carries “dark knowledge” — which wrong classes are nearly right — and the $T^2$ factor keeps gradient magnitudes comparable as $T$ changes. For language models the same idea appears as token-level KL to the teacher’s next-token distribution, or as sequence-level distillation (fine-tuning on teacher-generated text). Using the reverse KL $D_{\mathrm{KL}}(p_s \,|\, p_t)$, evaluated on the student’s own samples, is mode-seeking and often works better for generative students. See Model Compression for practice.

Language-Model and Preference Objectives

Next-Token Cross-Entropy

Pretraining an autoregressive language model minimizes the cross-entropy of each token given its prefix, averaged over all positions:

Its exponential, $\mathrm{PPL} = \exp(\mathcal{L}_{\text{LM}})$, is the perplexity. This single objective is what scaling laws describe. Common additions at scale are a small z-loss, $\lambda\,(\log Z)^2$ on the softmax normalizer $Z$ (used in PaLM with $\lambda = 10^{-4}$), which keeps logits from drifting and stabilizes low-precision training, and load-balancing auxiliary losses that keep mixture-of-experts routers from collapsing onto a few experts. During supervised fine-tuning the loss is usually masked so only the response tokens, not the prompt, contribute.

Preference Objectives

Post-training aligns a model with human or AI preferences over pairs of responses $(y_w, y_l)$ to a prompt $x$. Most methods start from the Bradley–Terry model of pairwise preference:

which trains a reward model $r_\phi$ for use with RL (PPO in classic RLHF). Direct Preference Optimization (Rafailov et al., 2023) substitutes the closed-form optimal policy into that loss, eliminating the separate reward model:

Structurally it is a binary cross-entropy on a margin between log-probability ratios, so the pairwise-ranking intuition above applies directly. For reasoning models trained with verifiable rewards (math answers, unit tests), policy-gradient objectives such as PPO and GRPO — which normalizes each sample’s reward against a group of samples for the same prompt, avoiding a learned value function — have become standard. Details and variants (IPO, KTO, ORPO) are in Fine-Tuning & Transfer Learning and Reinforcement Learning.

Generative Objectives

A generative model learns a distribution rather than a mapping, and each family is defined as much by its objective as by its architecture. The architectures themselves are covered in Generative Models.

flowchart TD
    G["Generative objective"] --> ML["Maximum likelihood"]
    G --> ADV["Adversarial"]
    G --> REG["Denoising / regression"]
    ML --> AR["Exact: autoregressive<br/>next-token cross-entropy"]
    ML --> ELBO["Lower bound: VAE ELBO"]
    ML --> NF["Exact: normalizing flows<br/>change of variables"]
    ADV --> GAN["GAN: JS divergence"]
    ADV --> WGAN["WGAN: Wasserstein-1"]
    REG --> DIFF["Diffusion: noise-prediction MSE"]
    REG --> FM["Flow matching: velocity MSE"]

Maximum Likelihood and the ELBO

The most direct objective is $\max_\theta \sum_i \log p_\theta(x_i)$. Autoregressive models optimize it exactly through the chain rule and next-token cross-entropy. For latent-variable models such as VAEs, $p_\theta(x) = \int p_\theta(x \mid z)\,p(z)\,dz$ is intractable, so an approximate posterior $q_\phi(z \mid x)$ is introduced and the evidence lower bound is maximized:

The first term rewards reconstruction; the second pulls the approximate posterior toward the prior. The gap to the true log-likelihood is exactly $D_{\mathrm{KL}}\left(q_\phi(z \mid x) \,|\, p_\theta(z \mid x)\right)$. Weighting the KL term by $\beta$ trades reconstruction fidelity for a more regular, sometimes more disentangled latent space; the VAEs inside latent diffusion models use a very small $\beta$ plus perceptual and adversarial terms.

Adversarial Objectives

GANs replace the likelihood with a learned discriminator:

At the optimal discriminator the generator minimizes the Jensen–Shannon divergence between the data and model distributions. When the two barely overlap, that divergence saturates and generator gradients vanish; in practice the generator uses the non-saturating loss $-\log D(G(z))$ instead. The Wasserstein GAN uses the earth-mover distance, which gives useful gradients even for disjoint supports:

with the critic $D$ constrained to be 1-Lipschitz, via a gradient penalty (WGAN-GP) or spectral normalization. The hinge adversarial loss is another common, stable choice. Changes to the objective, not the architecture, did most to make GAN training reliable.

Diffusion and Score Matching

Diffusion models corrupt data with a fixed Gaussian noising process and learn to reverse it. With $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon$, the variational bound reduces (after reweighting) to a regression on the added noise:

It is mean-squared error in disguise, which is why diffusion training is far more stable than adversarial training. Predicting the noise is equivalent, up to a scale factor $-1/\sqrt{1-\bar\alpha_t}$, to estimating the score $\nabla_x \log p_t(x)$ by denoising score matching. The choice of prediction target ($\varepsilon$, $x_0$, or velocity $v$) and of per-timestep weighting — e.g. min-SNR weighting, which caps the weight of low-noise steps — measurably affects convergence speed and sample quality.

Flow Matching

Flow matching and rectified flow regress a velocity field along a path from data to noise. For the straight path $x_t = (1-t)\,x_0 + t\,\varepsilon$:

It is again a simple MSE, but the straighter trajectories need fewer integration steps at sampling time. This objective, with a timestep distribution concentrated on intermediate noise levels, trains Stable Diffusion 3, FLUX, and most image and video models released since 2024.

Masked and Contrastive Pretraining

Self-supervised pretraining manufactures labels from unlabeled data:

• Masked modeling hides part of the input and reconstructs it — cross-entropy over masked tokens (BERT) or MSE over masked image patches (MAE). Joint-embedding predictive methods (I-JEPA, V-JEPA) predict the representation of the masked region instead of its pixels.
• Contrastive pretraining (SimCLR, MoCo, CLIP, SigLIP) uses InfoNCE or its sigmoid variant to align views or modalities.
• Self-distillation (BYOL, DINO/DINOv2) trains a student to match a slowly updated teacher on different augmentations, with no explicit negatives; collapse is avoided by the momentum teacher, centering, and sharpening.

Practical Considerations

• Numerical stability. Never apply softmax or sigmoid and then take a separate log. Use fused ops — nn.CrossEntropyLoss (which takes raw logits) and nn.BCEWithLogitsLoss — which use the log-sum-exp trick to avoid overflow and $\log 0$. Compute losses in float32 even under mixed-precision training.
• Class imbalance. Options include inverse-frequency class weights, resampling, focal loss, logit adjustment (subtracting $\tau \log$ of the class prior from the logits), and, for segmentation, Dice or IoU losses.
• Label noise. Symmetric or bounded losses (MAE-like, generalized cross-entropy) and label smoothing reduce the damage of mislabeled examples; cross-entropy fits noisy labels readily.
• Reduction. Summing versus averaging per-example losses interacts with the learning rate and batch size. Mean reduction keeps gradient scale roughly batch-size-independent; for sequence models, averaging per token rather than per sequence changes the effective weight of long examples.
• Composite losses. Real systems sum several terms — reconstruction, perceptual, adversarial, auxiliary, regularization — each weighted. The relative weights are hyperparameters and often matter more than the form of any single term; normalize each term’s scale and monitor them separately.
• Regularization is part of the objective. Weight decay (applied decoupled from the gradient in AdamW) and other penalties change the optimum. When training loss and validation behavior diverge, remember the true objective includes these terms.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision.ops import sigmoid_focal_loss

# Regression: robust to outliers, smooth near zero.
huber = nn.HuberLoss(delta=1.0)

# Multi-class: takes raw logits; fused log-softmax; optional label smoothing.
ce = nn.CrossEntropyLoss(label_smoothing=0.1)

# Binary / multi-label: fused sigmoid + BCE; pos_weight up-weights rare positives.
bce = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([4.0]))

# Dense detection with extreme imbalance.
def focal(logits, targets):
    return sigmoid_focal_loss(logits, targets, alpha=0.25, gamma=2.0, reduction="mean")

# Contrastive (CLIP-style, symmetric InfoNCE); positives on the diagonal.
def clip_loss(img_emb, txt_emb, temperature=0.07):
    img = F.normalize(img_emb, dim=-1)
    txt = F.normalize(txt_emb, dim=-1)
    logits = img @ txt.T / temperature
    labels = torch.arange(logits.size(0), device=logits.device)
    return 0.5 * (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels))

# Knowledge distillation: soft-target KL plus hard-label CE.
def kd_loss(student_logits, teacher_logits, labels, T=2.0, lam=0.5):
    soft = F.kl_div(
        F.log_softmax(student_logits / T, dim=-1),
        F.log_softmax(teacher_logits / T, dim=-1),
        reduction="batchmean",
        log_target=True,
    ) * T * T
    return (1 - lam) * F.cross_entropy(student_logits, labels) + lam * soft

Choosing a Loss

See Also

• Machine Learning Foundations — optimization, regularization, and maximum likelihood
• Generative Models — the architectures behind the ELBO, adversarial, diffusion, and flow-matching objectives
• Fine-Tuning & Transfer Learning — RLHF, DPO, and related preference methods
• Reinforcement Learning — policy-gradient objectives, PPO, and RLHF
• Frontier Research & Ethics — how the training loss scales with model and data size
• AI Mathematics — formal treatment of maximum likelihood, KL divergence, and variational bounds