Skip to content

Losses

Every loss takes embeddings and labels and returns a scalar tensor:

import torch
from pytorch_metric_learning import losses

embeddings = torch.randn(32, 128, requires_grad=True)
labels = torch.randint(0, 8, (32,))

loss_func = losses.TripletMarginLoss()
loss = loss_func(embeddings, labels)

With a miner, pass the mined pairs or triplets as the third argument:

from pytorch_metric_learning import miners

miner_func = miners.MultiSimilarityMiner()
miner_output = miner_func(embeddings, labels)
loss = loss_func(embeddings, labels, miner_output)

Tuple losses accept pair or triplet indices without labels. With ref_emb, positives and negatives index the reference set:

ref_emb = torch.randn(16, 128)
ref_labels = torch.randint(0, 8, (16,))
miner_func = miners.PairMarginMiner()
pairs = miner_func(embeddings, labels)
ref_pairs = miner_func(embeddings, labels, ref_emb=ref_emb, ref_labels=ref_labels)

loss = loss_func(embeddings, indices_tuple=pairs)
loss = loss_func(embeddings, indices_tuple=ref_pairs, ref_emb=ref_emb)

Tuple losses also accept a separate reference set. Anchors come from embeddings, and positives and negatives come from ref_emb:

loss = loss_func(embeddings, labels, ref_emb=ref_emb, ref_labels=ref_labels)

Forward arguments

loss_func(
    embeddings,
    labels=None,
    indices_tuple=None,
    ref_emb=None,
    ref_labels=None,
    *,
    mat=None,
)
  • embeddings: A (N, D) tensor of query embeddings.
  • labels: A (N,) tensor of labels on the embeddings' device. Labels on another device raise ValueError; the loss never copies them.
  • indices_tuple: Optional pair or triplet indices; see index tuples.
  • ref_emb: Optional (M, D) reference embeddings on the embeddings' device; see reference embeddings.
  • ref_labels: A (M,) tensor of reference labels. It requires ref_emb; passing it alone raises ValueError.
  • mat: Optional precomputed distance matrix used in place of the loss's distance call; see precomputed distance matrices.

The return value is a scalar tensor connected to the embeddings and to the loss's parameters. When there are no valid pairs, triplets, or queries, the result is a differentiable zero.

Reference embeddings

ref_emb=None compares the embeddings with each other, and a supplied ref_emb is a separate reference set, even when it is the same tensor. See reference sets for the indexing rule and the labels/ref_labels combinations that raise ValueError.

Precomputed distance matrices

mat lets a caller compute the matrix a loss would otherwise compute. Use it to reuse one computation across a miner and several losses, or to build a matrix from per-batch metadata such as sample ids, which a distance does not receive.

import torch
from pytorch_metric_learning import distances, losses, miners

distance = distances.CosineSimilarity()
miner = miners.MultiSimilarityMiner(distance=distance)
first = losses.MultiSimilarityLoss(distance=distance)
second = losses.ContrastiveLoss(distance=distance)

embeddings = torch.randn(32, 128, requires_grad=True)
labels = torch.arange(32) // 4

mat = distance(embeddings)
pairs = miner(embeddings, labels, mat=mat)
loss = first(embeddings, labels, pairs, mat=mat) + second(
    embeddings, labels, pairs, mat=mat
)
loss.backward()

mat has shape (len(embeddings), len(reference)), where the reference is ref_emb when supplied and embeddings otherwise. With the same inputs, a supported loss returns the same value and gradients as computing the matrix itself.

The rules are:

  • Compute mat once, outside any miner, with gradients enabled. A loss raises ValueError if embeddings or ref_emb require gradients, gradients are enabled, and mat has no graph, because the loss would not train anything.
  • Sum the losses and call backward() once. They share the graph of mat, so a second backward() needs retain_graph=True, as for any shared subgraph.
  • The loss's distance still sets the orientation and margin through is_inverted and margin, so mat must have the same meaning as that distance's output. A loss cannot detect a matrix with the wrong orientation or scale.
  • A wrong shape, an integer dtype, or another device raises ValueError.
  • Nothing is cached. Modules neither change mat nor keep a reference to it.
Loss mat Shape
CircleLoss, ContrastiveLoss, DecoupledContrastiveLoss, DynamicSoftMarginLoss, GeneralizedLiftedStructureLoss, IntraPairVarianceLoss, MarginLoss, MultiSimilarityLoss, MultiSupConLoss, NCALoss, NTXentLoss, RankNContrastLoss, SelectivelyContrastiveTripletLoss, SignalToNoiseRatioContrastiveLoss, SupConLoss, TripletMarginLoss, TupletMarginLoss Supported (N, N), or (N, M) with ref_emb
FastAPLoss, HistogramLoss, InstanceLoss, LiftedStructureLoss, PNPLoss, RankedListLoss, SmoothAPLoss Supported (N, N); these losses reject ref_emb
Proxy and classifier losses, AngularLoss, CenterLoss, CentroidTripletLoss, GE2ELoss, ManifoldLoss, NPairsLoss, ThresholdConsistentMarginLoss ValueError They compare embeddings with their own weights or subsets, or read embeddings directly
CrossBatchMemory, DistributedLossWrapper, MultipleLosses, SelfSupervisedLoss, VICRegLoss TypeError These wrappers build their own inputs or have no distance matrix, so their forward has no mat

TripletMarginLoss(swap=True) raises ValueError with mat, because swapping reads distances between references, which mat does not hold. Check type(loss).supports_mat to test a loss class.

To reduce memory on large batches, supported tuple losses can compute gradients in chunks of anchors with anchor_batched_backward, and every miner can cap its output with tuples_per_anchor.

Index tuples

indices_tuple takes one of two forms, matching what miners return:

  • types.PairIndices(pos_anchors, positives, neg_anchors, negatives): positive pairs (pos_anchors[i], positives[i]) and negative pairs (neg_anchors[j], negatives[j]).
  • types.TripletIndices(anchors, positives, negatives): triplets (anchors[i], positives[i], negatives[i]).

Plain tuples or lists of four or three tensors are accepted and converted to these named tuples. Every tensor must be one-dimensional, have dtype int64 or int32, and be on the embeddings' device. Anchors index embeddings; positives and negatives index ref_emb, or embeddings when ref_emb is None. A non-tensor entry or another index dtype, including bool and uint8, raises TypeError; a wrong length, dimension, or device raises ValueError. Indices must be in range: losses do not check bounds, because that would synchronize the device, so negative or out-of-range indices are a caller error with undefined results.

indices_tuple=None means no restriction: the loss uses every pair or triplet its labels define. A supplied tuple whose tensors are empty means that nothing was selected, which gives a differentiable zero.

Losses use the tuples in one of three ways:

  • Pair losses use the pairs directly and read triplets (a, p, n) as the pairs (a, p) and (a, n). Triplet losses use the triplets directly and form triplets from pairs by combining each anchor's positives with its negatives.
  • Classification, proxy, and some ranking losses use the tuples only to weight embeddings; see miner weights.
  • InstanceLoss, ManifoldLoss, P2SGradLoss, PNPLoss, and RankedListLoss raise ValueError when indices_tuple is supplied.

Supported arguments

Each loss accepts the arguments listed for it below. Passing an unsupported argument raises ValueError. These lists are verified by the test suite.

Losses that accept indices_tuple without labels

  • CircleLoss
  • ContrastiveLoss
  • DecoupledContrastiveLoss
  • DynamicSoftMarginLoss
  • GeneralizedLiftedStructureLoss
  • HistogramLoss
  • IntraPairVarianceLoss
  • LiftedStructureLoss
  • MarginLoss
  • MultiSimilarityLoss
  • NTXentLoss
  • SelectivelyContrastiveTripletLoss
  • SignalToNoiseRatioContrastiveLoss
  • SupConLoss
  • ThresholdConsistentMarginLoss
  • TripletMarginLoss
  • TupletMarginLoss

Losses that accept ref_emb and ref_labels

  • AngularLoss
  • CircleLoss
  • ContrastiveLoss
  • DecoupledContrastiveLoss
  • DynamicSoftMarginLoss
  • GeneralizedLiftedStructureLoss
  • IntraPairVarianceLoss
  • MarginLoss
  • MultiSimilarityLoss
  • MultiSupConLoss
  • NCALoss
  • NTXentLoss
  • RankNContrastLoss
  • SelectivelyContrastiveTripletLoss
  • SignalToNoiseRatioContrastiveLoss
  • SupConLoss
  • ThresholdConsistentMarginLoss
  • TripletMarginLoss
  • TupletMarginLoss

Use a loss from this list whenever queries are compared with a separate reference set, such as a gallery, a second augmented view, or a memory bank. CrossBatchMemory and SelfSupervisedLoss with symmetric=False pass a reference set to the wrapped loss, so they require a loss from this list. Every other loss raises ValueError when ref_emb is supplied.

Loss options

Most loss signatures end with **options, declared by pytorch_metric_learning.types.LossOptions. Losses that return one already-reduced value (DynamicSoftMarginLoss, HistogramLoss, ManifoldLoss, P2SGradLoss, PNPLoss, ThresholdConsistentMarginLoss) list distance (except P2SGradLoss), embedding_regularizer, and embedding_reg_weight explicitly and take no reducer. Wrappers (CrossBatchMemory, MultipleLosses, SelfSupervisedLoss) take the losses they wrap instead. The options are:

  • distance: A distance module. None selects the loss's default distance. A loss that supports only some distances raises TypeError for another distance class and ValueError for an incompatible setting.
  • reducer: A reducer module. None selects the loss's default reducer. The reducer object is used as given, not copied; a loss with several terms applies it to each term unless it is a MultipleReducers.
  • embedding_regularizer: A regularizer applied to the embeddings. None adds no penalty.
  • embedding_reg_weight: The weight multiplying that penalty. The default is 1.
from pytorch_metric_learning import distances, losses, reducers, regularizers

loss_func = losses.ContrastiveLoss(
    neg_margin=0.5,
    distance=distances.CosineSimilarity(),
    reducer=reducers.MeanReducer(),
    embedding_regularizer=regularizers.LpRegularizer(),
    embedding_reg_weight=0.1,
)

Proxy losses also accept the learnable weight arguments, declared by types.ProxyLossOptions.

Regularization terms

Losses with learnable weights also accept weight_regularizer and weight_reg_weight. The total loss is

\[ L = \operatorname{reduce}(\text{objective terms}) + w_e \, R_e(\text{embeddings}) + w_w \, R_w(\text{weights}), \]

where each penalty appears only when its regularizer is supplied. The reducer sees only the loss's own objective terms. Each regularizer applies its own reducer, and the weighted result is added after reduction.

Unreduced loss terms

loss_func.loss_terms(embeddings, labels, ...) takes the same arguments as forward and returns the pytorch_metric_learning.types.LossDict that the reducer would receive, plus the regularization terms embedding_reg_loss and weight_reg_loss when their regularizers are set. Regularization terms are already reduced and already multiplied by their weights. An objective term with the same name as a regularization term raises ValueError from both forward and loss_terms. Use loss_terms to implement a custom aggregation or to inspect per-pair values:

terms = loss_func.loss_terms(embeddings, labels)
pos_pair_losses = terms["pos_loss"]["losses"]

Each term follows the LossTerm contract: its indices say whether one value of losses is already reduced or belongs to an element, a pair, or a triplet, and a term with a divisor requires DivisorReducer. Each loss page lists its terms under Reducer input. A loss with no valid pairs, triplets, or queries returns {} or terms with empty losses, which reduce to a differentiable zero; an embedding regularizer still adds its penalty.

Miner weights

Losses that do not consume pairs or triplets directly can still use a miner's output. ArcFaceLoss, CosFaceLoss, DynamicArcFaceLoss, FastAPLoss, LargeMarginSoftmaxLoss, NCALoss, NormalizedSoftmaxLoss, ProxyAnchorLoss, ProxyNCALoss, SmoothAPLoss, SoftTripleLoss, SphereFaceLoss, and SubCenterArcFaceLoss turn indices_tuple into one weight per embedding:

\[ w_i = \frac{c_i}{\max_j c_j}, \]

where \(c_i\) counts the occurrences of embedding \(i\) in the tuples. Embeddings absent from the tuples get weight 0, and the most frequent embedding gets weight 1. With indices_tuple=None, every weight is 1. A supplied tuple that contains no indices, such as an empty miner result, gives every embedding weight 0, so the loss is a differentiable zero rather than the unmined loss. NCALoss counts only anchors when ref_emb is supplied, because positives and negatives then index the reference set. ProxyAnchorLoss multiplies each exponential term by its weight; the other losses multiply each embedding's loss by its weight. These losses require labels.

Learnable weight arguments

BaseProxyLoss is the base class of every loss that owns one learnable vector per class or per class center. It stores them in a parameter named weight with shape (num_weights, embedding_size); row r is one vector, as in torch.nn.Linear.weight. With one vector per class, row c belongs to class c. With K centers per class, rows c * K through (c + 1) * K - 1 belong to class c. Each subclass declares these keyword-only arguments:

  • weight_regularizer: A regularizer applied to weight. If None, no weight penalty is added. The penalty appears in loss_terms as weight_reg_loss.
  • weight_reg_weight: The weight multiplying that penalty before it is added to the reduced loss. The default is 1.0.
  • weight_init_func: A callable that initializes weight in place, such as torch.nn.init.xavier_uniform_ or functools.partial(torch.nn.init.kaiming_uniform_, a=1, mode="fan_out"). If None, the loss's default_weight_init(weight) is used, as listed below.
Loss weight shape Default initialization
ArcFaceLoss, CosFaceLoss, LargeMarginSoftmaxLoss, SphereFaceLoss (C, D) torch.nn.init.normal_
SubCenterArcFaceLoss, DynamicArcFaceLoss (C·S, D) torch.nn.init.normal_
NormalizedSoftmaxLoss (C, D) torch.nn.init.normal_
ProxyAnchorLoss (C, D) torch.nn.init.kaiming_normal_(mode="fan_out")
ProxyNCALoss (C, D) torch.nn.init.normal_
SoftTripleLoss (C·K, D) torch.nn.init.kaiming_uniform_(a=math.sqrt(5), mode="fan_out")

D is the embedding size, C the number of classes, S the sub-centers per class, and K the centers per class.

These parameters belong to the loss module. Move the loss with loss_func.to(device, dtype) and include loss_func.parameters() in an optimizer:

import torch
from pytorch_metric_learning import losses

loss_func = losses.ArcFaceLoss(num_classes=10, embedding_size=128)
loss_optimizer = torch.optim.SGD(loss_func.parameters(), lr=0.01)

embeddings = torch.randn(32, 128, requires_grad=True)
labels = torch.randint(0, 10, (32,))
loss = loss_func(embeddings, labels)
loss.backward()
loss_optimizer.step()

Outside autocast, the parameters must have the embeddings' device and dtype; otherwise the loss raises ValueError naming both placements. Under autocast, parameters may keep a higher precision than the embeddings. The loss never moves or casts its own parameters during a forward pass.

P2SGradLoss owns a weight parameter and ManifoldLoss owns proxies under the same placement rule. They do not derive from BaseProxyLoss and accept only the embedding regularization options.

Classification logits

These losses provide loss_func.get_logits(embeddings), which returns a (N, num_classes) tensor with one score per embedding and class. The scores omit the training margin, so they suit prediction and evaluation:

Loss get_logits returns
ArcFaceLoss, CosFaceLoss scale times the cosine similarity to each class weight
SubCenterArcFaceLoss, DynamicArcFaceLoss scale times the cosine similarity to each class's closest sub-center
LargeMarginSoftmaxLoss scale times the weight norm, the embedding norm, and the cosine
SphereFaceLoss scale times the embedding norm and the cosine
NormalizedSoftmaxLoss The similarity to each normalized class weight divided by temperature
ProxyAnchorLoss The distance-matrix value to each proxy, cosine similarity by default
ProxyNCALoss The distance-matrix value to each proxy, squared Euclidean distance by default
SoftTripleLoss The relaxed similarity \(\mathcal{S}'_{i,c}\) over each class's centers

For every loss except ProxyNCALoss with its default distance, a larger value means a closer class. ProxyNCALoss returns distances unless it is configured with a similarity, so its closest class is the smallest value. P2SGradLoss and ManifoldLoss do not provide get_logits.

Choose an objective

Each loss has a separate contract page with its signature, equation, default distance and reducer, the arguments it supports, and the paper's settings when the library default differs. Use the guidance below to pick a starting point.

Pair and triplet objectives

These compare embeddings with each other and need several examples per class in each batch, such as batches from MPerClassSampler.

  • AngularLoss: triplets constrained by the angle at the negative, which is invariant to the scale of the distances.
  • CenterLoss: an auxiliary term that pulls each embedding toward a class center, combined with a classification loss.
  • CentroidTripletLoss: triplets of an anchor and class centroids, which are less sensitive to a single unrepresentative sample.
  • CircleLoss: pairs weighted by how far each similarity is from its optimum, with one margin and one scale.
  • ContrastiveLoss: the simplest pair objective; pull positives within one margin and push negatives beyond another.
  • DynamicSoftMarginLoss: local descriptors and patch matching, where the margin adapts to a running histogram of distances.
  • GeneralizedLiftedStructureLoss: lifted structure per anchor, with support for a separate reference set.
  • HistogramLoss: no margin to tune; it penalizes the overlap of the positive and negative similarity distributions.
  • IntraPairVarianceLoss: an auxiliary term that reduces the variance of pair similarities, combined with TupletMarginLoss through MultipleLosses as in its paper.
  • LiftedStructureLoss: each positive pair against every negative of both of its members in the batch.
  • MarginLoss: a boundary between positive and negative distances, optionally learned per class, from the same paper as DistanceWeightedMiner.
  • MultiSimilarityLoss: a strong pair baseline that weights pairs by their relative similarities; pair it with MultiSimilarityMiner.
  • NCALoss: a softmax over distances to every reference, with no margin.
  • NPairsLoss: batches of one positive pair per class, scored as classification among the batch's positives.
  • RankedListLoss: each anchor's ranked list of references, separating positives from negatives by a margin.
  • SignalToNoiseRatioContrastiveLoss: the contrastive objective measured with the signal-to-noise ratio distance.
  • SelectivelyContrastiveTripletLoss: a triplet objective that only pushes the negative away on hard triplets, which avoids collapse under hard-negative mining.
  • TripletMarginLoss: the standard triplet baseline; combine it with a miner such as TripletMarginMiner.
  • TupletMarginLoss: one positive against all negatives of an anchor with scaled angular margins.

Classification and proxy objectives

These learn one vector per class, or per class center, so they need only labels and work with ordinary random batches. Train their parameters with the model; see learnable weight arguments.

  • ArcFaceLoss: an additive angular margin, widely used for face recognition and other tasks with many classes.
  • CosFaceLoss: an additive cosine margin.
  • DynamicArcFaceLoss: SubCenterArcFace with larger margins for smaller classes, for long-tailed class distributions.
  • LargeMarginSoftmaxLoss: a multiplicative angular margin that keeps the weight and embedding norms.
  • ManifoldLoss: proxies compared through a random-walk similarity over the batch and the proxies.
  • NormalizedSoftmaxLoss: a cosine softmax with a temperature and no margin; a strong, simple baseline.
  • P2SGradLoss: cosine classification without margin or scale hyperparameters.
  • ProxyAnchorLoss: proxies as anchors against all embeddings of the batch; fast to converge.
  • ProxyNCALoss: NCA against one learned proxy per class.
  • SoftTripleLoss: several centers per class, for classes with multiple modes.
  • SphereFaceLoss: a multiplicative angular margin on normalized weights.
  • SubCenterArcFaceLoss: ArcFace with sub-centers per class, for noisy labels; it can report outliers.

Ranking objectives

These optimize a retrieval ranking measure directly and benefit from large batches with several positives per class.

  • FastAPLoss: average precision through a histogram of distances.
  • PNPLoss: average precision that penalizes only negatives ranked before positives, with paper variants.
  • SmoothAPLoss: average precision with a sigmoid relaxation of the ranking.
  • ThresholdConsistentMarginLoss: an auxiliary term that makes one similarity threshold work across classes; add it to another loss with MultipleLosses.

Self-supervised and supervised contrastive objectives

  • DecoupledContrastiveLoss: InfoNCE without the positive in the denominator, which keeps small batches learning.
  • GE2ELoss: each embedding against every class centroid in the batch, with its own class's centroid computed without it.
  • InstanceLoss: classification of each instance against the batch, as in image-text instance retrieval.
  • MultiSupConLoss: SupCon for label sets, with positives chosen and weighted by Jaccard similarity.
  • NTXentLoss: the InfoNCE objective of SimCLR and MoCo; use SelfSupervisedLoss for unlabeled views.
  • RankNContrastLoss: supervised contrastive regression for continuous targets, which orders embeddings by target distance.
  • SupConLoss: the contrastive objective with every same-label sample as a positive.
  • VICRegLoss: two views without negatives, regularized by variance and covariance terms.

Composition and extension