Skip to content

ContrastiveLoss

Dimensionality Reduction by Learning an Invariant Mapping (Hadsell, Chopra, and LeCun, 2006)

losses.ContrastiveLoss(pos_margin=0.0, neg_margin=1.0, **options)

Equation:

With a distance such as LpDistance, the loss is:

\[ L_{contrastive} = \left[d_p - m_{pos}\right]_+ + \left[m_{neg} - d_n\right]_+ \]

Here \(d_p\) and \(d_n\) are positive-pair and negative-pair distances, \(m_{pos}\) and \(m_{neg}\) are pos_margin and neg_margin, and \([x]_+ = \max(0, x)\).

With a similarity such as CosineSimilarity, the loss is:

\[ L_{contrastive} = \left[m_{pos} - s_p\right]_+ + \left[s_n - m_{neg}\right]_+ \]

Here \(s_p\) and \(s_n\) are positive-pair and negative-pair similarities.

Implementation notes: The hinges are not squared, unlike the paper's \(\frac{1}{2} d_p^2\) and \(\frac{1}{2}[m - d_n]_+^2\). The default reducer averages the positive-pair losses greater than zero and, separately, the negative-pair losses greater than zero, then adds the two averages.

Paper settings: The paper squares both hinges, \(\frac{1}{2} d_p^2\) and \(\frac{1}{2}[m - d_n]_+^2\), on Euclidean distances with pos_margin=0. ContrastiveLoss has no squared-hinge option, so the paper's objective cannot be reproduced through its arguments; its margin \(m\) corresponds to neg_margin.

Example:

import torch
from pytorch_metric_learning import distances, losses

loss_func = losses.ContrastiveLoss(
    pos_margin=1, neg_margin=0, distance=distances.CosineSimilarity()
)
embeddings = torch.randn(32, 128, requires_grad=True)
labels = torch.randint(0, 8, (32,))
loss = loss_func(embeddings, labels)
loss.backward()

Parameters:

  • pos_margin: The distance (or similarity) over (under) which positive pairs contribute to the loss.
  • neg_margin: The distance (or similarity) under (over) which negative pairs contribute to the loss.

The defaults suit a distance such as LpDistance. With a similarity such as CosineSimilarity, use values such as pos_margin=1 and neg_margin=0.

Default distance:

Default reducer:

Reducer input:

  • pos_loss: One loss per (anchor, positive) pair, indexed by (anchors, positives).
  • neg_loss: One loss per (anchor, negative) pair, indexed by (anchors, negatives).