DynamicArcFaceLoss¶
Google Landmark Recognition 2020 Competition Third Place Solution
SubCenterArcFaceLoss with a margin for each class that grows as the class gets smaller, for long-tailed class distributions.
losses.DynamicArcFaceLoss(class_counts, embedding_size, margin_bounds=(0.05, 0.5), power=0.25, scale=64, sub_centers=3,
*, weight_regularizer=None, weight_reg_weight=1.0, weight_init_func=None, **options)
Equation:
Here \(n_{y_i}\) is the size of the class of embedding \(i\), and \(m_{y_i}\) replaces
the constant margin in the ArcFaceLoss equation. The
coefficients \(a\) and \(b\) follow from margin_bounds: the smallest class gets the
upper bound and the largest class gets the lower bound.
Implementation notes: class_counts is measured on the training set before
the loss is constructed. Its length sets the number of classes. The margins are
in radians and are stored in the class_margins buffer, which .to() moves
with the module and which is not saved in state_dict: reconstruct the loss
with the same arguments before loading a checkpoint. Forward passes do not change
the module. For a target angle above \(\pi - m_{y_i}\), the
target logit is \(s(\cos\theta_{y_i} - m_{y_i} \sin m_{y_i})\), as in
ArcFaceLoss. Everything else, including get_logits, get_outliers, and
miner weights, comes from SubCenterArcFaceLoss.
With sub_centers=1, this is ArcFaceLoss with class-dependent margins.
Counts that are not positive integers raise TypeError for a non-integer dtype
and ValueError otherwise. Identical counts define no margin range and raise
ValueError; use ArcFaceLoss for a constant margin.
Paper settings: The defaults are the paper's best margin function, \(n^{-1/4}\) with bounds 0.05 and 0.5, with \(K = 3\) sub-centers and the scale of the ArcFace paper. The paper reports bounds rather than values of \(a\) and \(b\), so the loss takes bounds; any \(a > 0\) and \(b\) correspond to one pair of bounds.
Example:
import torch
from pytorch_metric_learning import losses
class_counts = torch.tensor([3, 15, 200, 50, 1000])
loss_func = losses.DynamicArcFaceLoss(class_counts, embedding_size=128)
optimizer = torch.optim.SGD(loss_func.parameters(), lr=0.01)
embeddings = torch.randn(32, 128, requires_grad=True)
labels = torch.randint(0, 5, (32,))
loss = loss_func(embeddings, labels)
loss.backward()
optimizer.step()
Parameters:
- class_counts: A one-dimensional integer tensor or sequence with the
number of training examples in each class. Labels must be in
0..len(class_counts) - 1. - embedding_size: The size of the embeddings passed to the loss.
- margin_bounds: The
(lower, upper)margins in radians, with0 <= lower < upper. Another value raisesValueError. - power: The exponent \(\lambda\), which must be positive. The paper compares 1, 1/4, and 1/8.
- scale: The scale \(s\).
- sub_centers: \(K\), the number of sub-centers per class.
- weight_regularizer, weight_reg_weight, weight_init_func: See
learnable weight arguments. The
layout of
weightis that of SubCenterArcFaceLoss. Includeloss_func.parameters()in an optimizer.
Default distance:
CosineSimilarity(). This is the only compatible distance; another distance class raisesTypeError.
Default reducer:
Reducer input:
- loss: One loss per embedding, indexed by embedding position.
power must be finite and positive. The normalized power-law margins are
evaluated from relative class counts in the log domain. Integer differences are
formed before conversion to float64, so distinct large integer counts remain
distinct. Very small and very large finite powers retain finite margins and
the configured endpoints instead of a zero normalization denominator.