LargeMarginSoftmaxLoss¶
Large-Margin Softmax Loss for Convolutional Neural Networks
losses.LargeMarginSoftmaxLoss(num_classes, embedding_size, margin=4, scale=1,
*, weight_regularizer=None, weight_reg_weight=1.0, weight_init_func=None, **options)
Equation:
where
Here \(\theta_j\) is the angle between \(\mathbf{W}_j\) and \(\mathbf{x}_i\). The implementation computes \(\cos(m\theta)\) from \(\cos\theta\) with this expansion, equations 6 and 7 of the paper, and treats \(k\) as a constant for gradients.
Implementation notes: The paper's annealing of the target logit with a
decreasing \(\lambda\) is not implemented; schedule a combination with an ordinary
softmax loss in application code if needed. Every logit is multiplied by
scale. With an indices_tuple, each embedding's loss is multiplied by its
miner weight. get_logits(embeddings) returns
the logits with \(\cos\theta_j\) in place of \(\psi\); see
classification logits. This loss derives
from BaseProxyLoss.
For float16 and bfloat16 inputs, embedding and weight norms and magnitude
scaling use float32, including the final scale multiplication. The completed
logits keep their usual result dtype and autocast promotion. This preserves
representable scaled logits when an intermediate norm or product would
overflow or underflow in float16. Float64 inputs retain float64. The final
logits still follow their result dtype's range, while cosine and angular-margin
calculations retain their existing precision.
Example:
import torch
from pytorch_metric_learning import losses
loss_func = losses.LargeMarginSoftmaxLoss(num_classes=10, embedding_size=128)
optimizer = torch.optim.SGD(loss_func.parameters(), lr=0.01)
embeddings = torch.randn(32, 128, requires_grad=True)
labels = torch.randint(0, 10, (32,))
loss = loss_func(embeddings, labels)
loss.backward()
optimizer.step()
Parameters:
- num_classes: The number of classes in the training set. Labels must be in
0..num_classes - 1. - embedding_size: The size of the embeddings passed to the loss.
- margin: The integer angular margin \(m\). A value that is not an
intraisesTypeError, and a value below 1 raisesValueError. The paper findsm=4works best. - scale: A multiplier of every logit. The paper uses 1, so it does not appear in the equation.
- weight_regularizer, weight_reg_weight, weight_init_func: See
learnable weight arguments. The
class weights are
weight, shaped(num_classes, embedding_size)and initialized withtorch.nn.init.normal_. Includeloss_func.parameters()in an optimizer.
Default distance:
CosineSimilarity(). This is the only compatible distance; another distance class raisesTypeError.
Default reducer:
Reducer input:
- loss: One loss per embedding, indexed by embedding position.