NormalizedSoftmaxLoss¶
Classification is a Strong Baseline for Deep Metric Learning
losses.NormalizedSoftmaxLoss(num_classes, embedding_size, temperature=0.05,
*, weight_regularizer=None, weight_reg_weight=1.0, weight_init_func=None, **options)
Equation:
Here \(x\) is the L2-normalized embedding with label \(y\), \(p_z\) is the L2-normalized weight vector of class \(z\), and \(Z\) is the set of all classes.
Implementation notes: The class weights are L2-normalized at every forward
pass, exactly once, using the distance normalization and its epsilon. The
embeddings are normalized by the default distance. With a distance
where smaller means closer, the logits are negated distances. With an
indices_tuple, each embedding's loss is multiplied by its
miner weight. get_logits(embeddings) returns
the logits \(x^T p_z / \sigma\); see
classification logits. This loss derives
from BaseProxyLoss.
Example:
import torch
from pytorch_metric_learning import losses
loss_func = losses.NormalizedSoftmaxLoss(num_classes=10, embedding_size=128)
optimizer = torch.optim.SGD(loss_func.parameters(), lr=0.01)
embeddings = torch.randn(32, 128, requires_grad=True)
labels = torch.randint(0, 10, (32,))
loss = loss_func(embeddings, labels)
loss.backward()
optimizer.step()
Parameters:
- num_classes: The number of classes in the training set. Labels must be in
0..num_classes - 1. - embedding_size: The size of the embeddings passed to the loss.
- temperature: The temperature \(\sigma\). The paper uses 0.05.
- weight_regularizer, weight_reg_weight, weight_init_func: See
learnable weight arguments. The
class weights are
weight, shaped(num_classes, embedding_size)and initialized withtorch.nn.init.normal_. Includeloss_func.parameters()in an optimizer.
Default distance:
DotProductSimilarity(normalize_embeddings=True), which L2-normalizes the embeddings.
Default reducer:
Reducer input:
- loss: One loss per embedding, indexed by embedding position.
Temperature division uses float32 for float16/bfloat16 distance outputs.
get_logits returns these unshifted, temperature-scaled logits in the accumulation
dtype, and cross-entropy retains it internally. Per-embedding terms and the reduced nonempty loss retain the accumulation dtype,
so a representable batch mean cannot inherit an overflowing float16 term.
Training calls the public get_logits method, including subclass overrides.
Custom wider distances and float32/float64 distances retain their precision.
Gradients still accumulate in the embedding and parameter dtypes.