Label Smoothing Cross Entropy

Write cross entropy with label smoothing and padding, the loss used to train most classifiers and language models.

Hard labels say the true class has probability 1. Label smoothing moves a fraction smoothing of that mass evenly over all C classes:

q = (1 - smoothing) * one_hot(target) + smoothing / C

The loss is the cross entropy between q and the model's distribution p = softmax(logits). Because q is a mixture, the loss splits into two parts:

loss = (1 - smoothing) * (-log p[target]) + smoothing * mean over classes of (-log p)

Compute log p with torch.log_softmax, so that large logits do not overflow. Positions where the target equals ignore_index are padding. Leave them out, and average over the rest.

Function signature

def smoothed_cross_entropy(logits, targets, smoothing, ignore_index=-100):

Example

Input

logits  = torch.tensor([[2.0, 0.0, 0.0], [0.0, 0.0, 0.0]])
targets = torch.tensor([0, -100])
smoothed_cross_entropy(logits, targets, smoothing=0.3)

Output

tensor(0.6395)

Reasoning

Only the first row counts. log p = [-0.2395, -2.2395, -2.2395]. The hard part is 0.2395, and the mean of -log p is 1.5728. So the loss is 0.7 * 0.2395 + 0.3 * 1.5728 = 0.6395.

Constraints and notes

• logits has shape (..., C) and targets has shape (...), int64. A language model gives (batch, time, C).
• At least one target is not ignore_index. 0 <= smoothing < 1.
• Return a scalar tensor that autograd can differentiate. Do not use the label_smoothing argument of the built in loss.