Cross Entropy with Label Smoothing

Compute the mean cross entropy between logits and smoothed one-hot targets.

With plain one-hot targets, the loss only reaches zero when the correct logit is infinitely larger than the others, so the model is pushed to be overconfident. Label smoothing moves a small amount eps of the target probability away from the true class and spreads it evenly over all K classes:

target[c] = (1 - eps) + eps / K      if c is the true class
          = eps / K                  otherwise
loss_i    = -sum_c target[c] * log_softmax(logits_i)[c]

Return the mean of loss_i over the batch. Compute log_softmax directly as z - max(z) - log(sum(exp(z - max(z)))), so that large logits do not overflow.

Function signature

def label_smoothing_ce(logits, labels, eps=0.1):

Example

Input

label_smoothing_ce(np.array([[2.0, 1.0, 0.0]]), np.array([0]), eps=0.1)

Output

0.50760596

Reasoning

The log probabilities are [-0.408, -1.408, -2.408]. The smoothed target is [0.933, 0.033, 0.033]. The loss is 0.933 * 0.408 + 0.033 * (1.408 + 2.408) = 0.5076. Plain cross entropy would give 0.408. The extra cost comes from the small target on the wrong classes.

Constraints and notes

• logits has shape (batch, K) and labels has shape (batch,).
• With eps = 0 the result is ordinary cross entropy.
• Return a plain float.