Temperature Scaling

Fix an overconfident classifier with temperature scaling.

Modern networks are often right but too sure. They predict 99% when they are right only 80% of the time. Temperature scaling divides every logit by one number T and then applies softmax:

p = softmax(logits / T)

T > 1 makes the probabilities softer, and T < 1 makes them sharper. The largest logit stays the largest, so accuracy does not change. Only the confidence changes.

Find the T that minimises the negative log-likelihood (cross-entropy) of the labels on a held-out set. T must stay positive, so optimise log_t = log(T) and use T = exp(log_t). This is a one-parameter, smooth, convex problem in log_t, so it suits torch.optim.LBFGS.

Return T as a Python float. Grading checks it against the true minimum, and checks that the expected calibration error goes down.

Function signature

def fit_temperature(logits, labels):

Example

Input

logits = 3.0 * true_logits         # an overconfident model
labels = labels drawn from softmax(true_logits)
fit_temperature(logits, labels)

Output

2.98                               # approximately

Reasoning

The labels follow softmax(true_logits), and the model's logits are three times too large. Dividing by T = 3 gives back the true probabilities, so the NLL is lowest close to 3. The exact value moves a little because the labels are a random sample.

Constraints and notes

• logits has shape (n, k) and dtype float32. labels has shape (n,) and dtype int64.
• Do not change logits. The fitted T can be smaller than 1.
• The answer must be within 1% of the true minimum.