Expected Calibration Error

Compute the expected calibration error (ECE) of a classifier's confidences.

A model is calibrated when its confidence matches its accuracy: of all the predictions it makes with 80 percent confidence, about 80 percent should be right. Modern neural networks are often overconfident, so this matters for any decision that uses the probability itself.

ECE groups the predictions into n_bins equal-width confidence bins. Bin b holds confidences in (b / n_bins, (b + 1) / n_bins], and a confidence of exactly 0 goes in bin 0. For each non-empty bin, take the gap between its accuracy and its mean confidence, and weight it by the bin's share of the predictions:

ECE = sum_b (n_b / n) * |accuracy_b - mean_confidence_b|

Function signature

def expected_calibration_error(confidences, correct, n_bins=10):

Example

Input

confidences = np.array([0.9, 0.8, 0.3, 0.6])
correct     = np.array([1, 0, 0, 1])
expected_calibration_error(confidences, correct, n_bins=2)

Output

0.15

Reasoning

The bin (0, 0.5] holds only 0.3, which was wrong: accuracy 0, confidence 0.3, gap 0.3, weight 1/4. The bin (0.5, 1] holds 0.9, 0.8 and 0.6: accuracy 2/3, mean confidence 0.767, gap 0.1, weight 3/4. The total is 0.075 + 0.075 = 0.15.

Constraints and notes

• confidences are in [0, 1] and correct holds 0 or 1.
• Empty bins contribute nothing.
• Bin edges are closed on the right: a confidence of exactly 0.5 with two bins goes in the first bin.