Loading
How to score a model honestly. The course covers the confusion matrix and the metrics built on it, averaging over classes, threshold curves, overlap and regression metrics, calibration, and how to write an evaluation loop in PyTorch that gives the right numbers.
01
The confusion matrix counts each pair of true and predicted class. Accuracy, precision, recall and F1 all come from these counts, and accuracy alone can hide a bad model when the classes are imbalanced.
02
With many classes you must say how to combine the per-class scores. Macro averaging weights every class the same, and micro averaging weights every example the same. Top-k accuracy counts a prediction as right when the true class is in the k best guesses.
03
A score becomes a prediction only after you pick a threshold. The ROC curve and the precision-recall curve show every threshold at once, and AUC and average precision sum each curve up as one number.
Practise it
04
Jaccard and Dice measure how much two sets or masks overlap. They are the standard scores for segmentation and for matching boxes.
Practise it
05
Mean squared error and mean absolute error measure the size of the mistakes. R-squared compares the model with always predicting the mean.
Practise it
06
A calibrated model is right 80% of the time when it says 80%. Expected calibration error measures the gap, and temperature scaling closes most of it with one parameter.
07
An evaluation loop puts the model in eval mode, turns off gradient recording, and adds up counts over all batches before it computes any metric.
Practise it