Loading
How to combine many weak models into one strong model. The course starts with voting and the decision trees that most ensembles use, then covers bagging, AdaBoost and gradient boosting, and ends with deep ensembles of neural networks and the uncertainty they measure.
01
A vote of several models is more accurate than each model when their errors are not the same. Hard voting counts labels, and soft voting averages probabilities.
02
A tree splits the data on one feature at a time to make the groups purer. A stump is a tree with one split, the usual weak learner for boosting.
03
Bagging trains each model on a bootstrap sample and averages them, which lowers variance. Each model never sees about a third of the data, and those out-of-bag points give a free validation score.
Practise it
04
AdaBoost trains weak learners one after another. Each round gives more weight to the points the last learner got wrong, and the final vote weights each learner by its accuracy.
Practise it
05
Gradient boosting adds small trees fitted to the negative gradient of the loss. It is gradient descent in the space of predictions, and it works for any differentiable loss.
06
Neural networks trained from different random starts disagree away from the training data. Averaging them improves accuracy and calibration, and their disagreement measures model uncertainty.
Practise it