The guide
Combine many weak models and you often get a strong one. The idea works because different models make different mistakes, and when you average their answers the mistakes partly cancel out.
There are two main ways to get that variety. Bagging trains each model on a different bootstrap sample and lets them vote, which reduces variance and is the core of random forests. Boosting trains models one after another, each one on the mistakes of the ones before it, which reduces bias. AdaBoost does this by reweighting the samples. Gradient boosting does it by fitting each new model to the residuals.
On tabular data, gradient boosted trees are still the method to beat, and the libraries that win competitions run the same loop you write here.