Gradient Boosting with Any Loss

Write gradient boosting that works for any differentiable loss, with autograd supplying the gradients.

Gradient boosting builds a model F as a sum of small trees. Each round fits a tree to the negative gradient of the loss with respect to the current predictions, then adds a small step of that tree to F:

F = zeros(n)
repeat n_rounds times:
    g    = d loss_fn(F, y).sum() / dF        (one value per sample)
    tree = DecisionTreeRegressor(max_depth=max_depth, random_state=0)
    tree.fit(X, -g)
    F    = F + lr * tree.predict(X)

XGBoost and LightGBM let you plug in a custom objective, but you must write its gradient yourself. Here autograd does it: make F a float64 tensor with requires_grad=True and call torch.autograd.grad. The same code then works for squared error, log loss, quantile loss, or a loss you invent.

Return a predict(X_new) function that sums lr * tree.predict(X_new) over all trees and returns a float64 tensor of raw scores.

Function signature

def gradient_boost(X, y, loss_fn, n_rounds=100, lr=0.1, max_depth=3):

Example

Input

X = np.array([[0.], [1.], [2.], [3.]])
y = np.array([0., 0., 1., 1.])
squared = lambda F, y: 0.5 * (F - y) ** 2
predict = gradient_boost(X, y, squared, n_rounds=1, lr=0.5, max_depth=1)
predict(X)

Output

tensor([0.0000, 0.0000, 0.5000, 0.5000], dtype=torch.float64)

Reasoning

At F = 0 the gradient of 0.5 * (F - y)^2 is F - y = -y, so the tree fits y itself. It splits between 1 and 2 and predicts 0 and 1. Half a step of that is the output.

Constraints and notes

• X is a numpy array (n, d), y a numpy array (n,). loss_fn(F, y) takes two float64 tensors and returns the loss of each sample, shape (n,).
• Get the gradient with autograd. Do not write the derivative of any loss.
• Use sklearn.tree.DecisionTreeRegressor(max_depth=max_depth, random_state=0) so that the result is deterministic.
• With n_rounds=0 the model predicts zeros.