Ridge Regression

Add L2 regularisation to least squares.

Ridge penalises large weights, trading a little bias for a large drop in variance:

loss = ||Xw + b - y||^2 + alpha * ||w||^2

The closed form solution is

w = (X_c^T X_c + alpha * I)^-1 X_c^T y_c

where X_c and y_c are X and y with their column means subtracted, and

b = mean(y) - mean(X) . w

Two details that are easy to get wrong:

• The intercept is not penalised. Shrinking b towards zero would make the fit depend on where the target happens to be centred. Centring the data first and recovering b afterwards handles this cleanly.
• Adding alpha * I makes the matrix invertible even when X^T X is singular, which is the other reason ridge exists. With alpha = 0 you get ordinary least squares back.

Function signature

def fit_ridge(X, y, alpha):

Example

Input

X = np.array([[1.0], [2.0], [3.0]])
y = np.array([3.0, 5.0, 7.0])
fit_ridge(X, y, alpha=0.0)

Output

(array([2.]), 1.0)

Reasoning

With alpha = 0 ridge is ordinary least squares. The points lie exactly on y = 2x + 1, so w = [2] and b = 1.

Constraints and notes

• Do not penalise the intercept. Centre the data and recover b at the end.
• Return w with shape (d,) and b as a float.