L-BFGS with a Closure

Fit L2-regularised logistic regression to full precision with torch.optim.LBFGS.

The objective is the mean binary cross-entropy plus a penalty on the weights. The bias is not penalised:

z    = X @ w + b
loss = mean(BCE_with_logits(z, y)) + 0.5 * l2 * sum(w ** 2)

This problem is small, smooth and convex, which is where a quasi-Newton method is far better than plain gradient descent. L-BFGS builds a picture of the curvature from its last few gradients and finds the minimum in tens of iterations, not thousands.

L-BFGS is different from other torch.optim optimisers in one way. Each step must evaluate the loss many times, at points that the line search picks. So you do not call backward() yourself. You give step a closure, a function that clears the gradients, computes the loss, calls backward() and returns the loss.

Return the weights and the bias as detached tensors. Grading checks that you are at the true minimum: the gradient of the objective at your answer must be almost zero, and the weights must match scikit-learn to 1e-4.

Function signature

def fit_logistic_lbfgs(X, y, l2):

Example

Input

X = torch.tensor([[-1.0], [0.0], [1.0], [2.0]], dtype=torch.float64)
y = torch.tensor([0.0, 0.0, 1.0, 1.0], dtype=torch.float64)
w, b = fit_logistic_lbfgs(X, y, l2=0.1)

Output

w = tensor([1.5077], dtype=torch.float64)
b = tensor(-0.7538, dtype=torch.float64)

Reasoning

Without the penalty, separable data pushes w to infinity. With l2 = 0.1, the penalty stops it at a finite value. The data is symmetric about x = 0.5, so the boundary -b / w is at 0.5.

Constraints and notes

• X has shape (n, d) and y has shape (n,), both float64. Keep that dtype. In float32 you cannot reach a gradient below 1e-6.
• Return w with shape (d,) and b with shape ().
• Use line_search_fn="strong_wolfe", and set max_iter and the tolerances so that one call to step runs to convergence.