Logistic Regression by Gradient Descent

Fit a binary classifier by gradient descent on the mean binary cross entropy.

p    = sigmoid(Xw + b)
loss = -(1 / n) * sum_i [ y_i log(p_i) + (1 - y_i) log(1 - p_i) ]

The gradient of that loss has the same clean form as linear regression, which is the reason the logistic link is chosen in the first place:

dw = (1 / n) * X^T (p - y)
db = (1 / n) * sum(p - y)

Start from w = zeros(d) and b = 0.0 and run exactly epochs updates. Return (w, b).

Use a numerically stable sigmoid: 1 / (1 + exp(-z)) overflows for large negative z, so branch on the sign of z or use np.logaddexp.

Function signature

def fit_logistic_regression(X, y, lr, epochs):

Example

Input

X = np.array([[-2.0], [-1.0], [1.0], [2.0]])
y = np.array([0, 0, 1, 1])
fit_logistic_regression(X, y, lr=0.5, epochs=2000)

Output

(array([6.22942119]), 0.0)    # b is 0 up to rounding

Reasoning

The data is symmetric around 0, so b stays at 0. The classes can be split perfectly, so each step makes w larger to push the probabilities towards 0 and 1.

Constraints and notes

• Start from w = 0 and b = 0.
• Use a stable sigmoid, so large negative inputs do not overflow.