Logistic regression from scratch · AI Engineering

Logistic regression is a linear model with a sigmoid on the output. The output is the probability of class 1. Gradient descent on the binary cross-entropy loss trains it. The same loss and gradient return in every neural network classifier with two classes.

Task

Write four functions.

sigmoid(z) returns \sigma(z) = 1 / (1 + e^{-z}) for each element of z.
compute_loss(X, y, w, b) returns the mean binary cross-entropy of the model p = \sigma(X w + b).
compute_gradients(X, y, w, b) returns (dw, db), the gradient of that mean loss with respect to w and b.
fit(X, y, learning_rate, epochs) trains the model with full batch gradient descent and returns (w, b, loss_history).

The loss of one example is -[y \log p + (1 - y) \log(1 - p)]. With z = x \cdot w + b, the same value is \log(1 + e^z) - y z. The loss is the mean over the n examples.

The gradients are dw = X^T (p - y) / n and db = \frac{1}{n} \sum_i (p_i - y_i).

fit starts with w as a zero vector and b = 0.0. Each epoch does these steps:

Compute (dw, db) at the current w and b.
Move w and b one step against their gradients. The step is learning_rate times the gradient.
Append the loss at the new w and b to loss_history.

Large values of |z| overflow np.exp. The sigmoid and the loss must be correct for any finite z. Do not clip the probabilities.

Input and output

Name	Type	Shape	Meaning
`z`	float array	any shape	the input of `sigmoid`
`X`	float array	`(n, d)`	one example in each row
`y`	float array	`(n,)`	the label of each example, 0.0 or 1.0
`w`	float array	`(d,)`	the weights
`b`	float	scalar	the bias
`learning_rate`	float	scalar	the step size
`epochs`	int	scalar	the number of gradient steps, 0 or more

sigmoid returns a float array with the shape of z.
compute_loss returns a Python float.
compute_gradients returns a float array of shape (d,) and a Python float.
fit returns a float array of shape (d,), a Python float and a list of epochs floats.

Constraints

1 <= n <= 1000 and 1 <= d <= 50.
z can be any finite float, also values such as 1000.0 and -1000.0.
Do not change the input arrays.
Use numpy only.

Examples

python

sigmoid(np.array([-2.0, 0.0, 2.0, 1000.0, -1000.0]))

text

[0.11920292 0.5        0.88079708 1.         0.        ]

python

X = np.array([[1.0, 2.0], [2.0, 1.0], [3.0, 4.0], [4.0, 3.0]])
y = np.array([0.0, 0.0, 1.0, 1.0])
w = np.array([0.5, -0.25])
compute_loss(X, y, w, 0.1)
compute_gradients(X, y, w, 0.1)

text

0.6545645625913818
(array([ 0.00990023, -0.07111509]), 0.1663330660944403)

python

fit(X, y, 0.1, 3)

text

(array([0.10335357, 0.10335357]), -0.016102386795294854, [0.6521696016041545, 0.6346023820290841, 0.6258174296603579])