Linear regression by gradient descent

Linear regression predicts a number as a weighted sum of the features plus a bias. Gradient descent finds the weights. It computes the gradient of the mean squared error and takes a small step against it, again and again. The same loop trains every model later in the course.

Task

Write four functions.

1. predict(X, w, b) returns the prediction \hat{y} = Xw + b for each row of X.
2. compute_cost(X, y, w, b) returns the mean squared error J = \frac{1}{n} \sum_i (\hat{y}_i - y_i)^2.
3. compute_gradients(X, y, w, b) returns the pair (dw, db). Here dw = \frac{2}{n} X^\top (\hat{y} - y) and db = \frac{2}{n} \sum_i (\hat{y}_i - y_i).
4. fit(X, y, learning_rate, epochs) runs batch gradient descent and returns (w, b, cost_history).

The rules for fit:

• Start with w as a vector of zeros and b = 0.0.
• In each epoch, compute dw and db from the current w and b. Then update both: w \leftarrow w - \eta \, dw and b \leftarrow b - \eta \, db, where \eta is learning_rate.
• After the update, append the cost of the new w and b to cost_history.

Input and output

• predict returns a float array of shape (n,).
• compute_cost returns a Python float.
• compute_gradients returns a tuple: a float array dw of shape (d,) and a float db.
• fit returns a tuple: a float array w of shape (d,), a float b and a list of epochs floats.

Constraints

• 1 <= n <= 1000 and 1 <= d <= 50.
• 0 <= epochs <= 5000.
• Do not change the input arrays.
• Use numpy only. Do not use a closed form solution such as np.linalg.lstsq in fit.

Examples

X = np.array([[1.0], [2.0], [3.0], [4.0]])
y = np.array([3.0, 5.0, 7.0, 9.0])
compute_cost(X, y, np.zeros(1), 0.0)
compute_gradients(X, y, np.zeros(1), 0.0)

41.0
(array([-35.]), -12.0)

w, b, cost_history = fit(X, y, 0.05, 500)
w, b, len(cost_history), cost_history[0]

(array([2.00005173]), 0.9998479052895004, 500, 1.1287500000000006)

The data lies on the line y = 2x + 1, so w moves close to 2 and b moves close to 1.