CODEWITHSUNDEEP
linear-regressiongradient-descentderivative
csismylife
csismylife

Reputation: 71

how to implement gradient descent from scratch?

how do you find the derivative of the cost function of a linear regression problem ?

there were bunch of posts on the internet but weirdly none of them had shown how to compute the derivative.

for example I picked this from link of the website

def gradient_descent(x, y, theta, learning_rate=0.1, num_epochs=10):
m = x.shape[0]
J_all = []

for _ in range(num_epochs):
    h_x = h(x, theta)
    cost_ = (1/m)*(x.T@(h_x - y))
    theta = theta - (learning_rate)*cost_
    J_all.append(cost_function(x, y, theta))

return theta, J_all

but I don't see any derivative function being used. it's just the learning rate multiplied by the cost function.

can you show me please how to compute the derivative ?

Upvotes: 0

Views: 190

Answers (1)

Ionut-Alexandru Baltariu
Ionut-Alexandru Baltariu

Reputation: 420

You can calculate the derivative manually if the expression is simple enough and it will not change dynamically based on some input. However, if the mathematical function is complex, you might want to use the finite difference gradient approximation (central difference approximation).

def finite_diff_grad(f, x, delta=0.1):
    return (f(x+delta) - f(x-delta))/(2*delta)

For multivariable functions, you would have something like this:

def finite_diff_grad(f, x, y, delta=10e-4):
    der_part_x = (f(x+delta, y) - f(x-delta, y))/(2*delta)
    der_part_y = (f(x, y+delta) - f(x, y-delta))/(2*delta)

    return [der_part_x, der_part_y]

Where f is the function you want to find the derivative/gradient of.

Based on your problem's context, you might want to give different values to delta.

Upvotes: 1

Related Questions