CODEWITHSUNDEEP
lightgbmstochasticstochastic-processstochastic-gradient
Amie Johnson
Amie Johnson

Reputation: 57

What is the difference between Stochastic Gradient Descent and LightGBM?

Although I have individually researched these concepts, I am confused on whether one or the other can be chosen for a solution, or can both of these be used simultaneously to improve results? Any guidance you can provide will be much appreciated.

Upvotes: 0

Views: 167

Answers (1)

Soran
Soran

Reputation: 391

My understanding is that the cost function of gradient descent is based on the entire training set whereas stochastic gradient descent approximates the cost of the true gradient using much less than the entire training set.

The question of which to use and when is based on determining whether there is sufficient computing power to calculate the exact cost of the gradient. If there is sufficient computing power and time then calculate it exactly.

If the training set is too large, stochastic gradient descent is worth a try. Use both for testing the quality of the approximation.

In general, I would not use both for the same reason I would never average an exact value and it's approximation. (Ex: 1=1 but 1 is also approximately 0.99 so (1+0.99)/2 = 0.995)

Upvotes: 0

Related Questions