CODEWITHSUNDEEP
tensorflowgradient
Mahdi Amrollahi
Mahdi Amrollahi

Reputation: 3021

How Tensorflow knows how to calculate the derivative?

In this resource by Tensorflow, we can see how the derivative of a function can be calculated in one or any point, but how Tensorflow do it?

enter image description here

Upvotes: 0

Views: 1175

Answers (1)

David Kaftan
David Kaftan

Reputation: 2174

Tensorflow calculates derivatives using automatic differentiation. This is different from symbolic differentiation and numeric differentiation (aka finite differences). More than a smart math approach, it is a smart programming approach. When you evaluate a tensorflow variable, it stores not only the value of that variable, but also the value of its derivative. Also, all operators are overloaded to calculate both the value and derivative of a variable. Using the chain rule, we can traverse a graph of variables and calculate the derivative of each variable. The posted link will give a better description but this is the TLDR.

Benefits? Over numerical differentiation, this is much faster. You can calculate the derivative of all your weights with one pass through of your cost function. In numerical differentiation, you have to evaluate your cost function twice for each weight you calculate the derivative of. You also aren't getting an approximation.

Upvotes: 4

Related Questions