CODEWITHSUNDEEP
tensorflowconv-neural-network
Amit Gupta
Amit Gupta

Reputation: 2918

Differences in tensorflow prediction on CPU and GPU for CNN models

I have trained an FCN network on a GPU and have saved the model(.pb file). I am getting correct predictions on the GPU. However i am getting NAN for the same model file when I am running predictions on CPU. Are there any CPU/GPU flags that need to be set? Or are there any overflow issues with CPU?

Upvotes: 2

Views: 442

Answers (1)

Sorin
Sorin

Reputation: 11968

There are no special overflow condition on the CPU. Both should implement IEEE 754.

There are different ways some high level functions can be implemented (tanh, sigmoid) and they are implemented differently on GPU vs CPU to take advantage of the platform.

Whenever you get NaN from your model something is most likely broken. Don't try to patch it with some flag, but instead try to debug and see what's going on. In almost all cases you have a degenerate model that only works because of some corner case of some hardware.

Once you've found the problem, it's usually fixed by capping some values or by modifying the way data is represented (taking log of large numbers for example).

Upvotes: 2

Related Questions