CODEWITHSUNDEEP
tensorflowprecision
Maruf
Maruf

Reputation: 790

precision of floating point in tensorflow

While working with tensorflow (version 1.4) faced some problem while debugging my code. Tensorflow have some serious precision error fault. I have shown here an example. Is there any way to handle this type of issue in tensorflow?

>>> a = [18.00146484]
>>> b= [[18.00146484]]
>>> c = [[[18.00146484]]]
>>> d = [[18.00146484],[12.83231735]]
>>> e = [[[18.00146484],[12.83231735]]]
>>> q = tf.nn.sigmoid()
KeyboardInterrupt
>>> q = tf.nn.sigmoid(a)
>>> w = tf.nn.sigmoid(b)
>>> e = tf.nn.sigmoid(c)
>>> r = tf.nn.sigmoid(d)
>>> z = [[[18.00146484],[12.83231735]]]
>>> t = tf.nn.sigmoid(z)
>>> init = tf.global_variables_initializer()
>>> sess = tf.Session()
2017-12-12 15:41:53.766287: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
>>> qq,ww,ee,rr,tt = sess.run([q,w,e,r,t])
>>> qq,ww,ee,rr,tt
(array([ 1.], dtype=float32), array([[ 1.]], dtype=float32), array([[[ 1.]]], dtype=float32), array([[ 1.        ],
       [ 0.99999738]], dtype=float32), array([[[ 1.        ],
        [ 0.99999738]]], dtype=float32))
>>> qq
array([ 1.], dtype=float32)
>>> ww
array([[ 1.]], dtype=float32)
>>> ee
array([[[ 1.]]], dtype=float32)
>>> rrr
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'rrr' is not defined
>>> rr
array([[ 1.        ],
       [ 0.99999738]], dtype=float32)
>>> tt
array([[[ 1.        ],
        [ 0.99999738]]], dtype=float32)
>>> from math import exp
>>> a = 18.00146484
>>> b = (1/(1+exp(-a)))
>>> b
0.9999999847923136
>>>

Upvotes: 0

Views: 6282

Answers (1)

iga
iga

Reputation: 3633

First, looking at how many trials you have, maybe you felt that tensorflow gives different results for the same initial value. It does not appear to be the case. sigmoid(18.00146484) is always 1 and sigmoid(12.83231735) is always 0.99999738.

Maybe you feel that sigmoid(18.00146484) is sufficiently far from 1 and should not be rounded to 1, but that is not the case. sigmoid(18.00146484) = 0.99999998479231364... (https://www.wolframalpha.com/input/?i=sigmoid(18.00146484)) and this number is too close to 1 for float32 precision. You really need to use a double (tf.float64 in tensorflow) for this level of precision. Here is a simple C++ program that shows the closest representable numbers to 1 for float and double.

#include <limits>
#include <cmath>
#include <iostream>

int main() {
  typedef std::numeric_limits< double > dbl;
  typedef std::numeric_limits< double > flt;

  double sigmoid = 0.99999998479231364;

  float x = 1.0;
  double firstSmallerThanX = std::nextafter(x, 0.0f);
  std::cout.precision(flt::max_digits10);
  std::cout << std::fixed << firstSmallerThanX << std::endl;
  std::cout << "Sigmoid is bigger than firstSmallerThanX: " << (sigmoid > firstSmallerThanX) << std::endl;

  double y = 1.0;
  double firstSmallerThanY = std::nextafter(y, 0.0);
  std::cout.precision(dbl::max_digits10);
  std::cout << std::fixed << firstSmallerThanY << std::endl;
  std::cout << "Sigmoid is smaller than firstSmallerThanY: " << (sigmoid < firstSmallerThanY) << std::endl;

  return 0;
}

This Prints:

0.99999994039535522
Sigmoid is bigger than firstSmallerThanX: 1
0.99999999999999989
Sigmoid is smaller than firstSmallerThanY: 1

Upvotes: 2

Related Questions