CODEWITHSUNDEEP
pythonpython-3.xnumpyscipy
Fcoder
Fcoder

Reputation: 9226

Overflow Error in Python's numpy.exp function

I want to use numpy.exp like this:

cc = np.array([
    [0.120,0.34,-1234.1]
])

print 1/(1+np.exp(-cc))

But this gives me error:

/usr/local/lib/python2.7/site-packages/ipykernel/__main__.py:5: RuntimeWarning: overflow encountered in exp

I can't understand why? How can I fix this? It seems the problem is with third number (-1234.1)

Upvotes: 38

Views: 169775

Answers (7)

Talha Tayyab
Talha Tayyab

Reputation: 27830

This overflow can be handled mathematically; you may not need to convert your values to np.float128

A sigmoid function is defined by:

enter image description here

Now, when you rearrange this function by multiplying numerator and denominator by e**x.

You will get:

enter image description here

So, you can design your sigmoid function accordingly:

import numpy as np

def sigmoid(x):
  if x > 0:   
    z = np.exp(-x)
    return 1/(1+z)
  else:
    z = np.exp(x)
    return z/(1+z)

Coming back to the question:

cc = np.array([0.120,0.34,-1234.1])
print([sigmoid(x) for x in cc])

#output
[0.5299640517645717, 0.5841905229354074, 0.0]

Upvotes: 1

Maulik Madhavi
Maulik Madhavi

Reputation: 166

As mentioned earlier by Praveen, you can use expit from scipy

So the problem can be solved by using: 1 / (1+ exp(-x)) = exp(x) / (1+exp(x))

>>> import numpy as np
>>> cc = np.array([[0.120,0.34,-1234.1]])
>>> np.exp(cc) / (1 +  np.exp(cc))
array([[0.52996405, 0.58419052, 0.        ]])

Upvotes: 0

miku3920
miku3920

Reputation: 61

If you don’t care about precision, you can use numpy.clip.

In float64:

cc = np.clip(cc, -709.78, 709.78)

In float32:

cc = np.clip(cc, -88.72, 88.72)

Upvotes: 3

user1559897
user1559897

Reputation: 1484

exp(-1234.1) is too small for 32bit or 64bit floating-point numbers. Since it cannot be represented, numpy produces the correct warning.

Using IEEE 754 32bit floating-point numbers, the smallest positive number it can represent is 2^(-149), which is roughly 1e-45.

If you use IEEE 754 64 bit floating-point numbers, the smallest positive number is 2^(-1074) which is roughy 1e-327.

In either case, it cannot represent a number as small as exp(-1234.1) which is about 1e-535.

You should be using the expit function from scipy to compute the sigmoid function. This would give you better precision.

For practical purposes, exp(-1234.1) is a very small number. If rounding to zero makes sense in your use case, numpy produces benign results by rounding it to zero.

Upvotes: 3

Praveen
Praveen

Reputation: 7222

As fuglede says, the issue here is that np.float64 can't handle a number as large as exp(1234.1). Try using np.float128 instead:

>>> cc = np.array([[0.120,0.34,-1234.1]], dtype=np.float128)
>>> cc
array([[ 0.12,  0.34, -1234.1]], dtype=float128)
>>> 1 / (1 + np.exp(-cc))
array([[ 0.52996405,  0.58419052,  1.0893812e-536]], dtype=float128)

Note however, that there are certain quirks with using extended precision. It may not work on Windows; you don't actually get the full 128 bits of precision; and you might lose the precision whenever the number passes through pure python. You can read more about the details here.

For most practical purposes, you can probably approximate 1 / (1 + <a large number>) to zero. That is to say, just ignore the warning and move on. Numpy takes care of the approximation for you (when using np.float64):

>>> 1 / (1 + np.exp(-cc))
/usr/local/bin/ipython3:1: RuntimeWarning: overflow encountered in exp
  #!/usr/local/bin/python3.4
array([[ 0.52996405,  0.58419052,  0.        ]])

If you want to suppress the warning, you could use scipy.special.expit, as suggested by WarrenWeckesser in a comment to the question:

>>> from scipy.special import expit
>>> expit(cc)
array([[ 0.52996405,  0.58419052,  0.        ]])

Upvotes: 54

jmd_dk
jmd_dk

Reputation: 13120

A possible solution is to use the decimal module, which lets you work with arbitrary precision floats. Here is an example where a numpy array of floats with 100 digits precision is used:

import numpy as np
import decimal

# Precision to use
decimal.getcontext().prec = 100

# Original array
cc = np.array(
    [0.120,0.34,-1234.1]
)
# Fails
print(1/(1 + np.exp(-cc)))    

# New array with the specified precision
ccd = np.asarray([decimal.Decimal(el) for el in cc], dtype=object)
# Works!
print(1/(1 + np.exp(-ccd)))

Upvotes: 6

fuglede
fuglede

Reputation: 18221

The largest value representable by a numpy float is 1.7976931348623157e+308, whose logarithm is about 709.782, so there is no way to represent np.exp(1234.1).

In [1]: import numpy as np

In [2]: np.finfo('d').max
Out[2]: 1.7976931348623157e+308

In [3]: np.log(_)
Out[3]: 709.78271289338397

In [4]: np.exp(709)
Out[4]: 8.2184074615549724e+307

In [5]: np.exp(710)
/usr/local/bin/ipython:1: RuntimeWarning: overflow encountered in exp
  #!/usr/local/bin/python3.5
Out[5]: inf

Upvotes: 22

Related Questions