CODEWITHSUNDEEP
pythonmatrixmachine-learningneural-networksigmoid
Fcoder
Fcoder

Reputation: 9226

Sigmoid function for large numbers in Python

Im doing a Neural network task with Sigmoid activation function. My networks input is image(MNIST dataset) and because dimension of each image is 28*28, When I have to convert them to a vector, i will have N*784 matrix. Multiplication of this large matrix with weight matrix produce large positive and negative numbers for weights and i have to pass them to a Sigmoid function. I use expit() as sigmoid function and my problem is that:

numbers until 30 resulting near 1 in expit(). for example expit(28) results 0.99999999 and expit(29) results 1.0 and upper 29 also gets 1. But my new wights are upper 30 and because of that some of them gets 1 and some 0 in the first cycle of learning and in fact there is no any learning at all.

What i have to do? Sigmoid's upper bound is 29? i cant change it? I have to change my image dimension to overcome this?

Upvotes: 1

Views: 2400

Answers (1)

Lukasz Tracewski
Lukasz Tracewski

Reputation: 11397

As discussed in comments section, the real problem turned out to be using sigmoid itself, which is not suited for such cases. In any finite-precision calculations one will face the described problem, one system with 29, on other with 38.

One way to tackle the problem would be use softmax activation function, which is less susceptible to such issues. Mind that with cost function you might encounter similar challenges.

Slightly off-topic, ut you might want to check how the problem is resolved with e.g. tensorflow. It has some nice tutorials for beginners.

Upvotes: 2

Related Questions