CODEWITHSUNDEEP
pythonpython-2.7tensorflowdeep-learningconv-neural-network
whoisraibolt
whoisraibolt

Reputation: 2698

Methods of initializing weights in Convolutional Neural Networks - Python/TensorFlow

I initially set my weights as follows:

def new_weights(shape):
    return tf.Variable(tf.truncated_normal(shape, stddev=0.05))

But I realized that I had weights -0 as in the following figure:

Figure 1

Then I decided to define it that way, and for me the weights are more acceptable:

def new_weights(shape):
    return tf.Variable(tf.zeros(shape))

Figure 2

I want to know what the implication of using tf.truncated_normal is and what does the -zeros mean? And if you have any problems initializing them like this.

And between the two I introduced, what would be the best method of initializing weights?

Upvotes: 0

Views: 1444

Answers (1)

Nipun Wijerathne
Nipun Wijerathne

Reputation: 1829

tf.truncated_normal outputs random values from a truncated normal distribution. Therefore, have good convergence properties in DNN. Following graphs are the validation loss (left) and validation accuracy (right) of a CNN on MNIST dataset. Used weight initializations are as follows,

  1. Zero: When all weights are set to 0
  2. Random: When weights are set completely randomly
  3. Random between -1 to +1: Random weights on the scale of -1 to +1
  4. Xavier-Glorot Initialization

enter image description here

As you can see, random weights on the scale of -1 to +1 have shown good results. Therefore, it is the most commonly used weight initialization method in DNN.

References, https://medium.com/@amarbudhiraja/towards-weight-initialization-in-deep-neural-networks-908d3d9f1e02

Upvotes: 1

Related Questions