CODEWITHSUNDEEP
python-2.7neural-networktensorflow
Kalanit
Kalanit

Reputation: 875

How to manually initialize the values for the weights?

I would like to experiment the weights initialization recommended by Karpathy in his lecture notes,

the recommended heuristic is to initialize each neuron's weight vector as: w = np.random.randn(n) / sqrt(n), where n is the number of its inputs

source: http://cs231n.github.io/neural-networks-2/#init

I'm beginner in python, and I don"t know how to implement this :/

weights = tf.Variable(??)

Please help? ...

Upvotes: 2

Views: 2284

Answers (3)

O K
O K

Reputation: 868

I do it in the following way:

    self.w_full, self.b_full = [], []

    n_fc_layers = len(structure)
    structure.insert(0, self.n_inputs)

    with vs.variable_scope(self.scope):
        for lr_idx in range(n_fc_layers):
            n_in, n_out = structure[lr_idx], structure[lr_idx+1]
            self.w_full.append(
                vs.get_variable(
                   "FullWeights{}".format(lr_idx),
                    [n_in, n_out],
                    dtype=tf.float32,
                    initializer=tf.random_uniform_initializer(
                        minval=-tf.sqrt(tf.constant(6.0)/(n_in + n_out)),
                        maxval=tf.sqrt(tf.constant(6.0)/(n_in + n_out))
                    )
                )
            )

            self.b_full.append(
                vs.get_variable(
                    "FullBiases{}".format(lr_idx),
                    [n_out],
                    dtype=tf.float32,
                    initializer=tf.constant_initializer(0.0)
                )
            )

after

structure.insert(0, self.n_inputs)

you'll have [n_inputs, 1st FC layer size, 2nd FC layer size ... output layer size]

Upvotes: 0

Yaroslav Bulatov
Yaroslav Bulatov

Reputation: 57903

n = 10
init_x = np.random.randn(n)
x = tf.Variable(init_x)
sess = tf.InteractiveSession()
sess.run(tf.initialize_all_variables())
print(sess.run(x))

Upvotes: 0

Sung Kim
Sung Kim

Reputation: 8536

For a single value, use:

weights = tf.Variable(10)

For a vector with random values:

shape = [784, 625]
weights = tf.Variable(tf.random_normal(shape, stddev=0.01)/tf.sqrt(n))

Please note that you need to sess.run to evaluate the variables.

Also, please check out other Random Tensors: https://www.tensorflow.org/versions/r0.8/api_docs/python/constant_op.html#random-tensors

Upvotes: 2

Related Questions