tf.constant_initializer() vs tf.global_variables_initializer()

Question

In the following code from this website, what's the role of tf.constant_initializer(0.) and tf.global_variables_initializer()? Why do we need two initializers?

import tensorflow as tf

### build the graph
## first set up the parameters
m = tf.get_variable("m", [], initializer=tf.constant_initializer(0.))
b = tf.get_variable("b", [], initializer=tf.constant_initializer(0.))
init = tf.global_variables_initializer()

## then set up the computations
input_placeholder = tf.placeholder(tf.float32)
output_placeholder = tf.placeholder(tf.float32)

x = input_placeholder
y = output_placeholder
y_guess = m * x + b

loss = tf.square(y - y_guess)

## finally, set up the optimizer and minimization node
optimizer = tf.train.GradientDescentOptimizer(1e-3)
train_op = optimizer.minimize(loss)

### start the session
sess = tf.Session()
sess.run(init)

The original post I link has an explanation that goes:

The answer lies in the split between sessions and graphs. We’ve set the initializer property of get_variable to point at our const_init_node, but that just added a new connection between nodes in the graph. We haven’t done anything about the root of the exception: the memory associated with the variable node (which is stored in the session, not the graph!) is still set to “null”. We need the session to tell the const_init_node to actually update the variable.

... which still seems confusing as, in that case, why do we need the tf.constant_initializer(0.) at all? We might as well just leave our m and b uninitialized until we run tf.global_variables_initializer()...

Answer 1

This TensorFlow Guide about initializing variables might be helpful.

And below three things are from TensorFlow site and also useful.
tf.get_variable: Gets an existing variable with these parameters or create a new one.
tf.constant_initializer: Initializer that generates tensors with constant values.
tf.global_variables_initializer: Returns an Op that initializes global variables.

---

To put it simply, the role of tf.constant_initializer is to generate constant values.
The role of tf.global_variables_initializer is to initialize variables. In TensorFlow, you should initialize variables before creating graphs and sessions. Therefore you have to use tf.global_variables_initializer when using variables.

---

Actually, in your code, you don't have to use tf.constant_initializer because tf.get_variable uses default initializer as glorot_uniform_initializer.

But you should use tf.global_variables_initializer because tf.get_variable gets an variables or creates an variables.

---

Example code 1: If you do not use tf.global_variables_initializer then yon encounter an error message.

import tensorflow as tf

m = tf.get_variable("m", [])
b = tf.get_variable("b", [])

y = m + b

sess = tf.Session()
sess.run(y)

# Example code 1 result

FailedPreconditionError: Attempting to use uninitialized value m
     [[{{node m/read}}]]

Example code 2: You don't have to use tf.constant_initializer.

import tensorflow as tf

m = tf.get_variable("m", [])
b = tf.get_variable("b", [])
init = tf.global_variables_initializer()

y = m + b

sess = tf.Session()
sess.run(init)
sess.run(y)

# Example code 2 result

Call initializer instance with the dtype argument instead of passing it to the constructor
0.43568623

---

Finally, even if you use tf.constant_initializer in tf.get_variable you should use tf.global_variables_initializer. The reason is that tf.get_variable is a variable. So you initialize variables before tf.Session by using tf.global_variables_initializer.

Maybe you can think of initializer in tf.get_variable(e.g tf.constant_initializer, glorot_uniform_initializer) as initializing value by some kind of distributions when initializing variables by using tf.global_variables_initializer.

Answer 2

Let's assume that you want to create a Tensorflow graph and train it from scratch. Therefore, all of the tf.Variable in your TF Graph will be initialized randomly. For example, in the code bellow, the tf.Variable will be initialized with values from a normal distribution.

tf.reset_default_graph()
x = tf.get_variable(name="random", shape=[1, 5] initializer=tf.random_normal_initializer())
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(x))
    # [[-1.1984633  -0.42527035 -1.020795   -0.7023787   0.22350429]]

Now, let us assume that you again want to create a TF Graph and train it from scratch. However, right now, for some weird reason, you know the exact values that you want to use for some of the tf.Variable in your graph. Therefore:

tf.reset_default_graph()
x = tf.get_variable(name="constant", shape=[1,5], initializer=tf.constant_initializer(value=[[1,2,3,4,5]]))
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(x))
    # [[1. 2. 3. 4. 5.]]

you initialized those tf.Variable with a the values you wish.

As for the distinction between tf.global_variables_initializer and tf.constant_initializer, they are something completely different:

• tf.global_variables_initializer is an operation that you execute to initialize all variables in your graph. It doesn't matter whether the variables will get initialized with tf.constant_initializer, tf.random_normal_initializer or tf.glorot_uniform_initializer. You just pass that operation to a tf.Session so that the graph variables will get initialized.

• tf.constant_initializer on the other hand, is just an initializer that you pass to the tf.Variable of your graph. Then, when a tf.Session runs the operation tf.global_variables_initializer, the TF Graph will use the tf.constant_initializer to initialize the corresponding tf.Variable with the constant values provided.

Answer 3

The two are entirely different. The constant_initializer is basically an "instruction" how the variable should be initialized once this is actually done. I.e., whether it should use constant values, random uniform values, random normal values...
Passing this does not mean that the variable will be initialized with constant values at that time. If you don't pass an initializer, get_variable uses a default (glorot_uniform, I believe).

global_variables_initializer, on the other hand, is an op that will actually perform the initialization once it is run inside a session -- this will then call the execute the initializers you passed to the variables when you created them. You always need to run an op to initialize variables, since as mentioned, just passing the initializers in the constructors doesn't actually do anything.