Why the following Tensorflow code produces the shown result?

Question

I am following the tensorflow tutorial to do some tests. In the section of Variable of the official tutorial, there is a code snippet indicating how to use tensorflow to implement a counter. I changed it a little bit so that the result changed as follows. However, I don't know the logic behind.

The specific line of the code I changed is state = tf.add(state, one). I think when sess.run(state) is called, the add operation should be executed multiple times, producing the result 0 1 2 3. However, it was still 1 1 1 1. Any explanations?

import tensorflow as tf
state = tf.Variable(0, name="counter")
one = tf.constant(1)
state = tf.add(state, one)
init_op = tf.initialize_all_variables()

with tf.Session() as sess:
    sess.run(init_op)
    print(sess.run(state))
    for _ in range(3):
        print(sess.run(state))

The output is

1
1
1
1

---

Follow-ups

import tensorflow as tf
state = tf.Variable(0, name="counter")
one = tf.constant(1)
new_value = tf.add(state, one)
state = tf.assign(state, new_value) ## <== (1)
#state = tf.add(0, new_value) ## <== (2)

init_op = tf.initialize_all_variables()

with tf.Session() as sess:
    sess.run(init_op)
    for _ in range(4):
        print(state.eval())

If run with line (1) but no (2), the result is 1 2 3 4;

If run with line (2) but no (1), the result is 1 1 1 1;

Why?

Answer 1

The result is indeed correct.

When you define

state = tf.Variable(0, name="counter")
one = tf.constant(1)
state = tf.add(state, one)

You are defining a graph of ordered operations such as:

1. state = 0
2. one = 1
3. state = state+one

After the graph definition, state holds the the last operation of the graph, not the numerical value of tf.add(state, one).

Therefore, when you run sess.run(state) in a loop, you are asking Tensorflow to run the graph you have created many times, but the operations graph will be always the same, so the result is always the same.

---

Follow-ups

Ok, lets see what's happening in both scenarios (1) and (2) for the code you provided.

import tensorflow as tf
state = tf.Variable(0, name="counter")
one = tf.constant(1)
new_value = tf.add(state, one)
state = tf.assign(state, new_value) ## <== (1)
#state = tf.add(0, new_value) ## <== (2)

init_op = tf.initialize_all_variables()

with tf.Session() as sess:
    sess.run(init_op)
    for _ in range(4):
        print(state.eval())

Scenario (1)

In this scenario you are calling 4 times state.eval(). What is state? Well, it is a Variable because you have previously initialized it to:

state = tf.Variable(0, name="counter")

Then with the line (1) state = tf.assign(state, new_value) what you are doing is giving a value to this Variable. The documentation for tf.assign() says that it returns the same Variable, so after the line (1) state is still the same Variable as before, only that now it has the value new_value. Then what is new_value? It's just the result of adding 1 to the Variable state as you have defined in new_value = tf.add(state, one).

Now we can see what happens if you eval state four times:

Iteration 0:

• state has an initial value of 0
• new_value is an operation which defines state+1 so it has value 1 when evaluated.
• With tf.assign you set state to the value of new_value, so it is 1 now.
• print(state.eval()) prints 1 because this is the current value of state.

Iteration 1:

• state was 1 in the previous iteration.
• state has the same value of new_value because of tf.assign().
• As new_value is state+1, now it has a value of 2.
• print(state.eval()) prints 2 because this is the current value of state.

Iteration 2:

• state was 2 in the previous iteration.
• ...

Scenario (2)

What's going on here is that you are overriding the initial state which is a tensorflow Variable with the operation tf.add, so the code is no longer doing what you expect. Let me explain.

First state is a Variable with value 0 as initialized in:

state = tf.Variable(0, name="counter")

and new_value is an operation which defines state+1, so it is has a value of 1 when evaluated. Then you override state by assigning it to the operation tf.add(0, new_value). Why is that important? Well, because new_value depends on state, yes, but it depends on the content of state before overriding it, so it is using the "old" state which is a Variable with value 0.

Hence, when you call state.eval() four times you are evaluating tf.add(0, new_value) four times, and its result is always 1 because new_value is always 1, as the value of "old" state is not changing.