Working with multiple graphs in TensorFlow

Question

Can someone explain to me how name_scope works in TensorFlow?

Suppose I have the following code:

import tensorflow as tf

g1 = tf.Graph()
with g1.as_default() as g:
    with g.name_scope( "g1" ) as scope:
        matrix1 = tf.constant([[3., 3.]])
        matrix2 = tf.constant([[2.],[2.]])
        product = tf.matmul(matrix1, matrix2)

tf.reset_default_graph()

g2 = tf.Graph()
with g2.as_default() as g:
    with g.name_scope( "g2" ) as scope:
        matrix1 = tf.constant([[4., 4.]])
        matrix2 = tf.constant([[5.],[5.]])
        product = tf.matmul(matrix1, matrix2)

tf.reset_default_graph()

with tf.Session( graph = g1 ) as sess:
    result = sess.run( product )
    print( result )

When I run this code I get the following error message:

Tensor Tensor("g2/MatMul:0", shape=(1, 1), dtype=float32) is not an element of this graph.

I agree "g2/MatMul" is not an element of graph g1, but why is it selecting "g2/MatMul" when the session graph is set to g1? Why doesn't it select "g1/MatMul"?

---

Edit

The following code seems to work:

import tensorflow as tf

g1 = tf.Graph()
with g1.as_default() as g:
    with g.name_scope( "g1" ) as g1_scope:
        matrix1 = tf.constant([[3., 3.]])
        matrix2 = tf.constant([[2.],[2.]])
        product = tf.matmul( matrix1, matrix2, name = "product")

tf.reset_default_graph()

g2 = tf.Graph()
with g2.as_default() as g:
    with g.name_scope( "g2" ) as g2_scope:
        matrix1 = tf.constant([[4., 4.]])
        matrix2 = tf.constant([[5.],[5.]])
        product = tf.matmul( matrix1, matrix2, name = "product" )

tf.reset_default_graph()

use_g1 = False

if ( use_g1 ):
    g = g1
    scope = g1_scope
else:
    g = g2
    scope = g2_scope

with tf.Session( graph = g ) as sess:
    tf.initialize_all_variables()
    result = sess.run( sess.graph.get_tensor_by_name( scope + "product:0" ) )
    print( result )

By flipping the switch use_g1, graph g1 or g2 will run in the session. Is this the way name scoping was meant to work?

Answer 1

I have written two snippets to do what you are trying to do.

The first switches between graphs and does NOT use name_scope. The second uses the default_graph for both ops and uses name_scope to switch between them...

import tensorflow as tf

g1 = tf.Graph()
with g1.as_default() as g:
  matrix1 = tf.constant([[3., 3.]])
  matrix2 = tf.constant([[2.],[2.]])
  tf.matmul(matrix1, matrix2, name="productX")

g2 = tf.Graph()
with g2.as_default() as g:
  matrix1 = tf.constant([[4., 4.]])
  matrix2 = tf.constant([[5.],[5.]])
  tf.matmul(matrix1, matrix2, name="productX")

product_op = g1.get_tensor_by_name(  "productX:0" )
with tf.Session( graph = g1 ) as sess:
    result = sess.run( product_op )
    print( result )

product_op = g2.get_tensor_by_name(  "productX:0" )
with tf.Session( graph = g2 ) as sess:
  result = sess.run( product_op )
  print( result )

NOTE A) I have taken out the resetting of the default graph (the default graph is never used) and B) I have gotten rid of the 'product' variable and given the operation a name, instead

The important switching code is...

product_op = g2.get_tensor_by_name(  "productX:0" )

where we are using the variable name of the two graphs to distinguish between the two product ops.

NOTE the annoying ':0' that you have to put at the end of the variable name.

And now using name_scope...

import tensorflow as tf

g = tf.get_default_graph()

matrix1 = tf.constant([[3., 3.]])
matrix2 = tf.constant([[2.],[2.]])
with g.name_scope("prodA"):
  tf.matmul(matrix1, matrix2, name="productX")

matrix1 = tf.constant([[4., 4.]])
matrix2 = tf.constant([[5.],[5.]])
with g.name_scope("prodB"):
  tf.matmul(matrix1, matrix2, name="productX")

with tf.Session() as sess:

  product_op = g.get_tensor_by_name(  "prodA/productX:0" )
  result = sess.run( product_op )
  print( result )

  product_op = g.get_tensor_by_name(  "prodB/productX:0" )
  result = sess.run( product_op )
  print( result )

Now the switching line of code is this...

product_op = g.get_tensor_by_name(  "prodB/productX:0" )

NOTE A) There is no switching between graphs or sessions. B) name_scope give you a kind of directory structure looking name hierarchy.

What are the pros and cons of graph vs name_scope switching? If someone wrote a blog on that, I would read it!

I don't imagine that switching between graphs and sessions can be very fast, however.

Answer 2

Your problem is that you are calling the latest variable product which points to a tensor created on g2 - you overwrote it in your second scope. Just relabel all your variables and you should be good to go. The working code is below.

import tensorflow as tf

g1 = tf.Graph() with g1.as_default() as g:
    with g.name_scope( "g1" ) as scope:
        matrix11 = tf.constant([[3., 3.]])
        matrix21 = tf.constant([[2.],[2.]])
        product1 = tf.matmul(matrix11, matrix21)

tf.reset_default_graph()

g2 = tf.Graph() with g2.as_default() as g:
    with g.name_scope( "g2" ) as scope:
        matrix12 = tf.constant([[4., 4.]])
        matrix22 = tf.constant([[5.],[5.]])
        product2 = tf.matmul(matrix12, matrix22)

tf.reset_default_graph()

with tf.Session( graph = g1 ) as sess:
    result = sess.run( product1 )
    print( result )

Answer 3

This is quite the necro, but it's still a top search result for questions related to this and I thought it might help to make very explicit something that the prior answers (which are correct) note in passing:

The Q's variable product is a python variable. As such, it points to an object: When defined, it points to the tf.Tensor output of the matmul tf.Operation defined in the name_scope 'g1'. It is later redefined to point to a different object, the tf.Tensor output of 'g2'. This python variable has never heard of tf.name_scopes and doesn't care.

That is why you need to do lookups by the name attributes of the tensorflow objects... Those, by use of name_scopes, are unique and accessible. Or generate distinct python variables--that are unique and accessible according to python scoping rules--to point to each tf.Tensor object you want to reference.

Dunno if this is helpful for anybody else, but if I ever forget, I'll thank my past self for this.

Answer 4

You have 3 graphs , not 2 graphs ( g1 or g2 ) .

You can see 3 graphs by id()

...

print 'g1           ', id(g1)
print 'g2           ', id(g2)
print 'current graph', id ( tf.get_default_graph() )

with tf.Session( graph = g1 ) as sess:
    result = sess.run( product )
    print( result )

Using "with g1.as_default(): " make the same error

...

with g1.as_default() :
    with tf.Session( graph = g1 ) as sess:
        result = sess.run( product )
        print( result )

Because, you have two 'product' , not one.

g1 = tf.Graph()
with g1.as_default() as g:
    ...
    print 'g1 product', id ( product )

...

g2 = tf.Graph()
with g2.as_default() as g:
    ...
    print 'g2 product', id ( product )

with tf.Session( graph = g1 ) as sess:
    print 'last product', id(product)
    ...

last product == g2 product

...
    product = tf.matmul(matrix1, matrix2, name='g1_product')
...
with g1.as_default() as g:
    with tf.Session() as sess:
        product = g.get_tensor_by_name(  "g1_product:0" )
        result = sess.run( product )
        print( result )

Above code work.

But, two variables with the same name ( product )

Encapsulating with class is good?

Answer 5

I was having similar difficulties dealing with multiple graphs on a IPython notebook. What works for my purposes is to encapsulate each graph and its session in a function. I realize this is more of a hack I guess, I don't know anything about namespaces and I know OP wanted something along those lines. Maybe it will help someone I dunno, you can also pass results between computations.

import tensorflow as tf

def Graph1():
    g1 = tf.Graph()
    with g1.as_default() as g:
        matrix1 = tf.constant([[3., 3.]])
        matrix2 = tf.constant([[2.],[2.]])
        product = tf.matmul( matrix1, matrix2, name = "product")

    with tf.Session( graph = g ) as sess:
        tf.initialize_all_variables().run()
        return product


def Graph2(incoming):
    i = incoming
    g2 = tf.Graph()
    with g2.as_default() as g:
        matrix1 = tf.constant([[4., 4.]])
        matrix2 = tf.constant([[5.],[5.]])
        product = tf.matmul( matrix1, matrix2, name = "product" )

    with tf.Session( graph = g ) as sess:
        tf.initialize_all_variables().run()
        print product
        print i

print Graph1()

Graph2(Graph1())

Answer 6

Your product is a global variable, and you've set it to point to "g2/MatMul".

In particular

Try

print product

and you'll see

Tensor("g2/MatMul:0", shape=(1, 1), dtype=float32)

So the system takes "g2/MatMul:0" since that's the Tensor's name, and tries to find it in the graph g1 since that's the graph you set for the session. Incidentally you can see all nodes in the graph print [n.name for n in g1.as_graph_def().node]

Generally, using more than one graph is rarely useful. You can't merge them and can't pass tensors between them. I'd recommend just doing

tf.reset_default_graph()
a = tf.Constant(2)
sess = tf.InteractiveSession()
....

This way you'll have one default graph and one default session and you can omit specifying graph or session in most cases. If you ever need to refer to them explicitly, you can get them from tf.get_default_graph() or tf.get_default_session()