How to load several identical models from save files into one session in Tensorflow

Question

In my main code I create a model based on a config file like this

with tf.variable_scope('MODEL') as topscope:
    model = create_model(config_file)#returns input node, output node, and some other placeholders

Name of this scope is the same across all saves.

Then I define an optimizer and a cost function, etc.(they are outside of this scope)

Then I create a saver and save it:

saver = tf.train.Saver(max_to_keep=10)
saver.save(sess, 'unique_name', global_step=t)

Now I've created and saved 10 different models, and I want to load them all at once like this maybe:

models = []
for config, save_path in zip(configs, save_paths):
    models.append(load_model(config, save_path))

and be able to run them and compare their results, mix them, average etc. I don't need optimizer slot variables for these loaded models. I need only those variables that are inside 'MODEL' scope.

Do I need to create multiple sessions?

How can I do it? I don't know where to start. I can create a model from my config file, then load this same model using this same config file and a save like this:

saver.restore(sess, save_path)

But how do I load more than one?

Edit: I didn't know the word. I want to make an ensemble of networks. Question that asks it and is still not answered: How to create ensemble in tensorflow?

EDIT 2: Okay, so here's my workaround for now:

Here's my main code, it creates a model, trains it and saves it:

import tensorflow as tf
from util import *

OLD_SCOPE_NAME = 'scope1'

sess = tf.Session()

with tf.variable_scope(OLD_SCOPE_NAME) as topscope:
    model = create_model(tf, 6.0, 7.0)
    sc_vars = get_all_variables_from_top_scope(tf, topscope)

print([v.name for v in sc_vars])

sess.run(tf.initialize_all_variables())
print(sess.run(model))

saver = tf.train.Saver()
saver.save(sess, OLD_SCOPE_NAME)

Then I run this code creating the same model, loading its checkpoint save and renaming variables:

#RENAMING PART, different file
#create the same model as above here
import tensorflow as tf
from util import *
OLD_SCOPE_NAME = 'scope1'
NEW_SCOPE_NAME = 'scope2'

sess = tf.Session()

with tf.variable_scope(OLD_SCOPE_NAME) as topscope:
    model = create_model(tf, 6.0, 7.0)
    sc_vars = get_all_variables_from_top_scope(tf, topscope)

print([v.name for v in sc_vars])

saver = tf.train.Saver()
saver.restore(sess, OLD_SCOPE_NAME)
print(sess.run(model))


#assuming that we change top scope, not something in the middle, functionality can be added without much trouble I think
#not sure why I need to remove ':0' part, but it seems to work okay
print([NEW_SCOPE_NAME + v.name[len(OLD_SCOPE_NAME):v.name.rfind(':')] for v in sc_vars])
new_saver = tf.train.Saver(var_list={NEW_SCOPE_NAME + v.name[len(OLD_SCOPE_NAME):v.name.rfind(':')]:v for v in sc_vars})
new_saver.save(sess, NEW_SCOPE_NAME)

Then to load this model into a file containing additional variables and with a new name:

import tensorflow as tf
from util import *
NEW_SCOPE_NAME = 'scope2'
sess = tf.Session()

with tf.variable_scope(NEW_SCOPE_NAME) as topscope:
    model = create_model(tf, 5.0, 4.0)
    sc_vars = get_all_variables_from_top_scope(tf, topscope)
q = tf.Variable(tf.constant(0.0, shape=[1]), name='q')

print([v.name for v in sc_vars])

saver = tf.train.Saver(var_list=sc_vars)
saver.restore(sess, NEW_SCOPE_NAME)
print(sess.run(model))

util.py:

def get_all_variables_from_top_scope(tf, scope):
    #scope is a top scope here, otherwise change startswith part
    return [v for v in tf.all_variables() if v.name.startswith(scope.name)]

def create_model(tf, param1, param2):
    w = tf.get_variable('W', shape=[1], initializer=tf.constant_initializer(param1))
    b = tf.get_variable('b', shape=[1], initializer=tf.constant_initializer(param2))
    y = tf.mul(w, b, name='mul_op')#no need to save this
    return y

Answer 1

At the conceptual level:

• there are two separate things: the graph, and the session
• the graph is created first. It defines your model. There's no reason why you cant store multiple models in one graph. Thats fine. It also defines the Variables, but it doesnt actually contain their state
• a session is created after the graph
  • it is created from a graph
  • you can create as many session as you like from a graph
  • it holds the state of the different Variables in the graph, ie the weights in your various models

So:

• when you load just the model definition, all you need is: one or more graphs. one graph is sufficient
• when you load the actual weights for the model, the learned weights/parameters, you need to have created a session for this, from the graph. A single session is sufficient

Note that variables all have names, and they need to be unique. You can give them unique names, in the graph, by using variable scopes, like:

with tf.variable_scope("some_scope_name"):
    # created model nodes here...  

This will groups your nodes together nicely in the Tensorboard graph.

Ok, rereading your question a bit. It looks like you want to save/load single models at a time.

Saving/loading the parameters/weights of a model happens from the session, which is what contains the weights/parameters of each Variable defined in the graph.

You can refer to these variables by name, eg via the scope you created above, and just save a subset of these variables, into different files, etc.

By the way, its also possible to use session.run(...) to get the values o the weights/parameters, as numpy tensors, which you can then pickle, or whatever, if you choose.