Confused about saving/restoring trained weights and biases in Tensorflow

Question

I am training a convolutional neural network in Tensorflow. My code runs to completion without error. That said, I am having trouble understanding exactly how I can save the weights and biases the NN learns (this is important as I'm training on a server and would like to do easier visualization stuff locally).

I initialize my weights and biases thusly:

weights = {
'wConv1':  tf.Variable(tf.random_normal([5, 5, 1,   3],0,0.25),    name='wC1'),
'wConv2':  tf.Variable(tf.random_normal([5, 5, 3,  32],0,0.25),  name='wC2'),
'wConv3':  tf.Variable(tf.random_normal([5, 5, 32, 64],0,0.25),  name='wC3'),
'wConv4':  tf.Variable(tf.random_normal([5, 5, 64, 128],0,0.25), name='wC4'),
'wConv5':  tf.Variable(tf.random_normal([5, 5, 128, 64],0,0.25), name='wC5'),
'wConv6':  tf.Variable(tf.random_normal([5, 5, 64, 32],0,0.25),  name='wC6'),
'wConv7':  tf.Variable(tf.random_normal([5, 5, 32, 16],0,0.25),  name='wC7'),
'wOUT'  :  tf.Variable(tf.random_normal([5, 5, 16, 1],0,0.25),          name='wCOUT')
}

biases = {
'bConv1': tf.Variable(tf.random_normal([3]),   name='bC1'),
'bConv2': tf.Variable(tf.random_normal([32]),  name='bC2'),
'bConv3': tf.Variable(tf.random_normal([64]),  name='bC3'),
'bConv4': tf.Variable(tf.random_normal([128]), name='bC4'),
'bConv5': tf.Variable(tf.random_normal([64]),  name='bC5'),
'bConv6': tf.Variable(tf.random_normal([32]),  name='bC6'),
'bConv7': tf.Variable(tf.random_normal([16]),  name='bC7'),
'bOUT': tf.Variable(tf.random_normal([1]),     name='bCOUT')
 }

Then, once however-many epochs I run are complete, I save everything using the following:

 saver = tf.train.Saver({"weights": weights, "biases": biases})
 save_path = saver.save(sess, "./output/trained.ckpt")

Now, on my own machine I have an evaluation script, wherein I try to load the weights:

with sess.as_default():
          saver = tf.train.import_meta_graph('output.ckpt.meta')
          saver.restore(sess,tf.train.latest_checkpoint('./'))
          a= tf.all_variables()
          sess.run(tf.global_variables_initializer())
          b=sess.run(pred,feed_dict={x: input[:,:,:,30,:]})

Now, the issue is, when I load in "a" I get a mess, with what appears to be many copies of my bias and weight variables:

,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,


,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ,
 ]

My question is, how can I save ONLY the trained weights and biases in Tensorflow and then load them later on for testing purposes?

amirbar · Accepted Answer

Before answering the exact question, let me first address your concern:

the issue is, when I load in "a" I get a mess, with what appears to be many copies of my bias and weight variables

In your evaluation script you load your training metagraph:

saver = tf.train.import_meta_graph('output.ckpt.meta')

Inside that graph, during training, other than the explicit weight and biases variables you defined, there are variables related to the optimization process (i.e variables with suffix adam or beta1_power). Executing the line specified above, they are defined again in your evaluation script, although may not necessarily required for inference.

An alternative, would be to define the exact graph you want for inference, which may be a bit different from training. In your case - just not defining an optimizer.

Now to address your question:

My question is, how can I save ONLY the trained weights and biases in Tensorflow and then load them later on for testing purposes?

From your code, it seems like your'e essentially doing this. The other variables you see stems from the described above.

One thing to point out - make sure that you don't initialize the variables after restoring them. If you stay with the current code, first initialize and then restore. If you plan to change your inference graph and not include the optimizer you will not need to initialize any variable.

Confused about saving/restoring trained weights and biases in Tensorflow

Answers (1)

Related Questions