CODEWITHSUNDEEP
pythonmachine-learningtensorflowdeep-learning
Muhammad
Muhammad

Reputation: 96

Modifying the weights and biases of a restored CNN model in TensorFlow

I have recently started using TensorFlow (TF), and I have come across a problem that I need some help with. Basically, I've restored a pre-trained model, and I need to modify the weights and biases of one of its layers before I retest its accuracy. Now, my problem is the following: how can I change the weights and biases using the assign method in TF? Is modifying the weights of a restored modeled even possible in TF?

Here is my code:

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data # Imports the MINST dataset

# Data Set:
# ---------
mnist = input_data.read_data_sets("/home/frr/MNIST_data", one_hot=True)# An object where data is stored

ImVecDim = 784# The number of elements in a an image vector (flattening a 28x28 2D image)
NumOfClasses = 10

g = tf.get_default_graph()

with tf.Session() as sess:
  LoadMod = tf.train.import_meta_graph('simple_mnist.ckpt.meta')  # This object loads the model
  LoadMod.restore(sess, tf.train.latest_checkpoint('./'))# Loading weights and biases and other stuff to the model

  # ( Here I'd like to modify the weights and biases of layer 1, set them to one for example, before I go ahead and test the accuracy ) #

  # Testing the acuracy of the model:
  X = g.get_tensor_by_name('ImageIn:0')
  Y = g.get_tensor_by_name('LabelIn:0')
  KP = g.get_tensor_by_name('KeepProb:0')
  Accuracy = g.get_tensor_by_name('NetAccuracy:0')
  feed_dict = { X: mnist.test.images[:256], Y: mnist.test.labels[:256], KP: 1.0 }
  print( 'Model Accuracy = ' )
  print( sess.run( Accuracy, feed_dict ) )

Upvotes: 6

Views: 10842

Answers (4)

prerakmody
prerakmody

Reputation: 116

An update to this for Tensorflow 2.4 using a different example than OP's.

# Step 0 - Init
model        = # some tf.keras.Model
model_folder = # path to model files
    
ckpt_obj = tf.train.Checkpoint(model=model)
ckpt_obj.restore(save_path=tf.train.latest_checkpoint(str(model_folder))).expect_partial()

# Step 1 - Loop over all layers
for layer in model.layers:

    # Step 2 - Loop over submodules of a layer
    for submodule in layer.submodules:

        # Step 3 - Find a particular type of submodule (alternative use submodule.name=='SomeName')
        if type(submodule) == tfp.layers.Convolution3DFlipout: # kernel=N(loc,scale) --> N=Normal distro
            
            # Step 4 - Extract numpy weights using .get_weights()
            ## Note: Different compared to submodule.weights which returns a tensor that shall also have a name e.g. wc1:0
            weights = submodule.get_weights() # [scale, rho, bias] --> kernel=N(loc,scale=tfp.bijectors.Softplus(rho)) --> output=input*kernel + bias
            
            # Step 5 - Set weights as a new numpy array of your choice
            weights[1] = np.full(weights[1].shape, -np.inf) 

            # Step 6 - Update weights
            submodule.set_weights(weights)


input = tf.random.normal((1,100,100,100,1)) # 3D input with batch=1, channels=1
_ = model(input)

Upvotes: 1

Kilian Obermeier
Kilian Obermeier

Reputation: 7128

Yes it is possible. Your weights and biases are already loaded after you loaded the meta graph. You need to find their names (see the list_variables function) and then assign them to a Python variable.

For that, use tf.get_variable with the variable name. You might have to set reuse=True on your variable scope. See this answer for more detail on reusing variables.

Once you have them as a weights variable, you can call sess.run(weights.assign(...)).

Upvotes: 1

Muhammad
Muhammad

Reputation: 96

Thanks for everyone who responded. I'd just like to put the pieces together. This is the code the helped me accomplish what I want:

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data # Imports the MINST dataset

# Data Set:
# ---------
mnist = input_data.read_data_sets("/home/frr/MNIST_data", one_hot=True)# An object where data is stored

ImVecDim = 784# The number of elements in a an image vector (flattening a 28x28 2D image)
NumOfClasses = 10

g = tf.get_default_graph()

with tf.Session() as sess:
   LoadMod = tf.train.import_meta_graph('simple_mnist.ckpt.meta')  # This object loads the model
   LoadMod.restore(sess, tf.train.latest_checkpoint('./'))# Loading weights and biases and other stuff to the model

   wc1 = g.get_tensor_by_name('wc1:0')
   sess.run( tf.assign( wc1,tf.multiply(wc1,0) ) )# Setting the values of the variable 'wc1' in the model to zero.

   # Testing the acuracy of the model:
   X = g.get_tensor_by_name('ImageIn:0')
   Y = g.get_tensor_by_name('LabelIn:0')
   KP = g.get_tensor_by_name('KeepProb:0')
   Accuracy = g.get_tensor_by_name('NetAccuracy:0')
   feed_dict = { X: mnist.test.images[:256], Y: mnist.test.labels[:256], KP: 1.0 }
   print( 'Model Accuracy = ' )
   print( sess.run( Accuracy, feed_dict ) )

Upvotes: 2

Maxim
Maxim

Reputation: 53758

In addition to an existing answer, tensor update can be performed via tf.assign function.

v1 = sess.graph.get_tensor_by_name('v1:0')
print(sess.run(v1))   # 1.0
sess.run(tf.assign(v1, v1 + 1))
print(sess.run(v1))   # 2.0

Upvotes: 3

Related Questions