How can one set weights of a layer of model1 from the weights of a layer of trained model2?

Question

Consider two

model=Model(input,x)
model2=Model(input,x)

After training the model2,

model.weights[49]
#returned 
<tf.Variable 'dense_2/bias:0' shape=(28,) dtype=float32, numpy=
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>
#and 
model2.weights[49]
#returned
<tf.Variable 'dense_8/bias:0' shape=(28,) dtype=float32, numpy=
array([-0.05584828, -0.06524061,  0.02897491, -0.00907347,  0.01428275,
       -0.06510548,  0.02931084, -0.01003758, -0.01820367, -0.00648802,
        0.01412329, -0.06746212,  0.03339575,  0.05024051,  0.06549978,
       -0.04594664,  0.00359419, -0.01723421,  0.05004687, -0.0144965 ,
        0.02175333,  0.04898168,  0.04556213,  0.05112274,  0.03278055,
       -0.02069422, -0.00631414, -0.03294226], dtype=float32)>

However, if one tried to assign the weights from layer to layer

model.weights[49]=model2.weights[49]
#successfully ran
model.weights[49]
#returned 
<tf.Variable 'dense_2/bias:0' shape=(28,) dtype=float32, numpy=
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>

The weights of the dense_2 layer in model was unchanged. How could one assign model.weights[49]=model2.weights[49] yet not change the value? How to assign the weights of the layer in model2 to model?

Answer 1

What's the problem of using get_weight and set_weight? Here is an example to assign weights from one model to another model. Let's we have the following two models:

x = tf.keras.Input(shape=[4])
y = tf.keras.layers.Dense(15)(x)
modelA = tf.keras.Model(x, y)

x = tf.keras.Input(shape=[4])
y = tf.keras.layers.Dense(15)(x)
modelB = tf.keras.Model(x, y)

First, let's check their parameters.

print([np.all(w1 == w2) for w1, w2 in zip(modelA.get_weights(), modelB.get_weights())]) 

[False, True]

False refers to their weights are not matched (randomly initialized) and True refers to their corresponding biases (initialized with zeros). To check:

modelA.get_weights() 

[array([[-0.2183539 ,  0.37479502,  0.12565714, -0.09790632, -0.03254837,
         -0.19809991, -0.2914252 , -0.21781999, -0.46998876,  0.40218872,
         -0.2449336 , -0.1377019 ,  0.5495464 , -0.24009705,  0.43448   ],
        [-0.25566337,  0.31467468,  0.15949893,  0.3412726 , -0.09523717,
          0.10946101,  0.32810467, -0.29927975,  0.3723418 , -0.11723584,
         -0.01329237, -0.4096715 , -0.05762988,  0.5516278 ,  0.28062445],
        [ 0.12657142, -0.3244409 , -0.39447513,  0.27886128,  0.13312596,
         -0.5275597 ,  0.34260398, -0.14882985, -0.1329239 , -0.53033626,
          0.3994707 ,  0.1498189 , -0.32308942,  0.06801641,  0.1812048 ],
        [-0.11621088, -0.10942346, -0.07019302,  0.40427893,  0.04286063,
         -0.06250447, -0.00837666, -0.06694067, -0.4972997 ,  0.31986964,
          0.01897019, -0.20199525,  0.42970175, -0.26112786, -0.40973634]],
       dtype=float32),
 array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       dtype=float32)]

and

modelB.get_weights()

[array([[-0.03211719,  0.28092802,  0.33855444, -0.20934188,  0.20574439,
          0.35670918, -0.4615593 , -0.07423961, -0.08961365,  0.3235132 ,
         -0.05443043, -0.4499644 ,  0.534965  ,  0.23826486, -0.38260213],
        [ 0.44154066, -0.06687328, -0.5428564 ,  0.12680143, -0.31863016,
         -0.48609015,  0.30164462, -0.55543095,  0.1313321 ,  0.37449527,
         -0.03355491, -0.53522277, -0.47578162,  0.13009185,  0.53617924],
        [-0.11203861, -0.4966424 , -0.2357387 , -0.1968436 ,  0.40537828,
         -0.3463826 ,  0.07999241, -0.41489804,  0.26287907, -0.3327279 ,
          0.23491085,  0.23072398,  0.55298656,  0.5054632 , -0.16610476],
        [-0.42975706,  0.2832412 ,  0.385575  , -0.3906275 , -0.28473172,
          0.34568614,  0.41107148, -0.07780078,  0.08179295, -0.4420647 ,
          0.03800696, -0.3615641 ,  0.05891383, -0.13369054, -0.21534714]],
       dtype=float32),
 array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       dtype=float32)]

Now, we will set weights values of model modelB to modelA using get_weight and set_weight functions.

# assign values 
modelB.set_weights(modelA.get_weights())

# to check 
modelB.get_weights()

[array([[-0.2183539 ,  0.37479502,  0.12565714, -0.09790632, -0.03254837,
         -0.19809991, -0.2914252 , -0.21781999, -0.46998876,  0.40218872,
         -0.2449336 , -0.1377019 ,  0.5495464 , -0.24009705,  0.43448   ],
        [-0.25566337,  0.31467468,  0.15949893,  0.3412726 , -0.09523717,
          0.10946101,  0.32810467, -0.29927975,  0.3723418 , -0.11723584,
         -0.01329237, -0.4096715 , -0.05762988,  0.5516278 ,  0.28062445],
        [ 0.12657142, -0.3244409 , -0.39447513,  0.27886128,  0.13312596,
         -0.5275597 ,  0.34260398, -0.14882985, -0.1329239 , -0.53033626,
          0.3994707 ,  0.1498189 , -0.32308942,  0.06801641,  0.1812048 ],
        [-0.11621088, -0.10942346, -0.07019302,  0.40427893,  0.04286063,
         -0.06250447, -0.00837666, -0.06694067, -0.4972997 ,  0.31986964,
          0.01897019, -0.20199525,  0.42970175, -0.26112786, -0.40973634]],
       dtype=float32),
 array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       dtype=float32)]

Let's verify, it should be both True now.

print([np.all(w1 == w2) for w1, w2 in zip(modelA.get_weights()[:5], modelB.get_weights()[:5])]) 

[True, True]

Answer 2

In Keras/TensorFlow, layer weights are made of tf.Variable. To update a tf.Variable you need to call the assign method.

model.weights[49].assign(model2.weights[49])

You can read more about variables in the guide: Introduction to Variables.