CODEWITHSUNDEEP
pythontensorflow
Roman
Roman

Reputation: 131038

Is it possible to manually set values of model parameters without extending computational graph in TensorFlow?

I have created a computational graph for a model. It has nodes for input (X), target(Y), predictions (P), model parameters(P1, P2, ..., Pn), as well as costs (C), that should be minimised. In addition to that I have nodes representing gradients of costs with respect to model parameters (G1, G2, ...., Gn).

For a given X and Y and some initial values of model parameters (P1, P2, ..., Pn) I have calculated values gradients.

Now I want to scan over different learning rates and for all of them calculate new "candidate" values of model parameters, calculate the corresponding costs and then accept the best "candidate" parameters as new values of the model parameters.

I have tried to do it this way:

# get the current values of the model parameters
ini_vals = [s.run(param) for param in params]

# get the current values of the gradients
grad_vals = [s.run(grad, feed_dict = feed_dict) for grad in grads]

# start from the smallest allowed learning rate
lr = min_lr

best_score = None

while True:

    # get new "candidate" values of model parameters for a given learning rate
    new_vals = [ini_val - lr * grad for ini_val, grad in zip(ini_vals, grad_vals)]

    # assign the new values to the corresponding symbolic nodes
    for i in range(len(new_vals)):
        s.run(tf.assign(params[i], new_vals[i]))

    # get the corresponding score
    score_val = s.run(score, feed_dict = feed_dict)

    # if the current score is better than the previous one, accept it
    if best_score == None or score_val < best_score:
        best_score = score_val
        best_lr = lr
        best_params = new_vals[:]

    # if the new score is worse than the previous one, stop the search
    else:

        # use the best found parameters as new model paramers
        for i in range(len(new_vals)):
            s.run(tf.assign(params[i], best_params[i]))

        break

    # increase the learning rate
    lr *= factor

However, this procedure gets slower and slower when for each subsequent call. The problem is caused by the fact that by using tf.assign function I extend the computational graph (as I have learned from this answer).

Now, my question is how to achieve what I want. In more details, I want to be able manually update model parameters without extending the computational graph.

Upvotes: 1

Views: 118

Answers (1)

Vlad
Vlad

Reputation: 8585

You could use tf.Variable.load():

import tensorflow as tf
import numpy as np

w = tf.Variable(tf.ones((2,)))

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(w)) # [1. 1.]
    w.load(np.zeros((2,)))
    print(sess.run(w)) # [0. 0.]

Upvotes: 1

Related Questions