Multiple sequential Tensorflow operations in same Session.run() call

Question

As the title suggests, I want to run multiple Tensorflow operations in the same Session.run() call. Specifically, to make the problem more concrete, suppose I want to run multiple training iterations in a single call.

The standard way of doing that with multiple Session.run() calls would be something like this:

# Declare the function that we want to minimize
func = ...

# Create the optimizer which will perform a single optimization iteration
optimizer = tf.train.AdamOptimizer().minimize(func)

# Run N optimization iterations
N = 10
with tf.Session() as sess:

    sess.run( tf.global_variables_initializer() )
    for i in range(N):
        sess.run( optimizer )

However, this of course will have some overhead because we are making multiple session calls. I assume that we could remove some significant overhead by somehow grouping the operations. I assume that either the group or count_up_to are what I should be using, but I cannot find any examples demonstrating how to use them for this case. Could somebody please point me in the right direction?

The ultimate goal is to define some compound operation that would run N iterations in a single call, so that the above could be transformed to something like this:

# Declare the function that we want to minimize
func = ...

# Create the optimizer which will perform a single optimization iteration
optimizer = tf.train.AdamOptimizer().minimize(func)

# Create the compound operation that will run the optimizer 10 times
optimizeNIterations = ?????
with tf.Session() as sess:

    sess.run( tf.global_variables_initializer() )
    sess.run( optimizeNIterations )

EDIT::

As musically_ut points out, I could indeed chain the opertions together by forcing the problem to have feed dictionaries. But that feels like a solution to one very specific problem. My overall concern is how to sequentially execute operations in a single session run. I can given another example why you would want this....

Suppose now that in addition to wanting to run my optimizer, I want to retrieve the optimized values, let's say these lay in the variable X. If I want to optimize AND get the optimized values, I could try to do something like this

with tf.Session() as sess:

    sess.run( tf.global_variables_initializer() )
    o, x = sess.run( [ optimizer, X ] )

But in fact this will not work because the operations (optimizer,X) do not run sequentially. I fundamentally need to have 2 session calls:

with tf.Session() as sess:

    sess.run( tf.global_variables_initializer() )
    o = sess.run( optimizer )
    x = sess.run( X )

The question is how one can combine these two calls into one.

Answer 1

It sounds like you could put whichever operations you'd like to run multiple times in a tf.while_loop. If the operations are independent, you may have to either set parallel_iterations to 1 or (better) use control dependencies to sequence the optimizer calls. For example:

import tensorflow as tf

with tf.Graph().as_default():
  opt = tf.train.AdamOptimizer(0.1)
  # Use a resource variable for a true "read op"
  var = tf.get_variable(name="var", shape=[], use_resource=True)
  def _cond(i, _):
    return tf.less(i, 20)  # 20 iterations
  def _body(i, sequencer):
    with tf.control_dependencies([sequencer]):
      loss = .5 * (var - 10.) ** 2
      print_op = tf.Print(loss, ["Evaluating loss", i, loss])
    with tf.control_dependencies([print_op]):
      train_op = opt.minimize(loss)
    with tf.control_dependencies([train_op]):
      next_sequencer = tf.ones([])
    return i + 1, next_sequencer
  initial_value = var.read_value()
  with tf.control_dependencies([initial_value]):
    _, sequencer = tf.while_loop(cond=_cond, body=_body, loop_vars=[0, 1.])
  with tf.control_dependencies([sequencer]):
    final_value = var.read_value()
  init_op = tf.global_variables_initializer()
  with tf.Session() as session:
    session.run([init_op])
    print(session.run([initial_value, final_value]))

Prints:

2017-12-21 11:40:35.920035: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][0][46.3987083]
2017-12-21 11:40:35.920317: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][1][45.4404]
2017-12-21 11:40:35.920534: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][2][44.4923515]
2017-12-21 11:40:35.920715: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][3][43.55476]
2017-12-21 11:40:35.920905: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][4][42.6277695]
2017-12-21 11:40:35.921084: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][5][41.711544]
2017-12-21 11:40:35.921273: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][6][40.8062363]
2017-12-21 11:40:35.921426: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][7][39.9120026]
2017-12-21 11:40:35.921578: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][8][39.028965]
2017-12-21 11:40:35.921732: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][9][38.1572723]
2017-12-21 11:40:35.921888: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][10][37.2970314]
2017-12-21 11:40:35.922053: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][11][36.4483566]
2017-12-21 11:40:35.922187: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][12][35.6113625]
2017-12-21 11:40:35.922327: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][13][34.7861366]
2017-12-21 11:40:35.922472: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][14][33.9727631]
2017-12-21 11:40:35.922613: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][15][33.1713257]
2017-12-21 11:40:35.922777: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][16][32.3818779]
2017-12-21 11:40:35.922942: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][17][31.6044941]
2017-12-21 11:40:35.923115: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][18][30.8392067]
2017-12-21 11:40:35.923253: I tensorflow/core/kernels/logging_ops.cc:79] [Evaluating loss][19][30.0860634]
[0.36685812, 2.3390481]