TensorFlow tf.group ignoring dependencies?

Question

Following on from an earlier question, it seems tf.group is indeed ignoring dependencies. Here's a simple stand-alone example (I have run it on Python 2.7 with TensorFlow 1.1):

import tensorflow as tf
from tensorflow.python.ops import control_flow_ops

xs = [tf.constant(x) for x in range(10)]
xs = [tf.Print(x, [x]) for x in xs]
dependency = None
dxs = []

for x in xs:
    if dependency is None:
        dependency = x
    else:
        dependency = control_flow_ops.with_dependencies([dependency], x)

    dxs.append(dependency)

print_all_op = tf.group(*dxs)

with tf.Session() as session:
    session.run(print_all_op)

Expected output:

2017-05-29 15:11:53.961221: I tensorflow/core/kernels/logging_ops.cc:79] [0]
2017-05-29 15:11:53.961236: I tensorflow/core/kernels/logging_ops.cc:79] [1]
2017-05-29 15:11:53.961255: I tensorflow/core/kernels/logging_ops.cc:79] [2]
2017-05-29 15:11:53.961237: I tensorflow/core/kernels/logging_ops.cc:79] [3]
2017-05-29 15:11:53.961262: I tensorflow/core/kernels/logging_ops.cc:79] [4]
2017-05-29 15:11:53.961263: I tensorflow/core/kernels/logging_ops.cc:79] [5]
2017-05-29 15:11:53.961268: I tensorflow/core/kernels/logging_ops.cc:79] [6]
2017-05-29 15:11:53.961272: I tensorflow/core/kernels/logging_ops.cc:79] [7]
2017-05-29 15:11:53.961274: I tensorflow/core/kernels/logging_ops.cc:79] [8]
2017-05-29 15:11:53.961221: I tensorflow/core/kernels/logging_ops.cc:79] [9]

Actual output (different each time the code is run):

2017-05-29 15:16:26.279655: I tensorflow/core/kernels/logging_ops.cc:79] [0]
2017-05-29 15:16:26.279655: I tensorflow/core/kernels/logging_ops.cc:79] [9]
2017-05-29 15:16:26.279697: I tensorflow/core/kernels/logging_ops.cc:79] [3]
2017-05-29 15:16:26.279660: I tensorflow/core/kernels/logging_ops.cc:79] [1]
2017-05-29 15:16:26.279711: I tensorflow/core/kernels/logging_ops.cc:79] [8]
2017-05-29 15:16:26.279713: I tensorflow/core/kernels/logging_ops.cc:79] [4]
2017-05-29 15:16:26.279723: I tensorflow/core/kernels/logging_ops.cc:79] [5]
2017-05-29 15:16:26.279663: I tensorflow/core/kernels/logging_ops.cc:79] [2]
2017-05-29 15:16:26.279724: I tensorflow/core/kernels/logging_ops.cc:79] [7]
2017-05-29 15:16:26.279728: I tensorflow/core/kernels/logging_ops.cc:79] [6]

There's nothing in the tf.group documentation to indicate why dependencies are ignored.

Is there an alternative to tf.group that does consider dependencies?

Switching to use tf.control_dependencies instead of tensorflow.python.ops.control_flow_ops.with_dependencies doesn't help:

import tensorflow as tf

xs = [tf.constant(x) for x in range(10)]
xs = [tf.Print(x, [x]) for x in xs]
dependency = None
dxs = []

for x in xs:
    if dependency is None:
        dependency = x
    else:
        with tf.control_dependencies([dependency]):
            dependency = x

    dxs.append(dependency)

print_all_op = tf.group(*dxs)

with tf.Session() as session:
    session.run(print_all_op)

Answer 1

I think the problem is that the initial code creates dependencies between dummy identity ops implicitly created by control_flow_ops.with_dependencies and not the actual tf.Print ops. Tensorflow seems to only ensure that the ops in the dependency list have been already executed but the order of other preceding ops is not fixed. In the above example, the dependencies are created on the dummy identity ops created by control_flow_ops.with_dependencies:

    dependency = control_flow_ops.with_dependencies([dependency], x)

which should be equivalent to:

        with tf.control_dependencies([dependency]):
            dependency = tf.identity(x)

Thus, the dependencies here are created between the tf.identity ops and not the tf.Print ops. The tf.Print ops can be run in any order, the strict ordering is only on the tf.identity ops. I don't think it is possible to achieve the desired behavior with control_flow_ops.with_dependencies. Instead one has to use with tf.control_dependencies instead (as already suggested by the op):

xs = [tf.constant(x) for x in range(10)]
dependency = None
dxs = []

for x in xs:
    if dependency is None:
        dependency = tf.Print(x, [x])
    else:
        with tf.control_dependencies([dependency]):
            dependency = tf.Print(x, [x])

    dxs.append(dependency)

Answer 2

Using tf.control_dependencies correctly does solve this problem:

import tensorflow as tf

xs = [tf.constant(x) for x in range(10)]
dependency = None
dxs = []

for x in xs:
    if dependency is None:
        dependency = tf.Print(x, [x])
    else:
        with tf.control_dependencies([dependency]):
            dependency = tf.Print(x, [x])

    dxs.append(dependency)

print_all_op = tf.group(*dxs)

with tf.Session() as session:
    session.run(print_all_op)

Note that the Print operation needs to be created within the tf.control_dependencies context manager.

I am still unclear why the control_flow_ops.with_dependencies version fails.