CODEWITHSUNDEEP
pythonmachine-learningtensorflow
narner90
narner90

Reputation: 23

Tensorflow AdamOptimizer throws error when variable has validate_shape=False

Tensorflow's AdamOptimizer is throwing an error for me when any trainable variables have the "validate_shape" argument set to False. I set this argument to false to allow a dynamic batch size at runtime. 

~/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py in as_list(self)
    774     """
    775     if self._dims is None:
--> 776       raise ValueError("as_list() is not defined on an unknown TensorShape.")
    777     return [dim.value for dim in self._dims]
    778 

ValueError: as_list() is not defined on an unknown TensorShape.

I cannot reproduce this error with GradientDescentOptimizer but all other optimizers in Tensorflow throw the same error. 

opt = tf.train.AdamOptimizer()

X = tf.placeholder(tf.float64, shape=[None, None], name='X')
Y = tf.placeholder(tf.float64, shape=[None, 1], name='Y')
B = tf.Variable(initial_value=tf.zeros([tf.shape(X)[1], 1], dtype=tf.float64), validate_shape=False, name='B')

prediction = tf.matmul(X, B)
loss = tf.reduce_sum((Y- prediction) ** 2)
opt_op = opt.minimize(loss)

Is there a reason why GradientDescentOptimizer seems to support dynamic variable shapes at runtime but the other optimizers do not?

Full repro at: https://github.com/narner90/scratchpad/blob/master/AdamOptimizerValidateShape.ipynb

Upvotes: 2

Views: 1507

Answers (1)

yuefengz
yuefengz

Reputation: 3368

The AdamOptimizer needs to define several internal variables as documented in its __init__ function: https://www.tensorflow.org/versions/r0.11/api_docs/python/train.html#AdamOptimizer

These variables (moment vectors) require the same shape with variables to be updated and can not be dynamically shaped. They are initialized via tf.zeros which doesn't accept a unknown or a partially known shape because tensorflow doesn't know how many elements to set to zero if the shape is not known. Even if you could use tf.zeros_like instead, the complete shapes of these zero variables are still needed when you run session.run(init_op).

Upvotes: 5

Related Questions