Keras callbacks Tensorboard multi-output Error

Question

I can't figure out what is causing this error for multi-output Keras Model when using callbacks.TensorBoard.

tbCallBack = keras.callbacks.TensorBoard(log_dir = logdir, histogram_freq = 1, write_graph = 1, write_images = 0, write_grads = 1)
###No errror when not using callbacks
regr.fit(  Ax_train, [Ay_train_p, Ay_train_s], validation_data= (Ax_test, [Ay_test_p, Ay_test_s]), epochs = 500, batch_size = 10, verbose = 1)
###No errror when not using validation_data
regr.fit(  Ax_train, [Ay_train_p, Ay_train_s], epochs = 500, batch_size = 10, verbose = 1, callbacks=[tbCallBack])
###Error Occurred
regr.fit( Ax_train, [Ay_train_p, Ay_train_s], validation_data= (Ax_test, [Ay_test_p, Ay_test_s]), epochs = 500, batch_size = 10, verbose = 1, callbacks=[tbCallBack])

Error

Epoch 1/500
1280/1663 [======================>.......] - ETA: 0s - loss: 1.6230 - output_power_loss: 0.9627 - output_slack_loss: 0.66032017-07-11 03:17:27.964542: W tensorflow/core/framework/op_kernel.cc:1148] Invalid argument: Shape [-1] has negative dimensions
2017-07-11 03:17:27.964589: E tensorflow/core/common_runtime/executor.cc:644] Executor failed to create kernel. Invalid argument: Shape [-1] has negative dimensions
     [[Node: output_slack_sample_weights = Placeholder[dtype=DT_FLOAT, shape=[?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
2017-07-11 03:17:27.970690: W tensorflow/core/framework/op_kernel.cc:1148] Invalid argument: Shape [-1] has negative dimensions
2017-07-11 03:17:27.970735: E tensorflow/core/common_runtime/executor.cc:644] Executor failed to create kernel. Invalid argument: Shape [-1] has negative dimensions
     [[Node: output_power_sample_weights = Placeholder[dtype=DT_FLOAT, shape=[?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
2017-07-11 03:17:27.972004: W tensorflow/core/framework/op_kernel.cc:1148] Invalid argument: Shape [-1] has negative dimensions
2017-07-11 03:17:27.972026: E tensorflow/core/common_runtime/executor.cc:644] Executor failed to create kernel. Invalid argument: Shape [-1] has negative dimensions
     [[Node: output_power_sample_weights = Placeholder[dtype=DT_FLOAT, shape=[?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Traceback (most recent call last):
  File "tf_keras.py", line 183, in <module>
    regr.fit( Ax_train, [Ay_train_p, Ay_train_s], validation_data= (Ax_test, [Ay_test_p, Ay_test_s]), epochs = 500, batch_size = 10, verbose = 1, callbacks=[tbCallBack])
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/engine/training.py", line 1507, in fit
    initial_epoch=initial_epoch)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/engine/training.py", line 1176, in _fit_loop
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/callbacks.py", line 77, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/callbacks.py", line 768, in on_epoch_end
    result = self.sess.run([self.merged], feed_dict=feed_dict)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape [-1] has negative dimensions
     [[Node: output_power_sample_weights = Placeholder[dtype=DT_FLOAT, shape=[?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'output_power_sample_weights', defined at:
  File "tf_keras.py", line 180, in <module>
    regr = nn_model()
  File "tf_keras.py", line 177, in nn_model
    model.compile(optimizer = 'adam', loss ={'output_power': 'mean_squared_error', 'output_slack': 'binary_crossentropy'})
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/engine/training.py", line 870, in compile
    name=name + '_sample_weights'))
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 431, in placeholder
    x = tf.placeholder(dtype, shape=shape, name=name)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1530, in placeholder
    return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1954, in _placeholder
    name=name)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Shape [-1] has negative dimensions
     [[Node: output_power_sample_weights = Placeholder[dtype=DT_FLOAT, shape=[?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

What does "Shape [-1] has negative dimensions" means? I had also tried with each output with callbacks.Tensorboard and there wasn't any error occurred. Also search with "Node: output_power_sample_weights" but no results.

regr.summary()

____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
main_input (InputLayer)          (None, 5)             0                                            
____________________________________________________________________________________________________
Hidden (Dense)                   (None, 5)             30          main_input[0][0]                 
____________________________________________________________________________________________________
output_power (Dense)             (None, 1)             6           Hidden[0][0]                     
____________________________________________________________________________________________________
output_slack (Dense)             (None, 1)             6           Hidden[0][0]                     
====================================================================================================
Total params: 42
Trainable params: 42
Non-trainable params: 0

Answer 1

The "Shape [-1] has negative dimensions" error message is a (misleading) error printed when you don't feed a tf.placeholder() that has a partially-defined shape. This error message has been replaced with a more informative message in the nightly version of TensorFlow.

The error and the stack trace suggests that this Session.run() call is failing (source):

  File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/callbacks.py", line 768, in on_epoch_end
    result = self.sess.run([self.merged], feed_dict=feed_dict)

Looking at the implementation, it appears that this is only called when the TensorBoard callback is installed, and self.validation_data is not None. From the implementation, it looks like this callback is incompatible with models that depend on placeholders that are not one of (i) model.inputs, (ii) model.targets, or (iii) model.sample_weights. It might be best to open an issue on the Keras GitHub issue tracker about this.