keras for adding two dense layers

Question

There are two inputs, x, and u, that generate the output y. There is a linear relationship between x, u, and y, i.e. y = x wx + u wx. I'm trying to calculate wx and wu from data. Here is the code for model construction / fitting.

    n_train = 400
    n_val = 100
    train_u = u[:(n_train+n_val)]
    train_x = x[:(n_train+n_val)]
    train_y = y[:(n_train+n_val)]
    test_u = u[(n_train+n_val):]
    test_x = x[(n_train+n_val):]
    test_y = y[(n_train+n_val):]
    val_u = train_u[-n_val:]
    val_x = train_x[-n_val:]
    val_y = train_y[-n_val:]
    train_u = train_u[:-n_val]
    train_x = train_x[:-n_val]
    train_y = train_y[:-n_val]

    # RNN derived classes want a shape of (batch_size, timesteps, input_dim)
    # batch_size. One sequence is one sample. A batch is comprised of one or more samples.
    # timesteps. One time step is one point of observation in the sample.
    # input_dim. number of observation at a time step.
    # I believe n_train = one_epoch = batch_size * time_steps, features = nx_lags or nu_lags
    # I also thing an epoch is one pass through the training data

    n_batches_per_epoch = 8
    n_iterations_per_batch = round(n_train / n_batches_per_epoch)
    batch_size = n_batches_per_epoch
    time_steps = n_iterations_per_batch
    features_x = train_x.shape[1]
    features_u = train_u.shape[1]
    features_y = train_y.shape[1]

    keras_train_u = train_u.values.reshape((batch_size, time_steps, features_u))
    keras_train_x = train_x.values.reshape((batch_size, time_steps, features_x))
    keras_train_y = train_y.reshape((batch_size, time_steps, features_y))
    keras_val_u = val_u.values.reshape((2, time_steps, features_u))
    keras_val_x = val_x.values.reshape((2, time_steps, features_x))
    keras_val_y = val_y.reshape((2, time_steps, features_y))
    keras_test_u = test_u.values.reshape((1, test_u.shape[0], features_u))
    keras_test_x = test_x.values.reshape((1, test_u.shape[0], features_x))
    keras_test_y = test_y.reshape((1, test_u.shape[0], features_y))

    print('u.values.shape: ', u.values.shape)
    # Now try a tensorflow model
    # x_input = keras.Input(shape=(batch_size, time_steps, features_x), name='x_input')
    # u_input = keras.Input(shape=(batch_size, time_steps, features_u), name='u_input')
    x_input = keras.Input(shape=(time_steps, features_x), name='x_input')
    u_input = keras.Input(shape=(time_steps, features_u), name='u_input')
    da = layers.Dense(ny, name='dense_a', use_bias=False)(x_input)
    db = layers.Dense(ny, name='dense_b', use_bias=False)(u_input)
    output = layers.Add()([da, db])

    model = keras.Model(inputs=[x_input, u_input], outputs=output)

    model.compile(optimizer=keras.optimizers.RMSprop(),  # Optimizer
                  # Loss function to minimize
                  loss=keras.losses.SparseCategoricalCrossentropy(),
                  # List of metrics to monitor
                  metrics=[keras.metrics.SparseCategoricalAccuracy()])
    print(model.summary())
    print('keras_train_x.shape: ', keras_train_x.shape)
    print('keras_train_u.shape: ', keras_train_u.shape)
    print('keras_train_y.shape: ', keras_train_y.shape)
    print('keras_val_x.shape: ', keras_val_x.shape)
    print('keras_val_u.shape: ', keras_val_u.shape)
    print('keras_val_y.shape: ', keras_val_y.shape)
    history = model.fit([keras_train_x, keras_train_u], keras_train_y,
                        batch_size=64,
                        epochs=3,
                        # We pass some validation for
                        # monitoring validation loss and metrics
                        # at the end of each epoch
                        validation_data=([keras_val_x, keras_val_u], keras_val_y))

And, here is the output, with error.

Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
x_input (InputLayer)            [(None, 50, 7)]      0                                            
__________________________________________________________________________________________________
u_input (InputLayer)            [(None, 50, 7)]      0                                            
__________________________________________________________________________________________________
dense_a (Dense)                 (None, 50, 2)        14          x_input[0][0]                    
__________________________________________________________________________________________________
dense_b (Dense)                 (None, 50, 2)        14          u_input[0][0]                    
__________________________________________________________________________________________________
add (Add)                       (None, 50, 2)        0           dense_a[0][0]                    
                                                                 dense_b[0][0]                    
==================================================================================================
Total params: 28
Trainable params: 28
Non-trainable params: 0
__________________________________________________________________________________________________
None
keras_train_x.shape:  (8, 50, 7)
keras_train_u.shape:  (8, 50, 7)
keras_train_y.shape:  (8, 50, 2)
keras_val_x.shape:  (2, 50, 7)
keras_val_u.shape:  (2, 50, 7)
keras_val_y.shape:  (2, 50, 2)
Train on 8 samples, validate on 2 samples

Epoch 1/3
Traceback (most recent call last):
  File "arx_rnn.py", line 487, in <module>
    main()
  File "/arx_rnn.py", line 481, in main
    rnn_prediction = x.rnn_n_steps(y_measured, u_control, n_to_predict)
  File "arx_rnn.py", line 387, in rnn_n_steps
    validation_data=([keras_val_x, keras_val_u], keras_val_y))
  File "venv\lib\site-packages\tensorflow\python\keras\engine\training.py", line 780, in fit
    steps_name='steps_per_epoch')
  File "venv\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py", line 363, in model_iteration
    batch_outs = f(ins_batch)
  File "venv\lib\site-packages\tensorflow\python\keras\backend.py", line 3292, in __call__
    run_metadata=self.run_metadata)
  File "venv\lib\site-packages\tensorflow\python\client\session.py", line 1458, in __call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Can not squeeze dim[2], expected a dimension of 1, got 2
     [[{{node metrics/sparse_categorical_accuracy/Squeeze}}]]

Process finished with exit code 1

What is the error message telling me, and how to correct?

Answer 1

Keras categorical accuracy metrics expect the output, & labels, shape as (batch_size,num_classes). The dim[2] in error message indicates output shape is 3d: (None,50,2)

The simple fix is to ensure, by whatever means, that the output layer gives one prediction per class per batch - i.e. has shape (batch_size,num_classes) - which can be done via Reshape, or Flatten.

The better fix is to alter your input-output topology per design needs - namely, what, exactly, are you classifying? Your data dimensionality suggests that you seek to classify individual timesteps - in that case, feed data one timestep at a time: (batch_size,features). Alternatively, feed timesteps in the batch axis, one batch at a time, so 1000 timesteps will correspond to (1000,features) - but do not do this if model has any stateful layers, which treats each batch axis entry as an independent sequence.

To classify sequences with timesteps>1, again, ensure layer data flow ultimately yields a 2d output.