ValueError: Error when checking target: expected dense_3 to have 2 dimensions, but got array with shape (500, 10, 14)

Question

Keras: 2.1.6, python 3.6, tensorflow 1.8.0

I am trying to train a Sequence Model that has two LSTM layers and 3 dense layers. I had done some data preparation beforehand and setup my data in the format the LSTM layer requires it, i.e. (n_samples, n_timesteps, n_features). My data has 14 features and is actually a long sequence of 5000 steps thus I have broken it down to 500 samples to 10 time steps each. Once I completed that, I started with below model, but soon ran into the error of Input shape for the last layer. I tried using Sequential and Functional API both yield the same error.

import keras 
from keras import callbacks
import tensorflow as tf
from keras.models import Model
from keras.layers import Input, Dense
from keras.layers import LSTM


X_input = Input(X_train.shape[1:]);

## First LSTM Layer
X = LSTM(10, return_sequences=True, input_shape = (10,14), name = 'LSTM_1')(X_input);

## Second LSTM Layer
X = LSTM(10)(X);

## First Dense Layer
X = Dense(10, activation='relu', name = 'dense_1')(X)

## Second Dense Layer
X = Dense(5, activation='relu', name = 'dense_2')(X)

## Final Dense Layer
X = Dense(1, activation='relu', name = 'dense_3')(X)

    ##The model object
model = Model(inputs = X_input, outputs = X, name='LSTMModel')

model.compile(optimizer = "Adam" , loss = "mean_squared_error", metrics = ['mean_squared_error','cosine', 'mae']);

Model.fit(x = X_train, y = Y_train, epochs = 300, callbacks=[tensorboard], validation_data=(X_eval,Y_eval));

My data is of shape (500,10,14):

>>> X_train.shape
(500,10,14)

And my model summary looks like such:

model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 10, 14)            0
_________________________________________________________________
LSTM_1 (LSTM)                (None, 10, 10)            1000
_________________________________________________________________
LSTM_2 (LSTM)                (None, 10)                840
_________________________________________________________________
dense_1 (Dense)              (None, 10)                110
_________________________________________________________________
dense_2 (Dense)              (None, 5)                 55
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 6
=================================================================
Total params: 2,011
Trainable params: 2,011
Non-trainable params: 0
_________________________________________________________________

Although, I still get the error:

ValueError: Error when checking target: expected dense_3 to have 2 dimensions, but got array with shape (500, 10, 14)

The shape of my labels are as follows:

X_train = np.reshape(Train_data_scaled.values,(500,10,14));
Y_train = np.reshape(Train_labels_scaled.values,(500,10,1));
X_eval = np.reshape(Validation_data_scaled.values,(10,10,14));
Y_eval = np.reshape(Validation_labels_scaled.values,(10,10,1));

After adding the layer of RepeatVector I find another issue here is the stack trace of the same.

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 10, 14)            0
_________________________________________________________________
LSTM_1 (LSTM)                (None, 10)                1000
_________________________________________________________________
repeat_vector_1 (RepeatVecto (None, 10, 10)            0
_________________________________________________________________
LSTM_2 (LSTM)                (None, 10, 10)            840
_________________________________________________________________
dense_1 (Dense)              (None, 10, 10)            110
_________________________________________________________________
dense_2 (Dense)              (None, 10, 5)             55
_________________________________________________________________
dense_3 (Dense)              (None, 10, 1)             6
=================================================================
Total params: 2,011
Trainable params: 2,011
Non-trainable params: 0
_________________________________________________________________
Traceback (most recent call last):
  File ".\lstm.py", line 76, in <module>
    tf.app.run()
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
    _sys.exit(main(argv))
  File ".\lstm.py", line 67, in main
    Hist =  Model.fit(x = X_train, y = Y_train, epochs = 300,batch_size=10, callbacks=[tensorboard], validation_data=(X_eval,Y_eval));
  File "C:\Program Files\Python36\lib\site-packages\keras\engine\training.py", line 1630, in fit
    batch_size=batch_size)
  File "C:\Program Files\Python36\lib\site-packages\keras\engine\training.py", line 1480, in _standardize_user_data
    exception_prefix='target')
  File "C:\Program Files\Python36\lib\site-packages\keras\engine\training.py", line 123, in _standardize_input_data
    str(data_shape))
ValueError: Error when checking target: expected dense_3 to have shape (10, 1) but got array with shape (10, 14)

Answer 1

Since you want to predict value of a story 10 days in the future you need to set the return_sequences argument of second LSTM layer to True to make the output shape of the whole model (None, 10, 1):

## Second LSTM Layer
X = LSTM(10, return_sequences=True)(X)

Further, a more general solution to predict the values d days in the future is to use a RepeatVector layer right after the first LSTM layer. This time you need to set return_sequences argument of the first LSTM layer to False:

d = 5  # how many days in the future you want to predict?

## First LSTM Layer
X = LSTM(10, input_shape = (10,14), name = 'LSTM_1')(X_input);

X = RepeatVector(d)(X)

## Second LSTM Layer
X = LSTM(10, return_sequences=True)(X)

It's as if the first LSTM layer encodes the input data and the second LSTM layer predicts the future value(s) based on that encoded representation. Also, it is needless to say that the shape of label arrays (i.e. y_train) must also be (n_samples, d, n_feats).