recurrent neural network ValueError: Found array with dim 3. Estimator expected <= 2

Question

I am running an LSTM, GRU and bilstm model using the following code

# Create BiLSTM model
def create_model_bilstm(units):
    model = Sequential()
    model.add(Bidirectional(LSTM(units = units,                             
              return_sequences=True),
              input_shape=(X_train.shape[1], X_train.shape[2])))
    #model.add(Bidirectional(LSTM(units = units)))
    model.add(Dense(1))
    #Compile model
    model.compile(loss='mse', optimizer='adam')
    return model

# Create LSTM or GRU model
def create_model(units, m):
    model = Sequential()
    model.add(m (units = units, return_sequences = True,
                input_shape = [X_train.shape[1], X_train.shape[2]]))
    model.add(Dropout(0.1))
    #model.add(m (units = units))
    #model.add(Dropout(0.2))
    model.add(Dense(units = 1))
    #Compile model
    model.compile(loss='mse', optimizer='adam')
    return model

# BiLSTM
model_bilstm = create_model_bilstm(20)

# GRU and LSTM
model_gru = create_model(50, GRU)
model_lstm = create_model(20, LSTM)

# Fit BiLSTM, LSTM and GRU
def fit_model(model):
    early_stop = EarlyStopping(monitor = 'val_loss',
                                               patience = 100)
    
    history = model.fit(X_train, y_train, epochs = 700,  
                        validation_split = 0.2, batch_size = 32, 
                        shuffle = False, callbacks = [early_stop])
    return history

history_bilstm = fit_model(model_bilstm)
history_lstm = fit_model(model_lstm)
history_gru = fit_model(model_gru)

This all runs smoothly and prints out my loss graphs. but when it comes to predictions i run the following code

# Make prediction
def prediction(model):
    prediction = model.predict(X_test)
    prediction = scaler_y.inverse_transform(prediction)
    return prediction

prediction_bilstm = prediction(model_bilstm)
prediction_lstm = prediction(model_lstm)
prediction_gru = prediction(model_gru)

and i get the following error

   ValueError Traceback (most recent call last)
<ipython-input-387-9d45f01ae2a2> in <module>
      5     return prediction
      6 
----> 7 prediction_bilstm = prediction(model_bilstm)
      8 prediction_lstm = prediction(model_lstm)
      9 prediction_gru = prediction(model_gru)

<ipython-input-387-9d45f01ae2a2> in prediction(model)
      2 def prediction(model):
      3     prediction = model.predict(X_test)
----> 4     prediction = scaler_y.inverse_transform(prediction)
      5     return prediction

                         ...

    ValueError: Found array with dim 3. Estimator expected <= 2.

I am assuming this has something to do with my X_test shape based on other posts i have read so i tried to reshape it to 2d but got another error telling me "expected bidirectional_3_input to have 3 dimensions, but got array with shape (62, 36)" on line 7 again.

What am i doing wrong and how can i fix it?

Data Explanation: So I am trying to predict discharge rates (target variable) using groundwater levels (34 features), precipitation and temperature as input which gives me a total of 36 features. My data is in monthly resolution. I am using 63 observation for my test (5 year pred) and the rest for my train.

Answer 1

What are you doing wrong? Let's assume your input data has shape X_train.shape = [d0,d1,d2], then after setting up your BiLSTM-model like

from tensorflow.keras import Sequential
from tensorflow.keras.layers import Bidirectional,LSTM,Dense

model = tf.keras.Sequential()
model.add(
   tf.keras.layers.Bidirectional(
      tf.keras.layers.LSTM(
         units = 10,                             
         return_sequences=True),
         input_shape=(d1, d2)
         )
      )
model.add(Dense(1))
model.compile(loss='mse', optimizer='adam')

we can check the input- and output-shapes your model expects by

>>model.input.shape
TensorShape([None, d1, d2])
>>model.output.shape
TensorShape([None, d1, 1])

So your model expects input of shape (n_batch,d1,d2), where n_batch is the batch size of the data, and returns a shape (n_batch,d1,1), thus a 3d-tensor.

Now if you provide a 3d-tensor to your model, the model.prediction-method will succesfully return a 3d-tensor, however sklearn.preprocessing.StandardScaler.inverse_transform only works for 2d-data, thats why it says

ValueError: Found array with dim 3. Estimator expected <= 2.

On the other hand, if you first reshape your data to be 2d, then model.prediction complains, because it is set up to expect a 3d-tensor.

How can you fix it? For further help on how to fix your code, you will need to provide us with more detailled information on what you expect your model to do, especially what output-shape you want your BiLSTM-model to have. I assume you actually want your BiLSTM-model to return a scalar for each sample, so an additional Flatten-layer might do the trick:

from tensorflow.keras import Sequential
from tensorflow.keras.layers import Bidirectional,LSTM,Dense,Flatten

model = tf.keras.Sequential()
model.add(
   tf.keras.layers.Bidirectional(
      tf.keras.layers.LSTM(
         units = 10,                             
         return_sequences=True),
         input_shape=(d1, d2)
         )
      )
model.add(Flatten()) #<-- additional flatten-layer
model.add(Dense(1))
model.compile(loss='mse', optimizer='adam')