Tensorflow/keras error: ValueError: Error when checking input: expected lstm_input to have 3 dimensions, but got array with shape (4012, 42)

Question

I have a pandas dataframe that is being made by a train_test_split called x_train with 4012 rows all the values in the dataframe are either int's or floats (after train/test split) and 42 columns (after train/test split). I am trying to train a recursive neural network with LSTM cells, but my program keeps giving me an error when the model tries to input the data frame. I've tried reshaping the dataframe, that doesn't work too, but maybe I'm doing that wrong. Please let me know if you know how to fix it and/or know how I can make my model more effective.

Here is my code:

df = pd.read_csv("data.csv")

x = df.loc[:, df.columns != 'result']
y = df.loc[:, df.columns == 'result']

y = y.astype(int)

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, shuffle = False)

model = Sequential()

model.add(LSTM(128, input_shape = (4012, 42), activation = 'relu', return_sequences = True)) #This is the line where the error happens.
model.add(Dropout(0.2))

model.add(LSTM(128, activation = 'relu'))
model.add(Dropout(0.2))

model.add(Dense(32, activation = 'relu'))
model.add(Dropout(0.2))

model.add(Dense(2, activation = 'softmax'))

opt = tf.keras.optimizers.Adam(lr = 10e-3, decay=1e-5)

model.compile(loss = 'sparse_categorical_crossentropy', optimizer = opt, metrics = ['accuracy'])

model.fit(x_train, y_train, epochs = 5, validation_data = (x_test, y_test))

The error that I'm getting:

ValueError: Error when checking input: expected lstm_input to have 3 dimensions, but got array with shape (4012, 42)

Someone please help me.

Answer 1

The problem is that you are trying to feed 2 dimensional array for the model, but it is expecting 3 dimensional array. Instead of reshaping Dataframe convert it to array and then reshape according to the modified code below.

df = pd.read_csv("data.csv")

x = df.loc[:, df.columns != 'result']
y = df.loc[:, df.columns == 'result']

y = y.astype(int)

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, shuffle = False)
x_train = x_train.values.reshape((1, x_train.shape[0], x_train.shape[1]))
x_test = x_test.values.reshape((1, x_test.shape[0], x_test.shape[1]))



y_train = y_train.values.reshape((1, y_train.shape[0], y_train.shape[1]))
y_test = y_test.values.reshape((1, y_test.shape[0], y_test.shape[1]))