CODEWITHSUNDEEP
keraslstm
Sudipta Bhawmik
Sudipta Bhawmik

Reputation: 11

Keras LSTM training accuracy stuck at low levels for NLP task

I am using an LSTM architecture to create a chatbot. I am using GloVe embedding. During my training process, my Training accuracy gets stuck at very low values (0.1969) and no progress happens. I am attaching my code below. Can you tell me what can be done to improve the training?

from keras.models import Sequential
from keras.layers import Embedding, Flatten, Dense, LSTM
from keras.optimizers import Adam


model=Sequential()
model.add(Embedding(max_words,embedding_dim,input_length=maxlen))
model.add(LSTM(units=100,return_sequences=True, kernel_initializer="glorot_normal", recurrent_initializer="glorot_normal", activation='sigmoid'))
model.add(LSTM(units=100,return_sequences=True, kernel_initializer="glorot_normal", recurrent_initializer="glorot_normal", activation='sigmoid'))
model.add(LSTM(units=100,return_sequences=True, kernel_initializer="glorot_normal", recurrent_initializer="glorot_normal", activation='sigmoid'))
model.add(LSTM(units=100,return_sequences=True, kernel_initializer="glorot_normal", recurrent_initializer="glorot_normal", activation='sigmoid'))
model.summary()

model.layers[0].set_weights([embedding_matrix])
model.layers[0].trainable = False

model.compile(loss='cosine_proximity', optimizer='adam', metrics=['accuracy'])

model.fit(x_train, y_train,
epochs = 500,
batch_size = 32,
validation_data=(x_val,y_val))

Epoch 498/500
60/60 [==============================] - 0s 3ms/step - loss: -0.1303 - acc: 0.1969 - val_loss: -0.1785 - val_acc: 0.2909
Epoch 499/500
60/60 [==============================] - 0s 3ms/step - loss: -0.1303 - acc: 0.1969 - val_loss: -0.1785 - val_acc: 0.2909
Epoch 500/500
60/60 [==============================] - 0s 3ms/step - loss: -0.1303 - acc: 0.1969 - val_loss: -0.1785 - val_acc: 0.2909

Further training (on the same conversation data set ) does not improve accuracy.

Upvotes: 1

Views: 818

Answers (1)

Ruman Khan
Ruman Khan

Reputation: 61

Add BatchNormalization layer after Embedding and LSTM too. This helps in regularizing the learning. I added it and it helped me. Also, look at data. There might be some issue too. 

#clear session
keras.backend.clear_session()
model = Sequential()
#embedding layer
model.add(Embedding(vocab_size, 50, input_length=pad_length))

#Batch Norm layer
model.add(BatchNormalization())

#First LSTM layer
model.add(LSTM(units=100, return_sequences=True))

#Batch Norm layer
model.add(BatchNormalization())
#add dropout too
model.add(Dropout(0.25))

#Second LSTM layer
model.add(LSTM(units=100))

#Batch Norm
model.add(BatchNormalization())

model.add(Dense(1, activation="sigmoid"))

Upvotes: 1

Related Questions