CODEWITHSUNDEEP
sequencekeraspredictionlstm
Hima
Hima

Reputation: 12054

Accuracy/Loss doesn't change

It might be duplication of the previous posts but here is my code. My inputs X are sequences of characters each of length 10 encoded as 1-26 numbers with added random noise.output is the next word in the sequence.

from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.layers.recurrent import LSTM
import keras.optimizers

in_out_neurons = 1
hidden_neurons = 20

model = Sequential()

# n_prev = 100, 2 values per x axis
model.add(LSTM(hidden_neurons, input_shape=(10, 1)))
model.add(Activation('relu'))
model.add(Dense(in_out_neurons))
model.add(Activation("sigmoid"))
model.add(Activation("softmax"))
rms = keras.optimizers.RMSprop(lr=5, rho=0.9, epsilon=1e-08, decay=0.0)
sgd = keras.optimizers.SGD(lr=0.01, momentum=0.0, decay=0.001, nesterov=False)
model.compile(loss="binary_crossentropy",
    optimizer='adam',
    metrics=['accuracy'])

(X_train, y_train), (X_test, y_test) = train_test_split(data)

model.fit(X_train, y_train, batch_size=100, nb_epoch=50, validation_data=(X_test, y_test), verbose=1)
score = model.evaluate(X_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

predicted = model.predict(X_test, batch_size=700)

# and maybe plot it
pd.DataFrame(predicted).to_csv("predicted.csv")
pd.DataFrame(y_test).to_csv("test_data.csv")

Tried changing different loss functions and optimizers. No luck.

Upvotes: 1

Views: 583

Answers (1)

Nassim Ben
Nassim Ben

Reputation: 11553

Encoding characters by number is not a good way. It will be interpreted as numbers so it's like saying that Y and Z are close together which doesn't make sense. This is why the Embedding() layers exist. Or you might consider one-hot encoding . Characters are then one-hot vectors of length 26. "a" would become [1 0 0 0 0 0 0 0 0 ... 0] for example.

That being said, the reason it's not working is because you put a Softmax on a layer which has only one value... Softmax on one value will always give output 1, so your network can't learn since output is 1 whatever happens before.

Softmax is used to make a probability density out of a tensor, if there is only one possible value, it will get probability 1. If you want that one neuron to be a probability (between 0 and 1) use only the sigmoid, not the softmax.

I hope this helps :)

Upvotes: 1

Related Questions