CODEWITHSUNDEEP
machine-learningkerasdeep-learninglstmautoencoder
EvaHHHH
EvaHHHH

Reputation: 353

LSTM Accuracy unchanged while loss decrease

We put a sensor to detect anomalies in accelerometer.

There is only one sensor so my data is 1-D array.

I tried to use LSTM autoencoder for anomaly detection.

But my model didn't work as the losses of the training and validation sets were decreasing but accuracy unchanged.

Here is my Code and training log:

dim = 1
timesteps = 32
data.shape = (-1,timesteps,dim)

model = Sequential()
model.add(LSTM(50,input_shape=(timesteps,dim),return_sequences=True))
model.add(Dense(dim))

lr = 0.00001
Nadam = optimizers.Nadam(lr=lr)

model.compile(loss='mae', optimizer=Nadam ,metrics=['accuracy'])

EStop = EarlyStopping(monitor='val_loss', min_delta=0.001,patience=150, verbose=2, mode='auto',restore_best_weights=True)

history = model.fit(data,data,validation_data=(data,data),epochs=2000,batch_size=64,verbose=2,shuffle=False,callbacks=[EStop]).history

Trainging Log

Train on 4320 samples, validate on 4320 samples
Epoch 1/2000
 - 3s - loss: 0.3855 - acc: 7.2338e-06 - val_loss: 0.3760 - val_acc: 7.2338e-06
Epoch 2/2000
 - 2s - loss: 0.3666 - acc: 7.2338e-06 - val_loss: 0.3567 - val_acc: 7.2338e-06
Epoch 3/2000
 - 2s - loss: 0.3470 - acc: 7.2338e-06 - val_loss: 0.3367 - val_acc: 7.2338e-06
...
Epoch 746/2000
 - 2s - loss: 0.0021 - acc: 1.4468e-05 - val_loss: 0.0021 - val_acc: 1.4468e-05
Epoch 747/2000
 - 2s - loss: 0.0021 - acc: 1.4468e-05 - val_loss: 0.0021 - val_acc: 1.4468e-05
Epoch 748/2000
 - 2s - loss: 0.0021 - acc: 1.4468e-05 - val_loss: 0.0021 - val_acc: 1.4468e-05
Restoring model weights from the end of the best epoch
Epoch 00748: early stopping

Upvotes: 0

Views: 1049

Answers (3)

Nicolae Petridean
Nicolae Petridean

Reputation: 604

Early stopping is not the best technique for regularization while you are facing this problem. At least, while you are still struggling to fix it I would rather take it out or at replace it with other regularization method. to figure out what happens.

Also another suggestion. Can you change a bit the validation set and see what is the behavior ? How did you build the validation set ?

Did you normalize / standardize the data ? Please note normalization is even more important for LSTMs

the metric is definitely a problem. The above suggestions are good.

Upvotes: 0

Fabian
Fabian

Reputation: 802

As @MatiasValdenegro said you shouldn't use accuracy when you want to do regression. You can see that your model might be fine because your loss is decreasing over the epochs and is very low when early stopping. In Regression Problems normaly these Metrics are used:

  • Mean Squared Error: mean_squared_error, MSE or mse

  • Mean Absolute Error: mean_absolute_error, MAE, mae

  • Mean Absolute Percentage Error: mean_absolute_percentage_error, MAPE, mape

  • Cosine Proximity: cosine_proximity, cosine

Resource

To geht the right metrics you should change this (e.g. for "Mean Squared Error"):

model.compile(loss='mae', optimizer=Nadam ,metrics=['mse'])

As already said your model seems to be fine, you are just looking at the wrong metrics.

Hope this helps and feel free to ask.

Upvotes: 2

Kazesui
Kazesui

Reputation: 159

A couple of things

  • As Matias in the comment field pointed out, you're doing a regression, not a classification. Accuracy will not give expected values for regression. That said, you can see that the accuracy did improve (from 0.0000072 to 0.0000145). Check the direct output from your model to check how well it approximates to original time series.
  • You can safely omit the validation data when your validation data is the same as the training data
  • With autoencoders, you generally want to compress the data in some way as to be able to represent the same data in a lower dimension which is easier to analyze (for anomalies or otherwise. In your case, you are expanding the dimensionality instead of reducing it, meaning the optimal strategy for your autoencoder would be to pass through the same values it gets in (value of your timeseries is sent to 50 LSTM units, which send their result to 1 Dense unit). You might be able to combat this if you set return_sequence to False (i.e. only the result from the last timestep is returned), preferably into more than one unit, and you then try to rebuild the timeseries from this instead. It might fail, but is still likely to lead to a better model

Upvotes: 2

Related Questions