CODEWITHSUNDEEP
pythontensorflowneural-networkkeraslstm
hfjn
hfjn

Reputation: 57

Constant output value for multi-variate multi-timeseries regression with LSTMs in Keras

I'm currently working on a regressional problem with multiple time-series of similar machines and multiple features in Keras with the Tensorflow Backend.

The target is to predict a value at each time-step, which gives the remaining lifetime of the machine.

I normalized all values and padded the in- and output in front of all time sequences to make them as long as longest one. Then I added the masking layer to mask those zeros.

The problem I have is that the Network is giving me constant values for the output variables.

I already played with the hidden neurons, the batch, the activation functions, the epochs but nothing really changes. The loss slowly reduces but the output remains the same for all timesteps. Oh, and it kind of worked for one time-sequences.

For all zero-padded rows I get some value like 4.88323085e-02 (which is good I guess?) and for all others the output looks like this:

[...
8.72270355e+01,   8.72270355e+01,   8.72270355e+01,
8.72270355e+01,   8.72270355e+01,   8.72270355e+01,
8.72270355e+01,   8.72270355e+01,   8.72270355e+01,
8.72270355e+01,   8.72270355e+01,   8.72270355e+01,
8.72270355e+01,   8.72270355e+01,   8.72270355e+01,
8.72270355e+01,   8.72270355e+01,   8.72270355e+01,
8.72270355e+01,   8.72270355e+01,   8.72270355e+01,
8.72270355e+01,   8.72270355e+01]

My data is formed like:

[n_machines, n_timesteps, n_features]

And the output looks like:

[n_machines, n_timesteps, remaining_life]

My current model looks like this:

model = Sequential()
model.add(Masking(mask_value=0., input_shape=(None, n_features)))
model.add(LSTM(1000, return_sequences = True, unit_forget_bias=True))
model.add(TimeDistributed(Dense(1)))

model.compile(loss="mse", optimizer="adam", metrics=[RMSE])

model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=100, batch_size=1, shuffle=True)

Using something instead of Neural Nets is sadly not an option so I need to make this work. Would be happy if anyone could help me with this.

Upvotes: 2

Views: 1252

Answers (1)

hfjn
hfjn

Reputation: 57

After many days testing and playing around I figured out a working configuration. Apparently what I needed was a stateful LSTM. Here's my working solution:

model = Sequential()
model.add(Masking(mask_value=0., batch_input_shape=(1, 362, 105)))
model.add(
    LSTM(100, return_sequences=True, unit_forget_bias=True, stateful=True))
model.add(TimeDistributed(Dense(1, activation="linear")))

model.compile(loss="mse", optimizer="adam", metrics=[RMSE])

# Train the model
for epoch in range(200):
    print('Epoch %s' % epoch)
    train_loss=[]

    # Train
    for i in range(x_train.shape[0]):
        tr_loss, tr_RMSE = model.train_on_batch(x_train[i, :, :].reshape((1, x_train.shape[1], x_train.shape[2])), y_train[i, :, :].reshape(((1, y_train.shape[1], y_train.shape[2]))))
        train_loss.append(tr_loss)
        model.reset_states()
print('loss training = {}'.format(np.mean(train_loss)))

Upvotes: 1

Related Questions