Deep Learning (Neural Network) Challenge piece of data

Question

I have a difficult two datasets, I am trying to use Neural network to fit each data alone. Please find link to the datasets.It's seq2seq. Data#1 has 56 samples each with 3dims, and Data#2 has 56 samples each with 32dims. So, it's like many to many or seq2seq. https://drive.google.com/open?id=16VGoqKP1zjmlxK2u6RpVtkYgqWnlBwnl Data#1. Input : X1 Output: Y1 It's simply many to many fitting problem. each input is of length 32, and each output is of length 32

Data#2 Input : X2 Output: Y2 It's simply many to many fitting problem. each input is of length 3, and each output is of length 3.

I tried many NN (DNN,LSTM,Conv1d) to fit any of those datasets, but fitting is always bad. Below is one of the network I tried (LSTM)

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

# load
X1 = np.load('X1.npy')
Y1 = np.load('Y1.npy')
X2 = np.load('X2.npy')
Y2 = np.load('Y2.npy')

# Reshape data
X1 = X1.reshape(-1,3,1)
Y1 = Y1.reshape(-1,3,1)
X2 = X1.reshape(-1,32,1)
Y2 = Y1.reshape(-1,32,1)

# LSTM
model = tf.keras.models.Sequential()
# layre #1
model.add(tf.keras.layers.LSTM(units=50, return_sequences=True, input_shape=(X.shape[1],1)))
model.add(tf.keras.layers.Dropout(0.12))
#layer #2
model.add(tf.keras.layers.LSTM(units=50, return_sequences=True))
model.add(tf.keras.layers.Dropout(0.12))
# layer #3
model.add(tf.keras.layers.Dense(units=1))

model.compile(optimizer='adam', loss='mse')

model.fit(X1,Y1, epochs=150, batch_size=20, verbose=1)

pred = model.predict(X)

plt.plot(Y[30,:,0], 'r')
plt.plot(pred[30,:,0], 'b')

I tried normalization as well, but still fitting is not good. Can anyone please suggest, why fitting is not good, and is there any better NN architecture to use. If you could test it , will be better.

Thanks

Answer 1

First of all, your data reshaping is wrong. N.B.: I just noticed, you are mistaking X1, Y1 for X2, Y2.

print(X1.shape) -> (56,3) print(Y1.shape) -> (56,3)

Are you trying to reshape them to dimension (-1,30,1) or (-1,3,1). There's no 32 in the sequences. X1 and Y1 both have same dimension (56,3), so it's a seq2seq problem. But, you are using a 1 unit FC layer like in a regression manner which will give you error as your output has more dimensions.

Also, there are other bugs like X was not defined and still you have used it. Your code is completely broken, unfortunately.

Case #1, I'm assuming the data has 56 samples and each sample has 3 values (temporal dimension).

I have added a minimal code-base to get started.

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

# load
X1 = np.load('X1.npy')
Y1 = np.load('Y1.npy')
X2 = np.load('X2.npy')
Y2 = np.load('Y2.npy')

print(X1.shape)
print(Y1.shape)
# Reshape data
X = X1.reshape(-1,3,1)
Y = Y1.reshape(-1,3)

# LSTM
model = tf.keras.models.Sequential()
# layre #1
model.add(tf.keras.layers.LSTM(units=3, return_sequences=True, input_shape=(3,1), activation = 'relu'))
model.add(tf.keras.layers.Dropout(0.4))
#layer #2
model.add(tf.keras.layers.LSTM(units=3, return_sequences=False))
model.add(tf.keras.layers.Dropout(0.12))
# layer #3

model.summary()

Model: "sequential_6"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_9 (LSTM)                (None, 3, 3)              60        
_________________________________________________________________
dropout_8 (Dropout)          (None, 3, 3)              0         
_________________________________________________________________
lstm_10 (LSTM)               (None, 3)                 84        
_________________________________________________________________
dropout_9 (Dropout)          (None, 3)                 0         
=================================================================
Total params: 144
Trainable params: 144
Non-trainable params: 0
__________________________

model.compile(optimizer='adam', loss='mse')

model.fit(X,Y, epochs=2, batch_size=20, verbose=1)

pred = model.predict(X)

For X2, Y2:

X = X1.reshape(-1,32,1)
Y = Y1.reshape(-1,32)

# LSTM
model = tf.keras.models.Sequential()
# layre #1
model.add(tf.keras.layers.LSTM(units=32, return_sequences=True, input_shape=(3,1), activation = 'relu'))
model.add(tf.keras.layers.Dropout(0.4))
#layer #2
model.add(tf.keras.layers.LSTM(units=32, return_sequences=False))
model.add(tf.keras.layers.Dropout(0.12))

model.compile(optimizer='adam', loss='mse')

model.fit(X,Y, epochs=2, batch_size=20, verbose=1)

pred = model.predict(X)

Some ideas to improve performance: Use min-max normalization, in the last layer use sigmoid for activation. Add more dropout and recurrent_dropout. But, don't use dropout after last layer, it messes up your predictions.

Here's an updated code:

X = X1.reshape(-1,3,1)
Y = Y1.reshape(-1,3)

X = (X-np.min(X))/(np.max(X) - np.min(X))
Y = (Y-np.min(Y))/(np.max(Y) - np.min(Y))

# LSTM
model = tf.keras.models.Sequential()
# layre #1
model.add(tf.keras.layers.LSTM(units=8, return_sequences=True, input_shape=(3,1), activation = 'relu', recurrent_dropout = 0.2))
model.add(tf.keras.layers.Dropout(0.4))
model.add(tf.keras.layers.LSTM(units=8, return_sequences=True, input_shape=(3,1), activation = 'relu', recurrent_dropout = 0.2))
model.add(tf.keras.layers.Dropout(0.4))
#layer #2
model.add(tf.keras.layers.LSTM(units=3, return_sequences=False, activation = 'sigmoid'))

model.compile(optimizer='adam', loss='mse')

hist = model.fit(X,Y, epochs=50, batch_size=20, verbose=1)

import matplotlib.pyplot as plt

plt.plot(hist.history['loss'])
plt.show()

Case #2, You have only one sample with 56 temporal dimension and 3 filters. It's not a case of LSTM at all. You need to provide more details about your input data format.

Case #3, you have more data and code, which you have not added here. Kindly, add those snippets so that we can help.