InvalidArgumentError with RNN/LSTM in Keras

Question

I'm throwing myself into machine learning, and wish to use Keras for a university project that's time-critical. I realise it would be best to learn individual concepts and building blocks, but it's important that this is done soon.

I'm working with someone who has some experience and interest in machine learning, but we cannot seem to get further than this. The below code was adapted from GitHub code mentioned in a guide in Machine Learning Mastery.

For context, I've got data from multiple physical sensors (where each sensor is a column), with each sample from those sensors represented by one row. I wish to use machine learning to determine who the sensors were tracking at any given time. I'm trying to allocate approximately 80% of the rows to training and 20% to testing, and am creating my own "y" set of data (with the first 521,549 rows being from one participant, and the remainder from another). My data (training and test) has a total of 1,019,802 rows, and 16 columns (all populated), but the number of columns can be reduced if need be.

I would love to know the following:

1. What does this error mean in the context of what I'm trying to achieve, and how can I change my code to avoid it?
2. Is the below code suitable for what I'm trying to achieve?
3. Does this code represent any specific fundamental flaw in my understanding of what machine learning (generally or specifically) is designed to achieve?

Below is the Python code I'm trying to run to make use of machine learning:

x_all = pd.read_csv("(redacted)...csv",
                delim_whitespace=True, header=None, low_memory=False).values
y_all = np.append(np.full((521549,1), 0), np.full((498253,1),1))
limit = 815842
x_train = x_all[:limit]
y_train = y_all[:limit]
x_test = x_all[limit:]
y_test = y_all[limit:]

max_features = 16
maxlen = 80
batch_size = 32

model = Sequential()
model.add(Embedding(500, 32, input_length=max_features))
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=15,
          validation_data=(x_test, y_test))
score, acc = model.evaluate(x_test, y_test,
                            batch_size=batch_size)

This is an excerpt from the CSV referenced in the code:

6698.486328125  4.28260869565217    4.6304347826087 10.6195652173913    2.4392579293836 2.56134051466188    9.05326152004788    0.0 1.0812  924.898261191267    -1.55725190839695   -0.244274809160305  0.320610687022901   -0.122938530734633  0.490254872563718   0.382308845577211   
6706.298828125  4.28260869565217    4.58695652173913    10.5978260869565    2.4655894673848 2.50867743865949    9.04368641532017    0.0 1.0812  924.898261191267    -1.64885496183206   -0.366412213740458  0.381679389312977   -0.122938530734633  0.490254872563718   0.382308845577211   
6714.111328125  4.26086956521739    4.64130434782609    10.5978260869565    2.45601436265709    2.57809694793537    9.03411131059246    0.0 1.0812  924.898261191267    -0.931297709923664  -0.320610687022901  0.320610687022901   -0.125937031484258  0.493253373313343   0.371814092953523   

The following error occurs when running this:

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0,0] = 972190 is not in [0, 500)
         [[Node: embedding_1/embedding_lookup = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@training/Adam/Assign_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](embedding_1/embeddings/read, embedding_1/Cast, training/Adam/gradients/embedding_1/embedding_lookup_grad/concat/axis)]]

For reference, I'm on a 2017 27-inch iMac Retina 5K with 4.2 GHz i7, 32 GB RAM, with a Radeon Pro 580 8 GB.

Answer 1

There are some more tutorials on Machine Learning Mastery for what you want to accomplish https://machinelearningmastery.com/convert-time-series-supervised-learning-problem-python/ https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

And I'll give my own quick explanation of what you probably want to do.

Right now it looks like you are using the exact same data for the X and y inputs into your model. The y inputs are the labels which in your case is "who the sensors were tracking". So in the binary case of having 2 possible people it is set to 0 for the first person and 1 for the second person.

The sigmoid activation on the final layer will output a number between 0 and 1. If the number is bellow 0.5 then it is predicting that the sensor is tracking person 0 and if it above 0.5 then it is predicting person 1. This will be represented in the accuracy score.

You will probably not want to use an embedding layer, its possible that you might but I would drop it to start with. Normalize your data though before feeding it into the net to improve training. Scikit-Learn has good tools for this if you want a quick solution. http://scikit-learn.org/stable/modules/preprocessing.html

When working with time series data you often want to feed in a window of time points rather than a single point. If you send your time series to Keras model.fit() then it will use a single point as input.

In order to have a time window as input you need to reorganize each example in the data set to be a whole window, or you can use a generator if that will take up to much memory. This is described in the Machine Learning Mastery pages that I linked. Keras has a generator that you can use called TimeseriesGenerator

from keras.preprocessing.sequence import TimeseriesGenerator
timeseries_generator = TimeseriesGenerator(data, targets, length, sampling_rate)

where data is your time series of features and targets is your time series of labels. If you use the timeseries generator then when fitting you will have to use fit_generator

model.fit_generator(timeseries_generator)

same with evaluating using evaluate_generator()

If you have your data set up correctly then your model should work

model = Sequential()
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))

you could also try a simpler dense model

model = Sequential()
model.add(Flatten())
model.add(Dense(64, dropout=0.2, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

One more issue I see is that it appears you would be splitting off a test set that contains only one type of label which is not only bad practice but will also weight your training set towards the other label which might hurt your results.

Hopefully that gets you started. Make sure you get your data set up correctly!