CODEWITHSUNDEEP
neural-networklstm
user
user

Reputation: 199

How to handle extremely long LSTM sequence length?

I have some data that is sampled at at a very high rate (on the order of hundreds of times per second). This results in a sequence length that is huge (~90,000 samples) on average for any given instance. This entire sequence has a single label. I am trying to use an LSTM neural network to classify new sequences as one of these labels (multiclass classification).

However, using an LSTM with a such a large sequence length results in a network that is quite large.

What are some methods to effectively 'chunk' these sequences so that I could reduce the sequence length of the neural network, yet still maintain the information captured in the entire instance?

Upvotes: 14

Views: 12713

Answers (3)

user
user

Reputation: 199

Three years later, we have what seems to be the start of solutions for this type of problem: sparse transformers.

See

https://arxiv.org/abs/1904.10509

https://openai.com/blog/sparse-transformer/

Upvotes: 3

xgaox
xgaox

Reputation: 71

This post is from some time ago, but I thought I would chime in here. For this specific problem that you are working on (one-dimensional continuous-valued signal with locality, composition-ality, and stationarity), I would highly recommend a CNN convolutional neural network approach, as opposed to using an LSTM.

Upvotes: 7

Shamane Siriwardhana
Shamane Siriwardhana

Reputation: 4201

When you have very long sequences RNNs can face the problem of vanishing gradients and exploding gradients.

There are methods. The first thing you need to understand is why we need to try above methods? It's because back propagation through time can get real hard due to above mentioned problems.

Yes introduction of LSTM has reduced this by very large margin but still when it's is so long you can face such problems.

So one way is clipping the gradients. That means you set an upper bound to gradients. Refer to this stackoverflow question

Then this problem you asked

What are some methods to effectively 'chunk' these sequences?

One way is truncated back propagation through time. There are number of ways to implement this truncated BPTT. Simple idea is

  1. Calculate the gradients only for number of given time steps That means if your sequence is 200 time steps and you only give 10 time steps it will only calculate gradient for 10 time step and then pass the stored memory value in that 10 time step to next sequence(as the initial cell state) . This method is what tensorflow using to calculate truncated BPTT.

2.Take the full sequence and only back propagate gradients for some given time steps from selected time block. It's a continuous way

Here is the best article I found which explains these trunacated BPTT methods. Very easy. Refer to this Styles of Truncated Backpropagation

Upvotes: 12

Related Questions