CODEWITHSUNDEEP
tensorflowdeep-learningtime-serieslstmforecasting
Eghbal
Eghbal

Reputation: 3783

Does deeper LSTM need more units?

I'm applying LSTM on time series forecasting with 20 lags. Suppose that we have two cases. The first one just using five lags and the second one (like my case) is using 20 lags. Is it correct that for the second case we need more units compared to the former one? If yes, how can we support this idea? I have 2000 samples for training the model, so this is the main limitation for increasing number of units here.

Upvotes: 0

Views: 446

Answers (1)

thushv89
thushv89

Reputation: 11333

It is very difficult to give an exact answer as the relationship between timesteps and number of hidden units is not an exact science. For example, following factors can affect the number of units required.

  • Short term memory problem vs long-term memory problem

    • If your problem can be solved with relatively less memory (i.e. requires to remember only a few time steps) you wouldn't get much benefit from adding more neurons while increasing the number of steps.

  • The amount of data

    • If you don't have enough data for the model to learn from (which I feel like you will run into with 2000 data points - but I could be wrong), then increasing the number of timesteps won't help you much.

  • The type of model you use

    • Depending on the type of model you use (e.g. LSTM / GRU ) you might get different results (this is not always true but can happen for certain problems)

I'm sure there are other factors out there, but these are few that came to my mind.

Proving more units give better results while having more time steps (if true)

That should be relatively easy as you can try few different options,

  • 5 lags with 10 / 20 / 50 hidden units
  • 20 lags with 10 / 20 / 50 hidden units

And if you get better performance (e.g. lower MSE) with 20 lags problem than 5 lags problem (when you use 50 units), then you have gotten your point across. And you can reinforce your claims by showing results with different types of models (e.g. LSTMs vs GRUs).

Upvotes: 3

Related Questions