CODEWITHSUNDEEP
pythontime-seriesforecasting
R. Marolahy
R. Marolahy

Reputation: 1596

How to handle high frequency timeseries data?

I am working with a high frequency timeseries data:

        date           values
2019-12-30 21:00:00    7134.0
2019-12-30 21:00:27    7147.0
2019-12-30 21:04:27    7135.0
2019-12-30 21:08:27    7122.0
2019-12-30 21:12:27    7121.0
2019-12-30 21:16:27    7125.0
2019-12-30 21:20:27    7135.0
2019-12-30 21:24:27    7121.0
2019-12-30 21:28:27    7125.0
2019-12-30 21:32:27    7127.0
2019-12-30 21:36:27    7130.0

Can anyone help me to show how to handle such data to do forecasting? Every method that I have seen uses all low frequency data (daily, weekly, yearly). Assume that we cannot do averaging because that may change information.

Any help or suggestion is very welcome.

PS: I am using python.

Upvotes: 0

Views: 1897

Answers (1)

Rikki
Rikki

Reputation: 3528

Be aware of spread (if you are being charged for spread, always accept tick data as time, ask and offer - or at least discount spread from your price value).

If you need to work with timeseries data, and you only have the tick input, a few things might come in handy:

  • pandas.DataFrame - which is a tabular representation of your raw data. A lot of statistical analytics libraries out there accept pandas DataFrame format.
  • Familiarize yourself with window functions within pandas, and how you can transform a tick time series into OHLC (Open-High-Low-Close) data frame [should you need to do that for solving certain problems].
  • Libraries like StreamZ - which basically allows you to construct a pipeline for your data. (So you could stream your tick data through a pipe and transform it into what you want for a specific purpose, or deliver it to a handler who does certain part of your prediction for example)
  • Ta-Lib (Technical Analytics Library) also has a decent set of technical indicators, which can be used in pandas.DataFrame with ease.
  • Be aware, depending on the volatility of the market you are working with - you may want to avoid noise by choosing larger timeframes, or dive deep as much as you can into tick data - which contains a lot of noise.

If you are working to predict price, you will need to read A LOT about Regression Analysis (Subjects like ARIMA, SARIMA are what comes out of regressions, and you need to know the basics before starting to move further)

Again, forecasting can do so much. You will really never know what the direction of the market will be for sure. It's all a probability optimization problem, so if you can/or intend to - I strongly suggest utilizing an AI brain to do your trading.

For AI, keras and TF (tensorflow) come in handy, especially in Python. Keras sits on top of TF, and provides an extensive amount of features/capabilities for you to build networks to match your needs. TF also supports Nvidia's CUDA SDK - so you can utilize your GPU's processor for optimization and training. You can construct different flavors of Neural Networks, like RNN (where your ratio could be sort-of the reward for your trading system), DeepQ learning etc.

Now, not to leave you without a practical example (and mentioning a library I forgot to tell you about above.... statsmodels) - I think this would be a good place to start: Markov switching dynamic regression models (It uses a time series of Federal Funds - so no OHLC, just date and price - and it ends up plotting the probability of different regimes)

A more advanced example I would study is Time series forecasting on TF's website. It does it with weather data, but the process is interesting - from transforming the input data to pushing it through different models using LSTM or Conv NNs might give you some ideas. (:

Upvotes: 1

Related Questions