CODEWITHSUNDEEP
pythonu8darts
AyushBhardwaj261
AyushBhardwaj261

Reputation: 41

Difficulty in Implementing Model Re-Training for Time-Series Data with Pre-Trained Model

I'm encountering an issue with implementing incremental learning for time-series data using a pre-trained model, and I'm seeking guidance on how to address it. Library I'm using here is darts from unit8co LINK

Description:
I have a substantial dataset consisting of 2.5 million data points, structured as follows:

                                ds    y
0        2020-10-21 12:57:47+00:00  0.0
1        2020-10-21 12:57:48+00:00  0.0
2        2020-10-21 12:57:49+00:00  0.0
3        2020-10-21 12:57:50+00:00  0.0
4        2020-10-21 12:57:51+00:00  0.0
...                            ...  ...
2591996  2020-11-20 12:57:43+00:00  7.0
2591997  2020-11-20 12:57:44+00:00  7.0
2591998  2020-11-20 12:57:45+00:00  7.0
2591999  2020-11-20 12:57:46+00:00  6.0
2592000  2020-11-20 12:57:47+00:00  6.0

[2592001 rows x 2 columns]

Objective:
My aim is to train an initial model, let's call it Model_v1, on this historical data. However, due to the large size of the dataset, training the model from scratch is time-consuming. Therefore, I want to explore the possibility of using Model_v1 as a base model and incrementally updating it with new data obtained from a real-time source, which arrives approximately every minute. Ideally, I would like to re-train the model on a weekly or monthly basis to adapt to any changes in the underlying patterns.

Challenge:
Although I've managed to save the historical model (Model_v1), I'm encountering difficulties when attempting to load it and fit new data onto it. The incremental learning process seems to be failing, and I'm unsure about the correct approach to achieve this.

Request for Assistance:
I would greatly appreciate any insights, suggestions, or best practices on how to effectively implement incremental learning with pre-trained models for time-series data. Additionally, if anyone has encountered similar challenges or has successfully implemented such a solution, your input would be invaluable.

Thank you in advance for your assistance and guidance.

Upvotes: 0

Views: 54

Answers (0)

Related Questions