CODEWITHSUNDEEP
pythonmachine-learningkerasscikit-learntraining-data
hakuna_code
hakuna_code

Reputation: 793

Is there an option like generator in keras with scikit to process large records of data?

I have a training dataset of shape(90000,50) and I trying to fit this in model(Gaussian process regression). This errors out with memory error. I do understand the computation, but is there way to pass data in batches using scikit? I am using the scikit implementation of the GPR algorithm.

Upvotes: 1

Views: 567

Answers (2)

hakuna_code
hakuna_code

Reputation: 793

The Gaussian process implementation(Regression/classification) from scikit is'nt capable of handling big dataset. It can run only upto 15000 rows of data. So I decided to use a different algorithm instead as this seems to be a problem with algorithm.

Upvotes: 0

Sıddık Açıl
Sıddık Açıl

Reputation: 967

Keras has generator because, you can create checkpoints and resume from where you left off in Neural Networks. However, not all of trainable algorithms has this property. Take a look at incremental learning from Scikit-API docs.

Upvotes: 1

Related Questions