CODEWITHSUNDEEP
pythonpandascsvhdf5large-data
physicsfreak
physicsfreak

Reputation: 49

Updating large DataFrame objects not on disk.

I've been learning the ins and outs of Pandas by way of manipulating large csv files obtained online, the files are time-series of financial data. I have so far figured out how to use HDFStore to store and manipulate them, however I was wondering if there exists an easier way to update the files, without re-downloading the entire source file?

I ask because I'm working with 12 ~300+MB files, which update every 15mins. While I don't need the update to be continuous it'd be swell to not download what I already have.

Upvotes: 4

Views: 232

Answers (1)

hurrial
hurrial

Reputation: 514

The Blaze library from Continuum should help you. You can find an introduction here.

Upvotes: 1

Related Questions