CODEWITHSUNDEEP
ruby-on-railscsvherokuamazon-s3
AMBasra
AMBasra

Reputation: 969

Read Massive CSV File From S3

I have big csv files on my S3. I want to import their data without download(copying) them to my tmp folder on heroku. Plus i dont want to load whole file in memory. Can you guys suggest me how can i do that. Some thing like getting data chunk by chunk or anything like that.

Thanks in advance.

Upvotes: 0

Views: 1102

Answers (1)

Max Williams
Max Williams

Reputation: 32933

Ok, I thought of a way to do this, which is insanely inefficent and pretty stupid. But, if you are determined to not save the file on your server then here's a way.

You add code to your app which accepts some data (eg a line, or lots of lines worth) in some form, and then creates the records accordingly. Deploy this.

Then, on your LOCAL machine, save the file. Write a script which reads the file in (again, a line at a time is best to avoid memory issues), converts it into the format needed for action you wrote, and then send it as a GET or POST request to your production site. It will need to make lots of requests as it churns through the file.

Like I say, this is really stupid and a little insane: you really should just save the file on your server.

Upvotes: 1

Related Questions