CODEWITHSUNDEEP
ruby-on-railsrubycsvherokusql-update
user984621
user984621

Reputation: 48453

Heroku - how to update price list which has size 38MB?

My client has an e-commerce system with products (there is ~60k product items). The price list is placed on the URL address in ZIP format.

I need firstly download the price list, unzip that, then open the CSV file and start read line by line. In his database is updated price from the CSV (if is product's price different in the CSV than in his database, use the price from CSV and update database).

But, is possible to do this on Heroku? The download of almost 40MB file take some time, import data from CSV to database (the import on localhost takes about 17 minutes - but this is import of whole CSV to database) as well.

How would you solve this situation? What options do I have? Unfortunately, there is no way to change the CSV file, so I have to work with that file...

Thanks

Upvotes: 0

Views: 157

Answers (2)

hgmnz
hgmnz

Reputation: 13306

There's a common misconception about the file system on heroku. It's not that it's read-only, it's that it's ephemeral. You can start a process and start writing to the filesystem, but as soon as that process exits, the data is gone forever.

So, you can indeed download a zip file and then process it, but even if you run into an error while processing, causing your program to exit, the unzipped data will be lost. You might want to think about unzipping it, creating chunks of data that you put on S3, and then having any number of workers crunching that data and internalizing it with your own database.

Upvotes: 2

Kevin Sylvestre
Kevin Sylvestre

Reputation: 38012

This is possible by running the work on a delayed job on a worker process. For more information see: https://devcenter.heroku.com/articles/ps. Note that if this is a regular task you want to run on a scheduled time interval you can also use Scheduler (https://devcenter.heroku.com/articles/scheduler).

Upvotes: 1

Related Questions