CODEWITHSUNDEEP
postgresqlcsvherokuruby-on-rails-5sidekiq
crmmmc
crmmmc

Reputation: 13

CSV Importing With Rails, Postgres, and Sidekiq

I'm building a customer management system using Rails that requires CSV files containing customer information to be imported into/diffed with a Postgres database. I'm hosting the application on Heroku. I moved the database to the background with Sidekiq but need advice on where to upload the file to in the first place for importing. Is hosting the file on S3 really the best solution or is there a simpler solution without using a third party storage service? The application will be used daily but up 10 employees and the larges CSV file being upload is around 100,000 rows.

Thanks.

Upvotes: 1

Views: 1008

Answers (1)

Taufiq Muhammadi
Taufiq Muhammadi

Reputation: 352

Yes, I do think S3 is the best solution

We faced same problem at Storemapper (we use Resque instead of Sidekiq, but that's not a problem). The limiting factor here is the Heroku request timeout. You only have 30s to finish your upload to Heroku, which put hard limit on how big your csv can be. This is where S3 come. Basically what we do is:

  1. User upload csv directly to S3 via javascript, bypassing our app server on Heroku.

  2. Once the upload complete, the javascript makes a request to app server that will launch background worker, telling the worker where the file is at S3

  3. The worker download the csv from s3, then process it as necessary

I found carrierwave_direct gem to be very helpful for step 1 and 2. For step 3, I use smarter_csv gem. Checkout our complete story here: https://tylertringas.com/very-large-csv-import-in-rails-on-heroku/

Upvotes: 2

Related Questions