CODEWITHSUNDEEP
ruby-on-railsherokusidekiq
gemart
gemart

Reputation: 376

Rails + Sidekiq Csv Import Error

I just switched my CSV upload process to run on a worker. It works fine locally, but when I try to upload a file in production I'm getting this error. It seems to me that it just doesn't know where to grab the file from

017-02-22T16:32:48.914560+00:00 app[worker.1]: 4 TID-os5wk7tgo InventoryUploadWorker JID-f5be1032c019c28684582427 INFO: start
2017-02-22T16:32:49.224819+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.53973862.c2c36482-5d99-4a68-a399-0918d1ed36d2 sample#load_avg_1m=0.29 sample#load_avg_5m=0.07 sample#load_avg_15m=0.02
2017-02-22T16:32:49.224900+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.53973862.c2c36482-5d99-4a68-a399-0918d1ed36d2 sample#memory_total=144.37MB sample#memory_rss=134.18MB sample#memory_cache=6.66MB sample#memory_swap=3.54MB sample#memory_pgpgin=55377pages sample#memory_pgpgout=19323pages sample#memory_quota=512.00MB
2017-02-22T16:32:49.167416+00:00 app[worker.1]:   Company Load (0.6ms)  SELECT  "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT 1  [["id", 32]]
2017-02-22T16:32:49.246868+00:00 app[worker.1]: 4 TID-os5wk7tgo InventoryUploadWorker JID-f5be1032c019c28684582427 INFO: fail: 0.332 sec
2017-02-22T16:32:49.247408+00:00 app[worker.1]: 4 TID-os5wk7tgo WARN: {"class":"InventoryUploadWorker","args":["/tmp/RackMultipart20170222-4-1jaehp1.csv","32"],"retry":false,"queue":"default","jid":"f5be1032c019c28684582427","created_at":1487781168.915459,"enqueued_at":1487781168.9161458}
2017-02-22T16:32:49.247452+00:00 app[worker.1]: 4 TID-os5wk7tgo WARN: Errno::ENOENT: No such file or directory @ rb_sysopen - /tmp/RackMultipart20170222-4-1jaehp1.csv

Worker:

class InventoryUploadWorker
  include Sidekiq::Worker
  sidekiq_options retry: false

  Sidekiq.configure_server do |config|
    config.redis = { url: ENV["REDISTOGO_URL"], network_timeout: 5 }
  end

  Sidekiq.configure_client do |config|
    config.redis = { url: ENV["REDISTOGO_URL"], network_timeout: 5 }
  end

  def perform(file_path, company_id)
    CsvImport.csv_import(file_path, Company.find(company_id))
  end
end

Import Method:

class CsvImport

    def self.csv_import(filename, company)
        time = Benchmark.measure do
            File.open(filename) do |file|
                headers = file.first
                file.lazy.each_slice(150) do |lines|
                    Part.transaction do 
                        inventory = []
                        insert_to_parts_db = []
                        rows = CSV.parse(lines.join, write_headers: true, headers: headers)
                        rows.map do |row|
                            part_match = Part.find_by(part_num: row['part_num'])
                            new_part = build_new_part(row['part_num'], row['description']) unless part_match
                            quantity = row['quantity'].to_i
                            row.delete('quantity')
                            row["condition"] = match_condition(row)
                            quantity.times do 
                                part = InventoryPart.new(
                                    part_num: row["part_num"], 
                                    description: row["description"], 
                                    condition: row["condition"],
                                    serial_num: row["serial_num"],
                                    company_id: company.id,
                                    part_id: part_match ? part_match.id : new_part.id
                                    )           
                                inventory << part                   
                            end
                        end
                        #activerecord-import (bulk import)
                        InventoryPart.import inventory
                    end
                end
            end         
        end
        puts time
    end

Upvotes: 0

Views: 572

Answers (1)

Mike Perham
Mike Perham

Reputation: 22208

It's not a good idea for the sidekiq process to rely on a temporary file from the web process. What happens if the job fails and retries for the next week? What happens if your web and worker processes are on different machines or in different containers?

You should push the CSV contents as an argument or move the file to a well-known spot for the worker to pick up.

Upvotes: 4

Related Questions