Rails file uploading and Heroku

Question

I have a problem with file uploading on Heroku server. (Also a hint about the right way of doing such type of things with rails would be greatly appreciated - I'm very new in RoR).

All this code is about uploading some CSV file, then allowing user to tweak couple of settings, and parse file after all. This usually work on localhost (several times I get troubles with value stored in session), but on Heroku it is always die on upload.

In one of the neighborhood questions was written that Heroku store file only during singe instance run, but I still couldn't find anything about this in Heroku's docs. Should I store file data in the db right after upload, so in such case it always be available? The downside - files could be pretty big, about 10-20Mb, it isn't looks nice.

Heroku logs say:

 2012-05-21T19:27:20+00:00 app[web.1]: Started POST "/products/upload" for 46.119
.175.140 at 2012-05-21 19:27:20 +0000
2012-05-21T19:27:20+00:00 app[web.1]:   Processing by ProductsController#upload
as HTML
2012-05-21T19:27:20+00:00 app[web.1]:   Parameters: {"utf8"=>"тЬУ", "authenticit
y_token"=>"aqJFg3aqENfxS2lKCE4o4txxkZTJgPx36SZ7r3nyZBw=", "upload"=>{"my_file"=>
#<ActionDispatch::Http::UploadedFile:0x000000053af020 @original_filename="marina
-AutoPalmaPriceList_2011-07-30.txt", @content_type="text/plain", @headers="Conte
nt-Disposition: form-data; name=\"upload[my_file]\"; filename=\"marina-AutoPalma
PriceList_2011-07-30.txt\"\r\nContent-Type: text/plain\r\n", @tempfile=#<File:/t
mp/RackMultipart20120521-1-10g8xmx>>}, "commit"=>"Upload"}
2012-05-21T19:27:20+00:00 app[web.1]:
2012-05-21T19:27:20+00:00 app[web.1]: LoadError (no such file to load -- CSV):
2012-05-21T19:27:20+00:00 app[web.1]:   app/controllers/products_controller.rb:8
2:in `upload'
2012-05-21T19:27:20+00:00 app[web.1]:
2012-05-21T19:27:20+00:00 app[web.1]:
2012-05-21T19:27:20+00:00 app[web.1]: cache: [POST /products/upload] invalidate,
 pass

The code itself:

ProductsController:

def import
  respond_to do |format|
    format.html
  end
end

def import_adjust
  case params[:commit]
    when "Adjust"
      @col_default = params[:col_data]
      #abort @col_default.to_yaml
      #update csv reader with form data, restore filters  from params
    when "Complete"
      #all ok, read the whole file
      #abort params.to_yaml
      redirect_to import_complete
    else
      @col_default = nil
  end
  #read first part of the file
  @tmp = session[:import_file]
  @csv = []
  source = CSV.open @tmp, {col_sep: ";"}

  5.times do
    line = source.readline
    if line.size>0
      @line_size = line.size
      @csv.push line
    end
  end

  #generate a selection array
  #selection = select_tag 'col_data[]', options_for_select([['name','name'], ['brand','brand'], ['delivery_time','delivery_time'], ['price','price']])
  #@csv = [selection * line_size] + @csv
end

def import_complete
  #remove all items
  #todo check products with line items will not be destroyed.
  Product.destroy_all
  #abort params.to_yaml
  map = {}
  cnt = 0
  #todo check for params count.
  params[:col_data].each do |val|
    map[cnt] = val if val != 'ignore'
    cnt += 1
  end

  source = CSV.open session[:import_file], {col_sep: ';'}
  source.each do |row|
    cnt += 1
    if row.size > 0
      item = Product.new
      map.each do |col, attr|
        item[attr] = row[col]
      end
      item[:provider_id] = params[:adjust][:provider]
      item.save
      #abort item.to_yaml
    end
  end

  #abort map.to_yaml
  #todo response needed.
end

def upload
  require 'CSV' #looks like I dont need this in fact.
  @tmp = params[:upload][:my_file].path #tempfile

  @csv = []
  #source = CSV.open @tmp, {col_sep: ";"}

  session[:import_file] = params[:upload][:my_file].path

  respond_to do |format|
    format.html { redirect_to action: 'import_adjust' }
  end
end

upload.html.erb:

<h1>Uploaded</h1>
<%= @tmp %>

<% @csv.each do |val| %>
    <%= val %>
<% end %>

_form_import.html.erb:

<%= form_for :upload, :html => {:multipart => true}, :url => {action: "upload"} do |f| %>
<%= f.file_field :my_file %>
<%= f.submit "Upload" %>
<% end %>

import_adjust.html.erb:

<h1>New product</h1>


<%= form_for :adjust, :url => {action: "import_adjust"} do |f| %>
<% if @csv %>
<table>
  <tr>
    <% @line_size.times do |cnt| %>
        <td>
        <%= select_tag 'col_data[]',
               options_for_select([
                  ['--ignore--', 'ignore'],
                  ['name','name'],
                  ['brand','brand'],
                  ['delivery_time','delivery_time'],
                  ['price','price']
        ], @col_default!=nil ? @col_default[cnt] : nil) %>
        </td>
    <% end %>
  </tr>

  <% @csv.each do |val| %>
     <tr>
     <% val.each do |cell| %>
       <td>
         <%= cell %>
       </td>
     <% end %>
     </tr>
  <% end %>
</table>
<% end %>


  <%= f.label :delimiter, 'Разделитель' %>
  <%= f.text_field :delimiter %>
  <br>
  <%= f.label :provider, 'Поставщик' %>
  <%#todo default empty option needed! Human mistakes warning! %>
  <%= f.select :provider, Provider.all.collect { |item| [item.name, item.id] } %>
  <br>
  <%= f.label :delimiter, 'Разделитель' %>
  <%= f.text_field :delimiter %>
  <br>
  <%# Adjust for proceed adjusting or Complete  for parsing %>
  <%= f.submit "Adjust" %>
  <%= f.submit "Complete" %>
<% end %>


<%= link_to 'Back', products_path %>

Answer 1

I have an identical scenario as Lifecoder where a user uploads a file, names the columns using a map_fields plugin (by Andrew Timberlake), and then the file is parsed and processed. Here's how I handle it:

  file_field = params[options[:file_field]]
  map_fields_file_name = "map_fields_#{Time.now.to_i}_#{$$}"

  bucket = S3.buckets[CSV_COUPON_BUCKET_NAME]    # gets an existing bucket
  obj = bucket.objects[map_fields_file_name]
  obj.write( file_field.read )

  # Save the name and bucket to retrieve on second pass
  session[:map_fields][:bucket_name] = map_fields_file_name

Then on the second pass to process the file, I open the file and read it back into temp for the dyno to process:

    # Get CSV data out of bucket and stick it back into temp, so we pick up where
    # we left off as far as map_fields is concerned.
    bucket = S3.buckets[CSV_COUPON_BUCKET_NAME]
    obj = bucket.objects[session[:map_fields][:bucket_name]]
    temp_path = File.join(Dir::tmpdir, "map_fields_#{Time.now.to_i}_#{$$}")
    File.open(temp_path, 'wb') do |f|
      f.write obj.read
    end

I had to use the plugin so I could modify the code, as obviously the gem is handled by Heroku and doesn't allow for modifications.

Answer 2

Could you paste the entire controller code? The problem is on line #82, but I can't be 100% confident what line that is if you've stripped the class def and before_filters out.

That said, it looks like the problem is with one of the CSV.open lines. The way you're trying to set session[:import_file] is not guaranteed to work. If you ever run the app on more than one dyno you could have the first request served by your web.1 dyno and the second served by web.2, and they have different file systems and would not be able to see the same temp files.

I'd suggest one of the following:

• Do all the processing immediately on the upload and avoid the re-direct.
• An improvement on that would be to have the upload store the data somewhere shared and accessible (the database or S3) and have start a background job/process to do the processing.
• Best of all would be to upload directly to S3 (I believe the S3 Uploader library can do this, there are probably others) and issue a callback to create a background job to process.

That last option means your web dynos are never tied up handling massive uploads and you don't burden the user with waiting for the latency involved in upload to server->store in S3->schedule background job, it is reduced simply to store in S3 from their perspective.