Strange "ArgumentError: key may not be blank" on aws S3

Question

I have got a rails app which allow user to directly upload the images relating to a post to S3 then the image processing "delay" jobs will be run to resize it. Everything works well when using the web browser but recently I was writing some db populating tasks but when I tried to save the images and its related model, this error shows: ArgumentError: key may not be blank. Any idea why would this happen would be great!!

The fragment of the populate.rake code:

post = Post.new(user: user)
((rand*5).floor + 1).times do

      ## Here, you will need to upload the file to S3, get the direct_upload_url then save it for img.direct_upload_url
      # Upload to S3, then get the direct_upload_url
      file=File.open(Dir.glob(File.join(Rails.root, 'sampleimages/posts', '*')).sample)
      s3_obj = S3_BUCKET.objects["direct_uploads/#{Time.now.to_i}/#{File.basename(file)}"]
      puts "    Selected file: #{File.basename(file)}"
      puts "      --> Starting upload"

      s3_obj.write(file)
      #s3_obj.acl= :public_read
      obj_url = ""

      while obj_url.blank?
        obj_url = s3_obj.url_for(:read)
        sleep(3)
      end
      puts "    --> Loop completed with obj_url = #{obj_url}"
      img = post.images.build(direct_upload_url: obj_url.to_s)

      # puts "url_for #{obj_url}"

      puts "Image uploaded: #{s3_obj.key} URL: #{img.direct_upload_url}\n"
      # The below code is used for local file storage
      #img.image_file = File.open(Dir.glob(File.join(Rails.root, 'sampleimages/posts', '*')).sample)
    end

    post.save!

end

The error appears at post.save!.

Here's the Image.rb as well:

## S3_direct_upload post processing!
before_create :set_upload_attributes
### setting the image_to_process of this image.
before_create :inc_images_to_process
after_create :queue_processing

def direct_upload_url=(escaped_url)
  write_attribute(:direct_upload_url, (CGI.unescape(escaped_url) rescue nil))
end

# Determines if file requires post-processing (image resizing, etc)
# All files should be processable in my case, where only images can be stored.
def post_process_required?
  %r{^(image|(x-)?application)/(bmp|gif|jpeg|jpg|pjpeg|png|x-png)$}.match(image_file_content_type).present?
end



# Final upload processing step
def self.transfer_and_cleanup(id)
  begin
    ## This will raise exception, need to catch this.
    image = Image.find_by_id(id)
    ## Check if the image still exists when this is running, if not, then just skip this and minus one from post
    if image.nil? || image.processed
      ## Something's wrong, either the image is not present or the image is already processed
      Delayed::Worker.logger.debug "ERROR: Image with id: #{id} Not Found. This file will still up on aws temp!"
      ## The file is not here anymore.
      ## Ask the post to recalculate how many images it actually has!
        if image.present? && image.processed 
          direct_upload_url_data = DIRECT_UPLOAD_URL_FORMAT.match(image.direct_upload_url)
          S3_BUCKET.objects[direct_upload_url_data[:path]].delete
        end
    else
        ## With this match, path and filename will be extracted.
        direct_upload_url_data = DIRECT_UPLOAD_URL_FORMAT.match(image.direct_upload_url)

        if image.post_process_required?
          image.image_file = URI.parse(URI.escape(image.direct_upload_url))
        else
            # Somethings's wrong???!?!?!? It should be images!
          paperclip_file_path = "images/uploads/#{id}/original/#{direct_upload_url_data[:filename]}"
          S3_BUCKET.objects[paperclip_file_path].copy_from(direct_upload_url_data[:path])
      end

      image.processed = true

      if image.save
          ## To unset the image_to_process field of its parent - the Post
          if image.imageable.class.to_s == "Post"
            # logger.debug "I am here!!!"
            post = image.imageable
            post.images_to_process -= 1
            post.save!
          end
          Delayed::Worker.logger.debug "Image:#{id}:transfer_and_cleanup: It is now saved"

      else
          Delayed::Worker.logger.debug "Image:#{id}:transfer_and_cleanup: ERROR Occurs: #{image.errors.full_messages}"
      end
      ### End of image.save

      ## Delete the temp file.
      Delayed::Worker.logger.debug "Image:#{id}:transfer_and_cleanup: direct_upload_url: #{direct_upload_url} is going to deleted"
      S3_BUCKET.objects[direct_upload_url_data[:path]].delete

    end
    ## end of image.present? loop
  end




end ## End of transfer_and_cleanup



  # Set attachment attributes from the direct upload
  # @note Retry logic handles S3 "eventual consistency" lag.
  def set_upload_attributes
    logger.debug "You are now in set_upload_attributes"
    tries ||= 5
    direct_upload_url_data = DIRECT_UPLOAD_URL_FORMAT.match(direct_upload_url)

    direct_upload_head = S3_BUCKET.objects[direct_upload_url_data[:path]].head

    self.image_file_file_name     = direct_upload_url_data[:filename]
    self.image_file_file_size     = direct_upload_head.content_length
    self.image_file_content_type  = direct_upload_head.content_type
    self.image_file_updated_at    = direct_upload_head.last_modified


    ## Rescue the S3 Errors here

    rescue AWS::S3::Errors::NoSuchKey => e
      tries -= 1
      if tries > 0
        sleep(3)
        retry
      else
        false
      end
  end

  # Queue file processing
  def queue_processing
    Image.delay.transfer_and_cleanup(id)
    Delayed::Worker.logger.debug "queue_processing: Image #{id} added in the queue. At "
  end

  def inc_images_to_process
    # Increase the image required to process in Post.
    if self.imageable.class.to_s == "Post"

      post = self.imageable
      post.inc_images_to_process

      Delayed::Worker.logger.debug "inc_images_to_process: Post #{post.id} new images_to_process is now: #{post.images_to_process}"
    end
  end

The error messages are:

rake aborted!
ArgumentError: key may not be blank
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/s3/client.rb:384:in `validate!'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/s3/client.rb:390:in `validate_key!'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/s3/client.rb:542:in `block (2 levels) in object_method'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/s3/client.rb:1563:in `block (2 levels) in <class:V20060301>'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:560:in `build_request'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:491:in `block (3 levels) in client_request'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/response.rb:175:in `call'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/response.rb:175:in `build_request'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/response.rb:114:in `initialize'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:203:in `new'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:203:in `new_response'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:490:in `block (2 levels) in client_request'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:391:in `log_client_request'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:477:in `block in client_request'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:373:in `return_or_raise'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:476:in `client_request'
(eval):3:in `head_object'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/aws-sdk-v1-1.56.0/lib/aws/s3/s3_object.rb:296:in `head'
/Users/quindici/Documents/Career/QUIN CREATIVES/Local-say/Local-say-System/app/models/image.rb:134:in `set_upload_attributes'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activesupport-4.0.0/lib/active_support/callbacks.rb:377:in `_run__4088547230130556858__create__callbacks'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activesupport-4.0.0/lib/active_support/callbacks.rb:80:in `run_callbacks'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/callbacks.rb:303:in `create_record'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/timestamp.rb:57:in `create_record'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/persistence.rb:466:in `create_or_update'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/callbacks.rb:299:in `block in create_or_update'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activesupport-4.0.0/lib/active_support/callbacks.rb:383:in `_run__4088547230130556858__save__callbacks'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activesupport-4.0.0/lib/active_support/callbacks.rb:80:in `run_callbacks'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/callbacks.rb:299:in `create_or_update'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/persistence.rb:106:in `save'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/validations.rb:51:in `save'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/attribute_methods/dirty.rb:32:in `save'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:270:in `block (2 levels) in save'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:326:in `block in with_transaction_returning_status'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:200:in `transaction'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:209:in `transaction'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:323:in `with_transaction_returning_status'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:270:in `block in save'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:281:in `rollback_active_record_state!'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:269:in `save'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/associations/has_many_association.rb:39:in `insert_record'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/autosave_association.rb:348:in `block in save_collection_association'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/autosave_association.rb:339:in `each'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/autosave_association.rb:339:in `save_collection_association'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/autosave_association.rb:183:in `block in add_autosave_association_callbacks'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/autosave_association.rb:153:in `instance_eval'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/autosave_association.rb:153:in `block in define_non_cyclic_method'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activesupport-4.0.0/lib/active_support/callbacks.rb:383:in `_run__690630312330398441__update__callbacks'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activesupport-4.0.0/lib/active_support/callbacks.rb:80:in `run_callbacks'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/callbacks.rb:307:in `update_record'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/timestamp.rb:70:in `update_record'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/persistence.rb:466:in `create_or_update'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/callbacks.rb:299:in `block in create_or_update'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activesupport-4.0.0/lib/active_support/callbacks.rb:413:in `_run__690630312330398441__save__callbacks'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activesupport-4.0.0/lib/active_support/callbacks.rb:80:in `run_callbacks'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/callbacks.rb:299:in `create_or_update'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/persistence.rb:128:in `save!'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/validations.rb:57:in `save!'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/attribute_methods/dirty.rb:41:in `save!'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:275:in `block in save!'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:326:in `block in with_transaction_returning_status'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:202:in `block in transaction'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:210:in `within_new_transaction'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:202:in `transaction'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:209:in `transaction'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:323:in `with_transaction_returning_status'
/Users/quindici/.rvm/gems/ruby-2.0.0-p247@railstutorial_rails_4_0/gems/activerecord-4.0.0/lib/active_record/transactions.rb:275:in `save!'
/Users/quindici/Documents/Career-System/lib/tasks/populate.rake:139:in `block (4 levels) in <top (required)>'
/Users/quindici/Documents/Career-System/lib/tasks/populate.rake:56:in `times'
/Users/quindici/Documents/Career-System/lib/tasks/populate.rake:56:in `block (3 levels) in <top (required)>'
/Users/quindici/Documents/Career-System/lib/tasks/populate.rake:31:in `times'
/Users/quindici/Documents/Career-System/lib/tasks/populate.rake:31:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:db_populate

Answer 1

Thanks for all the helps and I have found the problem.

The Error is really because the key is not provided when accessing an object. This happened because the upload url is in different format as the regexp so the "key" extracted from the direct_upload_url is nil. As a result, when the Image object is trying to save the file details in set_upload_attributes, the S3_BUCKET.objects[] got no key and hence the ArgumentError!

Thank you for reading!