CODEWITHSUNDEEP
rubyamazon-web-services
koosa
koosa

Reputation: 3040

Ruby aws-sdk: "Unable to find marker in S3 list objects response"

I am trying to iterate over the children of a AWS::S3::Tree object, like so:

conn = AWS::S3.new(:access_key_id => 'ACCESSKEYID', :secret_access_key => 'ACCESSKEY')
bucket = conn.buckets['bucketname']
tree = bucket.objects.with_prefix('assets/images').as_tree
directories = tree.children.select(&:branch?).collect(&:prefix)

This works fine on most of the paths I use as a prefix, but one folder (that has tens of thousands of sub folders) returns this error:

/lib/aws/s3/object_collection.rb:311:in `next_markers': Unable to find marker in S3 list objects response (RuntimeError)

I have overridden the gem method to debug like so:

module AWS
class S3
    class ObjectCollection
        protected
        def next_markers page

            raise page.inspect

            marker = (last = page.contents.last and last.key)
            if marker.nil?
                raise 'Unable to find marker in S3 list objects response xxxxx'
            else
                { :marker => marker }
            end
        end
    end
  end
end

And this outputs the returned data:

{:delimiter=>"/", :contents=>[], :common_prefixes=>[{:prefix=>"assets/images/100/"}, {:prefix=>"assets/images/1000/"}, {:prefix=>"assets/images/1001/"}, etc etc

Why is the call returning an empty array for :contents ?

Upvotes: 0

Views: 582

Answers (1)

Trevor Rowe
Trevor Rowe

Reputation: 6528

This happens when the XML from S3 is not valid and the contents can not be parsed. If you enable wiretracing (add :http_wire_trace => true to AWS::S3.new) and you should be able to examine the offending http response body.

Upvotes: 1

Related Questions