CODEWITHSUNDEEP
python-2.7amazon-s3
Seema Mudgil
Seema Mudgil

Reputation: 385

Accessing S3 through smart_open & reading file contents line by line

I am fetching data from S3 using smart_open. But when i am reading file line by line its throwing error. I am iterating over buckets and fetching keys (file name). Now i need to read content of file line by line. 

for key, content in smart_open.s3_iter_bucket(bucket = bucket, prefix = prefix):

            print key
            with smart_open.smart_open(bucket+key) as fin:
                for line in fin:
                    print line

Its throwing Bad Request error. Pl suggest

Upvotes: 2

Views: 5863

Answers (1)

ECHOOooo... ECHOOooo...
ECHOOooo... ECHOOooo...

Reputation: 21

I believe your bucket name and key are stand-alone strings, while the argument passed to smart_open is a formatted URI that must include the S3 prefix. So...I would try:

for key, content in smart_open.s3_iter_bucket(bucket = bucket, prefix = prefix):

        print key
        s3_uri = 's3://' + bucket + '/' + key
        with smart_open.smart_open(s3_uri) as fin:
            for line in fin:
                print line

Upvotes: 2

Related Questions