CODEWITHSUNDEEP
pythonamazon-s3luigi
Z.G
Z.G

Reputation: 83

Luigi write file directly to S3

I'm creating a data pipeline with Luigi and I'm trying to write the processed data to S3 bucket directly. The code I used is: 

import luigi
from luigi.s3 import S3Target, S3Client

class myTask(luigi.Task):
    def requires(self):
        return otherTask()

    def output(self):
        client = S3Client('ACCESS_KEY', 'SECRET_KEY')
        return S3Target('s3.amazonaws.com/mybucket/myfolder/myfile.tsv', client=client)

    def run(self):
         fo = self.output().open('w')
         with self.input().open('r') as f:
            data = dosomething_to_input(f)
            fo.write(data)
         fo.close()

After I run the script, I got Error:

S3ResponseError: S3ResponseError: 405 Method Not Allowed

Can we directly write file into S3 bucket?

Upvotes: 4

Views: 1941

Answers (1)

Z.G
Z.G

Reputation: 83

Problem solved! It's because of the format of the s3 buckt. The correct format should be 's3://mybucket/myfile' The 405 ERROR is caused by boto not recognizing the bucket name. Also need to mention that boto does not recognize bucket name with '.' in it in Python 2.7.*, so you have to use a valid bucket name or change it in the config file.

Upvotes: 4

Related Questions