CODEWITHSUNDEEP
pythonamazon-s3scrapy
user2492364
user2492364

Reputation: 6713

scrapy store images to amazon s3

I store images in my local server then upload to s3
Now I want to edit it to stored images directly to amazon s3

But ther is error:

boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

here is my settings.py

AWS_ACCESS_KEY_ID = "XXXX"
AWS_SECRET_ACCESS_KEY = "XXXX"
IMAGES_STORE = 's3://how.are.you/'

Do I need to add something??

my scrapy edition: Scrapy==0.22.2

Please guide me,thank you!

Upvotes: 1

Views: 1369

Answers (3)

Wayne
Wayne

Reputation: 26

I found the cause of the problem is upload policy. The function Key.set_contents_from_string() takes argument policy, default set to S3FileStore.POLICY. So modify the code in scrapy/contrib/pipeline/files.py, change

 return threads.deferToThread(k.set_contents_from_string, buf.getvalue(),
                              headers=h, policy=self.POLICY)

to

 return threads.deferToThread(k.set_contents_from_string, buf.getvalue(),
                              headers=h)

Maybe you can try it, and share the result here.

Upvotes: 0

Tasawer Nawaz
Tasawer Nawaz

Reputation: 925

I think the problem is not in your code, actually the problem lies in permission, please check your credentials first and make sure your permissions to access and write on s3 bucket.

    import boto
    s3 = boto.connect_s3('access_key', 'secret_key')
    bucket = s3.lookup('bucket_name')
    key = bucket.new_key('testkey')
    key.set_contents_from_string('This is a test')
    key.delete()

If test run successfuly then look into your permission, for setting permission you can look at amazon configuration

Upvotes: 0

Neelam Sharma
Neelam Sharma

Reputation: 2855

AWS_ACCESS_KEY_ID = "xxxxxx"
AWS_SECRET_ACCESS_KEY = "xxxxxx"
IMAGES_STORE = "s3://bucketname/virtual_path/"

how.are.you should be a S3 Bucket that exist into your S3 account, and it will store the images you upload. If you want to store images inside any virtual_path then you need to create this folder into your S3 Bucket.

Upvotes: 1

Related Questions