CODEWITHSUNDEEP
pythonmongodbscrapy
user2492364
user2492364

Reputation: 6713

how to use scrapy download images and then upload to s3 server

I want to upload images to s3 when the spider closed,
My method now is sending all images in mongodb : upload_s3(ShA.objects.all())
But I want to edit it to send the images the scrapy download this time .

I need to send the variables sh.images from def process_item() to def close_spider() to let mongo filter the item the scrapy crawl this time

how can I edit to reach it?

here is my pipeline:

from mongo.models import ShA
from uploads3 import upload_s3 
class ShPipeline(object):
    def process_item(self, item, spider):
        if isinstance(item, ShaItem):  
            sh = item.save(commit=False)  
            sh_exist = ShA.objects.filter(link=sh.link)  
            if sh_exist:
                sh.id = sh_exist[0].id
            sh.save()
            #sh.images
            return item

    def close_spider(self, spider,item):
        if spider.name == "email":
            upload_s3(ShA.objects.all()) 
            #upload_s3(ShA.objects.get(images=sh.images)) no use,need to get sh.images from def process_item

Upvotes: 3

Views: 673

Answers (1)

nramirezuy
nramirezuy

Reputation: 171

You can simply use self, but I really recommend you using our pipeline.

Upvotes: 1

Related Questions