CODEWITHSUNDEEP
pythonscrapyscraper
zara
zara

Reputation: 1

download and save images from a website using scrapy

I am new to scrapy and Python, so my question may be a simple one. By using an existing website guide, I've written a scraper which scrapes a website's pages and shows the images URL, name and ... in a output file. I want to download the images in a directory but the output directory is empty!

Here is my code:

myspider.py

import scrapy
class BrickSetSpider(scrapy.Spider):
name = 'brick_spider`enter code here`'
start_urls = ['http://brickset.com/sets/year-2016']

def parse(self, response):
    SET_SELECTOR = '.set'
    for brickset in response.css(SET_SELECTOR):

        NAME_SELECTOR = 'h1 a ::text'
        PIECES_SELECTOR = './/dl[dt/text() = "Pieces"]/dd/a/text()'
        MINIFIGS_SELECTOR = './/dl[dt/text() = "Minifigs"]/dd[2]/a/text()'
        IMAGE_SELECTOR = 'img ::attr(src)'
        yield {
            'name': brickset.css(NAME_SELECTOR).extract_first(),
            'pieces': brickset.xpath(PIECES_SELECTOR).extract_first(),
            'minifigs': brickset.xpath(MINIFIGS_SELECTOR).extract_first(),
            'image': brickset.css(IMAGE_SELECTOR).extract_first(),
        }

    NEXT_PAGE_SELECTOR = '.next a ::attr(href)'
    next_page = response.css(NEXT_PAGE_SELECTOR).extract_first()
    if next_page:
        yield scrapy.Request(
            response.urljoin(next_page),
            callback=self.parse
        )

settings.py

ITEM_PIPELINES = {'brickset.pipelines.BricksetPipeline': 1}
IMAGES_STORE = '/home/nmd/brickset/brickset/spiders/output'


#items.py 
import scrapy
class BrickSetSpider(scrapy.Item):
image_urls = scrapy.Field()
images = scrapy.Field()
pass

Upvotes: 0

Views: 3033

Answers (1)

Tarun Lalwani
Tarun Lalwani

Reputation: 146580

Scrapy provides a media pipeline if your interested in downloading files or images

ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}

Then you need to add image_urls in your item for the pipeline to download the file, so change

    yield {
        'name': brickset.css(NAME_SELECTOR).extract_first(),
        'pieces': brickset.xpath(PIECES_SELECTOR).extract_first(),
        'minifigs': brickset.xpath(MINIFIGS_SELECTOR).extract_first(),
        'image': brickset.css(IMAGE_SELECTOR).extract_first(),
    }

to

    yield {
        'name': brickset.css(NAME_SELECTOR).extract_first(),
        'pieces': brickset.xpath(PIECES_SELECTOR).extract_first(),
        'minifigs': brickset.xpath(MINIFIGS_SELECTOR).extract_first(),
        'image_urls': brickset.css(IMAGE_SELECTOR).extract_first(),
    }

For more details refer to https://doc.scrapy.org/en/latest/topics/media-pipeline.html

Upvotes: 1

Related Questions