CODEWITHSUNDEEP
pythonscrapy
Oliver
Oliver

Reputation: 81

Diagnose why my scrapy spider isn't downloading images

I have a simple scrapy project. I am crawling www.anthropologie.com for sale items. I want to use the standard imagePipeline to download the sale items I am scraping. I have enabled the standard imagePipeline in the settings.py file as well as a valid directory for the IMAGE_STORE. I have the required fields image_url and images. My spider is scrapping the correct url for the images as validated by checking the urls in the browser. When I run the spider it is indicated that the pipeline is enabled. However I see no indication that images are being downloaded and I find no images in the correct directory.

below are examples of my code:

settings.py:

BOT_NAME = 'LTTcrawlers'

SPIDER_MODULES = ['LTTcrawlers.spiders']
NEWSPIDER_MODULE = 'LTTcrawlers.spiders'
ITEM_PIPELINES = {'scrapy.contrib.pipeline.images.ImagesPipeline': 1}
IMAGES_STORE = 'images'

items.py

from scrapy.item import Item, Field


class saleItem(Item):

    image_url = Field()
    images = Field()
    retailer = Field()
    name = Field()
    prev_price = Field()
    sale_price = Field()
    link = Field()
    url = Field()

anthro_spider.py:

from scrapy.spider import Spider
from scrapy.selector import Selector

from LTTcrawlers.items import saleItem

class AnthroSpider(Spider):

    name = "anthro"
    allowed_domains = ['www.anthropologie.com']
    start_urls = [
    'http://www.anthropologie.com/anthro/category/clothing/shopsale-clothing.jsp?&id=SHOPSALE-CLOTHING&facetSelected=true&itemCount=100&bucketPriceHigh=10.0&cm_sp=LEFTNAV-_-PRICE-_-BUCKETPRICE%3C25.0'
    ]

def parse(self, response):

    sel = Selector(response)
    items = sel.xpath('//div[@class="category-items"]/div')

    sale_items = []

    for item in items:
        sale_item = saleItem()
        sale_item["retailer"] = "Anthropologie"
        sale_item["name"] = item.xpath("./div[@class='item-description']/a/text()").extract()[0].encode('ascii','ignore')
        sale_item["sale_price"]= item.xpath("./div[@class='item-description']/div/span/text()").extract()[0].encode('ascii', 'ignore')
        sale_item["prev_price"] = item.xpath("./div[@class='item-description']/div/span/span/text()").extract()[0].encode('ascii', 'ignore')
        sale_item["url"] = item.xpath("./div[@class='item-description']/a/@href"
        ).extract()[0].encode('ascii', 'ignore')
        sale_item["image_url"] = item.xpath('.//img/@data-original').extract()

        sale_items.append(sale_item)

    return sale_items

there are no errors reported so I can't figure out what I'm missing.

Upvotes: 0

Views: 159

Answers (1)

sp1der
sp1der

Reputation: 176

In the items.py file, it should be image_urls = Field(). not image_url = Field()

Upvotes: 1

Related Questions