CODEWITHSUNDEEP
pythonscrapy
crawf
crawf

Reputation: 85

Scrapy returns image SRC URL but does not download image

I have looked at many examples and similar questions but am not able to resolve this, thank you if you are able to help.

I have Pillow installed and the following settings uncommented

SPIDER_MIDDLEWARES = {'takaratomy.middlewares.TakaratomySpiderMiddleware': 543,}
DOWNLOADER_MIDDLEWARES = {'takaratomy.middlewares.TakaratomyDownloaderMiddleware': 543,}

The terminal returns no error and correctly returns the direct link but no image is downloaded, the folder is created but no images are saved. I guess the items.py or pipeline has some missing info, I have retracted some of the image results for ease of viewing.

I have tried the default Item_Pipeline but that also gives the same result

#ITEM_PIPELINES = {'takaratomy.pipelines.TakaratomyPipeline': 300,}

 <GET https://takaratomymall.jp/shop/g/g4904810104957/> (referer: None)
2021-06-01 17:26:14 [scrapy.core.scraper] DEBUG: Scraped from <200 
https://takaratomymall.jp/shop/g/g4904810104957/>
{'img_urls': ['https://takaratomymall.jp/img/usr/header/tt_logo_tomy1_1.png', 
'https://takaratomymall.jp/img/goods/5/4904810104957_9ddae881c00e43b4866201ccb3c92abe.jpg', 
'https://takaratomymall.jp/img/goods/L/4904810104957_b6d7dcacc3e34ee9863dddd49e432190.jpg',
 'https://takaratomymall.jp/img/goods/C/4904810104957_bf967f02392c4e9190fad6e24f635b8b.jpg', 
'https://takaratomymall.jp/img/goods/1/4904810104957_ad5bfbb36fb6484dbfced3d1f16a17c4.jpg'],
 'images': []}
2021-06-01 17:26:14 [scrapy.core.engine] INFO: Closing spider (finished)

spider.py

import scrapy

class TakaratomyscraperSpider(scrapy.Spider):
name = 'takaratomyscraper'

start_urls = ['https://takaratomymall.jp/shop/g/g4904810104957']
             

def parse(self, response):
    dlimages = response.css('img ::attr(src)').getall()
    finalimages = []
    for img_urls in dlimages:
        finalimages.append(response.urljoin(img_urls))

    yield {
        'img_urls': finalimages
    }

items.py

import scrapy

class TakaratomyItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
pass

settings.py

BOT_NAME = 'tomybot'

SPIDER_MODULES = ['takaratomy.spiders']
NEWSPIDER_MODULE = 'takaratomy.spiders'

ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = 'finalimagesfolder'

 ROBOTSTXT_OBEY = False

pipelines.py

class TakaratomyPipeline:
 def process_item(self, item, spider):
    return item

Any ideas as Ive been trying for hours

Upvotes: 1

Views: 80

Answers (1)

stranac
stranac

Reputation: 28266

You are using an incorrect field name.
Scrapy's ImagesPipeline uses image_urls by default, you have img_urls.

If you really want to use your name, it can be changed using the IMAGES_URLS_FIELD setting.

Upvotes: 1

Related Questions