scrapy ERROR: Error processing {'image_urls':

Question

I'm setting up a simple spider to download images form xkcd, this is the code I currentyl have:

spider:

import scrapy
from scrapy.loader import ItemLoader

from test_im.items import TestImItem

class SpiderSpider(scrapy.Spider):
    name = 'spider_'
    allowed_domains = ['xkcd.com/']
    start_urls = ['http://xkcd.com//']

    def parse(self, response):
        test_item  = TestImItem()
        relative_url = response.xpath('//*[@id="comic"]//@src').extract_first()
        image_urls = (response.urljoin(relative_url)  )
        print (image_urls)
        test_item['image_urls'] = image_urls 
        yield test_item

items:

import scrapy

class TestImItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    image_urls = scrapy.Field()

setings:

ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = '/home/luis/Documentos/proyectos/test_im/test_im/images/'

I'm getting the following error:

2019-07-08 21:25:13 [scrapy.core.scraper] ERROR: Error processing {'image_urls': 'https://imgs.xkcd.com/comics/trained_a_neural_net.png'}
Traceback (most recent call last):
  File "/home/luis/anaconda3/lib/python3.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/luis/anaconda3/lib/python3.7/site-packages/scrapy/pipelines/media.py", line 79, in process_item
    requests = arg_to_iter(self.get_media_requests(item, info))
  File "/home/luis/anaconda3/lib/python3.7/site-packages/scrapy/pipelines/images.py", line 155, in get_media_requests
    return [Request(x) for x in item.get(self.images_urls_field, [])]
  File "/home/luis/anaconda3/lib/python3.7/site-packages/scrapy/pipelines/images.py", line 155, in <listcomp>
    return [Request(x) for x in item.get(self.images_urls_field, [])]
  File "/home/luis/anaconda3/lib/python3.7/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
    self._set_url(url)
  File "/home/luis/anaconda3/lib/python3.7/site-packages/scrapy/http/request/__init__.py", line 62, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: h

As I understand it, 'ValueError: Missing scheme in request url: h' means that the image url is wrong.

But I can open it in the browser without issues.

'image_urls': 'https://imgs.xkcd.com/comics/trained_a_neural_net.png'

Answer 1

Scrapy tryies to process your string as a list of image URLs:

test_item['image_urls'] = [image_urls]