CODEWITHSUNDEEP
pythonscrapy
Kurt Peek
Kurt Peek

Reputation: 57461

Scrapy spider does not yield feed output after making 'start_urls' variable

The following spider with fixed start_urls works:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from funda.items import FundaItem

class PropertyLinksSimpleSpider(CrawlSpider):

    name = "property_links_simple"
    allowed_domains = ["funda.nl"]

    # def __init__(self, place='amsterdam', page='1'):
    #     self.start_urls = ["http://www.funda.nl/koop/%s/p%s/" % (place, page)]
    #     self.le1 = LinkExtractor(allow=r'%s+huis-\d{8}' % self.start_urls[0])

    start_urls = ["http://www.funda.nl/koop/amsterdam/"]
    le1 = LinkExtractor(allow=r'%s+huis-\d{8}' % start_urls[0])
    # rules = (Rule(le1, callback='parse_item'), )

    def parse(self, response):
        links = self.le1.extract_links(response)
        for link in links:
            if link.url.count('/') == 6 and link.url.endswith('/'):
                item = FundaItem()
                item['url'] = link.url
                yield item

When I run it with feed output by the command scrapy crawl property_links_simple -o property_links.json, the resulting file contains links as expected:

[
{"url": "http://www.funda.nl/koop/amsterdam/huis-49708477-paul-schuitemahof-27/"},
{"url": "http://www.funda.nl/koop/amsterdam/huis-49826458-buiksloterdijk-270/"},
{"url": "http://www.funda.nl/koop/amsterdam/huis-49818887-markiespad-19/"},
{"url": "http://www.funda.nl/koop/amsterdam/huis-49801910-claus-van-amsbergstraat-86/"},
{"url": "http://www.funda.nl/koop/amsterdam/huis-49801593-jf-berghoefplantsoen-2/"},
{"url": "http://www.funda.nl/koop/amsterdam/huis-49800159-breezandpad-8/"},
{"url": "http://www.funda.nl/koop/amsterdam/huis-49805292-nieuwendammerdijk-21/"},
{"url": "http://www.funda.nl/koop/amsterdam/huis-49890140-talbotstraat-9/"},
{"url": "http://www.funda.nl/koop/amsterdam/huis-49879212-henri-berssenbruggehof-15/"},
{"url": "http://www.funda.nl/koop/amsterdam/huis-49728947-emmy-andriessestraat-374/"},
{"url": "http://www.funda.nl/koop/amsterdam/huis-49713458-jan-vrijmanstraat-29/"}
]

However, I'd like to be able to pass different start_urls to the spider, such as http://www.funda.nl/koop/rotterdam/p2/. To this end I tried to adapt it as follows:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from funda.items import FundaItem

class PropertyLinksSimpleSpider(CrawlSpider):

    name = "property_links_simple"
    allowed_domains = ["funda.nl"]

    def __init__(self, place='amsterdam', page='1'):
        self.start_urls = ["http://www.funda.nl/koop/%s/p%s/" % (place, page)]
        self.le1 = LinkExtractor(allow=r'%s+huis-\d{8}' % self.start_urls[0])

    # start_urls = ["http://www.funda.nl/koop/amsterdam/"]
    # le1 = LinkExtractor(allow=r'%s+huis-\d{8}' % start_urls[0])
    # rules = (Rule(le1, callback='parse_item'), )

    def parse(self, response):
        links = self.le1.extract_links(response)
        for link in links:
            if link.url.count('/') == 6 and link.url.endswith('/'):
                item = FundaItem()
                item['url'] = link.url
                yield item

However, if I run this using the command scrapy crawl property_links_simple -a place=amsterdam -a page=1 -o property_links2.json, I get an empty .json file:

[
[

Why is the spider no longer yielding any output?

Upvotes: 1

Views: 137

Answers (1)

Kurt Peek
Kurt Peek

Reputation: 57461

This turned out to be a simple human error: in the second example, start_urls[0] was no longer the same. I added a self.base_url to make it the same again:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from funda.items import FundaItem

class PropertyLinksSimpleSpider(CrawlSpider):

    name = "property_links_simple"
    allowed_domains = ["funda.nl"]

    def __init__(self, place='amsterdam', page='1'):
        self.start_urls = ["http://www.funda.nl/koop/%s/p%s/" % (place, page)]
        self.base_url = "http://www.funda.nl/koop/%s/" % place
        self.le1 = LinkExtractor(allow=r'%s+huis-\d{8}' % self.base_url)

    def parse(self, response):
        links = self.le1.extract_links(response)
        for link in links:
            if link.url.count('/') == 6 and link.url.endswith('/'):
                item = FundaItem()
                item['url'] = link.url
                yield item

This makes the spider produce the desired .json file.

Upvotes: 1

Related Questions