CODEWITHSUNDEEP
pythonscrapy
Vishal Khanna
Vishal Khanna

Reputation: 53

Custom parse callback request not working in Scrapy

I'm trying to fetch URLs for entries in parse_start_url method which yields a request with a callback to parse_link method but the callback doesn't seem to work. What am I getting wrong?

Code:

from scrapy import Request
from scrapy.selector import Selector 
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import Rule, CrawlSpider
from property.items import PropertyItem
import sys

reload(sys)
sys.setdefaultencoding('utf8')  #To prevent UnicodeDecodeError, UnicodeEncodeError.

class VivastreetSpider(CrawlSpider):
    name = 'viva'
    allowed_domains = ['chennai.vivastreet.co.in']
    start_urls = ['http://chennai.vivastreet.co.in/rent+chennai/']
    rules = [
        Rule(LinkExtractor(restrict_xpaths = '//*[text()[contains(., "Next")]]'), callback = 'parse_start_url', follow = True)
        ]   

    def parse_start_url(self, response):
        urls = Selector(response).xpath('//a[contains(@id, "vs-detail-link")]/@href').extract() 

        for url in urls:
            print('test ' + url)
            yield Request(url = url, callback = self.parse_link)

    def parse_link(self, response):
        #item = PropertyItem()
        print('parseitemcalled')
        a = Selector(response).xpath('//*h1[@class = "kiwii-font-xlarge kiwii-margin-none"').extract()
        print('test ' + str(a))

Upvotes: 1

Views: 196

Answers (1)

alecxe
alecxe

Reputation: 473763

You need to adjust your allowed_domains to allow the extracted URLs to be followed:

allowed_domains = ['vivastreet.co.in']

Then, you'll get into invalid expression errors, this is because //*h1[@class = "kiwii-font-xlarge kiwii-margin-none" is invalid and needs to be fixed.

Upvotes: 1

Related Questions