CODEWITHSUNDEEP
pythonscrapyweb-crawler
Beer
Beer

Reputation: 57

Why can i not crawl this website with Scrapy

Am i not able to crawl this website? :

http://www.itbanen.nl/vacature/zoeken/overzicht/wijzigingsdatum/query//distance/30/output/html/items_per_page/15/page/1/ignore_ids

i try a very easy scrapy code to see if i can get something from the website but whatever i try i get nothing at all..

from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import Selector
from scrapy.log import *
from vacatures.settings import *
from vacatures.items import *
from scrapy.http import Request

class VacaturesSpider(CrawlSpider):

    name = 'vacatures_spider'
    allowed_domains = ['www.itbanen.nl']
    start_urls = ['http://www.itbanen.nl/vacature/zoeken/overzicht/wijzigingsdatum/query//distance/30/output/html/items_per_page/15/page/1/ignore_ids']



    def parse(self, response):
        self.log('Nieuwe pagina! %s' % response.url)
        #hxs = HtmlXPathSelector(response)
        sel = Selector(response)
        # HXS to find url that goes to detail page
        test = sel.xpath('//div[@id="resultlist"]/div[@class="resultlist"]/h2/text()').extract()
        print test

        links = sel.xpath('//div[@class="container"]/h2/text()')
        print links
        for link in links:
            link_item = link.extract()
            print link_item
            #yield Request(complete_url(link_item), callback=self.parse_category)

Upvotes: 0

Views: 192

Answers (1)

Guy Gavriely
Guy Gavriely

Reputation: 11396

I used scrapy shell and gave it a try

>>> a = sel.xpath('//div[@class="result-item-header"]//h2/a')
>>> a.xpath('text()').extract()
[u'Service Desk Engineer (Unified C...', u'Virtualisatie specialist', 
                          u'Medior beheerder ICT', ... ]
>>> a.xpath('@href').extract()
[u'http://www.itbanen.nl/vacature/topbaan/3030450/Service+Desk+Engineer+%28Unified+Communications%29', 
 u'http://www.itbanen.nl/vacature/topbaan/3025022/Virtualisatie+specialist', 
 u'http://www.itbanen.nl/vacature/3043979/Medior+beheerder+ICT/0', 
 ...]

So I guess your request generation should look something like:

for link in a.xpath('@href').extract():
    yield Request(link, callback=self.parse_category)

Upvotes: 2

Related Questions