CODEWITHSUNDEEP
pythonxpathscrapy
Badshah
Badshah

Reputation: 431

Xpath selector in python Scrapy

Right now I am learning how to use Xpath to scrape websites in combination with python Scrapy. Right now I am stuck at the following:

I am looking at a dutch website http://www.ah.nl/producten/bakkerij/brood where I want to scrape the names of the products: enter image description here

So eventually I want a csv file with the names of the articles of all these breads. If I inspect elements, I get to see where these names are defined:

enter image description here

I need to find the right XPath to extract "AH Tijgerbrood bruin heel". So what I thought I should do in my spider is the following:

import scrapy
from stack.items import DmozItem

class DmozSpider(scrapy.Spider):
    name = "ah"
    allowed_domains = ["ah.nl"]
    start_urls = ['http://www.ah.nl/producten/bakkerij/brood']
    def parse(self, response):
        for sel in response.xpath('//div[@class="product__description small-7 medium-12"]'):
            item = DmozItem()
            item['title'] = sel.xpath('h1/text()').extract()
            yield item

Now, if I crawl with this spider, I dont get any result. I have no clue what I am missing here.

Upvotes: 0

Views: 563

Answers (2)

Aswin Sathyan
Aswin Sathyan

Reputation: 283

title = response.xpath('//div[@class="product__description small-7 medium-12"]./h1/text').extract()[0]

Upvotes: 0

heinst
heinst

Reputation: 8786

You would have to use selenium for this task since all the elements are loaded in JavaScript:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("http://www.ah.nl/producten/bakkerij/brood")
#put an arbitrarily large number, you can tone it down, this is to allow the webpage to load
driver.implicitly_wait(40) 
elements = driver.find_elements_by_xpath('//*[local-name()= "div" and @class="product__description small-7 medium-12"]//*[local-name()="h1"]')
for elem in elements:
    print elem.text

Upvotes: 1

Related Questions