CODEWITHSUNDEEP
pythonweb-scrapingscrapyconditional-statements
2one2
2one2

Reputation: 401

adding conditions when scraping

I'm trying to scrape a webpage, and it seems that each separation has a different div, depending in how much the user pays or the type of page it has.

Example:

<div class="figuration Web company-stats">
..information i want to scrap..
</div>

<div class="figuration Commercial" >
..information i want to scrap..
</div>

It seems to have more than 3 types of divs so I wanted to know if there's a way to just select every div that contains the first word figuration?

Here is my spider code:

import scrapy

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from pagina.items import PaginaItem
from scrapy.contrib.linkextractors import LinkExtractor


class MySpider(CrawlSpider):
    name = "pagina"
    allowed_domains = ["paginasamarillas.com.co"]
    start_urls = ["http://www.paginasamarillas.com.co/busqueda/bicicletas-medellin"]
    rules = (Rule(SgmlLinkExtractor( restrict_xpaths=('//ul[@class="paginator"]')), 
        callback='parse_item', follow=True, ),
)

    def parse_item(self, response):
        item = PaginaItem()

        for sel in response.xpath('//div[@class="figuration Web company-stats"]'):
            item = PaginaItem()
            item['nombre'] = sel.xpath('.//h2[@class="titleFig"]/a/text()').extract()
            #item['lugar'] = sel.xpath('.//div[@class="infoContact"]/div/h3/text()').extract()
            #item['numero'] = sel.xpath('.//div[@class="infoContact"]/span/text()').extract()
            #item['pagina'] = sel.xpath('.//div[@class="infoContact"]/a/@href').extract() 
            #item['sobre'] = sel.xpath('.//p[@class="CopyText"]/div/h3/text()').extract()
            yield item

Upvotes: 0

Views: 106

Answers (2)

Arijit C
Arijit C

Reputation: 96

CSS selector mentioned above will work, but if you wish to use xpath selector, you can use it like:

for each in response.xpath('//div[contains(@class,"figuration")]'):
    ...

Actually, response.xpath('//div[contains(@class,"figuration")]') can be used interchangeably with response.css('div.figuration')

Upvotes: 1

Blender
Blender

Reputation: 298166

Use a CSS selector:

for sel in response.css('div.figuration'):
    ...

Upvotes: 2

Related Questions