CODEWITHSUNDEEP
python-3.xweb-scrapingscrapy
gongarek
gongarek

Reputation: 1034

How to skip missing data in Scrapy

Let's say that I have html page something like this:

...
<a class="hehe"><span>joke23</span></a>
<a class="hrtojoke" href="link/to/joke23"></a>
<a class="hehe"><span>joke24</span></a>

<a class="hehe"><span>joke25</span></a>
<a class="hrtojoke" href="link/to/joke25"></a>
...

As You can see I don't have link to joke24 ;)

I want to that to every joke assign his link. If link does not exist exist I want assign it as None.

My code:

...
def parse(self, response):
    for joke, link in response.css(itertools.zip_longest(response.css('a.hehe'), response.css('a.hrtojoke')):
        yield {
            'name_joke': joke.xpath('span/text()').extract_first(),
            'link_joke': link.css('::attr(href)').extract_first(),
        } 
...

As You can guess, this code works, but no correctly

Current Output:

...
{'name_joke': 'joke23', 'link_joke': 'link/to/joke23'}
{'name_joke': 'joke25', 'link_joke': 'link/to/joke25'}
error..
...

Desired Output:

{'name_joke': 'joke23', 'link_joke': 'link/to/joke23'}
{'name_joke': 'joke24', 'link_joke': None}
{'name_joke': 'joke25', 'link_joke': 'link/to/joke25'}

How can I achieve my goal?

Upvotes: 0

Views: 412

Answers (2)

Konstantin
Konstantin

Reputation: 547

Try this one:

def parse(self, response):
    for item in response.xpath('//*[@class="hehe"]'):
        joke = item.xpath('./span/text()').extract_first() 
        link = item.xpath('./following-sibling::*[1][@class="hrtojoke"]/@href').extract_first()
        yield {'name_joke': joke, 'link_joke': link}

OUTPUT:

{'name_joke': 'joke23', 'link_joke': 'link/to/joke23'}
{'name_joke': 'joke24', 'link_joke': None}
{'name_joke': 'joke25', 'link_joke': 'link/to/joke25'}

Upvotes: 3

CK Chen
CK Chen

Reputation: 664

Just use a try-except to catch the exception.
*Remember to catch the exact exception.

def parse(self, response):
    for joke, link in response.css(itertools.zip_longest(response.css('a.hehe'), response.css('a.hrtojoke')):
        name_joke = joke.xpath('span/text()')extractfirst()
        try:
            link_joke = link.css('::attr(href)').extract_first()
        except:  # pls add the exact exception you want to catch.
            link_joke = None
        yield {
            'name_joke': name_joke,
            'link_joke': link_joke
        }

Upvotes: 0

Related Questions