CODEWITHSUNDEEP
pythonxpathweb-scrapingscrapy
Deoxyseia
Deoxyseia

Reputation: 1387

How to set a default value when Scrapy selector returns None

I was trying to set default value when the result of my xpath selector return None. This happens when in some pages the xpath node dont exist and I want to set for example 'N/A' or 'Not found'.

I used the following code but I think this isn't clean and efficient:

value = response.xpath(property.xpath).extract_first()

if(value != None):
    data[property.name] = response.xpath(property.xpath).extract_first()
else:
    data[property.name] = "N/A"

Any ideas? Thanks

Upvotes: 5

Views: 2667

Answers (1)

Padraic Cunningham
Padraic Cunningham

Reputation: 180441

There is no need to do the query twice, a simple solution is to pass a default value:

data[property.name] = response.xpath(property.xpath).extract_first(default='N/A')

For future reference, if you were to rewrite your own code without using the default keyword, I would query once and use an if/else:

value = response.xpath(property.xpath).extract_first()
data[property.name] = value if value else "N/A"

Upvotes: 14

Related Questions