CODEWITHSUNDEEP
pythonregexxpathscrapy
deekay
deekay

Reputation: 35

How to get <p> that contains text which matches regex

I am trying to scrape this website using scrapy, xpath and regex. I have checked and tried the answers to this question: xpath+ regex: matches text

I want to create a 'scrapy.selector.unified.SelectorList' of <p> that contain the text "11 (sun)" or "9 (fri)" and such, and loop through the list.

event = response.xpath('//p[matches(text(), "\d+\s\(\w{3}\)")]').extract()

does not work.

FYI, below does work.

event = response.xpath('//p[contains(text(), "11 (sun)")]').extract()

What am I missing here?

Upvotes: 2

Views: 1180

Answers (2)

stranac
stranac

Reputation: 28236

If you're only after text, Karan Verma's answer is sufficient.
If you're after the elements themselves, keep reading.

matches is only available in XPath 2.0 and higher (as are the other regex functions), and is not available in scrapy.

Scrapy uses parsel for parsing, which in turn uses lxml, which only supports XPath 1.0.
It does, however, support regular expressions in the EXSLT namespace

Since the regex namespace is enabled by default in scrapy, you can do this:

event = response.xpath('//p[re:match(text(), "\d+\s\(\w{3}\)")]')

Upvotes: 1

Karan Verma
Karan Verma

Reputation: 158

You can use re() instead of extract() Call the .re() method for each element in this list and return their results flattened, as a list of unicode strings. .re() returns a list of unicode strings. So you can’t construct nested .re() calls.

event = response.xpath('//p/text()').extract("\d+\s\(\w{3}\)")

Note: re() decode HTML entities (except < and &).

For more information please refer doc here : https://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.SelectorList.re

Upvotes: 1

Related Questions