CODEWITHSUNDEEP
scrapy
TechGeek
TechGeek

Reputation: 2212

Scrapy - Get Text from all matching divs

I need to get text from all divs which has class "feature has-feature"

<div class="features as-columns list">
    <div class="feature has-feature">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512">
            <path d="M504 256c0 136.967-111.033 248-248 248S8 392.967 8 256 119.033 8 256 8s248 111.033 248 248zM227.314 387.314l184-184c6.248-6.248 6.248-16.379 0-22.627l-22.627-22.627c-6.248-6.249-16.379-6.249-22.628 0L216 308.118l-70.059-70.059c-6.248-6.248-16.379-6.248-22.628 0l-22.627 22.627c-6.248 6.248-6.248 16.379 0 22.627l104 104c6.249 6.249 16.379 6.249 22.628.001z"></path>
        </svg>
        "What If" Scenarios
    </div>
    <div class="feature ">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512">
            <path d="M504 256c0 136.967-111.033 248-248 248S8 392.967 8 256 119.033 8 256 8s248 111.033 248 248zM227.314 387.314l184-184c6.248-6.248 6.248-16.379 0-22.627l-22.627-22.627c-6.248-6.249-16.379-6.249-22.628 0L216 308.118l-70.059-70.059c-6.248-6.248-16.379-6.248-22.628 0l-22.627 22.627c-6.248 6.248-6.248 16.379 0 22.627l104 104c6.249 6.249 16.379 6.249 22.628.001z"></path>
        </svg>
        Audit Trail
    </div>
    <div class="feature has-feature">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512">
            <path d="M504 256c0 136.967-111.033 248-248 248S8 392.967 8 256 119.033 8 256 8s248 111.033 248 248zM227.314 387.314l184-184c6.248-6.248 6.248-16.379 0-22.627l-22.627-22.627c-6.248-6.249-16.379-6.249-22.628 0L216 308.118l-70.059-70.059c-6.248-6.248-16.379-6.248-22.628 0l-22.627 22.627c-6.248 6.248-6.248 16.379 0 22.627l104 104c6.249 6.249 16.379 6.249 22.628.001z"></path>
        </svg>
        Balance Sheet
    </div>
    <div class="feature ">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512">
            <path d="M504 256c0 136.967-111.033 248-248 248S8 392.967 8 256 119.033 8 256 8s248 111.033 248 248zM227.314 387.314l184-184c6.248-6.248 6.248-16.379 0-22.627l-22.627-22.627c-6.248-6.249-16.379-6.249-22.628 0L216 308.118l-70.059-70.059c-6.248-6.248-16.379-6.248-22.628 0l-22.627 22.627c-6.248 6.248-6.248 16.379 0 22.627l104 104c6.249 6.249 16.379 6.249 22.628.001z"></path>
        </svg>
        Multi-Department / Project
    </div>
</div>

I am using below code but it returns me blank strings. May be because of svg tag in between:

product_features = ""
divs = response.xpath("//div[@class='feature has-feature']")
for div in divs:
    product_features = product_features + div.xpath("./text()").extract_first().strip() + "|"
product_features = product_features.strip("|")

Upvotes: 1

Views: 102

Answers (3)

nyov
nyov

Reputation: 1462

In [1]: [text.strip() for text in response.xpath("//div[@class='feature has-feature']/text()[normalize-space()]").extract()]
Out[1]: ['"What If" Scenarios', 'Balance Sheet']

(A nice XPath cheat-sheet can be found here: https://devhints.io/xpath )

Upvotes: 0

ThunderMind
ThunderMind

Reputation: 799

for div in divs:
    product_features = product_features + div.xpath("./text()").extract_first().strip() + "|"
product_features = product_features.strip("|")

You are using extract_first() which return the first element while if you check your extract() there are three values as [u'\n ', u'\n "What If" Scenarios\n ']

To get your value use,

txt = [val for val in div.xpath("./text()").extract() if val.strip()]
product_features = product_features + txt + '|'
product_features = product_features.strip('|')

Upvotes: 1

vezunchik
vezunchik

Reputation: 3717

You can use css selector for short. Check this, for example:

>>> [i.strip() for i in response.css("div.feature.has-feature ::text").extract() if i.strip()]
[u'"What If" Scenarios', u'Balance Sheet']

Upvotes: 1

Related Questions