CODEWITHSUNDEEP
htmlpython-3.xxpathweb-scrapinglxml
beginsql
beginsql

Reputation: 135

Search for element in lxml tree by xpath returns empty list

I'm trying to find element by xpath and then print that piece of information but unfortunately this code returns an empty list:

response = requests.get('https://justjoin.it/offers/ulam-labs-frontend-developer')
tree = html.fromstring(response.content)
result = tree.xpath('//div[contains(@class, "css-eytwkb") and text() = "EXP. lvl"]/preceding-sibling::div[1]')
print(result)

Above Xpath works great in selenium and firefox inspector so I don't really know what's wrong with this code. And this is HTML:

<div class="css-1uvpahd">
   <div class="css-1e6hsr3">
      <svg class="MuiSvgIcon-root" focusable="false" viewBox="0 0 24 24" aria-hidden="true">
         <path d="M3.5 18.49l6-6.01 4 4L22 6.92l-1.41-1.41-7.09 7.97-4-4L2 16.99z"></path>
      </svg>
   </div>
   <div class="css-1ji7bvd">senior</div>
   <div class="css-eytwkb">EXP. lvl</div>
</div>

I want to extract senior in css-1ji7bvd class.

Upvotes: 0

Views: 99

Answers (1)

baduker
baduker

Reputation: 20052

This simply happens becasue the content of the page is dynamically rendered by JS. If you turn it off, you won't see much. That's why your Xpath doesn't work, becasue there's no such element in the source.

However, the website provides an API that you can query. Here's how to get the field you're looking for.

To get the API request URL just examine the XHR tab in your Developer Tool console and try this:

import requests
url = "https://justjoin.it/api/offers/ulam-labs-frontend-developer"
requests.get(url).json()['experience_level']

This outputs senior.

Upvotes: 1

Related Questions