CODEWITHSUNDEEP
pythonpython-3.xxpathweb-scraping
Vinay Varma
Vinay Varma

Reputation: 75

getting text value using xpath with python

from lxml import html
import requests
url = 'https://www.bloomberg.com/quote/SPX:IND'
page = requests.get(url)
tree = html.fromstring(page.content)
num = tree.xpath('//*[@id="root"]/div/div/section[2]/div[1]/div/section[1]/section/section[2]/section/div[1]/span[1]/text()')
print (num)

this is the code I have written. I'm trying to get the string 2758.82,from this. but what I get is.

[]

I copied the xpath for that section from the website. I have seen similar questions here, but they didn't help. Is something wrong with my code?

Upvotes: 0

Views: 4188

Answers (1)

Arount
Arount

Reputation: 10431

It's not about the xpath. It's about how the page is generated.

If you check the content of page.content you will see there is no <div id="root" [..]> in the webpage's source. It's because the HTML content is mainly generated via Javascript.

But this is not something that should stop you, if you open the raw html source (from page.content) and look for the value you want (2759.81), you will find a tag: <meta itemprop="price" content="2759.82" /> and another <div class="price">2759.81</div>, you can use one of them:

print(tree.xpath('//*[@itemprop="price"]')[0].get('content'))

Gives

2759.82

Upvotes: 2

Related Questions