CODEWITHSUNDEEP
pythonxpathweb-scrapinglxml
Liam Flynn
Liam Flynn

Reputation: 2069

Scraping using python and xpath

I am trying to scrape data from the following website:

http://mozo.com.au/credit-cards/search#fetch/680

Using chrome's 'inspect element feature' I have been able to locate the element address I want as:

//*[@id="p-40"]/div[4]/table/tbody/tr/td[1]/text()

I was hoping using this code, I would be able to get the text "9.99%"

import requests
page = requests.get('http://mozo.com.au/credit-cards/search#fetch/680')
tree = html.fromstring(page.text)


tree.xpath('//*[@id="p-40"]/div[4]/table/tbody/tr/td[1]/text()')

However, the output is an empty array. Where am I going wrong?

Upvotes: 2

Views: 5765

Answers (1)

PascalVKooten
PascalVKooten

Reputation: 21481

Like tobifasc said, the page is loaded dynamically. Try selenium for example,

First install:

pip3 install selenium

Then:

import lxml.html
from selenium import webdriver
driver = webdriver.Firefox()
driver.get(url)

tree = lxml.html.fromstring(driver.page_source)

Now you can query:

# With your xpath there are 2 results...
results = tree.xpath('//*[@id="p-40"]/div[4]/table/tbody/tr/td[1]/text()')   
results[1].strip()
'9.99%'

Upvotes: 4

Related Questions