CODEWITHSUNDEEP
pythonxpathlxml
Raza Ul Haq
Raza Ul Haq

Reputation: 344

unable to extract text from href tag using xpath

I am trying to extract trends names from this page using following xpath

//div[@class ='table-responsive']/table[@class = 'table table-striped table-hover dataTable no-footer']/tbody/tr/th/a/text()

it gives 50 results while trying on web browser. But with following code

import requests
import lxml.html

html = requests.get('https://twitter-trends.iamrohit.in/')
doc = lxml.html.fromstring(html.content)
trends_name = doc.xpath("//div[@class = 'table-responsive']/table[@class = 'table table-striped table-hover dataTable no-footer']/tbody/tr/th/a/text()")

I get nothing in trends_name variable. I have tried to print html.content and it gives raw html content. Furthermore I have tried same xpath on online xapth selector using source code of same page and it gives 50 trends I am not sure what am I doing wrong with code as I have tried it with other sites with different xpaths and it is working, kindly help. Thank you

Upvotes: 1

Views: 39

Answers (1)

Parolla
Parolla

Reputation: 407

Simply remove "dataTable" and "no-footer" class names from predicate for table - these class names added when table rendered in Browser, but absent in page source:

trends_name = doc.xpath("//div[@class = 'table-responsive']/table[@class = 'table table-striped table-hover']/tbody/tr/th/a/text()")

Upvotes: 1

Related Questions