CODEWITHSUNDEEP
scrapy
dedi
dedi

Reputation: 3

XPath selector returns empty list

I'm trying to scrape data from store: https://www.tibia.com/charactertrade/?subtopic=currentcharactertrades&page=details&auctionid=12140&source=overview

There is no problem with getting data from 1st and 2nd table, but when I goes down, xpath returns only empty lists.

even tried to save response in file:

scrapy fetch --nolog "https://www.tibia.com/charactertrade/?subtopic=currentcharactertrades&page=details&auctionid=3475&source=overview" > response.html

for table with skills everything works good

sword = response.xpath('//div [@class="AuctionHeader"]/a/text()').get()

but when it comes to getting for example gold value, I get only empty list:

gold = response.xpath('/html/body/div[3]/div[1]/div[2]/div/div[2]/div/div[1]/div[2]/div[5]/div/div/div[3]/div[2]/div[2]/table/tbody/tr/td/div/table/tbody/tr[2]/td/div[2]/div/table/tbody/tr[3]/td/div/text()').get()

In chrome/firefox both selectors works smooth, but in scrapy only 1st one

I know there might be some problems with data updated by javascript, but it doesn't look like this case

Upvotes: 0

Views: 538

Answers (1)

AaronS
AaronS

Reputation: 2335

Doesn't look like it's a javascript problem. Think you're not getting your XPATH selectors correct. It's best to be as specific as possible and not to use multiple nodes down. Here we can select the attribute TableContent to get the tables you want. There you can select each individual table that you require if needed.

Code Example 

table = response.xpath('//table[@class="TableContent"]')[3] 
gold_title = table.xpath('tr/td/span/text()')[2].get()
gold_value = table.xpath('tr/td/div/text()')[2].get()

output 

'Gold: '
'31,030'

Explanation

Using the class attribute TableContent, you can select which table you want. Here I've selected the table with the gold values. I've then selected each row and the specific element which has the gold value. The values are hidden behind span and div elements. get() returns a string, getall() returns a list.

Upvotes: 2

Related Questions