CODEWITHSUNDEEP
pythonhtmlweb-scraping
Pruthviraj Acharya
Pruthviraj Acharya

Reputation: 49

Getting values from one specific table from a website

I am trying to get the average price from this website. From the "2021 Tin of Ancient Battles --MP21-EN143', the table is below the "Picture Not Available" image. Orange Box outlines the able I am interested in

When I use the following code, I just get content from the first table at the top of the URL, the able underneath "Tournament Eligibility".

import pandas as pd
from bs4 import BeautifulSoup
import requests

url = 'https://yugiohprices.com/card_price?name=Triple+Tactics+Talent'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'lxml')
table = soup.findAll('table',{'id':'item_stats'})
print(table)

Is it that my code is not waiting long enough for the entire website to load? I tried waiting for 5 seconds after I request the url, but it still shows the first table at the top. Any suggestions for how to get this value?

Upvotes: 0

Views: 49

Answers (1)

HedgeHog
HedgeHog

Reputation: 25048

What happens?

Site is dynamically loading the content you like to get - Cause requests could not simulate browser behavior, it do not work to wait?

How to fix?

Information comes from another url, you can choose this:

import pandas as pd
from bs4 import BeautifulSoup
import requests
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}
url = 'https://yugiohprices.com/get_card_prices/Triple%20Tactics%20Talent'
r = requests.get(url, headers= headers)
soup = BeautifulSoup(r.text, 'lxml')
soup.select('table#item_stats tr:nth-of-type(3) td')[1].text

Alternativ and maybe better solution if you like to go deeper is selenium so you can simulate a browser.

Upvotes: 1

Related Questions