CODEWITHSUNDEEP
pythonpandasbeautifulsoup
Cmagelssen
Cmagelssen

Reputation: 660

Failing to scrape a table with beautifulsoup

I am trying to scrape a table on a webpage, which I want to turn into a pandas DataFrame. The page I am trying to scrape requires authentication but I have managed to pass it using the request package. Next, I want to scrape the table and I found using the dev tools in Chrome. I have copied the selector and passed it to the soup selector() method. However, when I print it out, it returns an empty string. I have tried several different approaches, all of which fail to give me the table that I so desperately want. What am I doing wrong? This is my code:

import requests
from bs4 import BeautifulSoup as bs


cookies = {
#some information
}

headers = {
 #some information
}

params = (
    #some information
)
response = requests.get('http://www.hctiming.com/myphp/resources/login/browse_results.php?live_action=yes&smartphone_action=no', headers=headers, cookies=cookies, verify=False)

soup = bs(response.content, features="lxml")

test = soup.select("#fis_result_content0 > div.row.racers-list-tab.ranking > div > div > table")
print(test)

And this is a screenshot of my dev.tools in chrome, just to give you some idea about where my table sits:

enter image description here

Upvotes: 1

Views: 48

Answers (1)

Sãúrâßh Jáðhàv
Sãúrâßh Jáðhàv

Reputation: 112

Instead of this:

test = soup.select("#fis_result_content0 > div.row.racers-list-tab.ranking > div > div > table")

Try this :

soup.find("table",class_="ranking_table")

Upvotes: 2

Related Questions