CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouppython-requests
Trevor
Trevor

Reputation: 595

Using find_all to find a certain class in BS4 with BeautifulSoup

I'm trying to find all of the top players "arena pts" (the in game competative point system) from this website: https://www.epicgames.com/fortnite/competitive/en-US/hype-leaderboard?sessionInvalidated=true

I'm using requests to get the information and bs4 to sort through it, but I'm having trouble using bs4. Here's what I have so far:

page = requests.get('https://www.epicgames.com/fortnite/competitive/en-US/hype-leaderboard?sessionInvalidated=true')
soup = BeautifulSoup(page.content, 'html.parser')
results = soup.find_all('td', class_ = "asdt-points")
print(len(results))

When ran, the program outputs 0, meaning it didn't find anything.. from doing research online I haven't found any groundbreaking information that could help me, and I'm using stack as a last resort. Any help would be amazing! Thank you!

Upvotes: 0

Views: 120

Answers (2)

Pozdniakov Filipp
Pozdniakov Filipp

Reputation: 170

You wont able to do it using requests lib, because the page is asking api for those stats, which can be found in browser console "network" tab

import requests
from bs4 import BeautifulSoup
page = requests.get('https://www.epicgames.com/fortnite/competitive/en-US/hype-leaderboard?sessionInvalidated=true')
soup = BeautifulSoup(page.content,features="lxml")
soup.find("div", {"class": "ApiStandingsDetailsTables"}).text
'Loading'

So instead lets just ask the api

import requests
api_url='https://www.epicgames.com/fortnite/competitive/api/leaderboard/persistent/Hype_S14/undefined'
data_to_parse=requests.get(api_url).json()
resulting_list_of_dictionaries=[{'rank':i.get('rank'), 'name':i.get('players')[0]['displayName'], 'hype':i.get('pointsEarned')} for i in data_to_parse['entries']]

resulting_list_of_dictionaries
00:{'hype': 75434, 'name': 'Marzz_Ow', 'rank': 1}
01:{'hype': 50160, 'name': 'Its Filipе', 'rank': 2}
02:{'hype': 45821, 'name': 'たいちゃん0216', 'rank': 3}
03:{'hype': 42827, 'name': 'Fishy R2L2', 'rank': 4}
04:{'hype': 41806, 'name': 'Claw FA', 'rank': 5}
05:{'hype': 41379, 'name': '必ずアニキ救って来いやァァァァァ', 'rank': 6}
06:{'hype': 40511, 'name': 'えすぼん.', 'rank': 7}
07:{'hype': 39566, 'name': 'LG Slackes', 'rank': 8}

Upvotes: 3

Mario
Mario

Reputation: 573

This is a problem that would require using selenium, since the table is loaded dynamically with Javascript. Bs4 can only work, and would work in your case, if the data were already loaded. What is happening is that when requests gets the page, there are indeed no td.asdt-points elements. you can see that by using:

print(page.text)

Upvotes: 0

Related Questions