CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Alex is funny
Alex is funny

Reputation: 197

Getting none and "[]" as a response for my webscraping

So I am trying to scrape osu! stats from my friends profile, when I trying running the code I get "None" here is the code

from bs4 import BeautifulSoup
import requests



html_text = requests.get('https://osu.ppy.sh/users/17906919').text
soup = BeautifulSoup(html_text, 'lxml')
stats = soup.find_all('dl', class_ = 'profile-stats__entry')
print(stats)

Upvotes: 0

Views: 88

Answers (1)

αԋɱҽԃ αмєяιcαη
αԋɱҽԃ αмєяιcαη

Reputation: 11515

The desired data is actually presented within the HTML source under the following script tag"

<script id="json-user" type="application/json">

So we would need to pickup it and parse it as JSON below:

import requests
from bs4 import BeautifulSoup
from pprint import pprint as pp
import json


def main(url):
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'lxml')
    goal = json.loads(soup.select_one('#json-user').string)
    pp(goal['statistics'])


main('https://osu.ppy.sh/users/17906919')

Output:

{'country_rank': 18133,
 'global_rank': 94334,
 'grade_counts': {'a': 159, 's': 99, 'sh': 9, 'ss': 6, 'ssh': 2},
 'hit_accuracy': 97.9691,
 'is_ranked': True,
 'level': {'current': 83, 'progress': 84},
 'maximum_combo': 896,
 'play_count': 9481,
 'play_time': 347925,
 'pp': 3868.29,
 'rank': {'country': 18133},
 'ranked_score': 715205885,
 'replays_watched_by_others': 0,
 'total_hits': 1086843,
 'total_score': 3896191620}

Upvotes: 1

Related Questions