CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
ilmoi
ilmoi

Reputation: 2534

Beautiful Soup (Python) not seeing text inside of span

I can't figure out why BS4 is not seeing the text inside of the span in the following scenario:

My code:

stars = soup.find('span', {'class': 'github-repo-info__item', 'data-key': 'stargazers_count'}).text

also tried:

stars = soup.find('span', {'class': 'github-repo-info__item', 'data-key': 'stargazers_count'}).get_text()

Both return an empty string ''. The element itself seems to be located correctly (I can browse through parents / siblings in PyCharm debugger without a problem. Fetching text in other parts of the website also works perfectly fine. It's just the github-related stats that fail to fetch.

Any ideas?

Upvotes: 0

Views: 364

Answers (2)

RKS103
RKS103

Reputation: 1

Using bs4, we can't scrape stars rate.

After inspecting the site, please check response html. There, there is class information named "github-repo-info__item", but there is no text information.

in this case, use selenium.

Upvotes: 0

jizhihaoSAMA
jizhihaoSAMA

Reputation: 12672

Because this page use Javascript to load the page dynamically.So you couldn't get it directly by response.text

The source code of the page: enter image description here

You could crawl the API directly:

import requests

r = requests.get('https://api.github.com/repos/psf/requests')
print(r.json()["stargazers_count"])

Result:

43010

Upvotes: 2

Related Questions