CODEWITHSUNDEEP
pythonbeautifulsoupweb-crawler
Lukas Scholz
Lukas Scholz

Reputation: 55

Getting empty result when searching for <span> with bs4

Im want to use bs4 in my Flask-App for searching a specific span.

I never used bs4 before so I'm a little bit confused why I don't get any results for my search.

from bs4 import BeautifulSoup

url = "https://www.mcfit.com/de/fitnessstudios/studiosuche/studiodetails/studio/berlin-lichtenberg/"
html_content = requests.get(url).text
soup = BeautifulSoup(html_content, "lxml")

spans = soup.find_all('span', {'class': 'sc-fzoXWK hnKkAN'})
print(spans)

The class 'sc-fzoXWK hnKkAN' only contains 1 span. When I execute I only get a []as result.

Upvotes: 1

Views: 72

Answers (1)

Isma
Isma

Reputation: 15190

Those contents are dynamically generated using javascript so using requests to retrieve the HTML will just retrieve the static contents, you can combine BeautifulSoup with something like Selenium to achieve what you want:

Install selenium:

pip install selenium

And then retrieve the contents using the Firefox engine or any other that supports javascript:

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Firefox()
driver.get('https://www.mcfit.com/de/fitnessstudios/studiosuche/studiodetails/studio/berlin-lichtenberg/')


html_content = driver.page_source
soup = BeautifulSoup(html_content, "lxml")

elems = soup.find_all('div', {'class': 'sc-fzoXWK hnKkAN'})
print(elems)

If you use Firefox, the geckodriver needs to be accessible by your script, you can download it from https://github.com/mozilla/geckodriver/releases and put it in your PATH (or c:/windows if you are using this OS) so it is available from everywhere.

Upvotes: 1

Related Questions