Beautifulsoup parsing error

Question

I am trying to extract some information about an App on Google Play and BeautifulSoup doesn't seem to work.

The link is this(say): https://play.google.com/store/apps/details?id=com.cimaxapp.weirdfacts

My code:

url = "https://play.google.com/store/apps/details?id=com.cimaxapp.weirdfacts"
r = requests.get(url)
html = r.content
soup = BeautifulSoup(html)
l = soup.find_all("div", { "class" : "document-subtitles"})
print len(l)
0 #How is this 0?! There is clearly a div with that class

I decided to go all in, didn't work either:

i = soup.select('html body.no-focus-outline.sidebar-visible.user-has-no-subscription div#wrapper.wrapper.wrapper-with-footer div#body-content.body-content div.outer-container div.inner-container div.main-content div div.details-wrapper.apps.square-cover.id-track-partial-impression.id-deep-link-item div.details-info div.info-container div.info-box-top')
print i

What am I doing wrong?

Answer 1

You need to pretend to be a real browser by supplying the User-Agent header:

import requests
from bs4 import BeautifulSoup

url = "https://play.google.com/store/apps/details?id=com.cimaxapp.weirdfacts"
r = requests.get(url, headers={
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
})
html = r.content
soup = BeautifulSoup(html, "html.parser")

title = soup.find(class_="id-app-title").get_text()
rating = soup.select_one(".document-subtitle .star-rating-non-editable-container")["aria-label"].strip()

print(title)
print(rating)

Prints the title and the current rating:

Weird Facts
Rated 4.3 stars out of five stars

---

To get the additional information field values, you can use the following generic function:

def get_info(soup, text):
    return soup.find("div", class_="title", text=lambda t: t and t.strip() == text).\
        find_next_sibling("div", class_="content").get_text(strip=True)

Then, if you do:

print(get_info(soup, "Size"))
print(get_info(soup, "Developer"))

You will see printed:

1.4M
Email email@here.com