CODEWITHSUNDEEP
pythonhtmlbeautifulsouppython-requests
carusot42
carusot42

Reputation: 1211

Issues downloading full HTML of webpage with Python

I'm working on a project where I require the all of the game ID #'s found in the current scores section of http://www.nhl.com/ to download content/ parse stats for each game. I want to be able to get all current game ID's in one go, but for some reason, I'm unable to download the full HTML of the page, no matter how I try. I'm using requests and beautifulsoup4.

Here's my problem:

I've determined that the particular tags I'm interested in are div's where the CSS class = 'scrblk'. So, I wrote a function to pass into BeautifulSoup.find_all() to give me, specifically, blocks with that CSS class. It looks like this:

    def find_scrblk(css_class):
        return css_class is not None and css_class == 'scrblk'

so, when I actually went to the web page in Firefox and saved it, then loaded the saved file in beautifulsoup4, I did the following:

>>>soup = bs(open('nhl.html'))
>>>soup.find_all(class_=find_scrblk)
[<div class="scrblk" id="hsb2015010029"> <div class="defaultState"....]

and everything was all fine and dandy; I had all the info I needed. However, when I tried to download the page using any of several automated methods I know, this returned simply an empty list. Here's what I tried:

With all of the above, I was unable to parse out the data that I need from the HTML files; it's as if they weren't being completely downloaded or something, but I have no idea what that something is or how to fix it. What am I doing wrong or missing here? I'm using python 2.7.9 on Ubuntu 15.04.

All of the files can be downloaded here:

https://www.dropbox.com/s/k6vv8hcxbkwy32b/nhl_html_examples.zip?dl=0

Upvotes: 1

Views: 790

Answers (1)

GHajba
GHajba

Reputation: 3691

As the comments on your question state, you have to re-think your approach. What you see in the browser is not what the response contains. The site uses JavaScript to load the information you are after so you should look more carefully in the result what you get to find what you are looking for.

In the future to handle such problems try out Chrome's developer console and disable JavaScript and open a site such way. Then you will see if you are facing JS or the site would contain the values you are looking for.

And by the way what you do is against the Terms of Service of the NHL website (according to Section 2. Prohibited Content and Activities)

Engage in unauthorized spidering, scraping, or harvesting of content or information, or use any other unauthorized automated means to compile information;

Upvotes: 1

Related Questions