Trouble opening page with Python requests

Question

I am having some trouble opening a page with Python requests package. The page opens fine in a browser, but the program will just hang when attempting to get the site.

import bs4, requests

link = "http://s6.mediastreaming.it:8080/"
r = requests.get(link)
data=r.text

soup = bs4.BeautifulSoup(data)
soup.prettify()
print soup

Any help understanding why this site hangs would be greatly appreciated. The code works fine with http://google.com as the link.

Edit: Added info to help. This is a site that is streaming music. I just want to scrape the part of the page that says what the current song is. That is all. But maybe the fact that the page is a music streaming site is causing the issue? I just want the text of the source. Nothing else.

Edit 2: Tried the following to see if I was indeed getting a stream rather than the page I wanted.

import bs4, requests

link = "http://s6.mediastreaming.it:8080/"
r = requests.get(link,stream=True)

filename = "testfile.txt"

with open(filename,'wb') as fd:
    for chunk in r.iter_content(100):
        fd.write(chunk)

Here's what I got as the output:

ICY 200 OK
icy-notice1:
This stream requires Winamp

icy-notice2:SHOUTcast Distributed Network Audio Server/Linux v1.9.8

icy-name:Pig Radio - The Best Electronic & Indie Pop/Rock 24/7
icy-genre:Eclectic
icy-url:http://www.pigradio.com
content-type:audio/mpeg
icy-pub:1
icy-br:128

&Oç)goiYQŠ < 6Ê !‡À¡ö³ ‡/OFÌ)8…¨ÐU!ðiÁP¡¢ãÅ..........

When I open the page in a browser and view source I get the following:

SHOUTcast AdministratorU SHOUTcast D.N.A.S. Status
SHOUTcast Server Version 1.9.8/Linux
Status  |  Song History  |  Listen  |  Stream URL  |  Admin Login

Current Stream Information
Server Status: Server is currently up and public.
Stream Status: Stream is up at 128 kbps with 45 of 300 listeners (45 unique)
Listener Peak: 200
Average Listen Time: 7h 13m 12s
Stream Title: Pig Radio - The Best Electronic & Indie Pop/Rock 24/7
Content Type: audio/mpeg
Stream Genre: Eclectic
Stream URL: http://www.pigradio.com
Stream AIM: N/A
Stream IRC: N/A
**Current Song: Midlake - Young Bride (Cassettes Won't Listen Remix)**

 Written by Stephen 'Tag Loomis, Tom Pepper and Justin Frankel
Copyright Nullsoft Inc. 1998-2004

Obviously it's this last bit of HTML that I want to scrape for the "Current Song" How do I get just this HTML??

Edit3: Solved it! I used Wireshark to capture the GET that the browser was sending and added all the parameters I saw there to the header of my Python GET. It looks like this:

import bs4, requests, urllib2

link = "http://173.255.137.244:8080/"

filename = "testfile.txt"

payload = {'Host':'173.255.137.244:8080','User-Agent':'Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0','Accept-Encoding':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language':'en-US,en;q=0.5','Accept-Encoding':'gzip, deflate'}
r = requests.get(link,headers=payload)
data = r.text

soup = bs4.BeautifulSoup(data)
soup.prettify()
print soup

Moral of the story. Wireshark is cool.

flyer · Accepted Answer

It seems like the site has some approaches preventing spiders from crawling.

If there's no authentication needed to crawl a page and the spider has something wrong when crawling, then there're usually two ways to do:

Add http request headers when emitting a request which makes it like visiting the page by a web browser. Some sites usually check the User-Agent field to see if it's a spider.
Add some cookies needed when emitting a request. Some sites need some information from cookies when responding.

Apart from wireshark which is a little complex, observing the network requests and its headers by the chrome developer tools in chrome is a good choice.

Trouble opening page with Python requests

Answers (1)

Related Questions