CODEWITHSUNDEEP
python-3.xurllib
Evg
Evg

Reputation: 3080

hangs on open url with urllib (python3)

I try to open url with python3:

import urllib.request
fp = urllib.request.urlopen("http://lebed.com/")

mybytes = fp.read()    
mystr = mybytes.decode("utf8")
fp.close()

print(mystr)

But it hangs on second line. What's the reason of this problem and how to fix it?

Upvotes: 9

Views: 6147

Answers (2)

Mark T
Mark T

Reputation: 105

Agree with Arpit Solanki. Shown output for a failed request vs successful.

Failed
    GET / HTTP/1.1
    Accept-Encoding: identity
    Host: www.lebed.com
    Connection: close
    User-Agent: Python-urllib/3.5

Success
    GET / HTTP/1.1
    Accept-Encoding: identity
    Host: www.lebed.com
    Connection: close
    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36

Upvotes: 4

Arpit Solanki
Arpit Solanki

Reputation: 9931

I suppose the reason is that the url does not support robot visiting a site visit. You need to fake a browser visit by sending browser headers along with your request

import urllib.request
url = "http://lebed.com/"
req = urllib.request.Request(
    url, 
    data=None, 
    headers={
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
    }
)
f = urllib.request.urlopen(req)

Tried this one on my system and it works.

Upvotes: 6

Related Questions