CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouppython-requestsurllib
Sourabh Khandelwal
Sourabh Khandelwal

Reputation: 43

Why am I getting connection refused exception for this Python script?

I was writing a Python script to grab lyrics of a song from azlyrics using the request module. This is the script I wrote:

import requests, re
from bs4 import BeautifulSoup as bs
url = "http://search.azlyrics.com/search.php"
payload = {'q' : 'shape of you'}
r = requests.get(url, params = payload)
soup = bs(r.text,"html.parser")
try:
    link = soup.find('a', {'href':re.compile('http://www.azlyrics.com/lyrics/edsheeran/shapeofyou.html')})['href']
    link = link.replace('http', 'https')
    print(link)
    raw_data = requests.get(link)
except Exception as e: 
    print(e)

but I got an exception stating :

Max retries exceeded with url: /lyrics/edsheeran/shapeofyou.html (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fbda00b37f0>: Failed to establish a new connection: [Errno 111] Connection refused',))

I read on the internet that I am probably trying to send too many requests. So I made the script sleep for some time :

import requests, re
from bs4 import BeautifulSoup as bs
from time import sleep
url = "http://search.azlyrics.com/search.php"
payload = {'q' : 'shape of you'}
r = requests.get(url, params = payload)
soup = bs(r.text,"html.parser")
try:
    link = soup.find('a', {'href':re.compile('http://www.azlyrics.com/lyrics/edsheeran/shapeofyou.html')})['href']
    link = link.replace('http', 'https')
    sleep(60)
    print(link)
    raw_data = requests.get(link)
except Exception as e: 
    print(e)

but no luck!

So I tried the same with urllib.request 

import requests, re
from bs4 import BeautifulSoup as bs
from time import sleep
from urllib.request import urlopen
url = "http://search.azlyrics.com/search.php"
payload = {'q' : 'shape of you'}
r = requests.get(url, params = payload)
soup = bs(r.text,"html.parser")
try:
    link = soup.find('a', {'href':re.compile('http://www.azlyrics.com/lyrics/edsheeran/shapeofyou.html')})['href']
    link = link.replace('http', 'https')
    sleep(60)
    print(link)
    raw_data = urlopen(link).read()
except Exception as e: 
    print(e)

but then got different exception stating :

<urlopen error [Errno 111] Connection refused>

Can anyone one tell me whats wrong with it and how do I fix it?

Upvotes: 2

Views: 3571

Answers (1)

Ari Cooper-Davis
Ari Cooper-Davis

Reputation: 3485

Try it in your web browser; when you try to visit http://www.azlyrics.com/lyrics/edsheeran/shapeofyou.html it'll work fine, but when you try to visit https://www.azlyrics.com/lyrics/edsheeran/shapeofyou.html it won't work.

So remove your link = link.replace('http', 'https') line and try again.

Upvotes: 3

Related Questions