CODEWITHSUNDEEP
pythonbeautifulsouppython-requests
user8569142
user8569142

Reputation:

'set' object has no attribute 'setdefault'. Error in scraping data using using Requests

I am trying to hit a website using python Requests, but it's giving me error. 

import requests 
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36'}
URL = ""
PROXY = {'https://surfproxy.de.db.com:8080' }
response = requests.get(URL , proxies = PROXY, headers: headers)

The error logs:

File "", line 1, in runfile('C:/Users/vermanjb/JiraScrapping.py', wdir='C:/Users/vermanjb')

File "C:\Program Files\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 699, in runfile execfile(filename, namespace)

File "C:\Program Files\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 88, in execfile exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)

File "C:/Users/vermanjb/JiraScrapping.py", line 12, in response = requests.get(URL , proxies = PROXY)

File "C:\Program Files\Anaconda3\lib\site-packages\requests\api.py", line 67, in get return request('get', url, params=params, **kwargs)

File "C:\Program Files\Anaconda3\lib\site-packages\requests\api.py", line 53, in request return session.request(method=method, url=url, **kwargs)

File "C:\Program Files\Anaconda3\lib\site-packages\requests\sessions.py", line 459, in request prep.url, proxies, stream, verify, cert

File "C:\Program Files\Anaconda3\lib\site-packages\requests\sessions.py", line 619, in merge_environment_settings proxies.setdefault(k, v)

AttributeError: 'set' object has no attribute 'setdefault'

Upvotes: 1

Views: 4358

Answers (1)

ron rothman
ron rothman

Reputation: 18127

There are at least two problems you should fix:

  1. The proxies parameter should be a dict, not a set.
  2. You have a syntax error in your headers parameter.

Try this instead:

headers = {'User-Agent': 'Mozilla/5.0...'}
url = 'http://www.yoursite.com/'
proxies = {
    'http': 'http://surfproxy.de.db.com:8080',
    'https': 'http://surfproxy.de.db.com:8080'
}
response = requests.get(url , proxies=proxies, headers=headers)

Upvotes: 1

Related Questions