CODEWITHSUNDEEP
pythonproxyurllib
gm1
gm1

Reputation: 265

Setting proxy to urllib.request (Python3)

How can I set proxy for the last urllib in Python 3. I am doing the next

from urllib import request as urlrequest
ask = urlrequest.Request(url)     # note that here Request has R not r as prev versions
open = urlrequest.urlopen(req)
open.read()

I tried adding proxy as follows :

ask=urlrequest.Request.set_proxy(ask,proxies,'http')

However I don't know how correct it is since I am getting the next error:

336     def set_proxy(self, host, type):
--> 337         if self.type == 'https' and not self._tunnel_host:
    338             self._tunnel_host = self.host
    339         else:

AttributeError: 'NoneType' object has no attribute 'type'

Upvotes: 14

Views: 69516

Answers (5)

Pierz
Pierz

Reputation: 8168

Urllib will automatically detect proxies set up in the environment - so one can just set the HTTP_PROXY variable either in your environment e.g. for Bash:

export HTTP_PROXY=http://proxy_url:proxy_port

Or if you're worried about the polluting the environment you can set the variable per process:

HTTP_PROXY=http://proxy_url:proxy_port your_python_app

or using Python e.g.

import os
os.environ['HTTP_PROXY'] = 'http://proxy_url:proxy_port'

Note from the urllib docs: "HTTP_PROXY[environment variable] will be ignored if a variable REQUEST_METHOD is set; see the documentation on getproxies()"

Upvotes: 5

adramazany
adramazany

Reputation: 674

It is possible to setting global proxy for any request on your python code by setting it like below code :

import os
proxy = 'http://<user>:<pass>@<proxy>:<port>'
os.environ['http_proxy'] = proxy 
os.environ['HTTP_PROXY'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTPS_PROXY'] = proxy

Upvotes: 1

abdallah jarwan
abdallah jarwan

Reputation: 21

import urllib.request
def set_http_proxy(proxy):
    if proxy == None: # Use system default setting
        proxy_support = urllib.request.ProxyHandler()
    elif proxy == '': # Don't use any proxy
        proxy_support = urllib.request.ProxyHandler({})
    else: # Use proxy
        proxy_support = urllib.request.ProxyHandler({'http': '%s' % proxy, 'https': '%s' % proxy})
    opener = urllib.request.build_opener(proxy_support)
    urllib.request.install_opener(opener)

proxy = 'user:pass@ip:port'
set_http_proxy(proxy)

url  = 'https://www.httpbin.org/ip'
request = urllib.request.Request(url)
response = urllib.request.urlopen(request)
html = response.read()
html

Upvotes: 1

Alexander Taubenkorb
Alexander Taubenkorb

Reputation: 3329

I needed to disable the proxy in our company environment, because I wanted to access a server on localhost. I could not disable the proxy server with the approach from @mhawke (tried to pass {}, None and [] as proxies).

This worked for me (can also be used for setting a specific proxy, see comment in code).

import urllib.request as request

# disable proxy by passing an empty
proxy_handler = request.ProxyHandler({})
# alertnatively you could set a proxy for http with
# proxy_handler = request.ProxyHandler({'http': 'http://www.example.com:3128/'})

opener = request.build_opener(proxy_handler)

url = 'http://www.example.org'

# open the website with the opener
req = opener.open(url)
data = req.read().decode('utf8')
print(data)

Upvotes: 15

mhawke
mhawke

Reputation: 87134

You should be calling set_proxy() on an instance of class Request, not on the class itself:

from urllib import request as urlrequest

proxy_host = 'localhost:1234'    # host and port of your proxy
url = 'http://www.httpbin.org/ip'

req = urlrequest.Request(url)
req.set_proxy(proxy_host, 'http')

response = urlrequest.urlopen(req)
print(response.read().decode('utf8'))

Upvotes: 23

Related Questions