CODEWITHSUNDEEP
pythonweb-scrapingpython-requestsipproxies
Ritik Sahu Student
Ritik Sahu Student

Reputation: 13

Python requests session does not rotate proxies

I am using private rotating proxy provided by (https://proxy.webshare.io/proxy/rotating?) in which each request to rotating proxy receives a new IP address. when I am using requests.get('https://httpbin.org/get', headers=headers, proxies=get_proxy()) it returns a new IP each time whenever I make request. but when using

session = requests.Session()
session.headers = headers
session.proxies = get_proxy()

session.get('https://httpbin.org/get')

It returns same IP each time whenever I make request. How does session object behaves different from requests.get() function in case of proxies.

Upvotes: 1

Views: 2608

Answers (3)

Student
Student

Reputation: 1

I finally switch to another rotating proxy provider (https://www.proxyegg.com) and the issue has been resolved now.

Upvotes: -3

Lews
Lews

Reputation: 526

The following code works, and it take a proxylistfile.txt file to check every proxy:

from requests import *
import bs4 
import sys 

if len(sys.argv) < 2:
    print('Usage: ./testproxy.py <proxylistfile.txt>')
    sys.exit()

ifco = 'http://ifconfig.co'
PROXIES_FILE = sys.argv[1]
proxy = dict()

with open(PROXIES_FILE) as file:
    for line in file:
        if line[0] == '#' or line == "\n":
            continue
        line_parts = line.replace('\n', '').split(':')
        proxy['http'] = f'{line_parts[0]}://{line_parts[1]}:{line_parts[2]}'
        try:
            i = get(ifco, proxies=proxy, timeout=11)
            print(f"{proxy['http']} - successfull - IP ---> ", end='')
            zu = bs4.BeautifulSoup(i.text, 'html.parser')
            testo = zu.findAll('p', text=True)[0].get_text()
            print(testo)
        except:
            print(f"{proxy['http']} - unsuccessfull")
            pass

It connect ot ifconfig.co site and return its real ip to check if the proxy works. The output will be something like:

http://proxy:port - successfull - IP ---> your.real.ip

the input file format should be like:

http:1.1.1.1:3128

Upvotes: 0

aneroid
aneroid

Reputation: 16007

Session uses previously set up variables/values for each subsequent request, like Cookies. If you want to change the proxy for each request in the session, then use Prepared Requests to set it each time or just put it in a function:

def send(session, url):
    return session.get(url, proxy=get_proxy())

sess = requests.Session()
sess.headers = headers
resp = send(sess, 'https://httpbin.org/get')
print(resp.status_code)

But if you're trying to hide your origin IP for scraping or something, you probably don't want to persist cookies, etc. so you shouldn't use sessions.

Upvotes: 2

Related Questions