CODEWITHSUNDEEP
pythonseleniumproxyscrapy
Gemcode
Gemcode

Reputation: 121

Using ProxyMesh (https://proxymesh.com ) IP in selenium chrome driver for web scraping

I have performed web-scraping using python-scrapy framework with a Proxy Mesh IP. If the proxy requires authentication I use the following code :

import base64

# Start your middleware class
class ProxyMiddleware(object):
    # overwrite process request
    def process_request(self, request, spider):
        # Set the location of the proxy
        request.meta['proxy'] = "http://....."

        # Use the following lines if your proxy requires authentication
        proxy_user_pass = "username:pwd"
        # setup basic authentication for the proxy
        encoded_user_pass = base64.encodestring(proxy_user_pass)
        request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

When I want to do the same while scraping using selenium chrome driver what is the appropriate technique that can be used. I find examples using firefox but no luck in chrome driver. Please share your ideas.

Upvotes: 2

Views: 2017

Answers (1)

Zubair Alam
Zubair Alam

Reputation: 8947

Going through documenation on proxymesh how to configure http client, you can set your profile object as follow

profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", "http://username:[email protected]")
profile.set_preference("network.proxy.http_port", "portnumber")
profile.update_preferences()

driver = webdriver.Firefox(firefox_profile=profile)

or you can use webdriver.Proxy object to set ssl credentials. Also there are options available on proxymesh configuration panel where you can add your ip address/hostname [of your server which is using proxymesh service].

Upvotes: 4

Related Questions