CODEWITHSUNDEEP
pythonscrapy
booleantrue
booleantrue

Reputation: 129

How to implement proxyrack with scrapy

Recently I bought IP rotation service from proxyrack and I want to use with Scrapy. But as their python example, I'm getting confused to implement with Scrapy. Please help me. Here is their code but I want to apply with scrapy

import requests

username = "vranesevic"
password = "svranesevic"

PROXY_RACK_DNS = "megaproxy.rotating.proxyrack.net:222"

urlToGet = "http://ip-api.com/json"

proxy = {"http":"http://{}:{}@{}".format(username, password, PROXY_RACK_DNS)}

r = requests.get(urlToGet , proxies=proxy)

print("Response:\n{}".format(r.text))

Upvotes: 1

Views: 992

Answers (1)

just don't be user123
just don't be user123

Reputation: 380

you can follow scrapy documentation that how to set up a custom proxy and if you are not familiar with then here are the steps...

Step 1 - Go to Middlewares.py file and paste this. Change the URL 1st and use that provided from proxyrack and keep the HTTP. Also, set the proxy rack user and password inside basic_auth_header.

from w3lib.http import basic_auth_header

class CustomProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta[“proxy”] = "http://192.168.1.1:8050"
        request.headers[“Proxy-Authorization”] = 
                          basic_auth_header(“<proxy_user>”, “<proxy_pass>”)

Step 2 - Go to settings.py file and Enable Downloader_middleware or paste this at the bottom. Also, make sure you replace the word myproject and set your project name.

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.CustomProxyMiddleware': 350,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}

That's it and you are ready to go.

Upvotes: 1

Related Questions