CODEWITHSUNDEEP
pythonscrapy
nivekdrol
nivekdrol

Reputation: 41

rotating proxies with scrapy with authentication

just a noob question but I can't seem to find the answer googling. how can i use this package https://pypi.org/project/scrapy-rotating-proxies/ if the proxy requires a user/password? do I just put it in the rotating list like that?

ROTATING_PROXY_LIST = [
    'https://username:password@proxy:port',
    'https://username:password@proxy:port',
    # ...
]

thanks

Upvotes: 3

Views: 1168

Answers (1)

Mubashir
Mubashir

Reputation: 31

what worked for me is using the rotating proxy list path:

ROTATING_PROXY_LIST_PATH = 'path/to/proxylist.txt'

and then in the text file, I have my list formatted like this:

USERNAME:PASSWORD@HOST:PORT
USERNAME:PASSWORD@HOST:PORT
USERNAME:PASSWORD@HOST:PORT

Upvotes: 3

Related Questions