CODEWITHSUNDEEP
scrapy
bayayaqd
bayayaqd

Reputation: 59

how can i use rotated proxy with scrapy?

i wrote this in setting.py after pip install scrapy-rotating-proxies

ROTATING_PROXY_LIST = ['http://209.50.52.162:9050']
 
DOWNLOADER_MIDDLEWARES = {
 
'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
'rotating_proxies.middlewares.BanDetectionMiddleware': 620
 }

then if i run spider like this scrapy crawl test it show this.

2021-05-03 15:03:32 [rotating_proxies.middlewares] WARNING: No proxies available; marking all proxies as unchecked
2021-05-03 15:03:50 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-05-03 15:03:50 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 0, unchecked: 0, reanimated: 1, mean backoff time: 0s)
2021-05-03 15:03:53 [rotating_proxies.expire] DEBUG: Proxy <http://209.50.52.162:9050> is DEAD
2021-05-03 15:03:53 [rotating_proxies.middlewares] DEBUG: Retrying <GET https://www.google.com> with another proxy (failed 3 times, max retries: 5)

how can i solve this issue ?

Upvotes: 1

Views: 2253

Answers (1)

Upendra
Upendra

Reputation: 726

Notice the message in your logs:

DEBUG: Proxy <http://209.50.52.162:9050> is DEAD

You need to add more proxies as shown in the documentation:

ROTATING_PROXY_LIST = [
    'proxy1.com:8000',
    'proxy2.com:8031',
    # ...
]

You can get a list of proxies from many sites.

Upvotes: 1

Related Questions