CODEWITHSUNDEEP
pythonweb-scrapingscrapy
surendhar selvam
surendhar selvam

Reputation: 11

Scrapy get's redirected to follow 302 and it does not crawl the site

Scrapy gets 302 redirect to another link. In the link 'https://xxxxxx.queue-it.net?c.....com' Scrapy does not add the '/'. It should be'https://xxxxxx.queue-it.net/?c.....com'.

I have tried adding '/' in middleware.py. Under downloaderMiddleware function. But, it does not work. Scrapy crawls when I manually add the redirect link with '/'. However, it is not very dynamic.

Upvotes: 0

Views: 125

Answers (1)

Arun Augustine
Arun Augustine

Reputation: 1766

Set 'dont_redirect': True in the specific request or disable redirect globally by setting REDIRECT_ENABLED setting to False.

Upvotes: 1

Related Questions