CODEWITHSUNDEEP
pythonscrapy
user709203
user709203

Reputation: 81

Follow to the redirect links in scrapy

How i can set this? i have written handle_httpstatus_list = [301, 302, 303], but scrapy does not follow to the new links, it just has begun to save empty files of pages.

P.S. English is not my native language. Exuse me for that.

Upvotes: 8

Views: 11733

Answers (2)

Steven Almeroth
Steven Almeroth

Reputation: 8192

Scrapy does handle redirects by default, meaning that the RedirectMiddleware is activated in the DOWNLOADER_MIDDLEWARES setting and the REDIRECT_ENABLED setting is set to True.

Upvotes: 7

Orochi
Orochi

Reputation: 397

You will also need to mention dont_redirect = true in your request as: 

meta = {'dont_redirect': True, "handle_httpstatus_list" : [301, 302, 303]}

Regards

Upvotes: 6

Related Questions