CODEWITHSUNDEEP
pythonpaginationscrapyxmlhttprequest
Rahul Sharma
Rahul Sharma

Reputation: 857

Scrapy Pagination XHR 400 Bad Request

I am trying to fetch all url from https://www.magzter.com/magazines/listAllIssues/503

In one set, Page show 12 magzines and scroll paginate and proceed with next 12 magazines

After Debugging, Upcoming request are as follows 

https://www.magzter.com/magazines/listAllIssues/503/12
https://www.magzter.com/magazines/listAllIssues/503/24

xml Request

But get request to https://www.magzter.com/magazines/listAllIssues/503/12 through 

400 Bad Request

Is there any implementation of this scenario in scrapy please provide a sample script.

or any other library which stimulate infinite scrolling and work with scrapy framework

Upvotes: 0

Views: 419

Answers (1)

Tarun Lalwani
Tarun Lalwani

Reputation: 146510

The issue is that the request is a AJAX request and not sending it X-Requested-With: XMLHttpRequest header makes it a 400 bad request. There is no way to send headers directly from shell command line, so you need to launch shell and type commands to fetch the request with headers

$ scrapy shell --nolog

>>> from scrapy import Request
>>> req = Request("https://www.magzter.com/magazines/listAllIssues/146/12", headers = {"X-Requested-With" : "XMLHttpRequest"})
>>> fetch(req)
>>> response.body
b'\r\n\t<div class="mag-wrap">\r\n    \t<h3></h3>\r\n    \t<ul class="mag-list main-magz">\r\n            <li>\r\n                        <div class="cover-wrap"><a href="https://www.magzter.com/IN/EFY-Enterprises-Pvt-Ltd/Electronics-For-You/Science/183025" onclick="ga(

Upvotes: 1

Related Questions