CODEWITHSUNDEEP
pythonajaxxmlhttprequestscrapy
SreeSindhu Sruthi
SreeSindhu Sruthi

Reputation: 21

How to get data for XMLHTTP requests using scrapy

A web page where the data I am intersted in has some XHR requests. I have got the url of the XHR request using developer tools. Now how to implement the same in my spider.py file and how to get all the values in that.I have started with scrapy unaware of working with ajax post requests as this. Please guide me on how to get the data and want the fields available to be stored in an excel file.

Upvotes: 2

Views: 1539

Answers (1)

Muttonchop
Muttonchop

Reputation: 392

You need to simulate a web request from inside your script. To do this, you can do something like this:

url = 'your url string'
req = scrapy.Request(url,
                     method='POST',
                     body='{"filters": []}',
                     headers={'X-Requested-With': 'XMLHttpRequest',
                              'Content-Type': 'application/json; charset=UTF-8'},
                     callback=self.parser2)
yield req

This is just an example and will probably not correspond to your exact use case. You'll need to investigate the actual request to determine what headers to send, the body text, and whether you want to do a callback or not.

Upvotes: 3

Related Questions