CODEWITHSUNDEEP
pythonjsonweb-scrapingscrapyscrapyd
ChristianTL
ChristianTL

Reputation: 125

Custom JSON Response from Scrapy Spider Deployed via Scrapyd

I need to find a way to make my Scrapy spider return a custom JSON response. It is deployed via scrapyd using schedule.json. Schedule.json responds with JobID and Status, but I'd like to add some more data to that response.

If there's a way I could do it by creating a separate response and then make a separate request that would work as well.

Much appreciated!

Upvotes: 1

Views: 577

Answers (1)

alecxe
alecxe

Reputation: 473893

enter image description here

(taken from Learn to Read the Source, Luke).

It is doable.

Implement your own resource based on scrapyd.webservice.WsResource, write a render_POST() method. Then, tell scrapyd to use your schedule.json resource implementation:

[services]
schedule.json = project.module.MySchedule

Upvotes: 1

Related Questions