CODEWITHSUNDEEP
pythonscrapyscrapinghub
Sam Lee
Sam Lee

Reputation: 10463

How to pass data to scrapinghub?

I'm trying to run a scrapy spider on scrapinghub, and I want to pass in some data. I'm using their API to run the spider:

http://doc.scrapinghub.com/api/jobs.html#jobs-run-json

They have an option for job_settings, which seems relevant, but I can't figure out how to access the job_settings data in my Spider class. What is the correct approach here?

Upvotes: 2

Views: 268

Answers (1)

starrify
starrify

Reputation: 14751

This job_settings shall be merged directly into the Scrapy settings, with a higher precedence (of 40, IIRC).

The Scrapy settings could be accessed via a .settings attribute of a spider instance, e.g. you could use self.settings if self is a scrapy.Spider instance.

Upvotes: 5

Related Questions