CODEWITHSUNDEEP
jsonscrapyparameter-passingscrapyd
eng_mazzy
eng_mazzy

Reputation: 1049

Passing json arguments to a spider in scrapy

I should pass to a spider some parameters taken from a json file. I have read that it is possible through scrapyd using schedule.json but I don't understand how to pass the json file. Someone of you have any experience?

Upvotes: 4

Views: 1543

Answers (2)

heamon7
heamon7

Reputation: 1

I had the same question(I wanted to pass a json file to the spiders to implement a simple distributed crawl system.

And I simply solved it by converting the json file to a string as a argument in scrapyd.

Upvotes: 0

marven
marven

Reputation: 1846

You don't pass the arguments using a JSON file. Scrapyd has a JSON API where you can pass arguments along with it. (e.g. $ curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider -d myargument="value")

You can handle the arguments passed through kwargs:

class MySpider(Spider):

    name = 'somespider'

    def __init__(self, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.myargument = kwargs.get('myargument', '')

See http://scrapyd.readthedocs.org/en/latest/api.html for more info.

Upvotes: 8

Related Questions