CODEWITHSUNDEEP
scrapyscrapy-shell
Computer's Guy
Computer's Guy

Reputation: 5363

Set headers for scrapy shell request

I know that you can scrapy shell -s USER_AGENT='custom user agent' 'http://www.example.com' to change the USER_AGENT, but how do you add request headers?

Upvotes: 20

Views: 13807

Answers (1)

eLRuLL
eLRuLL

Reputation: 18799

there is no current way to add headers directly on cli, but you could do something like:

$ scrapy shell
...
...
>>> from scrapy import Request
>>> req = Request('yoururl.com', headers={"header1":"value1"})
>>> fetch(req)

This will update the current shell information with that new request.

Upvotes: 46

Related Questions