CODEWITHSUNDEEP
pythonscrapyweb-crawlerscreen-scrapinguser-agent
B.Mr.W.
B.Mr.W.

Reputation: 19628

Scrapy Python Set up User Agent

I tried to override the user-agent of my crawlspider by adding an extra line to the project configuration file. Here is the code:

[settings]
default = myproject.settings
USER_AGENT = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"


[deploy]
#url = http://localhost:6800/
project = myproject

But when I run the crawler against my own web, I notice the spider did not pick up my customized user agent but the default one "Scrapy/0.18.2 (+http://scrapy.org)". Can any one explain what I have done wrong.

Note:

(1). It works when I tried to override the user agent globally: 

scrapy crawl myproject.com -o output.csv -t csv -s USER_AGENT="Mozilla...."

(2). When I remove the line "default = myproject.setting" from the configuration file, and run scrapy crawl myproject.com, it says "cannot find spider..", so I feel like the default setting should not be removed in this case.

Thanks a lot for the help in advance.

Upvotes: 43

Views: 57276

Answers (2)

Bletch
Bletch

Reputation: 516

Just in case anyone lands here that manually controls the scrapy crawl. i.e. you do not use the scrapy crawl process from the shell...

$ scrapy crawl myproject

But insted you use CrawlerProcess() or CrawlerRunner()...

process = CrawlerProcess()

or 

process = CrawlerRunner()

then the user agent, along with other settings, can be passed to the crawler in a dictionary of configuration variables.

Like this...

    process = CrawlerProcess(
            {
                'USER_AGENT': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
            }
    )

Upvotes: 5

paul trmbrth
paul trmbrth

Reputation: 20748

Move your USER_AGENT line to the settings.py file, and not in your scrapy.cfg file. settings.py should be at same level as items.py if you use scrapy startproject command, in your case it should be something like myproject/settings.py

Upvotes: 54

Related Questions