CODEWITHSUNDEEP
virtualenvcronscrapy
Mutant
Mutant

Reputation: 3821

Scrapy crawl on crontab under virtual environment

Trying to run scrapy crawl command using crontab under virtual environment, and getting below error while running scrapy command from cron job -

     UserWarning: Cannot import scrapy settings module myspider.settings warnings.warn("Cannot import scrapy settings module
 %s" % scrapy_module) 
     ..... 
     raise KeyError("Spider not found: %s" % spider_name) KeyError: 'Spider not found: myspider'

Any help or suggestion?

Upvotes: 3

Views: 1752

Answers (1)

Mutant
Mutant

Reputation: 3821

It was missing PYTHONPATH while running under crontab.

I have added it before my following cron job:

*/40 * * * * source /home/water/.virtualenvs/water/bin/activate && cd $HOME/water2012/ && scrapy crawl water2012 >> $HOME/water2012/log/log_$(date +\%Y\%m\%d).log 2>&1

Upvotes: 2

Related Questions