CODEWITHSUNDEEP
pythoncronscrapy
user891260
user891260

Reputation: 15

scrapy script called from cron only constructor called

scrapy crawler is called through a shell script, which is used as the command line in a crontab entry. The shell script looks like:

scrapy crawl targethost.com

when time is due and it did execute, but seems only the constructor is called (I verified with debug output). The problem is solved by re-write the shell script as:

scrapy crawl targethost.com &> cronlog.log

I just don't know why.

Upvotes: 0

Views: 366

Answers (1)

naeg
naeg

Reputation: 4002

Scrapy executes correctly, but doesn't output all its messages to STDOUT, so the simple pipe (>) doesn't redirect everything into your file, only that stuff that goes to STDOUT (which as you say, seems to be the constructor only).

With &> it fetches all messages from scrapy and puts them into your log.

Upvotes: 1

Related Questions