CODEWITHSUNDEEP
scrapycron
Aliyar Özercan
Aliyar Özercan

Reputation: 13

Setting up a Crontab for scrapy

I am trying to set up a crontab for scraping something. So far, I wrote

23 18 * * * cd PycharmProjects/untitled/Project1 && scrapy crawl xx -o test.csv

But when I do that I get this: 

/bin/sh: scrapy: command not found.

What should I do?
I tried to locate the scrapy in my mac but couldn't find it. But I am able run the second part of the crontab task from terminal.

Upvotes: 1

Views: 1801

Answers (1)

Granitosaurus
Granitosaurus

Reputation: 21436

Since crontab doesn't set up PATH variable for you, it doesn't know what scrapy is.

The easy way to remedy is to use full path of scrapy:

$ which scrapy
/usr/bin/scrapy

Then use that instead of just scrapy:

23 18 * * * cd PycharmProjects/untitled/Project1 && /usr/bin/scrapy crawl xx -o test.csv

Another way of doing this is to set the PATH environment in your crontab:

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# or your custom path, check your `.bashrc` for PATH you have set in your shell
23 18 * * * cd PycharmProjects/untitled/Project1 && scrapy crawl xx -o test.csv

Sidenote:
Also it's very common in cron to wrap your command in some sort of script that populates the PATH and other configurations and calling that script in cron instead of calling the commands directly.

Upvotes: 2

Related Questions