CODEWITHSUNDEEP
linuxscrapy
Umair Ayub
Umair Ayub

Reputation: 21271

How to create file name of Feed Export dynamically using scrapy command

This is command to run my scraper.

scrapy crawl monitor -a filename="ScrapeProject.csv" -o filename.csv

It runs and saves the scraped data into filename.csv

I want to schedule as cronjob I want filename.csv to be the current datetime the scraper ran.

I tried with back-ticks but didnt work

scrapy crawl monitor -a filename="ScrapeProject.csv" -o `date`.csv

Also tried like that

scrapy crawl monitor -a filename="ScrapeProject.csv" -o "date".csv

EDIT:

Below is the command I ran upon @dps recommendation but it prompts me to enter something? 

root@ubuntu:/home/mani/pricemonitor# scrapy crawl monitor -a filename="ScrapeProject.csv" -o `date +\%m`.`date +\%d`.`date +\%y`.csv`
> 
>

Upvotes: 1

Views: 1440

Answers (2)

paul trmbrth
paul trmbrth

Reputation: 20748

Scrapy Feed Exports also understand (some built-in) storage URI parameters out of the box.

%(time)s is one of them.

So you can do something like:

scrapy crawl monitor -a filename="ScrapeProject.csv" -o '%(time)s.csv'

which will create output files in the form YYYY-mm-ddTHH-MM-SS, e.g. 2017-05-11T12-12-18.csv.

Internally, time is converted using datetime.utcnow().replace(microsecond=0).isoformat().replace(':', '-').

Note: you can use any spider attribute in your Feed URI (what you set with -o). Remember that any spider argument (stuff you can add to the command line with -a key=value) will be available as spider argument (as strings).

Upvotes: 4

a2f0
a2f0

Reputation: 1855

You aren't trying to apply any formatting to the date command in your cron job are you? You need to escape percentage signs for cron, i.e.

`date +\%m`.`date +\%d`.`date +\%y`.tar.gz

See: Percent sign % not working in crontab

Also, does it work from the command line when you don't use cron (with the ticked `date` )?

Upvotes: 4

Related Questions