CODEWITHSUNDEEP
pythonmiddlewarescrapyhttp-cachingscrapyd
Acorn
Acorn

Reputation: 50557

HTTPCACHE not working when deploying scrapy project to scrapyd

When I try to use HTTPCACHE with scrapyd I get the following error:

[scrapy] WARNING: Disabled Httpcache Middlware: unable to find scrapy.cfg file to infer project data dir

Upvotes: 1

Views: 1198

Answers (1)

Acorn
Acorn

Reputation: 50557

The problem is that http caching defaults to using the relative path httpcache.

This works when you run the spider normally from the command line, but not when running it as a service.

The solution is to set the HTTPCACHE_DIR[docs] setting to an absolute path.

Upvotes: 3

Related Questions