CODEWITHSUNDEEP
web-scrapingscrapysh
StarLord1123
StarLord1123

Reputation: 27

Problem with launching scrapy spider from sh script

When I try to run my sh script with a spider, it displays only one warning and does not parse, but when I run the spider on my own, the parsing goes fine

my sh file

#!/bin/bash
# shellcheck disable=SC2164
cd /var/www/scrapy_parser/avito/avito/spiders
scrapy crawl avito -L WARNING
cd /var/www/scrapy_parser/info/info/spiders
scrapy crawl info_v1 -L WARNING.

sh output:

WARNING: /usr/local/lib/python3.6/site-packages/scrapy/extensions/feedexport.py:210: 
ScrapyDeprecationWarning: The `FEED_URI` and `FEED_FORMAT` settings have been deprecated in 
favor of the `FEEDS` setting. Please see the `FEEDS` setting docs for more details
exporter = cls(crawler)

What can I do to fix this?

Upvotes: 1

Views: 218

Answers (1)

StarLord1123
StarLord1123

Reputation: 27

In general, I solved this problem by logging, in the end it turned out that the site that I parse blocked me as a bot and the sh script did not read the pre-installed proxies in the system, I solved everything by adding the proxy setting directly to the sh script before starting the spider

Upvotes: 1

Related Questions