CODEWITHSUNDEEP
python-3.xscrapyweb-crawlerhttp-status-code-504
Raúl Bernardo
Raúl Bernardo

Reputation: 1

Error Getting Website with Scrapy

I'm trying to get a website through Scrapy in the shell, 

$ scrapy shell -s NAME="Mozilla/5.0" "http://www.yapo.cl/chile/inmuebles?ca=15_s&l=0&cmn=&st=a"

2017-08-21 20:55:07 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.yapo.cl/chile/inmuebles?ca=15_s&l=0&cmn=&st=a> (failed 3 times): 504 Gateway Time-out

but Scrapy through me a 504 error which I cannot figure out, any guess what could it be?.

Upvotes: 0

Views: 142

Answers (2)

Tom&#225;š Linhart
Tom&#225;š Linhart

Reputation: 10220

You are probably trying to set user agent string on command line, but using wrong setting (NAME). Try it with:

$ scrapy shell -s USER_AGENT="Mozilla/5.0" "http://www.yapo.cl/chile/inmuebles?ca=15_s&l=0&cmn=&st=a"

Like that, I get:

2017-08-22 07:40:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.yapo.cl/chile/inmuebles?ca=15_s&l=0&cmn=&st=a> (referer: None)

Upvotes: 1

Hebmaster
Hebmaster

Reputation: 27

You were banned or something like that. Try using another ip address. On my computer it gives this:

2017-08-22 00:07:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.yapo.cl/chile/inmuebles?ca=15_s&l=0&cmn=&st=a> (referer: None) ['partial']

Upvotes: 0

Related Questions