Reputation: 1429
Splash doesn't work with Privoxy/Tor. Although
yield SplashRequest(url,
self.parse_func, args={'wait': 2.5, 'proxy':
'http://a_proxy_address:port', })
. yield
scrapy.Request(url, callback=self.parse_func, meta={'proxy':
'http://127.0.0.1:8118'})
). In a script Splash give error 502.
If try in browser open Splash page localhost:8050
, it give errror page:
Privoxy was unable to socks5t-forward your request http://localhost:8050/ through localhost: SOCKS5 request failed
/etc/privoxy/config:
forward-socks5 / localhost:9050 .
forward-socks4 / localhost:9050 .
forward-socks4a / localhost:9050 .
forward-socks5t / localhost:9050 .
I also tried to add in /etc/privoxy/config
the following line, but it didn't help.
forward localhost .
Upvotes: 0
Views: 1182
Reputation: 146510
Your problem is that you are using splash and passing it a localhost proxy. When splash tries to use 127.0.0.1:8118
, this refers to the splash container itself and there is nothing running there. So it fails.
Two possible solutions
Run splash on host network
sudo docker run --net host scrapinghub/splash
Give proxy IP
If you laptop/PC has a 192.168.0.101
IP from the router then use
yield SplashRequest(url,
self.parse_func, args={'wait': 2.5, 'proxy':
'http://192.168.0.101:8118', })
Upvotes: 2