CODEWITHSUNDEEP
pythondockerscrapytwistedscrapy-splash
Peter K.
Peter K.

Reputation: 597

Scrapy, Splash and Connection was refused by other side: 10061

I am using scrapy with splash on a Javascript driven site. However, I can't get passed a Connection was refused by other side: 10061 error.

I get logs like this: 

[scrapy.downloadermiddlewares.retry] DEBUG: Retrying 
 <GET https://www2.deloitte.com/ch/en/misc/search.html#country=All#qr=accounting     
 via http://localhost:8050/render.html> (failed 1 times): Connection 
 was refused by other side: 10061: No connection could be made because 
 the target machine actively refused it..

and a traceback pointing to twisted:

twisted.internet.error.ConnectionRefusedError: Connection was refused 
by other side: 10061: No connection could be made because the target 
machine actively refused it..

I have checked all the entries in settings, did try various USER_AGENTS and ROBOT entries, but no luck. Also tried to use --disable-private-mode to start splash, but no effect.

Strangely, just copy-pasting the same url into the browser works perfectly.

I used normal command line scrapy, as well as via the API. Interestingly, when using the API, of course, clicking the url of the target in the error message within PyCharm, the hashtag # is replaced by its escape-code. So I am confused whether under the hud this is another issue or whether the two are related together.

Even tried to look at the packages sent via both Wireshark and Fiddler, but was not able to understand the results well enough, as I never used these tools before.

Any suggestions would be greatly appreciated.

Upvotes: 2

Views: 3496

Answers (1)

Peter K.
Peter K.

Reputation: 597

Finally, managed to identify the culprit. It was indeed the connection to the docker container.

First, I had to retrieve the docker container IP using

docker-machine ip

in the docker terminal. Next, I had to adjust SPLASH_URL in the scrapy settings.py file to point to the docker-machine ip instead of localhost:8050, and voila ... it works.

Unfortunately, the sources I have seen so far have been rather unclear about this, so I hope this will be of some use for other poor souls setting splash up for the first time.

Upvotes: 5

Related Questions