CODEWITHSUNDEEP
javaweb-crawlercrawler4j
Ali Hashemi
Ali Hashemi

Reputation: 3368

crawler4j always returns fatal transport error

This is what I get for any seed I add to crawler4j.

ERROR [Crawler 1] Fatal transport error: Connection to http://example.com refused while fetching http://example.com/page.html (link found in doc #0)

This is really weird for me. I don't know what causes it.

Upvotes: 4

Views: 957

Answers (1)

Peter Duan
Peter Duan

Reputation: 56

try to increase the politeness setting of the crawler. It always happens to me when the server that I want to crawler is not good.

Upvotes: 3

Related Questions