CODEWITHSUNDEEP
javaproxyjsoup
Tushar
Tushar

Reputation: 1176

JSoup Set Proxy IP during crawling

I have written a code that crawls through the site using Jsoup and the code is working absolutely fine.

The website which I am crawling has different information on the page when accessed on my Local (India IP) and some different information when accessed via my hosted server (US Ip). Is there any way I can set the proxy setting so that it appears to be hitting the site from India from the server too.

Upvotes: 0

Views: 1839

Answers (1)

Stuart Caie
Stuart Caie

Reputation: 2913

JSoup uses the java.net.HttpURLConnection class to create web connections. That class uses system properties to configure proxy connections.

Assuming you have an HTTP/HTTPS web proxy running on myproxyhost port 80, you could do the following:

System.setProperty("http.proxyHost", "myproxyhost");
System.setProperty("http.proxyPort", "80");
System.setProperty("https.proxyHost", "myproxyhost");
System.setProperty("https.proxyPort", "80");

This will cause subsequent Jsoup.connect() calls to make the HTTP/HTTPS request via the proxy, instead of connecting directly.

Upvotes: 3

Related Questions