CODEWITHSUNDEEP
javaurlproxyweb-crawlerjsoup
Vito
Vito

Reputation: 299

Setting proxy to use JSOUP fails

I try to set the proxy to conduct the scrapping google new search .

However, it appears the errors:

Exception in thread "main" java.lang.RuntimeException: Uncompilable source code - Erroneous sym type: org.jsoup.Connection.proxy.userAgent.ignoreHttpErrors.followRedirects.timeout.ignoreContentType.get
    at javaapplication27.JavaApplication27.main(JavaApplication27.java:47)

RED LINE ERROR-->

cannot find symbol .Symbol :method proxy( proxy )

location:interface connection

on this line:

 Document document = Jsoup.connect(string+"&start="+(j+0)*10)
        .proxy(proxy)
        .userAgent(userAgent)
        .ignoreHttpErrors(true)
        .followRedirects(true)
        .timeout(100000)
        .ignoreContentType(true)
        .get();

_

Proxy proxy = new Proxy(                                      //
    Proxy.Type.HTTP,                                      //
    InetSocketAddress.createUnresolved("127.0.0.1", 8080) //
);
for (int j=0;j<3;j++) {
    Document document = Jsoup.connect(string+"&start="+(j+0)*10)
        .proxy(proxy)
        .userAgent(userAgent)
        .ignoreHttpErrors(true)
        .followRedirects(true)
        .timeout(100000)
        .ignoreContentType(true)
        .get();
    Elements links = document.select( ".r>a");
    ......
 }

My imports

import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.net.URLDecoder;
import java.net.URLEncoder;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import static java.util.concurrent.TimeUnit.*;

How to fix?

Upvotes: 0

Views: 1849

Answers (1)

Jonathan Hedley
Jonathan Hedley

Reputation: 10522

The .proxy() method was first available in jsoup 1.9.1. What version are you using?

Also when I copied your code to test it, I found there are invisible zero-width spaces throughout which could be causing the syntax errors you're getting (your question talks about both missing interfaces and syntax errors).

Upvotes: 1

Related Questions