CODEWITHSUNDEEP
javaweb-scrapingjsoup
dani
dani

Reputation: 34

Why I cant get all page

On page https://www.jogossantacasa.pt/web/Placard/placard, I am trying get the links of Futebol->.... This I can but this only scrapes one page on the for loop. Thanks to all.

public class main {

    static List<String> links=new ArrayList<>();
    static List<String> ligas=new ArrayList<>();
    static String url="https://www.jogossantacasa.pt"; //main link

    public static void main(String[] args) {
        // TODO Auto-generated method stub
        Document doc;
        // Here i get the links
        try {
            doc = Jsoup.connect(url+"/web/Placard/placard").get();
            Elements a = doc.getElementsByClass("width9");
            boolean qwerty = true;
            for(Element ele : a) {
                Elements k = ele.select("li");      
                for(Element d : k)
                {   
                    String hj = d.select("a").text();
            
                    if(hj.contains("Ténis")) qwerty = false;
                    if(qwerty) {
                        if(!hj.contains("Futebol")) {
                            links.add(d.select("a").attr("href"));
                            ligas.add(hj);
                        }
                    }
                }
            }
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        // Here I try to scrape each country page and error is only the last page is scraped
        for(int i = 0 ; i < links.size() ; i++) {

            String urlEach=url+links.get(i);
            Document docEach;

            try {
                docEach = Jsoup.connect(urlEach).get();
                System.out.println(docEach.toString());
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }       
        }
    }
}

Upvotes: 0

Views: 74

Answers (1)

Krystian G
Krystian G

Reputation: 2941

The first page (/web/Placard/eventos?id=23316) is big, over 3MB. Jsoup downloads only the first 1MB of this file. To overcome this limit set higher maxBodySize on connection or 0 to disable the limit.

docEach = Jsoup.connect(urlEach).maxBodySize(10*1024*1024).get(); // 10MB

Upvotes: 1

Related Questions