CODEWITHSUNDEEP
javaweb-scrapingjsoup
mashal
mashal

Reputation: 37

Scraping multiple pages with jsoup

I am trying to scrap links in pagination of GitHub repositories I have scraped them separately but what Now I want is to optimize it using some loop. Any idea how can i do it? here is code

ComitUrl= "http://github.com/apple/turicreate/commits/master";

 Document document2 = Jsoup.connect(ComitUrl ).get();

 Element pagination = document2.select("div.pagination a").get(0);
 String Url1    =   pagination.attr("href");
 System.out.println("pagination-link1 = " + Url1);


 Document document3 = Jsoup.connect(Url1).get();
 Element pagination2 = document3.select("div.pagination a").get(1);
 String Url2 = pagination2.attr("href");

 System.out.println("pagination-link2 = " + Url2);
 Document document4 = Jsoup.connect(Url2).get();

 Element check = document4.select("span.disabled").first();

 if (check.text().equals("Older")) {
     System.out.println("No pagination link more"); 
 }
 else { Element pagination3 = document4.select("div.pagination a").get(1);
        String Url3 = pagination3.attr("href");
        System.out.println("pagination-link3 = " + Url3);

 }

Upvotes: 0

Views: 1730

Answers (1)

Eritrean
Eritrean

Reputation: 16498

Try something like given below:

public static void main(String[] args) throws IOException{
    String url  = "http://github.com/apple/turicreate/commits/master";
    //get first link
    String link = Jsoup.connect(url).get().select("div.pagination a").get(0).attr("href");
    //an int just to count up links
    int i = 1;
    System.out.println("pagination-link_"+ i + "\t" + link);
    //parse next page using link
    //check if the div on next page has more than one link in it
    while(Jsoup.connect(link).get().select("div.pagination a").size() >1){
        link = Jsoup.connect(link).get().select("div.pagination a").get(1).attr("href");
        System.out.println("pagination-link_"+ (++i) +"\t" + link);
    }
}

Upvotes: 2

Related Questions