JSoup not able to get links from html

Question

I'm trying to get links from html of a site but unable to do so using Jsoup.

This is the HTML:


    
  Domain
Link
  
    Server m1
Watch This Link
  
    
  
    Server m2
Watch This Link
  
  
              
      Xstreamcdn
Watch This Link
    
          
      Mixdrop
Watch This Link
    
          
      StreamSBWatch This Link
    
          
      DoodstreamWatch This Link

This is the android code that I wrote which doesn't seem to work:

try {
                Document doc = Jsoup.connect(URL).get();
                Elements content = doc.getElementsByClass("anime_muti_link");
                Elements links = content.select("a");

                String[] urls = new String[links.size()];
                for (int i = 0; i < links.size(); i++) {
                    urls[i] = links.get(i).attr("data-video");
                    if (!urls[i].startsWith("https://")) {
                        urls[i] = "https:" + urls[i];
                    }
                }
                arrayList.addAll(Arrays.asList(urls));
                Log.d("CALLING_URL", "Links: " + Arrays.toString(urls));

            } catch (IOException e) {
                e.getMessage();
            }

Can someone please help me with this? Thanks

Edit: Basically I'm trying to get those 6 links and add them to my list to use it within the app.

Edit 2:

So I found another HTML that can seems better:


      Servers
     
      Netu
      VideoVard
      Doodstream
      Okstream

jccampanero · Accepted Answer

As you can see, in this li definition you are including a nested div:


      XstreamcdnWatch This Link

This is causing that the variable content, the HTML fragment with class anime_muti_link, to look like:

 
  
  
   
    Domain
   
   
    Link
    
   
    
     Server m1
    
Watch This Link  
   
    
     Server m2
    
Watch This Link  
   Xstreamcdn

A similar result will be obtained even if you tidy your HTML. I used this code from one of my previous answers:

Tidy tidy = new Tidy();
tidy.setXHTML(true);
tidy.setIndentContent(true);
tidy.setPrintBodyOnly(true);
tidy.setInputEncoding("UTF-8");
tidy.setOutputEncoding("UTF-8");
tidy.setSmartIndent(true);
tidy.setShowWarnings(false);
tidy.setQuiet(true);
tidy.setTidyMark(false);

org.w3c.dom.Document htmlDOM = tidy.parseDOM(new ByteArrayInputStream(html.getBytes()), null);

OutputStream out = new ByteArrayOutputStream();
tidy.pprint(htmlDOM, out);
String tidiedHtml = out.toString();
// System.out.println(tidiedHtml);

Document document = Jsoup.parse(tidiedHtml);
Elements content = document.getElementsByClass("anime_muti_link");
System.out.println(content);

And this is why you are finding only three anchors.

Please, try correcting your HTML or selecting the anchor tag as the document level instead:

Document document = Jsoup.parse(html);
// Elements content = document.getElementsByClass("anime_muti_link");
// System.out.println(content);
Elements links = document.select("a");

String[] urls = new String[links.size()];
for (int i = 0; i < links.size(); i++) {
  urls[i] = links.get(i).attr("data-video");
  if (!urls[i].startsWith("https://")) {
    urls[i] = "https://" + urls[i];
  }
}
System.out.println(Arrays.asList(urls));

If the result obtained contains undesired links, perhaps you can try narrowing the selector used, something like:

document.select(".anime_muti_link a")

If this doesn't work, another possible alternative could be selecting the anchor elements with a data-video attribute, a[data-video]:

Document document = Jsoup.parse(html);
Elements videoLinks = document.select("a[data-video]");

String[] urls = new String[videoLinks.size()];
for (int i = 0; i < videoLinks.size(); i++) {
  urls[i] = videoLinks.get(i).attr("data-video");
  if (!urls[i].startsWith("https://")) {
    urls[i] = "https://" + urls[i];
  }
}
System.out.println(Arrays.asList(urls));

With your new test case, you can obtain the desired information with a very similar code:

String html = "
" +
    "      Servers
" +
    "     
" +
    "      Netu
" +
    "      VideoVard
" +
    "      Doodstream
" +
    "      Okstream
" +
    "     
" +
    "    ";

Document document = Jsoup.parse(html);
Elements videoLinks = document.select("div.heading-servers ul.servers li.server");

String[] urls = new String[videoLinks.size()];
for (int i = 0; i < videoLinks.size(); i++) {
  urls[i] = videoLinks.get(i).attr("data-vs");
  if (!urls[i].startsWith("https://")) {
    urls[i] = "https://" + urls[i];
  }
}

System.out.println(Arrays.asList(urls));

The most important part is the definition of the selector that should be applied to the parsed document, div.heading-servers ul.servers li.server in our case.

I provided a selector with many fragments, but depending on the actual use HTML it could be simplified with ul.servers li.server or even li.server.

JSoup not able to get links from html

Answers (1)

Related Questions