CODEWITHSUNDEEP
javaurl
Wai Loon II
Wai Loon II

Reputation: 259

how to return a list of url from the coding below

as title shows, how do i return a list of urls under (a href) reference and display it in a text file ? The code below return the html form a a website. 

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;

public class Main {
    public static void main(String[] args)  {
        try {
            URL my_url = new URL("http://www.placeofjo.blogspot.com/");
            BufferedReader br = new BufferedReader(
               new InputStreamReader(my_url.openStream()));
            String strTemp = "";
            while(null != (strTemp = br.readLine())){
                System.out.println(strTemp);
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

Upvotes: 0

Views: 414

Answers (3)

no.good.at.coding
no.good.at.coding

Reputation: 20371

Edit (2)

If you're looking for a robust solution (or might need to extend to parsing more HTML), then check out one of the other answers here. If you just want a quick-and-dirty, one time solution you might consider regex.

If I understand you correctly, you want to extract the href values for all <a> tags in the HTML you're fetching.

You can use regular expressions. Something like 

String regex = "<a\s.*href=['\"](.*?)['\"].*?>";
Pattern p = Pattern.compile(regex);
Matcher m = p.matcher(text);
while (m.find())
{
    String urlStr = m.group();
}

Edit (1)

Corrected the regex - we want reluctant quantifiers otherwise we'll end up getting everything!

Upvotes: -1

Matthew Gilliard
Matthew Gilliard

Reputation: 9498

You sound like you want to be using an HTML parsing library like HtmlUnit, rather than getting into the hassle of parsing the HTML yourself. The HtmlUnit code would be as simple as:

final WebClient webClient = new WebClient();
webClient.setJavaScriptEnabled(false);
final HtmlPage page = webClient.getPage("http://www.placeofjo.blogspot.com/");

//  Then iterate through
for (DomElement element : page.getElementsByTagName("a")){
    String link = ((HtmlAnchor)element).getHrefAttribute();
    System.out.println(link);
}

Gives output of:

http://www.twitter.com/jozefinfin/
http://www.facebook.com/jozefinfin/
http://placeofjo.blogspot.com/2008_08_01_archive.html
... etc etc
http://placeofjo.blogspot.com/2011_02_01_archive.html
http://endlessdance.blogspot.com
http://blogskins.com/me/aaaaaa
http://weheartit.com

Upvotes: 3

Jeremy
Jeremy

Reputation: 22435

You might want to try parsing the HTML with jsoup and collect all the anchor tags from the page.

Upvotes: 1

Related Questions