Java - XML Parser & Downloader

Question

I am creating a application that will download a lot of files from my own web server. But idk why it is not working. It dont have any response..

Here is some part of my code

Downloader.class

private Proxy proxy = Proxy.NO_PROXY;
    public void downloadLibrary()
        {
            System.out.println("Start downloading libraries from server...");
            try
            {
                URL resourceUrl = new URL("http://www.example.com/libraries.xml");
                DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
                DocumentBuilder db = dbf.newDocumentBuilder();
                Document doc = db.parse(resourceUrl.openConnection(proxy).getInputStream());
                NodeList nodeLst = doc.getElementsByTagName("Contents");
                for (int i = 0; i < nodeLst.getLength(); i++)
                {
                    Node node = nodeLst.item(i);

                    if (node.getNodeType() == 1)
                    {
                        Element element = (Element)node;
                        String key = element.getElementsByTagName("Key").item(0).getChildNodes().item(0).getNodeValue();
                        File f = new File(launcher.getWorkingDirectory(), key);
                        downloadFile("http://www.example.com/" + key, f, "libraries");
                    }
                }
            }
            catch(Exception e)
            {
                System.out.println("Error was found when trying to download libraries file " + e);
            }

        }

        public void downloadFile(final String url, final File path, final String fileName)
        {
            SwingWorker<Void, Void> worker = new SwingWorker<Void, Void>()
            {
                @Override
                protected Void doInBackground() throws Exception
                {
                    launcher.println("Downloading file " + fileName + "...");
                    try
                    {
                        URL fileURL = new URL(url);
                        ReadableByteChannel rbc = Channels.newChannel(fileURL.openStream());
                        FileOutputStream fos = new FileOutputStream(path);
                        fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
                    }
                    catch(Exception e)
                    {
                        System.out.println("Cannot download file : " + fileName + " " + e);
                    }
                    return null;
                }
                @Override
                public void done()
                {
                    System.out.println(fileName + " had downloaded sucessfully");

                }
            };
            worker.execute();
        }

Here is some part of my xml file(libraries.xml)

<Key>libraries/org/lwjgl/lwjgl/lwjgl/2.9.0/lwjgl-2.9.0.jar</Key>

My idea is, my application will read the XML file. Then it will download file from the server and save to computer. For example, my application download http://www.example.com/libraries/org/lwjgl/lwjgl/lwjgl/2.9.0/lwjgl-2.9.0.jar then it will save to C://WorkingDir/libraries/org/lwjgl/lwjgl/lwjgl/2.9.0/lwjgl-2.9.0.jar There is ton of <Key></Key> in my XML file and i have to download all of it.

Is that any code wrong? Thanks for helping.

Answer 1

Try consuming the connection directly through a reader of some sort to a String first, then you can manipulate that however you need.

package come.somecompany.somepackage.utils;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;

public class WebUtils {

    /**
     * Gets the HTML value of a website and
     * returns as a string value.
     * 
     * @param website website url to get.
     * @param ssl True if website is SSL.
     * @param useragent Specified User-Agent (empty string "" means use system default).
     * @return String value of website.
     */
    public String getHTML(String website, boolean ssl, String useragent) {
        String html = "";
        String temp;
        String prefix;
        if (ssl) {
            prefix = "https://";
        } else {
            prefix = "http://";
        }
        try {
            URL url = new URL(prefix + website);
            URLConnection con = url.openConnection();
        if (!(useragent.equalsIgnoreCase(""))) {
            con.setRequestProperty("User-Agent", useragent);
        }
        BufferedReader in = new BufferedReader(
                new InputStreamReader(con.getInputStream()));
        while((temp = in.readLine()) != null) {
            html += temp + "\n";
        }
        in.close();
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
        return html;
    }
}

Also, it sort of looks like you are attempting to parse HTML using an XML pattern... which may give you difficulties. You could try out JSoup - it's a java HTML parser, and works pretty well and is easy: http://jsoup.org/

It may help with consuming documents from your website without needing to build your own downloader too.

UPDATE --

Try reading into a BufferedReader, perhaps your program isn't receiving the full document, buffered reader may help.

BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));

So with your first method, something like:

public void downloadLibrary()
    {
        System.out.println("Start downloading libraries from server...");
        try
        {
            URL resourceUrl = new URL("http://www.example.com/libraries.xml");
            DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
            DocumentBuilder db = dbf.newDocumentBuilder();

            // change here
            URLConnection con = resourceUrl.openConnection();
            BufferedReader bfr = new BufferedReader(new InputStreamReader(con.getInputStream()));

            String tempDoc = "";
            String tempStr;
            while (tempStr = bfr.readLine()) != null) {
                tempDoc += tempStr + System.getProperty("line.separator");
            }

            Document doc = db.parse(tempDoc);

            NodeList nodeLst = doc.getElementsByTagName("Contents");
            for (int i = 0; i < nodeLst.getLength(); i++)
            {
                Node node = nodeLst.item(i);

                if (node.getNodeType() == 1)
                {
                    Element element = (Element)node;
                    String key = element.getElementsByTagName("Key").item(0).getChildNodes().item(0).getNodeValue();
                    File f = new File(launcher.getWorkingDirectory(), key);
                    downloadFile("http://www.example.com/" + key, f, "libraries");
                }
            }
        }
        catch(Exception e)
        {
            System.out.println("Error was found when trying to download libraries file " + e);
        }

    }