CODEWITHSUNDEEP
javaxmlweb-servicesinputstream
John
John

Reputation: 1922

How do I save a large file from a web service using Java?

I have to call a rest web service that returns a large amount of data as xml. The data is about 490m in size. Every time I try to call the service I run out of memory. All I want to do is write this data to a file.

Is there a way to read and write the data in small chunks to avoid running out of memory?

Here is what I tried;

public class GetWs {

   private static String url ="http://somewebservice";
   public static void main(String[] args) {

    InputStream in;
    OutputStream out;
    try {
          out = new FileOutputStream("testoutfile.txt");
          in = new URL(url).openStream();
          int b;
          do {
               b = in.read();
               if (b != -1) {
            out.write(b);
                 out.flush();
               }
           } while (b != -1);
            in.close();out.close();     
    } catch (Exception e) {
        e.printStackTrace();
     }

   }

}

Upvotes: 0

Views: 1778

Answers (3)

Philipp Reichart
Philipp Reichart

Reputation: 20961

If you really only want to download the contents of that URL into a File, try Google Guava and it's awesome helper methods:

URL url = ...
File file = ...
ByteStreams.copy(
    Resources.newInputStreamSupplier(url),
    Files.newOutputStreamSupplier(file));

This saves you from writing yet another copy loop with proper exception handling. There's even no need to close any streams, the ByteStreams.copy() does it for you.

If you want to store the data as UTF-16, use something like this:

Charset charsetFromServer = ...; // See notes below.

CharStreams.copy(
    Resources.newReaderSupplier(url, charsetFromServer),
    Files.newWriterSupplier(file, Charsets.UTF_16));

There's several ways to set charsetFromServer:

  • If you can trust the server to always use the same charset, manually set it by using Charset.forName(String) or one of the constants in Guava's Charsets class. Be really, really sure that the server will never use any other encoding, though, or this will break.

  • A more elaborate way is to determine the character encoding used by the server at runtime by looking at the Content-Type header. I suggest you take a look at how Apache's HttpClient does it or just use HttpClient to begin with, making this as easy as ContentType.getOrDefault(response.getEntity()).getCharset().

Upvotes: 2

yasth
yasth

Reputation: 171

If you really are just using inputstream just use 

byte[] buff = new byte[5000];
int num = 1;
while(num>1){
   num = inputStream.read(buff);
   outputStream.write(buff,0,num);
}

Though you'll need to add some code to detect when you hit the end of a file ~~~(implentation of inputstream dependent)~~~Edit no you won't, and fixed code some

Upvotes: 1

duffymo
duffymo

Reputation: 308848

Try compression and streaming to the file output stream, preferrably using NIO.

IF you have to parse and validate the XML, try a STAX parser.

Upvotes: 2

Related Questions