HttpURLConnection getting disconnected unexpectedly in java

Question

Why does it not complete the download? It stops after downloading a few bytes. Why is this happening?

import java.io.BufferedInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.Authenticator;
import java.net.HttpURLConnection;
import java.net.InetSocketAddress;
import java.net.PasswordAuthentication;
import java.net.Proxy;
import java.net.URL;
import java.util.Scanner;

/*
 * To change this license header, choose License Headers in Project Properties.
 * To change this template file, choose Tools | Templates
 * and open the template in the editor.
 */
/**
 *
 * @author prashanth
 */

public class downloadManager {

public static void main(String[] args) {
    String url;
    Proxy myproxy;
    HttpURLConnection uc;
    String downloadDir, file, filename;
    byte[] data;
    Scanner inp = new Scanner(System.in);
    System.out.println("Enter the URL of the file to be downloaded: ");
    url = inp.nextLine();
    if (url.isEmpty()) {
        System.out.println("URL is not provided, please enter the url and try again");
        System.exit(-1);
    }
    try {
        myproxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("172.31.102.14", 3128));
        Authenticator auth = new Authenticator() {
            @Override
            public PasswordAuthentication getPasswordAuthentication() {
                return (new PasswordAuthentication("edcguest", "edcguest".toCharArray()));
            }
        };
        Authenticator.setDefault(auth);
        URL u = new URL(url);
        uc = (HttpURLConnection) u.openConnection(myproxy);
        uc.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");
        downloadDir = System.getProperty("user.home") + System.getProperty("file.separator") + "Downloads" + System.getProperty("file.separator");
        file = u.getFile();
        filename = file.substring(file.lastIndexOf('/') + 1);
        System.out.println("Waiting for connection...");
        if (uc.getResponseCode() == 200) {
            System.out.println("Connection OK!\nFile: " + filename + "\nFile Size: " + uc.getContentLength() + " bytes\nDownloading file...");
            data = new byte[uc.getContentLength()];
            int bytesRead = 0;
            int offset = 0;
            InputStream in = new BufferedInputStream(uc.getInputStream());
            while (offset < uc.getContentLength()) {
                bytesRead += in.read(data, offset, data.length - offset);
                if (bytesRead == -1) {
                    break;
                }
                offset += bytesRead;
                System.out.println("Finished downloading: " + bytesRead + " Bytes");
            }
            in.close();
            if (offset != uc.getContentLength()) {
                throw new IOException("Only read " + offset + " bytes. Expected " + uc.getContentLength() + " bytes");
            }
            System.out.println("Finishing Download...");

            FileOutputStream out = new FileOutputStream(downloadDir + filename);
            out.write(data);
            System.out.println("Download Completed! Find the file in your Downloads folder");
            out.flush();
            out.close();
        } else {
            System.out.println("Download failed! Error: " + uc.getResponseCode());
        }

    } catch (Exception e) {
        System.err.println("Exception: "+e.getMessage());
    }
}

}

Where did I go wrong? The while loop doesn't execute completely and the code execution stops there without going further.

Edit: Here is the example

Enter the URL of the file to be downloaded: 
http://psthomas.com/solutions/Liar_Truth.pdf
Waiting for connection...
Connection OK!
File: Liar_Truth.pdf
File Size: 60326 bytes
Downloading file...
Finished downloading: 3709 Bytes
Finished downloading: 8053 Bytes
Finished downloading: 12397 Bytes
Finished downloading: 13845 Bytes
Finished downloading: 16741 Bytes
Finished downloading: 20093 Bytes
Exception: Only read 74838 bytes. Expected 60326 bytes

And that's it, I'm back to the prompt.

Answer 1

It looks like your byteRead is adding to each other (using += operator).

        while (offset < uc.getContentLength()) {
            bytesRead += in.read(data, offset, data.length - offset);
            if (bytesRead == -1) {
                break;
            }
            offset += bytesRead;
            System.out.println("Finished downloading: " + bytesRead + " Bytes");
        }

here is your code, and here is the actual values of bytesRead and offset for each loop:

Loop count       bytesRead         offset
1                3709 Bytes        3709
2                8053 Bytes        11762
3                12397 Bytes       24159
4                13845 Bytes       38004
5                16741 Bytes       54745
6                20093 Bytes       74838

You can see that offset is adding the bytesRead to itself each loop and it's why you are seeing the big number jumps.

Change the first line of the while loop to so:

bytesRead = in.read(data, offset, data.length - offset);

So bytesRead is no longer appending to itself and offset is adding the correct values to itself.

In the current implementation, because bytesRead is adding to itself, it will never have the value of -1 when you are at EOF.

Answer 2

Not sure if this will help. I was able to read your file using the following:

/*
    From the command line use:

    a) java UFile http://forum.java.sun.com ForumsHomePage.html
    b) java UFile someFile.java someOtherFile.java

    or create your own input and output streams and invoke the
    copy(InputStream, OutputStream) method directly
*/
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URL;
import java.net.URLConnection;

public class UFile
{
    public static void main(String arg[])
     throws IOException
    {
        InputStream is = getInputStream( arg[0] );
        OutputStream os = getOutputStream( arg[1] );
        long start = System.currentTimeMillis();
        System.out.println( UFile.copy(is, os) + " bytes copied" );
        System.out.println( System.currentTimeMillis() - start );

//      new java.io.File(arg[0]).delete();

        System.exit(0);
    }

    public static InputStream getInputStream(String fileName)
        throws IOException
    {
        InputStream input;

        if (fileName.startsWith("http:"))
        {
            URL url = new URL( fileName );
            URLConnection connection = url.openConnection();
            input = connection.getInputStream();
        }
        else
        {
            input = new FileInputStream( fileName );
        }

        return input;
    }

    public static OutputStream getOutputStream(String fileName)
        throws IOException
    {
        return new FileOutputStream( fileName );
    }

    public static int copy(InputStream in, OutputStream out)
        throws IOException
    {
        int bytesCopied = 0;
        byte[] buffer = new byte[4 * 1024];
        int bytes;

        try
        {
            while ( (bytes = in.read( buffer )) != -1 )
            {
                out.write( buffer, 0, bytes );
                bytesCopied += bytes;
            }
        }
        finally
        {
            in.close();
            out.close();
        }

        return bytesCopied;
    }
}

It doesn't use a proxy so the code is simple.

I got the following output:

C:\Java>java UFile http://psthomas.com/solutions/Liar_Truth.pdf test.pdf
60326 bytes copied

Anyway, the idea is to start with working code and then make a change. If it doesn't work, then you at least know what you changed and can provide more specific information in your question.

Answer 3

Can you try in this way?

int bytesRead = -1;
byte[] buffer = new byte[4098]; // you can change it
while ((bytesRead = inputStream.read(buffer)) != -1) {
    outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();

Have a look at this article once