File Downloader with Java Commons IO stuck at 1MB/s+

Question

Lately I've been experimenting with Java and Commons IO trying to create a web file downloader, however I've encountered a problem, in fact it seems that the file download speed does not exceed 1MB/s while the same download from the browser runs smoothly at 3MB /s. Could you help me? I would be really grateful.

This is my downloader code:

package com.application.steammachine;

import com.github.junrar.Archive;
import com.github.junrar.Junrar;
import com.github.junrar.exception.RarException;
import com.github.junrar.rarfile.FileHeader;
import com.github.junrar.volume.FileVolumeManager;
import javafx.beans.property.SimpleStringProperty;
import javafx.concurrent.Task;
import org.apache.commons.io.IOUtils;
import org.ini4j.Wini;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.io.*;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.net.URL;

public class Downloader extends Task<Void> {

    private URL url;
    private String fileName;
    private Game game;

    public Downloader(URL url, String fileName, Game game) {
        this.url = url;
        this.fileName = fileName;
        this.game = game;
    }

    public class ProgressListener implements ActionListener {

        private double bytes = 0;
        private double mbDownloaded = 0;
        private double fileSize = 0;
        private double lastMB = 0;
        private long initialTime;
        private double speed = 0;
        private String downloadedText = "";
        private String sizeText;

        public ProgressListener(double fileSize){
            this.fileSize = fileSize;
            initialTime = System.nanoTime();
        }

        @Override
        public void actionPerformed(ActionEvent e) {
            bytes = ((DownloadCountingOutputStream) e.getSource()).getByteCount();
            updateProgress(bytes, fileSize);
            mbDownloaded = round(bytes/1e+6, 2);

            if(fileSize >= 1073741824){ //>= 1GB
                double temp = ((fileSize/1e+6)/1024);
                sizeText = round(temp,2) + " GB";
            }else {
                double temp = (fileSize/1e+6);
                sizeText = round(temp,2) + " MB";
            }

            if(mbDownloaded >= 1024){
                downloadedText = String.valueOf(round(mbDownloaded/1024,2));
            }else{
                downloadedText = String.valueOf(mbDownloaded);
            }

            if((System.nanoTime() - initialTime) >= (Math.pow(10, 9))){
                speed = round((mbDownloaded - lastMB), 3);
                initialTime = System.nanoTime();
                lastMB = mbDownloaded;
            }

            updateMessage(String.valueOf(speed)+"MB/s,"+String.valueOf(downloadedText + "/" + sizeText));
        }
    }

    @Override
    protected Void call() throws Exception {

        URL dl = this.url;
        File fl = null;
        String x = null;
        OutputStream os = null;
        InputStream is = null;

        try {

            updateMessage("Searching files...,---/---");

            fl = new File(Settings.getInstallPath() +"/"+ this.fileName);
            os = new FileOutputStream(fl);
            is = dl.openStream();

            DownloadCountingOutputStream dcount = new DownloadCountingOutputStream(os);

            double fileSize = Double.valueOf(dl.openConnection().getHeaderField("Content-Length"));

            ProgressListener progressListener = new ProgressListener(fileSize);
            dcount.setListener(progressListener);

            IOUtils.copy(is, dcount, 512000 );

            updateMessage("Concluding...,Almost finished");

        } catch (Exception e) {

            System.out.println(e);
            IOUtils.closeQuietly(os);
            IOUtils.closeQuietly(is);
            updateMessage(",");
            this.cancel(true);
            return null;

        } finally {

            IOUtils.closeQuietly(os);
            IOUtils.closeQuietly(is);


            updateMessage(",");
            updateProgress(0, 0);
            this.cancel(true);
            return null;

        }
    }


    protected static double round(double value, int places) {
        if (places < 0) throw new IllegalArgumentException();

        BigDecimal bd = new BigDecimal(Double.toString(value));
        bd = bd.setScale(places, RoundingMode.HALF_UP);
        return bd.doubleValue();
    }

}

This is the DownloadCountingOutputStream class, which i use for keeping track of the download status:

import java.awt.event.ActionListener;
import java.io.IOException;
import java.io.OutputStream;

public class DownloadCountingOutputStream extends CountingOutputStream {

    private ActionListener listener = null;

    public DownloadCountingOutputStream(OutputStream out) {
        super(out);
    }

    public void setListener(ActionListener listener) {
        this.listener = listener;
    }

    @Override
    protected void afterWrite(int n) throws IOException {
        super.afterWrite(n);
        if (listener != null) {
            listener.actionPerformed(new ActionEvent(this, 0, null));
        }
    }

}

Thanks in adavance

Answer 1

First, there are a couple of simple things you could try to speed up the transfers:

1. Try using a larger transfer buffer size. Change the 8K buffer size to 64K or 512K.

2. Get rid of the DownloadCountingOutputStream and transfer directly to the FileOutputStream.

These should be simple to try ... and they may help a bit.

---

On a Linux system, it may also be worthwhile to replace Apache IOUtils.copy call with code that uses the kernel's zero-copy transfer support. See Efficient data transfer through zero copy for an explanation.

The example in the article is for uploading using transferTo, but downloading using transferFrom should be analogous. The article claims a 65% speedup for large file transfers compared with conventional Java I/O. But that is likely to depend on the characteristics of your network connection.