Limiting InputStream

Question

I'm quite new to Java. I wrote an Swing application which downloads specific files over internet.

What I'd like to do is to limit InputStream to read n (lets say 10240 bytes, this might change though) bytes per second.

Once the whole file is read, it should save the file into a local directory such as C:\Downloads\

I'm guessing I should create a class which extends to InputStream and override its methods but I'm not exactly sure how and which methods to achieve my goal.

I use this example for HTTP downloads and this example for FTP.

HTTP Example;

package net.codejava.networking;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;

/**
 * A utility that downloads a file from a URL.
 * @author www.codejava.net
 *
 */
public class HttpDownloadUtility {
    private static final int BUFFER_SIZE = 4096;

    /**
     * Downloads a file from a URL
     * @param fileURL HTTP URL of the file to be downloaded
     * @param saveDir path of the directory to save the file
     * @throws IOException
     */
    public static void downloadFile(String fileURL, String saveDir)
            throws IOException {
        URL url = new URL(fileURL);
        HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
        int responseCode = httpConn.getResponseCode();

        // always check HTTP response code first
        if (responseCode == HttpURLConnection.HTTP_OK) {
            String fileName = "";
            String disposition = httpConn.getHeaderField("Content-Disposition");
            String contentType = httpConn.getContentType();
            int contentLength = httpConn.getContentLength();

            if (disposition != null) {
                // extracts file name from header field
                int index = disposition.indexOf("filename=");
                if (index > 0) {
                    fileName = disposition.substring(index + 10,
                            disposition.length() - 1);
                }
            } else {
                // extracts file name from URL
                fileName = fileURL.substring(fileURL.lastIndexOf("/") + 1,
                        fileURL.length());
            }

            System.out.println("Content-Type = " + contentType);
            System.out.println("Content-Disposition = " + disposition);
            System.out.println("Content-Length = " + contentLength);
            System.out.println("fileName = " + fileName);

            // opens input stream from the HTTP connection
            InputStream inputStream = httpConn.getInputStream();
            String saveFilePath = saveDir + File.separator + fileName;

            // opens an output stream to save into file
            FileOutputStream outputStream = new FileOutputStream(saveFilePath);

            int bytesRead = -1;
            byte[] buffer = new byte[BUFFER_SIZE];
            while ((bytesRead = inputStream.read(buffer)) != -1) {
                outputStream.write(buffer, 0, bytesRead);
            }

            outputStream.close();
            inputStream.close();

            System.out.println("File downloaded");
        } else {
            System.out.println("No file to download. Server replied HTTP code: " + responseCode);
        }
        httpConn.disconnect();
    }
}

FTP Example;

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.apache.commons.net.ftp.FTP;
import org.apache.commons.net.ftp.FTPClient;

/**
 * A program demonstrates how to upload files from local computer to a remote
 * FTP server using Apache Commons Net API.
 * @author www.codejava.net
 */
public class FTPDownloadFileDemo {

    public static void main(String[] args) {
        String server = "www.myserver.com";
        int port = 21;
        String user = "user";
        String pass = "pass";

        FTPClient ftpClient = new FTPClient();
        try {

            ftpClient.connect(server, port);
            ftpClient.login(user, pass);
            ftpClient.enterLocalPassiveMode();
            ftpClient.setFileType(FTP.BINARY_FILE_TYPE);

            // APPROACH #1: using retrieveFile(String, OutputStream)
            String remoteFile1 = "/test/video.mp4";
            File downloadFile1 = new File("D:/Downloads/video.mp4");
            OutputStream outputStream1 = new BufferedOutputStream(new FileOutputStream(downloadFile1));
            boolean success = ftpClient.retrieveFile(remoteFile1, outputStream1);
            outputStream1.close();

            if (success) {
                System.out.println("File #1 has been downloaded successfully.");
            }

            // APPROACH #2: using InputStream retrieveFileStream(String)
            String remoteFile2 = "/test/song.mp3";
            File downloadFile2 = new File("D:/Downloads/song.mp3");
            OutputStream outputStream2 = new BufferedOutputStream(new FileOutputStream(downloadFile2));
            InputStream inputStream = ftpClient.retrieveFileStream(remoteFile2);
            byte[] bytesArray = new byte[4096];
            int bytesRead = -1;
            while ((bytesRead = inputStream.read(bytesArray)) != -1) {
                outputStream2.write(bytesArray, 0, bytesRead);
            }

            success = ftpClient.completePendingCommand();
            if (success) {
                System.out.println("File #2 has been downloaded successfully.");
            }
            outputStream2.close();
            inputStream.close();

        } catch (IOException ex) {
            System.out.println("Error: " + ex.getMessage());
            ex.printStackTrace();
        } finally {
            try {
                if (ftpClient.isConnected()) {
                    ftpClient.logout();
                    ftpClient.disconnect();
                }
            } catch (IOException ex) {
                ex.printStackTrace();
            }
        }
    }
}

What I'd like is to limit the download rate of these 2 input streams so downloading speed would be limited while getting these files from remote location.

I'd really appreciate if you could show me an example with a short explanation.

Answer 1

You can actually extend java.util.TimerTask instead of extending InputStream. Then using a java.util.Timer you can schedule a task which is executed once per second until the whole file downloaded. So simply you can define the number of bytes you want to be downloaded in a second. Sometimes the download may be lower than the limit you provided (due to a network delay or so) but this is not a conflict because in every situation your download rate would be below the limit you've provided. Moreover you can change the download rate whenever you want, even during the download process. A sample of HTTP/HTTPS implementation is as follow:

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.text.NumberFormat;
import java.util.Timer;
import java.util.TimerTask;

public class DownloaderWithLimit extends TimerTask
{
    private InputStream inputStream;
    private OutputStream outputStream;

    private int bytePerSecondLimit;
    private Timer timer;
    private float contentLength;
    private float downloadedLength;

    public DownloaderWithLimit (String urlToDownload, String destFileFullPath, int bytePerSecondLimit) throws IOException
    {
        this.bytePerSecondLimit = bytePerSecondLimit;
        //
        inputStream = createInputStreamFromUrl ( urlToDownload );
        outputStream = new FileOutputStream ( new File ( destFileFullPath ) );
    }

    public void start(){
        timer = new Timer();
        timer.scheduleAtFixedRate(this, 1000, 1000);
    }

    public void run () 
    {
        try
        {
            byte[] buffer = new byte[bytePerSecondLimit];
            int bytesRead = inputStream.read(buffer);
            if(bytesRead != -1){
                outputStream.write(buffer, 0, bytesRead);
            } else {
                outputStream.close();
                inputStream.close();
                timer.cancel();
            }
            downloadedLength += bytesRead;
            System.out.println (bytesRead + " bytes per second. "+ NumberFormat.getPercentInstance ().format ( downloadedLength/contentLength )+" completed...");
        }
        catch ( IOException e )
        {
            e.printStackTrace();
            throw new RuntimeException ( "Error During Download..." , e.getCause () );
        }
    }

    private InputStream createInputStreamFromUrl ( String fileUrl ) throws IOException
    {
        URL url = new URL ( fileUrl );
        HttpURLConnection httpConn = (HttpURLConnection) url.openConnection ();
        int responseCode = httpConn.getResponseCode ();

        // always check HTTP response code first
        if ( responseCode == HttpURLConnection.HTTP_OK )
        {
            String fileName = "";
            String disposition = httpConn
                    .getHeaderField ( "Content-Disposition" );
            String contentType = httpConn.getContentType ();
            contentLength = httpConn.getContentLength ();

            if ( disposition != null )
            {
                // extracts file name from header field
                int index = disposition.indexOf ( "filename=" );
                if ( index > 0 )
                {
                    fileName = disposition.substring ( index + 10 , disposition.length () - 1 );
                }
            }
            else
            {
                // extracts file name from URL
                fileName = fileUrl.substring ( fileUrl.lastIndexOf ( "/" ) + 1 , fileUrl.length () );
            }

            System.out.println ( "Content-Type = " + contentType );
            System.out.println ( "Content-Disposition = " + disposition );
            System.out.println ( "Content-Length = " + contentLength );
            System.out.println ( "fileName = " + fileName );

            // opens input stream from the HTTP connection
            InputStream inputStream = httpConn.getInputStream ();

            return inputStream;
        }

        return null;
    }

    /**
     * @return the bytePerSecondLimit
     */
    public int getBytePerSecondLimit ()
    {
        return bytePerSecondLimit;
    }

    /**
     * @param bytePerSecondLimit the bytePerSecondLimit to set
     */
    public void setBytePerSecondLimit ( int bytePerSecondLimit )
    {
        this.bytePerSecondLimit = bytePerSecondLimit;
    }

    public static void main ( String[] args ) throws IOException
    {
        DownloaderWithLimit d = new DownloaderWithLimit ( "https://download.mozilla.org/?product=firefox-46.0.1-SSL&os=win64&lang=en-US" , "c:/firefox-46.0.1_x64.exe" , 10240);//10Kb/s
        d.start ();
    }
}

And the FTP sample with rate limit is as follow:

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.text.NumberFormat;
import java.util.Timer;
import java.util.TimerTask;

import org.apache.commons.net.ftp.FTP;
import org.apache.commons.net.ftp.FTPClient;


public class FtpDownloaderWithLimit extends TimerTask
{
    private InputStream inputStream;
    private OutputStream outputStream;

    private int bytePerSecondLimit;
    private Timer timer;
    private float contentLength;
    private float downloadedLength;
    private FTPClient ftpClient;

    public FtpDownloaderWithLimit (String ftpServer, int port, String username, String password, String srcFileRelativePath, String destFileFullPath, int bytePerSecondLimit) throws IOException
    {
        this.bytePerSecondLimit = bytePerSecondLimit;
        //
        inputStream = createInputStreamFromUrl ( ftpServer, port, username, password, srcFileRelativePath );
        outputStream = new FileOutputStream ( new File ( destFileFullPath ) );
    }

    public void start(){
        timer = new Timer();
        timer.scheduleAtFixedRate(this, 1000, 1000);
    }

    public void run () 
    {
        try
        {
            byte[] buffer = new byte[bytePerSecondLimit];
            int bytesRead = inputStream.read(buffer);
            if(bytesRead != -1){
                outputStream.write(buffer, 0, bytesRead);
            } else {
                boolean success = ftpClient.completePendingCommand();
                if (success) {
                    System.out.println("File #2 has been downloaded successfully.");
                }
                outputStream.close();
                inputStream.close();
                timer.cancel();
            }
            downloadedLength += bytesRead;
            System.out.println (bytesRead + " bytes per second. "+ NumberFormat.getPercentInstance ().format ( downloadedLength/contentLength )+" completed...");
        }
        catch ( IOException e )
        {
            e.printStackTrace();
            throw new RuntimeException ( "Error During Download..." , e.getCause () );
        }
    }

    private InputStream createInputStreamFromUrl(String ftpServer, int port,
            String username, String password, String srcFileRelativePath) throws IOException{

        ftpClient = new FTPClient();
        ftpClient.connect(ftpServer, port);
        ftpClient.login(username, password);
        ftpClient.enterLocalPassiveMode();
        ftpClient.setFileType(FTP.BINARY_FILE_TYPE);

        InputStream inputStream = ftpClient.retrieveFileStream(srcFileRelativePath);

        return inputStream;
    }

    /**
     * @return the bytePerSecondLimit
     */
    public int getBytePerSecondLimit ()
    {
        return bytePerSecondLimit;
    }

    /**
     * @param bytePerSecondLimit the bytePerSecondLimit to set
     */
    public void setBytePerSecondLimit ( int bytePerSecondLimit )
    {
        this.bytePerSecondLimit = bytePerSecondLimit;
    }

    public static void main ( String[] args ) throws IOException
    {
        FtpDownloaderWithLimit d = new FtpDownloaderWithLimit ( "www.myserver.com" , 9111 /*sample port*/, "USERNAME", "PASSWORD", "/My/File/To/Downlaod.xxx", "c:/your-path-to-dest-file" , 10240);//10Kb/s
        d.start ();
    }

}

Hope this would be helpful.

Answer 2

you could try the read(byte[] b, int off, int len) of any InputStream.len is the maximum bytes read. off is the position the stream starts inserting and b is the byte[] in which stream writes so if you want to read only 1000 bytes you could try using

int whereYouWantToStartInserting = 0;
int maximumOfBytes;
read(buffer,whereYouWantToStartInserting,maximumOfBytes);

I hope this code might help you.

Answer 3

This is a bit primitive, but it should do what you're asking for (didn't test though).

int bytesReadSinceSleep = 0;
long lastSleepTime = System.currentTimeMillis();

while ((bytesRead = inputStream.read(buffer)) != -1) {
    outputStream.write(buffer, 0, bytesRead);

    // increment bytes read this second
    bytesReadSinceSleep += bytesRead;

    // if we've passed the throttling point, sleep for the remainder of the second
    if (bytesReadSinceSleep >= MAX_BYTES_PER_SECOND) {

        // calculate time elapsed since last sleep
        long timeElapsed = System.currentTimeMillis() - lastSleepTime;

        // sleep for the remainder of 1 second (if there is a remainder)
        Thread.sleep(Math.max(1000 - timeElapsed, 0));

        // reset byte count
        bytesReadSinceSleep = 0;

        // reset sleep time
        lastSleepTime = System.currentTimeMillis();
    }
}

If BUFFER_SIZE is not a factor of MAX_BYTES_PER_SECOND, and you're particular about the throttle rate, you may want to use the read() overload that accepts offset and limit parameters to read exactly the right amount.

---

For a cleaner solution, you can use Guava's RateLimiter:

RateLimiter limiter = RateLimiter.create(MAX_BYTES_PER_SECOND);
while ((bytesRead = inputStream.read(buffer)) != -1) {
    outputStream.write(buffer, 0, bytesRead);
    limiter.acquire(bytesRead);
}