CODEWITHSUNDEEP
javafileftp
PAUL WHITE
PAUL WHITE

Reputation: 333

how to upload multiple files using java

I am hoping someone can help me (once again).

I have a very large number of smmll files (over 4000) each only a few K. I have writen an FTP program in java which will transfer each file individually but it is taking a very long time. Also the handshaking overhead seems to make the problem worse.

What I would like to be able to do is open the FTP connection send all the files then close it again.

I know that this is possible in FTP but quite how to acheive this in java is beyond me.

I currently have the filenames in an array so parsing through them is no problem. I have tried calling the following class and passing it the filename but after several hours it was still moving about 1 file per second.

    package website;

    import java.io.BufferedOutputStream;
    import java.io.FileInputStream;
    import java.io.IOException;
    import java.net.MalformedURLException;
    import java.net.URL;
    import java.net.URLConnection;

    public class ftpUpload {

public ftpUpload(String target, String savename, String localFilePath) {

    URL url;
    try {
        url = new URL(target + savename + ";type=i");

        URLConnection con = url.openConnection();
        BufferedOutputStream out = 
            new BufferedOutputStream(con.getOutputStream());
        FileInputStream in = 
            new FileInputStream(localFilePath + savename);

        int i = 0;
        byte[] bytesIn = new byte[1024];
        while ((i = in.read(bytesIn)) >= 0) {
            out.write(bytesIn, 0, i);
        }
        out.close();
        in.close();

    } catch (MalformedURLException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }

}

    }

Is there a way I can open the connection with the ftp site username and password, then send it the files and finally close the connection?

This would seem to me easier than creating multiple threads to send files concurrently.

Any advice greatfully received.

Paul

Upvotes: 4

Views: 10061

Answers (2)

PAUL WHITE
PAUL WHITE

Reputation: 333

After plenty of testing I have found reliability issues with multiple ftp threads to public servers which is what I need in this case. Most (if not all) ftp servers limit the maximum number of connections and also limit the maximum number of concurrent connections from the same IP address. Two concurrent connections from the same IP seems to be the only guaranteed maximum you are allowed. The realistic option as suggested above is to zip the files and ftp a single file. You can unzip the file when it gets there using an php script (as long as the server supports unzip you will need to check if this included in the php build). Finally if like me you need to upload in excess of 10,000 files many ftp servers will not show more than 9998 files (10,000 inlcuding . and ..) If anyone knows of an ftp server free or cheap that supports ZipArchive in the php build and will list more than 9998 files when requesting a file listing in ftp can you please let me know.....

Upvotes: 0

the_ien
the_ien

Reputation: 84

I don't think it's possible to send multiple files in one session using URLConnection, this means you get the overhead of opening and closing the session for every file.

FTPClient from commons net does support multiple operations in one session. For example (exception handling ommitted):

FTPClient ftp = new FTPClient();
ftp.connect("ftp.example.com");
ftp.login("admin", "secret");
ftp.setFileType(FTPClient.BINARY_FILE_TYPE);

for(File file : files) {
    InputStream in = new FileInputStream(file);
    ftp.storeFile(file.getName(), in);
    in.close();
}

ftp.disconnect();

This should help.

If you still need better performance, I don't see any other alternative than using multiple threads.

Upvotes: 1

Related Questions