CODEWITHSUNDEEP
javafile-upload
Satya
Satya

Reputation: 19

how to upload >50gb file using java

We have a requirement to upload a >=50GB file to a server, probably to a file system. To achieve this we can split the file into chunks and merge them after upload.This is what I came up with after long googling.

Is there any best way to upload such big files? (keeping in mind performance issues).

Upvotes: 1

Views: 1520

Answers (2)

Stephen C
Stephen C

Reputation: 719641

Here are some options to consider:

  • HTTP is a probably the most commonly used protocol for uploads and downloads.

  • FTP can be a bit faster, but that is more of a reflection on web server design than on the protocol.

  • Explicit compressing might improve upload times.

  • There are a few less well known protocols and frameworks that offer faster data transfer by various means. This page describes 4 open source alternatives.

  • There are numerous commercial tools for "fast" file transfer.

Note however:

  1. The "faster" mechanisms tend to achieve their speed by monopolizing network bandwidth; i.e. they are typically not "good citizens" in the network world.
  2. There is typically a bottleneck in any network communication path that places an upper bound on your upload times. If you are talking about uploading from people's home systems, the bottleneck is likely to be the upload data rate on the user's ISP link ... which is typically a lot slower than the download data rate.

Upvotes: 2

mbarthelemy
mbarthelemy

Reputation: 12913

Using the FTP protocol may prove to have only advantages:

-it's a standard one

-I'm sure there is a lot of libraries in java (my answer is based on what exists for .NET)

-it has an "append" method which allows you to restart your transfer if it was interrupted (you can first check the size of your partially uploaded file, and then know where to re-start reading to complete the missing part)

Upvotes: 3

Related Questions