CODEWITHSUNDEEP
javanetwork-programmingdata-transfer
Vitek
Vitek

Reputation: 628

Large amount of data - what is the best way to send them?

we have this scenario:

A server which contains needed data and client component which these data wants.

On the server are stored 2 types of data: - some information - just a couple of strings basically - binary data

We have a problem with getting binary data. Both sides are written in Java 5 so we have couple of ways....

Web Service is not the best solution because of speed, memory etc...

So, What would you prefer?

I would like to miss low level socket connection if possible...

thanks in advance

Vitek

Upvotes: 4

Views: 12212

Answers (8)

Turnkey
Turnkey

Reputation: 9406

Take a look at the W3C standard MTOM to transfer binary data as part of a SOAP service. It is efficient in that it sends as a binary and can also send as buffered chunks. It will also interop with other clients or providers:

How to do MTOM Interop

Server Side - Sending Attachments with SOAP

Upvotes: 5

plinth
plinth

Reputation: 49189

Consider GridFTP as your transport layer. See also this question.

Upvotes: 0

zenazn
zenazn

Reputation: 14345

Is sneakernet an option? :P

RMI is well known for its ease-of-use and its memory leaks. Be warned. Depending on just how much data we're talking about, sneakernet and sockets are both good options.

Upvotes: 0

Toby Hede
Toby Hede

Reputation: 37133

I think the only way to do LARGE amounts of data is going to be with raw socket access.

You will hit the Out of Memory issues on large files with most other methods.

Socket handling is really pretty straight forward in Java, and it will let you stream the data without loading the entire file into memory (which is what happens behind the scenes without your own buffering).

Using this strategy I managed to build a system that allowed for the transfer of arbitrarily large files (I was using a 7+ GB DVD image to test the system) without hitting memory issues.

Upvotes: 6

Maurizio
Maurizio

Reputation: 81

What about the old, affordable and robust FTP? You can for example easily embed an FTP server in your server-side components and then code a FTP client. FTP was born exactly for that (File Transfer Protocol, isn't it?), while SOAP with attachments was not designed with that stuff in mind and can perform very badly. For example you could have a look at:

http://mina.apache.org/ftpserver/

But there are other implementations out there, Apache Mina is just the first one I can recall.

Good luck & regards

Upvotes: 0

frankodwyer
frankodwyer

Reputation: 14048

Some options:

  • You could use RMI which will hide the socket level stuff for you, and perhaps gzip the data...but if the connection fails it won't resume for you. Probably will encounter memory issues too.

  • just HTTP the data with a binary mime type (again perhaps configuring gzip on the webserver). similar problem on resume.

  • spawn something like wget (I think this can do resume)

  • if the client already has the data (a previous version of it), rsync would copy only the changes

Upvotes: 1

martinus
martinus

Reputation: 18013

You might want to have a look at protobuf, this is the library that google uses to exchange data. Its very efficient and extensible. On a sidenote, Never underestimate the bandwidth of a station wagon full of 1TB harddisks!

Upvotes: 2

Chris Bunch
Chris Bunch

Reputation: 89823

I've tried converting the binary data to Base64 and then sending it over via SOAP calls and it's worked for me. I don't know if that counts as a web service, but if it does, then you're pretty much stuck with sockets.

Upvotes: 1

Related Questions