CODEWITHSUNDEEP
javasocketslarge-data
David K
David K

Reputation: 1346

How to send very large objects over socket

My system involves sending very large amounts of data and metadata over TCP sockets. I am sending a single object which contains various metadata, as well as twenty byte arrays. I've been working on small examples (130kB arrays) so far using ObjectOutputStream enclosing a BufferedOutputStream. I tried getting up to my goal of 67MB arrays, but ObjectOutputStream can't handle that much data. I was able to send 1MB arrays, but writeObject() hung up at 2MB.

I've heard that DataOutputStream is faster, but I would rather not have to encode my metadata as bytes and decode on the other side. Is there a good alternative to be able to send very large objects quickly?

EDIT:

I am already using reset() after each write method. I am currently sending a single SignedObject which encloses my own Message object which encloses the 20 arrays as well as a number of metadata fields. I would prefer to keep this all packaged since it is so convenient, but I understand that separation may be the only way to go. That being said, I would still need to make sure that all transmissions are secure.

UPDATE:

So I went to lunch, came back, and decided I'd try to figure out whether things were hanging on the send or receive end. And then it never got hung up. I could continue to successfully send up to 4MB arrays until I ran out of heap (which is unrelated and just means I need a better computer or more efficient processing). So I guess the problem went away? Doesn't mean I don't need a better way of transmitting this though, so I'll take more ideas if people have them. Also, I'm assuming as soon as I update this, it will stop working again...

Upvotes: 1

Views: 4017

Answers (1)

Peter Lawrey
Peter Lawrey

Reputation: 533820

Is there a good alternative to be able to send very large objects quickly?

Use ObjectOutputStream with reset() between objects sent. Unless you only have a few MB of memory (inwhich cas eyou should get more) you won't have a problem.

ObjectOutputStream is pretty slow compared to encoding the bytes yourself, but it can handle the encoding just fine. It's more likely that you network upload speed is your limiting factor. In this example it is sending about 600 Mb/s whereas most braodband connection are around 500 kb/s (about 1000x slower)

ServerSocket ss = new ServerSocket(0);
Socket s = new Socket("localhost", ss.getLocalPort());
final Socket s2 = ss.accept();
ss.close();

new Thread(new Runnable() {
    @Override
    public void run() {
        int count = 0;
        long start = System.nanoTime();
        try {
            ObjectInputStream in = new ObjectInputStream(
                    new InflaterInputStream(s2.getInputStream()));
            while (in.readObject() != null) count++;
            s2.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
        long time = System.nanoTime() - start;
        System.out.printf("Read %,d objects in %.3f seconds%n", count, time / 1e9);
    }
}, "reader").start();
ObjectOutputStream oos = new ObjectOutputStream(
        new DeflaterOutputStream(s.getOutputStream()));
for (int i = 0; i < 20; i++) {
    byte[] bytes = new byte[67 * 1000 * 1000];
    oos.writeObject(bytes);
    oos.reset();
}
oos.writeObject(null); // poison pill
oos.close();
s.close();

prints

Read 20 objects in 21.644 seconds

Upvotes: 2

Related Questions