CODEWITHSUNDEEP
javaarraysperformanceserializationnative
Eeth
Eeth

Reputation: 39

Effiecient serialization of native java arrays with java.io

I gotta question about efficient serialization with java.io. Efficient mainly in terms of computation-time. What is the fastest way to serialize and deserialize native arrays like long[], int[], short[] etc. to disk?

My arrays are of size between 100mb upto 600mb. What i can do is of course to use my outstream with out.writeInt(...) and out.readInt(...), but i guess there must be a faster (buffered?!) way, as i know i am reading in a complete array.

All i know, is that the framework gives me a DataInput which can be casted to InputStream. The rest is handled by my framework.

Thanks in advance,

Eeth

Upvotes: 1

Views: 418

Answers (2)

MoveFast
MoveFast

Reputation: 3025

In case you don't want to stick with Java Serialization, for this particular use case Ascii Serialization also sounds good.

Decide on a simple format for data, and Use a Buffered Writer/Reader to Serialize/Deserialize. I bet that will be faster than using Java Serialization.

Plus the added advantage you get is you can see your serialized data.

Upvotes: 0

Peter Lawrey
Peter Lawrey

Reputation: 533660

Unfortunately there is only one method to use for each type with DataInput which is readLong etc.

If you could use memory mapped files and native byte ordering that would be much more efficient all round.

In previous tests I have done, reading long with a memory mapped ByteBuffer (with native byte ordering) was 6x faster than using DataInputStream.

In fact parsing text from a ByteBuffer was faster than using DataInputStream to read binary. :P

http://vanillajava.blogspot.com/2011/06/writing-human-readable-data-faster-than.html

Upvotes: 1

Related Questions