CODEWITHSUNDEEP
c#serialization
TrustyCoder
TrustyCoder

Reputation: 4789

serializing and deserializing large files

Is there any good approach to serialzing and de-serializing large files (>10M) in c#.

Thanks in advance.

Upvotes: 5

Views: 7419

Answers (4)

Lasse V. Karlsen
Lasse V. Karlsen

Reputation: 391366

Are you sure serialization/deserialization is the right approach for that much data? Would perhaps a client-side database, like SQLite be a better solution, where you can query it for the exact data you need, instead of just loading everything into memory?

Upvotes: 1

GalacticJello
GalacticJello

Reputation: 11445

You can check out my answer here to this question (there are all kinds of other relevant answers there too).

My method uses BinaryReader and BinaryWriter for performance.

I have used this method to deserialize 50MB files in a recent project, and it does it quite quickly (under 5 seconds) compared to the built-in serialization or Xml serialization (10 minutes for my data set).

Upvotes: 1

Matthias
Matthias

Reputation: 12259

If you really have large files (let's say larger than 100 MB), the best thing is to load only the things you need at the moment.

Let's say you have a list of 10.000 customers - each with an image. It makes no sense to keep this list in the memory.

For example, you could load all lastnames and the position of the person in the file. So the user could search for a person and you could load exactly that person.

Another possibility would be loading the first ten and display them to the user. As soon as he clicks at a "Next" button you could load the next ten - just plan how to organize the information.

Instead of very large files, databases can bring some advantages. They can abstract the large amout of work required to navigate within the file.

"Single-Line-Serialization" using BinaryFormatter etc., however, reaches its limits at files of that size in my opinion. You have to think of other concepts.

Upvotes: 1

Hans Passant
Hans Passant

Reputation: 941545

There isn't any difference between de/serializing small or large files. You would just have to make sure that you don't deserialize very large files to memory, that's going to buy you OOM.

And large files are going to take more time of course. If that makes your user interface unresponsive then you'll want to do this processing in a background thread. BackgroundWorker is a typical solution for that.

Random shots in the dark here btw, your question is far too vague.

Upvotes: 1

Related Questions