Possible Solutions to Poor Serialization Performance

I recently did some performance testing and analysis of an ASP.NET application using out-of-process session state - this is necessary when using session state on a web farm so that state can be retrieved on any of the web servers, e.g. if subsequent HTTP requests arrive at a different server because the sessions aren't 'sticky' or the original server is down, etc.

What surprised me was that when I ran the web servers at full load and profiled the CPU usage something like 99% of CPU time was spent serializing and deserializing session state. Subsequently we implemented a customised 'caching' state server; this always serializes state but also keeps state in-memory so that if you use sticky sessions the state doesn't have to be deserialized most of the time. This improved server throughput by a factor of 2; However, serialization still makes up 98% or more of CPU time.

We obtained some further improvements in speed by 'trimming' unnecessary object references between objects in the session state prior to serialization - fixing up the references manually upon desrialization. This improved speed by another 10-20% or so. The reasoning here is that some of the performance loss is due to the built in serialization having to walk the graph of object pointers, which becomes a more complex task with more pointers.

Continuining the investigation we wrote customized serialization routines for some of our classes rather than relying on .Net's built-in serialization. What we found was that performance was greatly improved, by a factor of about 50x. It seems the bulk of the CPU load is caused by built in .Net serialization, which in turn is slow due to reliance on using Reflection to walk the object pointers/graph and extract field data.

It's very tempting to boost our performance by 50x, thus reducing the web server hardware requirements by a large factor (and power requirements by a lesser but still significant factor). The options currently are:

1) Write customized serialization. This is an issue due to the complexity of the task and the maintenance overhead it generates, that is, any change to class state requires a change to the serialization/deserialization routines.

2) Some third party solution. Perhaps some product that automatically generates state save/load code at build time, thus eliminating the need to use Reflection.

I'd be very interested to know if anyone knows of a third party solution, or has encountered this issue as I haven't found any mention of it from internet searches.

UPDATE: Some have suggested a sort of halfway solution between the default built-in serialization and pure customized serialization routines. The idea is that you implement customized serialization for the classes that affect performance the most by, e.g. overiding ISerializable. This is an interesting and promising approach; However I still think there's scope for a complete replacement for built-in serialization without having to write and maintain any custom code - this can't be done at runtime because Reflection is needed to query objects and access private data. But it is theoretically possible to post-process already built assemblies and inject new methods as an additional build step. Some profilers use this approach to inject profiling code into assemblies after they have been built by the C# compiler. Also I /think/ I read somewhere that the .Net framework supports injecting methods into classes - thus all the messing around with IL is potentially taken care of.

Upvotes: 14

Answers (5)

Peter Mortensen

Reputation: 31616

There is a third-party solution:

Simon Hewitt's excellent open source library, see Optimizing Serialization in .NET - part 2.

I am using it in my application and got a similar speed-up as you, 20-40 times.

It eliminates the reflection that is the cause of the slowdown, but for lists it only supports a few native types. So for Genreric.List's of your own types there needs to be explicit code somewhere or the other. E.g. explicit loops or something smarter that automates this. In any case it is pretty simple and should not be hindrance for the huge benefits.

Upvotes: 1

Hugh Moran

Reputation: 1

We have encountered similar issues and have devised several ways of improving performance. We use some of these in our Persistore memory mapping product (currently beta). In our case we can simply access persisted data "in situ" because it is always in a memory mapped heap.

However, one 'trick' is to define your session state data (if possible) as marshalable class/struct and 'serialize' that using .NET marshalling support, this can be very fast indeed but won't handle 'object' graphs.

We also support a special persistence based on Binary serialization, but we also extract and persist metadata so that managed code can set/get fields within a memory persisted datum at runtime without the need to deserialize the whole object, this is useful in some settings (e.g securities and stock price updates etc). Our latest beta supports serialization across a network, of LINQ anonymous types, this is a first so far as I am aware.

Anyway, we'd be interested to have some new beta customers who are pushing ASP.NET and web performance issues, our latest beta is very impressive (but not ready until nex week).

If anyone is curious just contact us for the latest info on the product.

Hugh Moran

PS: Website is out of date, the product goes far beyond what is described there.

Upvotes: 0

Brian

Reputation: 25844

You can partially customize your serialization by implementing ISerializable. If you do this for the worst offenders, you won't increase maintenance as much but will still get some of the speedup.

Upvotes: 1