Lots of database access makes a WCF service unresponsive

Question

I have a WCF duplex service (session instancing with net.tcp binding) that can potentially do a lot of database access. When it goes into its high-db-access routine, it become unresponsive to any WCF service call.

I've checked the CPU utilization and, while high, it is not 100%. This is on a 4 core box.

It seems that no actual requests are getting handled by WCF. Not only are no service operations called, but I have some security custom behaviors further down the WCF stack that are not getting invoked either. This makes me think that somehow WCF is getting starved of threads and so is not able to allocate new threads for access.

I don't think this is a throttling problem, since I can make this happen when there are a few current sessions with the service (less than 5).

The DB operations in question are thousands of single row inserts. This is done with a single entity framework context that is disposed and reconstructed every 500 calls to reduce the memory accumulation of entity frameworks internal caches. The target database is SQL Express 2008 R2.

There are a fair number of worker threads in this service. All are instantiated by the Task Parallel Library... some as regular short-lived tasks (which uses the CLR threadpool), others are long-running tasks (get their own CLR thread), but non of these are I/O threads (unless the CLR is doing some magic that I don't know about). The DB writes are happening on a long-running Task.

Are there any WCF or debugging diagnostic tools that can report or visualize the current state of WCF threads, worker threads, I/O threads and WCF throttling?

Environment Summary

• WCF service is hosted inside a Windows service
• WCF service uses a net.tcp duplex binding
• WCF service is session based (required by duplex contract)
• There is only a single instance of the Windows service, but multiple instances of the WCF service object are instantiated as required by WCF.
• WCF service operations quickly delegate work to background threads that are persistent inside the Windows service. All WCF operations quickly return and then trickle additional results generated by the background threads down the WCF duplex callback channel.
• All DB access is done on background worker threads. No WCF threads are used for long-running work.
• WCF service has a single custom security behavior. There are no other behaviors (such as reliable messaging, etc)

Answer 1

If you're using session instancing, then each each client connection has a host service. If the host is engaged in accessing the database, then I'd say it's blocking & won't be able to handle any other calls from its client until it's done. You might be able to change the way it works to be asynchronous. Call a start method which kicks off the database activity on a worker thread and returns immediately. Either have a progress method to check the status, or since you're using duplex, have the host signal the client when it's done.

Or is the wcf service not accepting new clients?