Freezing managed threads

Question

I'm working on a quite big application that is in charge of doing real-time motion tracking and camera movement controlling. Its tasks are:

• Motion tracking (done by a native module, which decodes a video stream from a network camera and supplies both image buffers as big as 1280x720 pixels and the tracking results to the managed application via callbacks)
• Receiving positioning feedback data from and sending movement data to a pan/tilt hardware about 20 times a second as well as zoom commands from/to the camera
• Displaying the image data including live visualizations
• Encoding and writing of the image and session data
• Automatic postprocessing of the video is done by another process

The application uses .NET 4.0 and has a WPF user interface.

Managed threads freezing

From the beginning we had to face managed threads that freeze for between 500 to 1500ms, which is really much for a real-time application like this.

To find out when these hangs occur I created a thread whose only task is to do a sleep for 100ms all the time. I then calculated how long the sleep really took and got exactly the times when the camera movement stopped. It works very reliably, the threads all hang at the same time!

Unmanaged threads don't freeze

While all the managed threads freeze the unmanaged threads work without any problem. We check that by logs that are written independent from the managed part of the application.

Analysis

I tried to figure out with phenomenons could maybe cause this behaviour:

• The whole machine slows down when we encounter these problems: Windows is responding very slowly (e.g. directory listings hang for half a minute in both Windows Explorer and my application, or launching applications takes incredibly long)
• We read and write thousands of files at the same time (the tracking and postprocessing application), maybe this overcharges windows
• The response of the GUI becomes very slow
• The app uses about 1.3GB of virtual memory (according to Process.VirtualMemory64) / 500MB of working set memory (Process.WorkingSet64) - could it be that some of that is swapped onto the harddisk? (How can that be checked or solved?)
• Of course if we kill the process Windows is responding fast again, but it takes Windows a while to be responding normally again

Hints on how to investigate into this would be highly appreciated. Thank you very much!

Answer 1

I would suggest taking a process dump as you are encountering the performance problem. You can do this several ways (taskmgr.exe or procdump.exe from SysInternals). Take a full memory dump.

Once you have the .dmp file, you can analyze it with windbg (or Visual Studio 2010). For managed processes you need to load the sos.dll extensions.

There are a lot of good windbg resources out there, but here are a few that have helped me:

1) Tess Fernandez video (ASP.NET process, but the techniques are the same)

2) WinDbg cheatsheet

The memory analysis will be able to give you the stack (!clrstack) while you are encountering the problem and tell you the exact culprit.

Answer 2

Maybe GC is working? See the great article: http://samsaffron.com/archive/2011/10/28/in-managed-code-we-trust-our-recent-battles-with-the-net-garbage-collector