WCF client getting a TimeoutException (and I AM closing my connections)

Question

Here's the thing... I've been working on this issue for a long time now with no luck

I have a WCF service hosted on IIS (I also tried in a console host and I get the same behavior) which I call thru my scvutil created proxy, and the service fails to respond after a few (very few calls).

But, here are a few clues:
1- It only happens on the testing environment (not with the services hosted on my dev machine), figures!
2- I call the service asynchronously (client.MyServiceAsync)
3- this is the trickiest one... if I run fiddler to see what's going on, the client works and never fails, unitl of course, I close fiddler (without even restarting the client)

I have no clue so far what's going on. I started to think maybe the server (a Win2003) has some kind of "defender" which blocks further connections, could that be possible? Please trust me when I say I do close the clients after every call, actually after I get the response from the server since I'm using ascynchronous calls.

Any tip or advice would be more than appreciated. regards,

Edit: I thought this could be usefull info: I have enabled tracing at both ends with no luck.
At the server side I see the connections coming and going... and at some point there are no more connection coming in. While at the client side I see connection working until one of them throws a TimeoutException... that's all I get

Answer 1

WCF supports service throttling. This allows you to limit how many messages/connections/instances that can run at the same time. Search for ServiceThrottle on MSDN.

You specify your ServiceThrottle in app config like so:

 <behaviors>
      <serviceBehaviors>
        <behavior  name="Throttled">
          <serviceThrottling 
            maxConcurrentCalls="1" 
            maxConcurrentSessions="2200" 
            maxConcurrentInstances="1"
          />
        </behavior>
      </serviceBehaviors>
    </behaviors>

If you suspect throttling is the problem, look for the ServiceThrottle object in the debugger, or in a dump. Here's an example using cdb on a dump:

Dump ServiceHost

0:000> !do 0000000001647fc0
Name: System.ServiceModel.ServiceHost
MethodTable: 000007feef98a7f0
EEClass: 000007feef352d60
Size: 248(0xf8) bytes
 (D:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef23d5918  400098a       4c       System.Boolean  1 instance                0 aborted
000007fef23d5918  400098b       4d       System.Boolean  1 instance                0 closeCalled
000007feef93aa00  400098c        8 ...ct+ExceptionQueue  0 instance 0000000000000000 exceptionQueue
000007fef23d6038  400098d       10        System.Object  0 instance 0000000001648188 mutex
000007fef23d5918  400098e       4e       System.Boolean  1 instance                0 onClosingCalled
000007fef23d5918  400098f       4f       System.Boolean  1 instance                0 onClosedCalled
000007fef23d5918  4000990       50       System.Boolean  1 instance                1 onOpeningCalled
000007fef23d5918  4000991       51       System.Boolean  1 instance                1 onOpenedCalled
000007fef23d5918  4000992       52       System.Boolean  1 instance                0 raisedClosed
000007fef23d5918  4000993       53       System.Boolean  1 instance                0 raisedClosing
000007fef23d5918  4000994       54       System.Boolean  1 instance                0 raisedFaulted
000007fef23d5918  4000995       55       System.Boolean  1 instance                1 traceOpenAndClose
000007fef23d6038  4000996       18        System.Object  0 instance 0000000001647fc0 eventSender
000007feef929a60  4000997       48         System.Int32  1 instance                2 state
000007fef23ce158  4000998       20  System.EventHandler  0 instance 0000000000000000 Closed
000007fef23ce158  4000999       28  System.EventHandler  0 instance 0000000000000000 Closing
000007fef23ce158  400099a       30  System.EventHandler  0 instance 0000000001648660 Faulted
000007fef23ce158  400099b       38  System.EventHandler  0 instance 0000000000000000 Opened
000007fef23ce158  400099c       40  System.EventHandler  0 instance 0000000000000000 Opening
000007fef23d5918  4002e78       56       System.Boolean  1 instance                1 initializeDescriptionHasFinished
000007feef981300  4002e79       58 ...meKeyedCollection  0 instance 00000000016481a0 baseAddresses
000007feef981500  4002e7a       60 ...patcherCollection  0 instance 0000000001648220 channelDispatchers
000007fef2416b80  4002e7b       c0      System.TimeSpan  1 instance 0000000001648080 closeTimeout
000007feef980e70  4002e7c       68 ...erviceDescription  0 instance 0000000001656e28 description
0000000000000000  4002e7d       70                       0 instance 0000000001648290 extensions
0000000000000000  4002e7e       78                       0 instance 0000000000000000 externalBaseAddresses
0000000000000000  4002e7f       80                       0 instance 000000000166cf80 implementedContracts
000007feef989770  4002e80       88 ...nceContextManager  0 instance 0000000001648300 instances
000007fef2416b80  4002e81       c8      System.TimeSpan  1 instance 0000000001648088 openTimeout
000007feef8e9890  4002e82       90 ...rformanceCounters  0 instance 0000000000000000 servicePerformanceCounters
000007feef98b708  4002e83       98 ...rformanceCounters  0 instance 000000000171c590 defaultPerformanceCounters
000007feef981a18  4002e84       a0 ...r.ServiceThrottle  0 instance 0000000001648340 serviceThrottle
000007feef980c20  4002e85       a8 ...erviceCredentials  0 instance 0000000001800ed0 readOnlyCredentials
000007feef980d58  4002e86       b0 ...orizationBehavior  0 instance 0000000001801240 readOnlyAuthorization
0000000000000000  4002e87       b8                       0 instance 0000000000000000 UnknownMessageReceived
000007fef53bc640  4002e77      f80           System.Uri  0   static 0000000001648110 EmptyUri
000007fef23d6038  40032cc       d0        System.Object  0 instance 00000000015be070 singletonInstance
000007fef23d8420  40032cd       d8          System.Type  0 instance 00000000016486a0 serviceType
000007feef981ee0  40032ce       e0 ...ontractCollection  0 instance 0000000001657468 reflectedContracts
000007fef23d1000  40032cf       e8   System.IDisposable  0 instance 0000000000000000 disposableInstance

Dump Throttling Info for the ServiceHost

0:000> !do 0000000001648340
Name: System.ServiceModel.Dispatcher.ServiceThrottle
MethodTable: 000007feef981a18
EEClass: 000007feef34c2d8
Size: 72(0x48) bytes
 (D:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007feef981a90  400371d        8 ...cher.FlowThrottle  0 instance 00000000016484a8 calls
000007feef981a90  400371e       10 ...cher.FlowThrottle  0 instance 00000000016485d8 sessions
000007feef902140  400371f       18 ...her.QuotaThrottle  0 instance 0000000000000000 dynamic
000007feef981a90  4003720       20 ...cher.FlowThrottle  0 instance 0000000001779f80 instanceContexts
000007feef9808c8  4003721       28 ...l.ServiceHostBase  0 instance 0000000001647fc0 host
000007fef23d5918  4003722       38       System.Boolean  1 instance                1 isActive
000007fef23d6038  4003723       30        System.Object  0 instance 0000000001648410 thisLock

Dump Session throttling information

0:000> !do 00000000016485d8
Name: System.ServiceModel.Dispatcher.FlowThrottle
MethodTable: 000007feef981a90
EEClass: 000007feef34c390
Size: 64(0x40) bytes
 (D:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef23dd9b8  40034f5       30         System.Int32  1 instance             2220 capacity
000007fef23dd9b8  40034f6       34         System.Int32  1 instance             **2220 count
000007fef23d6038  40034f7        8        System.Object  0 instance 0000000001648618 mutex
000007fef23c4d70  40034f8       10 ...ding.WaitCallback  0 instance 0000000001648598 release
0000000000000000  40034f9       18                       0 instance 0000000001648630 waiters
000007fef23d6748  40034fa       20        System.String  0 instance 00000000016483c8 propertyName
000007fef23d6748  40034fb       28        System.String  0 instance 0000000001648550 configName
0:000>

Answer 2

Ok, so here's how I fixed the issue in case somebody is facing the same problem.
My services, called asynchronously, were returning a stream and this stream was not being disposed on the client, thus, the proxies were not being freeded.

Thanks everybody for the comments. They kinda led me to the solution since there was nothing NEW I hadn't heard before.

Answer 3

Your best bet would be to enable tracing on both the client and the service to sort of see the flow of control. At least then you will get some visibility into the actions being taken by WCF.

Here's info on enabling tracing: http://msdn.microsoft.com/en-us/library/ms733025.aspx

And I'd recommend loading the client-side and server-side traces into the Service Trace Viewer tool so you can see the events correlated together. Here's some more info on that tool and how to read the output: http://msdn.microsoft.com/en-us/library/aa751795.aspx

That tool is invaluable for debugging this kind of thing. Hopefully you will see what's going on right away.

If you find you still aren't able to see the information you need, you can also try message logging (the service trace viewer tool can read those files too).

I do think you are on the right track thinking it's connections not closing properly, though. Especially with async operations, it's easy to do this without realizing it. This happened to me in the past when I was recreating my client object before the end request fired. This caused me to propely close the second object's connection, but not the first. The trace viewer pointed this out to me pretty quickly.