Diagnosing "RPC Server Unavailable Error" which is cause by Windows Service calling WCF

Question

I have a windows service that uses WCF to connect to other services. It checks that they are alive, gets any error messages that these services have, and reports on these. This is checked every 30 seconds using a channel factory where proxies are created for each service found within the configuration that conforms to an interface. After a few days of running fine the server becomes unresponsive and starts reporting an "RPC Server Unavailable Error". I can use computer management to connect to it and it the memory foot print doesn't appear to be climbing, though if I stop the service it completely fixes the problem. I have attached the channel factory manager I am using though if anything else is needed please let me know. Could it be that the service channels are not being freed correctly? What can I do to diagnose this? Has anyone come across this before?

public class ChannelFactoryManager : IDisposable
{
    private static Dictionary<Tuple<Type, string>, ChannelFactory> _factories = new Dictionary<Tuple<Type, string>, ChannelFactory>();
    private static readonly object _syncRoot = new object();

    public virtual T CreateChannel<T>() where T : class
    {
        return CreateChannel<T>("*", null);
    }

    public virtual T CreateChannel<T>(string endpointConfigurationName) where T : class
    {
        return CreateChannel<T>(endpointConfigurationName, null);
    }

    public virtual T CreateChannel<T>(string endpointConfigurationName, string endpointAddress) where T : class
    {
        T local = GetFactory<T>(endpointConfigurationName, endpointAddress).CreateChannel();
        ((IClientChannel)local).Faulted += ChannelFaulted;
        return local;
    }

    protected virtual ChannelFactory<T> GetFactory<T>(string endpointConfigurationName, string endpointAddress) where T : class
    {
        lock (_syncRoot)
        {
            ChannelFactory factory;
            if (!_factories.TryGetValue(new Tuple<Type, string>(typeof(T), endpointConfigurationName), out factory))
            {
                factory = CreateFactoryInstance<T>(endpointConfigurationName, endpointAddress);
                _factories.Add(new Tuple<Type, string>(typeof(T), endpointConfigurationName), factory);
            }
            return (factory as ChannelFactory<T>);
        }
    }

    private ChannelFactory CreateFactoryInstance<T>(string endpointConfigurationName, string endpointAddress)
    {
        ChannelFactory factory = null;
        if (!string.IsNullOrEmpty(endpointAddress))
        {
            factory = new ChannelFactory<T>(endpointConfigurationName, new EndpointAddress(endpointAddress));
        }
        else
        {
            factory = new ChannelFactory<T>(endpointConfigurationName);
        }
        factory.Faulted += FactoryFaulted;
        factory.Open();
        return factory;
    }

    private void ChannelFaulted(object sender, EventArgs e)
    {
        IClientChannel channel = (IClientChannel)sender;
        channel.Abort();
    }

    private void FactoryFaulted(object sender, EventArgs args)
    {
        ChannelFactory factory = (ChannelFactory)sender;
        factory.Abort();            

        Type[] genericArguments = factory.GetType().GetGenericArguments();
        if ((genericArguments != null) && (genericArguments.Length == 1))
        {
            Type type = genericArguments[0];
            string endPointName = factory.Endpoint.Name;
            Tuple<Type, string> key = new Tuple<Type, string>(type, endPointName);

            if (_factories.ContainsKey(key))
            {
                _factories.Remove(key);
            }
        }
    }

    public void Dispose()
    {
        Dispose(true);
    }

    protected virtual void Dispose(bool disposing)
    {
        if (disposing)
        {
            lock (_syncRoot)
            {
                foreach (Tuple<Type, string> type in _factories.Keys)
                {
                    ChannelFactory factory = _factories[type];
                    try
                    {
                        factory.Close();
                        continue;
                    }
                    catch
                    {
                        factory.Abort();
                        continue;
                    }
                }
                _factories.Clear();
            }
        }
    }
}

Thanks Rob

Answer 1

If you go with the instantiate service proxies as needed route, the answers in this SO question give some options and rationale for disposing of the proxy instance. As a basic start I'd recommend:

//Your client type could be ICommunicationObject or ClientBase:
var client = new YourServiceProxyType();
try {
    var result = client.MakeCall();
    //do stuff with result...

    //Done with client. Close it:
    client.Close();
}
catch (Exception ex) {
    if (client.State != System.ServiceModel.CommunicationState.Closed)
        client.Abort();
}

The fundamental issue in designing a good WCF proxy disposal pattern is that the WCF team at Microsoft decided to implement Dispose in a way that can throw exceptions thus preventing release of unmanaged resources until Abort() is called or the proxy instance is completely garbage collected. They wrote the framework so they get to make the choices, unfortunately we have to suffer the consequences.