Performance Counter - Instances; Create/Update without error but not visible in PerfMon

Question

When invoking UpdatePerformanceCounters: In this updater all the counter names for the category and instance counters are the same - they are always derived from an Enum. The updater is passed a "profile" typically with content such as:

{saTrilogy.Core.Instrumentation.PerformanceCounterProfile}
    _disposed: false
    CategoryDescription: "Timed function for a Data Access process"
    CategoryName: "saTrilogy<Core> DataAccess Span"
    Duration: 405414
    EndTicks: 212442328815
    InstanceName: "saTrilogy.Core.DataAccess.ParameterCatalogue..ctor::[dbo].[sp_KernelProcedures]"
    LogFormattedEntry: "{\"CategoryName\":\"saTrilogy<Core> DataAccess ... 
    StartTicks: 212441923401

Note the "complexity" of the Instance name.

The toUpdate.AddRange() of the VerifyCounterExistence method always succeeds and produces the "expected" output so the UpdatePerformanceCounters method continues through to the "successful" incrementing of the counters.

Despite the "catch" this never "fails" - except, when viewing the Category in PerfMon, it shows no instances or, therefore, any "successful" update of an instance counter.

I suspect my problem may be that my instance name is being rejected, without exception, because of its "complexity" - when I run this through a console tester via PerfView it does not show any exception stack and the ETW events associated with counter updates are successfully recorded in an out-of-process sink. Also, there are no entries in the Windows Logs.

This is all being run "locally" via VS2012 on a Windows 2008R2 server with NET 4.5.

Does anyone have any ideas of how else I may try this - or even test if the "update" is being accepted by PerfMon?

public sealed class Performance {

    private enum ProcessCounterNames {
        [Description("Total Process Invocation Count")]
        TotalProcessInvocationCount,
        [Description("Average Process Invocation Rate per second")]
        AverageProcessInvocationRate,
        [Description("Average Duration per Process Invocation")]
        AverageProcessInvocationDuration,
        [Description("Average Time per Process Invocation - Base")]
        AverageProcessTimeBase
        }

    private readonly static CounterCreationDataCollection ProcessCounterCollection = new CounterCreationDataCollection{
                new CounterCreationData(
                    Enum<ProcessCounterNames>.GetName(ProcessCounterNames.TotalProcessInvocationCount),
                    Enum<ProcessCounterNames>.GetDescription(ProcessCounterNames.TotalProcessInvocationCount),
                    PerformanceCounterType.NumberOfItems32),
                new CounterCreationData(
                    Enum<ProcessCounterNames>.GetName(ProcessCounterNames.AverageProcessInvocationRate),
                    Enum<ProcessCounterNames>.GetDescription(ProcessCounterNames.AverageProcessInvocationRate),
                    PerformanceCounterType.RateOfCountsPerSecond32),
                new CounterCreationData(
                    Enum<ProcessCounterNames>.GetName(ProcessCounterNames.AverageProcessInvocationDuration),
                    Enum<ProcessCounterNames>.GetDescription(ProcessCounterNames.AverageProcessInvocationDuration),
                    PerformanceCounterType.AverageTimer32),
                new CounterCreationData(
                    Enum<ProcessCounterNames>.GetName(ProcessCounterNames.AverageProcessTimeBase),
                    Enum<ProcessCounterNames>.GetDescription(ProcessCounterNames.AverageProcessTimeBase),
                    PerformanceCounterType.AverageBase),
            };

    private static bool VerifyCounterExistence(PerformanceCounterProfile profile, out List<PerformanceCounter> toUpdate) {
        toUpdate = new List<PerformanceCounter>();
        bool willUpdate = true;
        try {
            if (!PerformanceCounterCategory.Exists(profile.CategoryName)) {
                PerformanceCounterCategory.Create(profile.CategoryName, profile.CategoryDescription, PerformanceCounterCategoryType.MultiInstance, ProcessCounterCollection);
                }
            toUpdate.AddRange(Enum<ProcessCounterNames>.GetNames().Select(counterName => new PerformanceCounter(profile.CategoryName, counterName, profile.InstanceName, false) { MachineName = "." }));
            }
        catch (Exception error) {
            Kernel.Log.Trace(Reflector.ResolveCaller<Performance>(), EventSourceMethods.Kernel_Error, new PacketUpdater {
                Message = StandardMessage.PerformanceCounterError,
                Data = new Dictionary<string, object> { { "Instance", profile.LogFormattedEntry } },
                Error = error
            });
            willUpdate = false;
            }
        return willUpdate;
        }

    public static void UpdatePerformanceCounters(PerformanceCounterProfile profile) {
        List<PerformanceCounter> toUpdate;
        if (profile.Duration <= 0 || !VerifyCounterExistence(profile, out toUpdate)) {
            return;
            }
        foreach (PerformanceCounter counter in toUpdate) {
            if (Equals(PerformanceCounterType.RateOfCountsPerSecond32, counter.CounterType)) {
                counter.IncrementBy(profile.Duration);
                }
            else {
                counter.Increment();
                }
            }
        }
    }

From MSDN .Net 4.5 PerformanceCounter.InstanceName Property (http://msdn.microsoft.com/en-us/library/system.diagnostics.performancecounter.instancename.aspx)...

Note: Instance names must be shorter than 128 characters in length.
Note: Do not use the characters "(", ")", "#", "\", or "/" in the instance name. If any of these characters are used, the Performance Console (see Runtime Profiling) may not correctly display the instance values.

The instance name of 79 characters that I use above satisfies these conditions so, unless ".", ":", "[" and "]" are also "reserved" the name would not appear to be the issue. I also tried a 64 character sub-string of the instance name - just in case, as well as a plain "test" string all to no avail.

Changes...

Apart from the Enum and the ProcessCounterCollection I have replaced the class body with the following:

    private static readonly Dictionary<string, List<PerformanceCounter>> definedInstanceCounters = new Dictionary<string, List<PerformanceCounter>>();

    private static void UpdateDefinedInstanceCounterDictionary(string dictionaryKey, string categoryName, string instanceName = null) {
        definedInstanceCounters.Add(
            dictionaryKey,
            !PerformanceCounterCategory.InstanceExists(instanceName ?? "Total", categoryName)
                ? Enum<ProcessCounterNames>.GetNames().Select(counterName => new PerformanceCounter(categoryName, counterName, instanceName ?? "Total", false) { RawValue = 0, MachineName = "." }).ToList()
                : PerformanceCounterCategory.GetCategories().First(category => category.CategoryName == categoryName).GetCounters().Where(counter => counter.InstanceName == (instanceName ?? "Total")).ToList());
        }


    public static void InitialisationCategoryVerify(IReadOnlyCollection<PerformanceCounterProfile> etwProfiles){
        foreach (PerformanceCounterProfile profile in etwProfiles){
            if (!PerformanceCounterCategory.Exists(profile.CategoryName)){
                PerformanceCounterCategory.Create(profile.CategoryName, profile.CategoryDescription, PerformanceCounterCategoryType.MultiInstance, ProcessCounterCollection);
            }
            UpdateDefinedInstanceCounterDictionary(profile.DictionaryKey, profile.CategoryName);
        }
    }

    public static void UpdatePerformanceCounters(PerformanceCounterProfile profile) {
        if (!definedInstanceCounters.ContainsKey(profile.DictionaryKey)) {
            UpdateDefinedInstanceCounterDictionary(profile.DictionaryKey, profile.CategoryName, profile.InstanceName);
            }
        definedInstanceCounters[profile.DictionaryKey].ForEach(c => c.IncrementBy(c.CounterType == PerformanceCounterType.AverageTimer32 ? profile.Duration : 1));
        definedInstanceCounters[profile.TotalInstanceKey].ForEach(c => c.IncrementBy(c.CounterType == PerformanceCounterType.AverageTimer32 ? profile.Duration : 1));
        }
    }

In the PerformanceCounter Profile I've added:

    internal string DictionaryKey {
        get {
            return String.Concat(CategoryName, " - ", InstanceName ?? "Total");
            }
        }

    internal string TotalInstanceKey {
        get {
            return String.Concat(CategoryName, " - Total");
            }
        }

The ETW EventSource now does the initialisation for the "pre-defined" performance categories whilst also creating an instance called "Total".

    PerformanceCategoryProfile = Enum<EventSourceMethods>.GetValues().ToDictionary(esm => esm, esm => new PerformanceCounterProfile(String.Concat("saTrilogy<Core> ", Enum<EventSourceMethods>.GetName(esm).Replace("_", " ")), Enum<EventSourceMethods>.GetDescription(esm)));
    Performance.InitialisationCategoryVerify(PerformanceCategoryProfile.Values.Where(v => !v.CategoryName.EndsWith("Trace")).ToArray());

This creates all of the categories, as expected, but in PerfMon I still cannot see any instances - even the "Total" instance and the update always, apparently, runs without error.

I don't know what else I can "change - probably "too close" to the problem and would appreciate comments/corrections.

Answer 1

These are the conclusions and the "answer" insofar as as it explains, to the best of my ability, what I believe is happening and posted by myself - given my recent helpful use of Stack Overflow this, I hope, will be of use to others...

Firstly, there is essentially nothing wrong with the code displayed excepting one proviso - mentioned later. Putting a Console.ReadKey() before program termination and after having done a PerformanceCounterCategory(categoryKey).ReadCategory() it is quite clear that not only are the registry entries correct (for this is where ReadCategory sources its results) but that the instance counters have all been incremented by the appropriate values. If one looks at PerfMon before the program terminates the instance counters are there and they do contain the appropriate Raw Values.

This is the crux of my "problem" - or, rather, my incomplete understanding of the architecture: INSTANCE COUNTERS ARE TRANSIENT - INSTANCES ARE NOT PERSISTED BEYOND THE TERMINATION OF A PROGRAM/PROCESS. This, once it dawned on me, is "obvious" - for example, try using PerfMon to look at an instance counter of one of your IIS AppPools - then stop the AppPool and you will see, in PerfMon, that the Instance for the stopped AppPool is no longer visible.

Given this axiom about instance counters the code above has another completely irrelevant section: When trying the method UpdateDefinedInstanceCounterDictionary assigning the list from an existing counter set is pointless. Firstly, the "else" code shown will fail since we are attempting to return a collection of (instance) counters for which this approach will not work and, secondly, the GetCategories() followed by GetCounters() or/and GetInstanceNames() is an extraordinarily expensive and time-consuming process - even if it were to work. The appropriate method to use is the one mentioned earlier - PerformanceCounterCategory(categoryKey).ReadCategory(). However, this returns an InstanceDataCollectionCollection which is effectively read-only so, as a provider (as opposed to a consumer) of counters it is pointless. In fact, it doesn't matter if you just use the Enum generated new PerformanceCounter list - it works regardless of whether the counters already exist or not.

Anyway, the InstanceDataCollectionCollection (this is essentially that which is demonstrated by the Win32 SDK for .Net 3.5 "Usermode Counter Sample") uses a "Sample" counter which is populated and returned - as per the usage of the System.Diagnostics.PerformanceData Namespace whichi looks like part of the Version 2.0 usage - which usage is "incompatible" with the System.Diagnostics.PerformanceCounterCategory usage shown.

Admittedly, the fact of non-persistance may seem obvious and may well be stated in documentation but, if I were to read all the documentation about everything I need to use beforehand I'd probably end up not actually writing any code! Furthermore, even if such pertinent documentation were easy to find (as opposed to experiences posted on, for example, Stack Overflow) I'm not sure I trust all of it. For example, I noted above that the instance name in the MSDN documentation has a 128 character limit - wrong; it is actually 127 since the underlying string must be null-terminated. Also, for example, for ETW, I wish it were made more obvious that keyword values must be powers of 2 and opcodes with value of less than 12 are used by the system - at least PerfView was able to show me this.

Ultimately this question has no "answer" other than a better understanding of instance counters - especially their persistence. Since my code is intended for use in a Windows Service based Web API then its persistence is not an issue (especially with daily use of LogMan etc.) - the confusing thing is that the damn things didn't appear until I paused the code and checked PerfMon and I could have saved myself a lot of time and hassle if I knew this beforehand. In any event my ETW event source logs all elapsed execution times and instances of what the performance counters "monitor" anyway.