CODEWITHSUNDEEP
azureazure-service-fabricetwazure-eventhubazure-diagnostics
Declan McNulty
Declan McNulty

Reputation: 3374

Is ETW streaming from Service Fabric cluster Diagnostic extension to Event Hubs currently possible?

We are currently working to implement the sinking of Diagnostic streams into Event Hubs:

enter image description here

I have added the diagnostics extension to the service fabric cluster using the ARM template, and this is currently outputting to Azure storage, what I need now is set up the sink to the Event hub.

I followed this post , which describes how to add a SinksConfig node to the diagnostics.wadcfgx file:

enter image description here

It seems however that this is only valid for Azure Cloud Service projects? In a cloud service project, it looks quite easy to add the config file by going to properties on the Role, and clicking enable Diagnostics to create the file:

enter image description here

With a Service Fabric project however, it is not possible to do this as there is no notion of a Web Role and there does not seem to be anywhere in the properties of the SF project where Diagnostics can be enabled/wadcfgx file added.

I have added the diagnostics extension to the VMs in the scale set by adding the following to the ARM template:

enter image description here

I can see where the storage account (where the ETW is currently being streamed to) is defined here, but I couldn’t find guidance on how to specify an Event Hub?

I did wonder whether this was just because it is not documented anywhere, or because it is not possible with Service Fabric yet. This article , dated in July, states the following:

enter image description here

This seems to indicate this functionality may or may not be currently available for Service Fabric.

Summary

There are 2 main questions then:

  1. Is it currently possible to set up ETW streaming from a Service Fabric cluster (with Diagnostics extension enabled) to an Event Hub? If not, how far away is it?
  2. If it is possible, how should it be set up? If it is to be defined in the ARM template as defined above, what is the syntax for it?

Upvotes: 2

Views: 871

Answers (2)

Declan McNulty
Declan McNulty

Reputation: 3374

After much trial and error, it appears there are two ways to do this.

The steps from Visual Studio:

  1. Locate your Scale Set in Cloud Explorer

  2. Choose Update diagnostics

  3. Record the current Storage Account being used
  4. Edit your Public and Private config files for Event Hub sinks and storage account
  5. Update diagnostics and choose to update based on the new config files

The public config is (only showing SinksConfig node for brevity)

{
 "WadCfg": {
     "DiagnosticMonitorConfiguration": {

     *** config for performance counters and ETW ***

       "SinksConfig": {
            "Sink": [
             {
               "name": "eventhub",
               "EventHub": {
               "Url": "https://myhub.servicebus.windows.net/mycompanyapplication",
               "SharedAccessKeyName": "RootManageSharedAccessKey"
             }
           }
         ]
       }
     },
    "StorageAccount": "<storageaccount>"
}

and the private config:

{
   "storageAccountName": "<storageaccountname>",
   "storageAccountKey": "<storageaccountkey>",
   "storageAccountEndPoint": "https://core.windows.net",
   "EventHub": {
         "Url": "https://myhub.servicebus.windows.net/mycompanyapplication",
         "SharedAccessKeyName": "RootManageSharedAccessKey",
         "SharedAccessKey": "<sharedaccesskey>"
   }
}

Or this configuration can be made part of your ARM template, with the json from the public config copied into the "settings" node of the VMDiagnosticsSettings resource (example resource definition here), and the private config copied into the "protectedsettings" node.

Upvotes: 1

LoekD
LoekD

Reputation: 11470

Have a look at this example. Vaclav has created event listeners for various types of stores. Make sure you include the Nuget package 'Microsoft.Diagnostics.Tracing.EventSource' and replace 'System.Diagnostics.Tracing' with 'Microsoft.Diagnostics.Tracing' in your ServiceEventSource classes. Register event providers in your ARM template like shown here.

Upvotes: 1

Related Questions