What is causing monitoring agent errors in the Azure emulator?

Question

When I run an application in the Azure emulator using SDK 2.0 I get the following errors repeated again and again in the console output. The error below states "Insufficient system resources..." but it's the only thing running on my dev machine which has a 6 core Xeon, 32 GB Ram, and plenty of HDD space.Also, I don't see anything in the Windows event viewer. What is causing these errors?

[MonAgentHost] Error: MA EVENT: 2013-07-13T12:49:53.944Z
[MonAgentHost] Error:     2
[MonAgentHost] Error:     12640
[MonAgentHost] Error:     12644
[MonAgentHost] Error:     EtwListener
[MonAgentHost] Error:     0
[MonAgentHost] Error:     12c939f8-b82a-48fc-bb30-cc66717
[MonAgentHost] Error:     etw.cpp
[MonAgentHost] Error:     EtwListener::StartRealTimeSession
[MonAgentHost] Error:     3605
[MonAgentHost] Error:     0
[MonAgentHost] Error:     5aa
[MonAgentHost] Error:     Insufficient system resources exist to complete the requested service.
[MonAgentHost] Error:     ERROR:EnableTraceEx() to enable provider failed
[MonAgentHost] Error: MA EVENT: 2013-07-13T12:49:53.944Z
[MonAgentHost] Error:     2
[MonAgentHost] Error:     12640
[MonAgentHost] Error:     12644
[MonAgentHost] Error:     EtwListener
[MonAgentHost] Error:     0
[MonAgentHost] Error:     12c939f8-b82a-48fc-bb30-cc66717
[MonAgentHost] Error:     etw.cpp
[MonAgentHost] Error:     EtwListener::SetupEventing
[MonAgentHost] Error:     1876
[MonAgentHost] Error:     ffffffff80010008
[MonAgentHost] Error:     0
[MonAgentHost] Error:     
[MonAgentHost] Error:     Failed to start the etw real time user session
[MonAgentHost] Error: MA EVENT: 2013-07-13T12:49:53.944Z
[MonAgentHost] Error:     2
[MonAgentHost] Error:     12640
[MonAgentHost] Error:     12644
[MonAgentHost] Error:     EtwListener
[MonAgentHost] Error:     0
[MonAgentHost] Error:     12c939f8-b82a-48fc-bb30-cc66717
[MonAgentHost] Error:     etw.cpp
[MonAgentHost] Error:     EtwListener::Start
[MonAgentHost] Error:     1758
[MonAgentHost] Error:     ffffffff80010008
[MonAgentHost] Error:     0
[MonAgentHost] Error:     
[MonAgentHost] Error:     Failed to start event tracing
[MonAgentHost] Error: MA EVENT: 2013-07-13T12:49:53.944Z
[MonAgentHost] Error:     2
[MonAgentHost] Error:     12640
[MonAgentHost] Error:     12644
[MonAgentHost] Error:     MonAgent
[MonAgentHost] Error:     0
[MonAgentHost] Error:     12c939f8-b82a-48fc-bb30-cc66717
[MonAgentHost] Error:     monagent.cpp
[MonAgentHost] Error:     MonAgent::InitializeEtw
[MonAgentHost] Error:     1303
[MonAgentHost] Error:     ffffffff80010008
[MonAgentHost] Error:     0
[MonAgentHost] Error:     
[MonAgentHost] Error:     Error: Failed to start ETW providers
[MonAgentHost] Error: MA EVENT: 2013-07-13T12:49:53.944Z
[MonAgentHost] Error:     2
[MonAgentHost] Error:     12640
[MonAgentHost] Error:     12644
[MonAgentHost] Error:     MonAgent
[MonAgentHost] Error:     0
[MonAgentHost] Error:     12c939f8-b82a-48fc-bb30-cc66717
[MonAgentHost] Error:     monagent.cpp
[MonAgentHost] Error:     MonAgent::InitializeForStart
[MonAgentHost] Error:     221
[MonAgentHost] Error:     ffffffff80010008
[MonAgentHost] Error:     0
[MonAgentHost] Error:     
[MonAgentHost] Error:     Failed to start the ETW providers
[MonAgentHost] Error: MA EVENT: 2013-07-13T12:49:53.944Z
[MonAgentHost] Error:     2
[MonAgentHost] Error:     12640
[MonAgentHost] Error:     12644
[MonAgentHost] Error:     MonAgent
[MonAgentHost] Error:     0
[MonAgentHost] Error:     12c939f8-b82a-48fc-bb30-cc66717
[MonAgentHost] Error:     monagent.cpp
[MonAgentHost] Error:     MonAgent::Start
[MonAgentHost] Error:     2215
[MonAgentHost] Error:     ffffffff80010008
[MonAgentHost] Error:     0
[MonAgentHost] Error:     
[MonAgentHost] Error:     Failed initialize agent sub-components

Here is my service definition:

<WorkerRole name="SPS.Azure.FooService" vmsize="Small">
    <Imports>
      <Import moduleName="Diagnostics" />
      <Import moduleName="RemoteAccess" />
      <Import moduleName="RemoteForwarder" />
    </Imports>
    <ConfigurationSettings>      
      <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.Level" />
    </ConfigurationSettings>
</WorkerRole>

Here is my service configuration:

<Role name="SPS.Azure.FooService">
    <Instances count="1" />
    <ConfigurationSettings>      
      <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true" />
      <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.Level" value="Verbose" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.Enabled" value="true" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountUsername" value="ausername" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountEncryptedPassword" value="apassword" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountExpiration" value="2014-05-10T23:59:59.0000000-04:00" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteForwarder.Enabled" value="true" />
    </ConfigurationSettings>  
</Role>

Here is my wadcfg file:

<?xml version="1.0" encoding="utf-8"?>
<DiagnosticMonitorConfiguration configurationChangePollInterval="PT1M" overallQuotaInMB="4096" xmlns="http://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration">
  <DiagnosticInfrastructureLogs />
  <Directories>
    <IISLogs container="wad-iis-logfiles" />
    <CrashDumps container="wad-crash-dumps" />
  </Directories>
  <Logs bufferQuotaInMB="1024" scheduledTransferPeriod="PT1M" scheduledTransferLogLevelFilter="Verbose" />
  <WindowsEventLog bufferQuotaInMB="1024" scheduledTransferPeriod="PT1M" scheduledTransferLogLevelFilter="Verbose">
    <DataSource name="Application!*" />
  </WindowsEventLog>
</DiagnosticMonitorConfiguration>

And here is my diagnostics config in my App.config file:

<system.diagnostics>
    <trace>
      <listeners>
        <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" name="AzureDiagnostics">
          <filter type=""/>
        </add>
      </listeners>
    </trace>
  </system.diagnostics>

Answer 1

Clearing the ETW event sessions seemed to fix the problem for me.

1) To see active sessions run:

Logman -ets

2) Stop main devFabric session:

Logman stop devFabric.mainSession -ets

3) For each MA_ETWSESSION entry from first query run something similar to this:

logman stop MA_ETWSESSION_MACHINE23_deployment23(1).CloudProductName.ServicePortal_IN_1 -ets

Answer 2

I had been having the same exact problem all week. As a shot in the dark, I deleted the entire contents of AppData\Local\dftmp under c:\Users\MyAccount, and I haven't seen the error message again yet.

If you click on a deployment in Azure Compute Emulator and click "Tools" -> "Open local store...", AppData\Local\dftmp is the folder it takes you to.