Agendum
Agendum

Reputation: 2021

TelemetryClient.Flush() takes about 2 minutes to run during application exit

I have a .NET Core 3.1 console application running as a hosted service. When the application shuts down, it hangs for about 2 minutes. When breaking in I can see that it is hanging on TelemetryClient.Flush(), specifically inside InMemoryChannel.Flush().

So I watch the result in Fiddler and I can see the application is making a request out to https://dc.services.visualstudio.com/v2/track to report the telemetry, but that service doesn't respond. Eventually after 2 minutes the response fails with 500 and in the web response: "IIS 10.0 Detailed Error - 500.1013 - Internal Server Error".

I cannot tell if this is something I am doing wrong or not. So I shortened the application to the bare minimum.

using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;

namespace DotNetCoreConsoleAppTest
{
    public class Program
    {
        public static async Task Main()
        {
            await Host.CreateDefaultBuilder()
                .ConfigureServices(services =>
                {
                    services
                        .Configure<TelemetryConfiguration>(options =>
                        {
                            options.InstrumentationKey = "<put your key here>";
                        })
                        .AddSingleton<TelemetryWriter>()
                        .AddHostedService<ProgramHost>();
                })
                .RunConsoleAsync()
                .ConfigureAwait(false);
        }
    }

    public class TelemetryWriter : IDisposable
    {
        private readonly TelemetryClient _telemetryClient;

        public TelemetryWriter(IOptions<TelemetryConfiguration> telemetryConfiguration)
        {
            _telemetryClient = new TelemetryClient(telemetryConfiguration.Value);
        }

        public void WriteEvent() => _telemetryClient.TrackEvent("test");

        public void Dispose() => _telemetryClient.Flush();
    }

    public class ProgramHost : IHostedService
    {
        private readonly TelemetryWriter _telemetryWriter;
        private readonly IHostApplicationLifetime _hostApplicationLifetime;

        public ProgramHost(
            TelemetryWriter telemetryWriter,
            IHostApplicationLifetime hostApplicationLifetime)
        {
            _telemetryWriter = telemetryWriter;
            _hostApplicationLifetime = hostApplicationLifetime;
        }

        public Task StartAsync(CancellationToken cancellationToken)
        {
            _telemetryWriter.WriteEvent();
            _hostApplicationLifetime.StopApplication();
            return Task.CompletedTask;
        }

        public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    }
}

Some content from the web response:

<div class="content-container"> 
 <fieldset><h4>Detailed Error Information:</h4> 
  <div id="details-left"> 
   <table border="0" cellpadding="0" cellspacing="0"> 
    <tr class="alt"><th>Module</th><td>&nbsp;&nbsp;&nbsp;iisnode</td></tr> 
    <tr><th>Notification</th><td>&nbsp;&nbsp;&nbsp;ExecuteRequestHandler</td></tr> 
    <tr class="alt"><th>Handler</th><td>&nbsp;&nbsp;&nbsp;iisnode</td></tr> 
    <tr><th>Error Code</th><td>&nbsp;&nbsp;&nbsp;0x0000006d</td></tr> 

   </table> 
  </div> 
  <div id="details-right"> 
   <table border="0" cellpadding="0" cellspacing="0"> 
    <tr class="alt"><th>Requested URL</th><td>&nbsp;&nbsp;&nbsp;https://dc.services.visualstudio.com:443/InputServer.js</td></tr> 
    <tr><th>Physical Path</th><td>&nbsp;&nbsp;&nbsp;E:\sitesroot\0\InputServer.js</td></tr> 
    <tr class="alt"><th>Logon Method</th><td>&nbsp;&nbsp;&nbsp;Anonymous</td></tr> 
    <tr><th>Logon User</th><td>&nbsp;&nbsp;&nbsp;Anonymous</td></tr> 
    <tr class="alt"><th>Request Tracing Directory</th><td>&nbsp;&nbsp;&nbsp;C:\Resources\directory\f3eec886680f474eb56deb0e59f20036.Breeze.DiagnosticStore\FailedReqLogFiles\Web</td></tr> 
   </table> 
   <div class="clear"></div> 
  </div> 
 </fieldset> 
</div> 

The hanging stack trace is:

System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)    Unknown
System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)  Unknown
System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)  Unknown
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task)    Unknown
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.GetResult()  Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.DequeueAndSend(System.TimeSpan timeout) Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.Flush(System.TimeSpan timeout)  Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Channel.InMemoryChannel.Flush(System.TimeSpan timeout)  Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Channel.InMemoryChannel.Flush() Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.TelemetryClient.Flush() Unknown
DotNetCoreConsoleAppTest.dll!DotNetCoreConsoleAppTest.TelemetryWriter.Dispose() Line 43 C#
Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.DisposeAsync()   Unknown
Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.DisposeAsync()  Unknown
Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(Microsoft.Extensions.Hosting.IHost host, System.Threading.CancellationToken token)    Unknown
Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.RunConsoleAsync(Microsoft.Extensions.Hosting.IHostBuilder hostBuilder, System.Threading.CancellationToken cancellationToken) Unknown
DotNetCoreConsoleAppTest.dll!DotNetCoreConsoleAppTest.Program.Main() Line 16    C#
DotNetCoreConsoleAppTest.dll!DotNetCoreConsoleAppTest.Program.<Main>()  Unknown

Despite all of this, the events DO appear in the logs inside of Application Insights. The only problem is my host application hangs. Is this a problem with the way I am trying to Flush()? Or is this an Application Insights service problem?

Upvotes: 1

Views: 1437

Answers (1)

Agendum
Agendum

Reputation: 2021

ZakiMa answered my question -- App Insights was having an outage.

Upvotes: 1

Related Questions