CODEWITHSUNDEEP
.net-coresystemdbackground-service
ghallas
ghallas

Reputation: 305

.NET core BackgroundService does not shut down gracefully as daemon

I am working on a .NET Core 3.1 background service to be installed as a daemon on an Debian AWS EC2 instance.
It is important to gracefully shut down the daemon to stop running tasks and finalize a number of tasks to be handled (sending some events, etc). The basic implementation looks like this:

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace MyApp.WorkerService
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).UseSystemd().Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureServices((hostContext, services) =>
                {
                    services.AddHostedService<Worker>();
                });        
    }
}

You can see I am using the SystemdLifetime here.

The worker is as follows:

using System;
using System.Threading;
using System.Threading.Tasks;
using AdClassifier.App.Audit;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using NLog;

namespace MyApp.WorkerService
{
    public class Worker : BackgroundService
    {
        private static readonly ILogger Logger = LogManager.GetLogger(typeof(Worker).FullName);

        private readonly int _jobPollIntervalMilliseconds;

        public IServiceProvider Services { get; }

        public Worker(IServiceProvider services, IConfiguration configuration)
        {
            Services = services;
            _jobPollIntervalMilliseconds = configuration.GetValue<int>("JobPollIntervalMilliseconds");
        }

        protected override Task ExecuteAsync(CancellationToken stoppingToken)
        {
            Logger.Info("Worker running.");
            var task = new Task(o => DoWork(stoppingToken), stoppingToken);
            task.Start();
            return task;
        }

        public override async Task StopAsync(CancellationToken cancellationToken)
        {
            Logger.Info("Worker stopping");
            await base.StopAsync(cancellationToken);
            Logger.Info("Worker stopped");
        }

        private void DoWork(CancellationToken stoppingToken)
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                using (var scope = Services.CreateScope())
                {
                    // do some work
                }

                Thread.Sleep(_jobPollIntervalMilliseconds);
            }
            Logger.Info("cancellation requested!");
        }        
    }
}

The problem
As I mentioned, we are setting this up as a daemon, like this

[Unit]
Description=my worker
Requires=deploy-my-worker.service
After=multi-user.target deploy-my-worker.service
ConditionFileIsExecutable=/home/my-worker/current/myworker
[Service]
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false
Environment=DOTNET_CLI_TELEMETRY_OPTOUT=true
Environment=ASPNETCORE_URLS=http://*:5000
Environment=DOTNET_ENVIRONMENT=Staging
Environment=ASPNETCORE_ENVIRONMENT=Staging
WorkingDirectory=/home/my-worker/current
ExecStart=/home/my-worker/current/myworker
SyslogIdentifier=my-worker
Restart=always
RestartSec=10
KillSignal=SIGTERM
User=usr
Group=usrgroup
[Install]
WantedBy=multi-user.target

The problem is that the worker will not stop gracefully. I am checking logs for the following log entries, but they do not appear:
Worker stopping, cancellation requested!, Worker stopped
Note that the application does shut down. What we have tried in order to shut down the service are the following:

What Works

If I start the worker like this: usr@ip-10-168-19-126:~/my-worker/current$ ./myworker and then press Ctrl-C (which should be a SIGINT), the application stops, and in my logs I can see the correct messages: 

2020-05-21 16:16:57.9676|INFO|MyApp.WorkerService.Worker|Worker stopping 
2020-05-21 16:16:57.9937|INFO|MyApp.WorkerService.Worker|cancellation requested! 
2020-05-21 16:16:57.9937|INFO|MyApp.WorkerService.Worker|Worker stopped 
2020-05-21 16:16:57.9980 Info AppDomain Shutting down. Logger closing...

Any ideas how I can get the daemon to work as expected?

NOTE:
I have good reason to believe that the problem lies somewhere in the daemon setup, or UseSystemd().
I replaced UseSystemd() with UseWindowsService() and installed it as a Windows service on a windows machine. Then went forward with starting and stopping the service via the Services panel, and saw shutdown logging as expected.
So, I am tempted to assume that there is no problem in the implementation, but rather somewhere in the setup.

Upvotes: 4

Views: 4361

Answers (4)

thomasgalliker
thomasgalliker

Reputation: 1847

Following service definition did solve the problem in my case. I paste the whole service definition, so don't get confused.

These two attributes made the ASP.NET Core service shutdown gracefully:

  • KillSignal=SIGTERM
  • ExecStop=/bin/kill ${MAINPID}

Let me know if it solves your problem too.

[Unit]
Description=WeatherDisplay.Api Service
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/home/pi/WeatherDisplay.Api
ExecStart=/home/pi/WeatherDisplay.Api/WeatherDisplay.Api
ExecStop=/bin/kill ${MAINPID}
KillSignal=SIGTERM
SyslogIdentifier=WeatherDisplay.Api

User=pi
Group=pi

Restart=always
RestartSec=5

Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false
Environment=DOTNET_ROOT=/home/pi/dotnet

[Install]
WantedBy=multi-user.target

You can find the full project here: https://github.com/thomasgalliker/PiWeatherStation

Upvotes: 0

ghallas
ghallas

Reputation: 305

It seems that the problem lay with NLog's shutdown. This was fixed by doing the following: LogManager.AutoShutdown = false; and in Worker::StopAsync adding LogManager.Shutdown();

Upvotes: 3

Aurelien BOUDOUX
Aurelien BOUDOUX

Reputation: 306

I'm facing the same problem, and it seems to be because systemctl stop is not sending the SIGTERM signal, so to make the service work as expected, I configured the service file with the following parameters:

[Service]
Type=simple
ExecStop=/bin/kill -s TERM $ MAINPID

Now, StopAsync and Dispose are called when I run a systemctl stop MyService

Upvotes: 1

jimnkey
jimnkey

Reputation: 432

It sounds like a threading issue ... where the application exits before completion of the task(s).

You might give this a try:

        protected override Task ExecuteAsync(CancellationToken stoppingToken)
        {
            System.Threading.Thread.CurrentThread.IsBackground = false;

            Logger.Info("Worker running.");
            var task = Task.Run(() => DoWork(stoppingToken), stoppingToken);
            task.Wait();
            return task;
        }

The key is to make your thread that is executing the task, wait for the task and NOT be in the background. I believe the application will exit after completion of any calls to ExecuteAsync.

Upvotes: 0

Related Questions