CODEWITHSUNDEEP
c#asp.net-corehangfire
A. Hasemeyer
A. Hasemeyer

Reputation: 1734

Hangfire Schedule Background tasks for different server

I have a .Net application that utilizes multiple Hangfire servers.

I want to be able to have one Hangfire RecurringJob trigger multiple BackgroundJobs that can be picked up by any available server. Currently whenever I schedule Background Jobs from a Hangfire Job only the server that scheduled them will process them.

For example, I have 5 Hangfire Servers and 10 tasks. I would want there to be 2 tasks on each Hangfire server, instead I am seeing 1 server with 10 tasks and 4 with 0.

So again I have 5 Hangfire servers, all using the same database, and 1 RecurringJob, this RecurringJob just reads some files and enqueues several background jobs.

 foreach (var file in reportSourceSetFileList)
 {
      _logger.LogInformation($"Queuing Background job for: {file}");

      var backgroundJobId = BackgroundJob.Enqueue<IJobHandler>(job => job.ProcessFile(file, files, null));
 }

However, only the Hangfire Server that ran the RecurringJob will process the Enqueued job.

How can I have those Enqueued jobs be processed by any of my 5 Hangfire Servers and not just the one that queued them?

Upvotes: 2

Views: 2877

Answers (1)

A. Hasemeyer
A. Hasemeyer

Reputation: 1734

There is no built in functionality in Hangfire to use a round robin type load balancer between multiple hangfire servers.

My solution was to use the Queuing system. When each Hangfire server starts they are given a task identifier, which is a GUID, I also add a unique queue to that server which uses the same GUID as its name.

So each server will look at 2 queues, Default and GUID.

Then I use the following code to find which server has the least jobs currently processing.

private string GetNextAvailableServer()
{
    var serverJobCounts = new Dictionary<string, int>();

    //get active servers
    var serverList = JobStorage.Current.GetMonitoringApi().Servers();
    foreach (var server in serverList)
    {
        if (server.Heartbeat.Value > DateTime.Now.AddMinutes(-1))
        {
            serverJobCounts.Add(server.Name, 0);
            foreach (var queue in server.Queues)
            {
                var currentQueues = JobStorage.Current.GetMonitoringApi().Queues();
                serverJobCounts[server.Name] += (int?)currentQueues.FirstOrDefault(e => e.Name == queue)?.Length ?? 0;
            }
        }
    }

    var jobs = JobStorage.Current.GetMonitoringApi().ProcessingJobs(0, int.MaxValue);
    foreach (var job in jobs)
    {
        if (serverJobCounts.ContainsKey(job.Value.ServerId))
        {
            serverJobCounts[job.Value.ServerId] += 1;
        }
    }

    var nextServer = serverJobCounts.OrderBy(e => e.Value).FirstOrDefault().Key;
    return nextServer.Split(':')[0].Replace("-", string.Empty, StringComparison.InvariantCulture);
}

This returns the GUID of the server that has the least jobs, which is also the name of the Queue. Therefore you can schedule the next job to the specific queue with the least jobs currently processing. 

var nextServer = GetNextAvailableServer();

var client = new BackgroundJobClient();
var state = new EnqueuedState(nextServer);
var enqueueJob = client.Create<IJobHandler>(job => job.ProcessFile(file, files, null), state);

Additionally when I wrote this Hangfire didn't allow for hyphens in a queue name, hence my string manipulation to make the GUIDs work. I think the newest version of hangfire lets you use hyphens in the name.

One thing to look out for, this solution breaks when one of you server dies. Since a job is given a unique Queue if the server watching that queue dies before processing the job it will never be picked up.

Upvotes: 1

Related Questions