CODEWITHSUNDEEP
c#batch-processinghangfire
TMan
TMan

Reputation: 4122

How to handle batch jobs without Hangfire Pro

I have implemented Hangfire into my project. I now have the need to enqueue multiple jobs that will technically be part of a batch. Company won't buy Pro version of Hangfire that offers batch functionality. Is there a workaround for me to be able to tell when all related jobs are complete so that I can call another function at the very end of each batch?

Example:

Batch A:
{
  BackgroundJob.Enqueue(jobA1);
  BackgroundJob.Enqueue(jobA2);
  BackgroundJob.Enqueue(jobA3);
}

When Batch A is all done: 
  BackgroundJob.Enqueue(createReportForBatchA);

Batch B:
{
  BackgroundJob.Enqueue(jobB1);
  BackgroundJob.Enqueue(jobB2);
  BackgroundJob.Enqueue(jobB3);
}

When Batch B is all done: 
  BackgroundJob.Enqueue(createReportForBatchB);

The only thing I can think of is to set a flag 'Done' for each job within a batch and at the end of each job within a batch check if all jobs are complete by checking flag for all batch related rows in table, then if so Enqueue the createReportForBatch. Seems kind of hackish though to do that and then I'd have to raise the question could I Enqueue a background job within another BackgroundJob (nested basically). Thanks for any input or advice.

Upvotes: 5

Views: 5816

Answers (1)

odinserj
odinserj

Reputation: 964

1. Implementing a filter

Batches use the extensibility API available for anyone. You can create a filter that adds a background job id to a some persisted set during the creation phase, sets the "processed" status for an item in the set during the state change process, checks for other pending jobs when a job is processed, and launches a new background job, if it was the last one in the batch.

This is a high level overview of batches. There are a lot of things to be considered, to avoid different race conditions and different scenarios (job re-queue, deletion and so on).

2. Using Hangfire + TPL

Alternatively, you can use Hangfire + TPL instead to perform calculations in parallel, if you have relatively low number of background jobs in a batch. So Batch A is a simple background job, Batch B is its continuation. Methods of Batch A and Batch B use TPL for methods jobAN with Task.WaitAll waiting for their completion.

Upvotes: 3

Related Questions