Start extremely long running processes through a REST request

Question

I'm working at an automation firm so we create processes for industrial automation. Previously this automation was done on the machine side of things, but we're slowly transitioning to controlling the machines with c#.

On my current project the production for one day takes about 2 hours. The operators of the factory have a web interface that we created in c# using asp.net core MVC in which they can start/pause/stop this production process.

When starting the process we await a function in our controller that is basically a while loop that controls this 2h long production process.

The problem is now that when I send out the REST request to start the production this request takes 2h to complete, I would prefer this request immediately completes and the production process starts on the background of my asp.net core application.

First I thought I could just leave out the await and simply do this in my controller (simplified code):

_ = _productionController.StartLongProcess(); // This contains the while loop
return Ok();

But since _productionController is scoped and all its dependencies are as well, these immediately get disposed of when the method returns and I can't access my database anymore for example.

The process should be able to continuously talk to our database to save information about the production process in case something fails, that way we can always pick off where we left off.

My question to you is now, are we tackling this the wrong way? I imagine it's bad practice to start these long running processes in the asp.net controller.

How do I make sure I always have access to my DatabaseContext in this long running process even though the REST request has already ended. Create a separate scope only for this method?

Answer 1

My question to you is now, are we tackling this the wrong way? I imagine it's bad practice to start these long running processes in the asp.net controller.

Generally, yes. Ideally an ASP.NET service does not have any long-running processes inside it - or at the very least, no long-running processes that can't be easily and quickly shut down.

Doing work outside of an HTTP request (i.e., request-extrinsic code) is most reliably achieved by adding a durable queue with a separate background processor. The "background processor" can be a Win32 service, possibly even on the same machine. With this architecture, the HTTP request places a request on the queue, and the processor picks up requests from the queue and executes them.

Answer 2

Starting ASP.NET Core 2.1, the right way to do this (within asp.net) is to extend BackgroundService (or implement IHostedService).

Incoming requests can tell the background service to start the long-running operation and return immediately. You'll of course need to handle cases where duplicate requests are sent in, or new requests are sent in before the existing request is completed.

The documentation page has an example where the BackgroundService reads commands of a queue and processes them one at a time.

How do I make sure I always have access to my DatabaseContext in this long running process even though the REST request has already ended. Create a separate scope only for this method?

Yes, create a separate scope.

My question to you is now, are we tackling this the wrong way? I imagine it's bad practice to start these long running processes in the asp.net controller.

We've done something similar in the past. As long as fault-tolerance (particularly w.r.t. app restarts) and idempotence are built into the long-running-operation's logic, you should be good to go.

Answer 3

REST requests are expected to be short, a few seconds at maximum. So best practice here would be to offload a long running task to a background service and return a token where you can poll the service if the operation has already finished.

The background service could be a BackGroundWorker in Net Core. This is easy but not really fault tolerant, so some sort of db and retry logic could be good.

If you are in an intranet, you could also move to an inherently asynchronous protocol like RabbitMQ, where you send a StartOperation Message and then receive a Started Message when the process has completed.

Answer 4

Another option would be to use Hangfire. It will allow you to Enqueue the work that you want to execute to a persistent store e.g. SQL Server, MSMQ, Redis depending on what you have in your infrastructure. The job will then be picked up by a worker which can also run in the ASP.NET process or a windows service. It's distributed too so you can have a number of instances of the workers running. Also supports retrying failed jobs and has a dashboard to view the jobs. Best of all, it's free!

var jobId = BackgroundJob.Enqueue(() => ExecuteLongRunningProcess(parameter1));

https://www.hangfire.io/

Answer 5

Following is my understanding of the issue that you have posted:

1. You want to initiate a long running call, via Rest api call
2. You want to use the Async call, but not sure how to maintain the DB context for a long running call which is used for db communication on regular basis during the operation

Couple of important points:

1. Mostly you are not clear regarding working of the Async calls
2. When you make an Async call, then it stores the current thread synchronization context for the continuity using state machine, it doesn't block any thread pool thread, it utilize the hardware based concurrency
3. Can use ConfigureAwait(false) on backend to avoid explicit reentry in the current synchronization context, which is better for performance
4. Only challenge with Async calls to be genuine async the complete chain need to be Async enabled from the entry point, else the benefits can't be reaped, if you use Task.Wait or Task.Result anywhere, infact may even cause a deadlock in the ASP.Net

Regarding the long running operation, following are the options

1. A Simple async call as suggested above, though it can help you can make large number of async calls (thus scalability) but context will be lost if the client goes away and no way to reap the status of operation back
2. You can make a fire and forget call, and use a mechanism like ASP.Net SignalR, which is like IObservable over the network and can help in notifying the client when the processing finish
3. Best option would be using a messaging queue like Rabbit MQ, which doesn't run the risk of client going down, it acts a producer consumer and can notify when the client comes up, in this case MQ can be notified when the process finish and thus client can be informed. MQ can be used for both incoming and response message in an async manner

In case, where client wants to periodically come up and check the status of the request, then DB persistence is important, which can be updated at regular intervals and it can be checked what's the status of the long running process.