CODEWITHSUNDEEP
c#multithreadingazureazure-cloud-services
Jerad Rose
Jerad Rose

Reputation: 15503

RoleEnvironment.RequestRecycle() not triggering a reboot

We have an Azure Cloud Serivce that has multiple instances. There is a fatal state that these instances can fall into, at which point we want to trigger an instance reboot.

We've tried calling:

RoleEnvironment.RequestRecycle();

But according to the Azure Portal, the instance continues to remain up and running without any sign of a reboot.

We've even tried to put it into a faulty state by issuing:

Environment.FailFast("Failed.");

But that also doesn't seem to trigger a reboot (again, at least according to Azure Portal).

Should we see this happen in the Azure Portal, and if so, what might prevent it from rebooting?

More Details

Basically, as part of our logic, we're spinning a separate thread that does some heavy processing, and this processing has a timeout, at which point we need to do a reboot to kill the thread and free up all consumed resources. I know ideally we would do more proper handling of the thread and cancel it, but this isn't an option at this point. So we just want to do a reboot.

A simplified version looks something like this:

var mainTask = Task.Run(() => DoHeavyProcessing());
var timeoutTask = Task.Delay(TimeSpan.FromMinutes(10));

if (Task.WaitAny(mainTask, timeoutTask) == 1)
{
    RoleEnvironment.RequestRecycle();
}

But again, RequestRecycle() doesn't seem to be triggering a reboot.

Upvotes: 2

Views: 831

Answers (2)

Jerad Rose
Jerad Rose

Reputation: 15503

So we did some further testing on this, and found that -- at least based on what we're experiencing -- that RequestRecyle does not reboot the Cloud Service VM, but rather just recycles the role instance within the VM.

Same goes with Environment.FailFast -- it was recycling the role instance, and not doing a complete reboot.

So the net effect was the same, if not a little better, since this was likely a bit quicker than a full reboot.

For those looking for a complete reboot (which was not necessary in our case), it may be necessary to do a shell shutdown:

Process.Start("shutdown","/r /t 0");

Upvotes: 5

Simon W
Simon W

Reputation: 5496

I would add a handler to your project to catch the OnStop event to check that the recycle is being requested. Also, if you look at the MSDN page for RequestRecycle it can throw an exception if the request fails so perhaps catch and log that as well.

I can imagine that if you underlying code is really smashing the compute resource that the recycle request is being queued and may take some time to trigger. As such you may be better off controlling this from outside of the instance and using the Azure Server Management API to force the instance to reboot. Maybe log to a queue and after a defined period if an instance / job is still logging then kill the instance.

Note that long running intensive processes may actually result in an unscheduled instance recycle anyway as the Azure Fabric might mark the host as unhealthy and reboot it.

Upvotes: 0

Related Questions