CODEWITHSUNDEEP
azureazure-storageazure-table-storageazure-blob-storageazure-worker-roles
RobVious
RobVious

Reputation: 12925

Azure: Worker role looping through "recycling"

I'm currently working on an Azure project that works 100% locally with emulator resources. I'm now trying to deploy a worker role, but I'm running into an issue that I'm not sure how to troubleshoot.

Upon deploying the worker role in my Azure portal, the two instances continually loop through "recycling".

I can try to RDP into the role, but I only have about a minute to look around before the connection closes, I'm assuming due to the recycling.

After some searching it doesn't seem like this is a super common problem. Is there something trivial I'm overlooking that could be causing this issue? How would you go about troubleshooting this? Thank you for your time :)

Upvotes: 9

Views: 6611

Answers (7)

SliverNinja - MSFT
SliverNinja - MSFT

Reputation: 31651

Also been pestered by this role recycle issue numerous times. Here is the sequence of steps to debug persistent role recycles:

Debugging Azure Role Recycles

  1. Enable Remote Access to your role - RDP login
  2. Check eventvwr.msc (Windows Logs -> Application, App & Service Logs->Windows Azure)
  3. Review the Azure text file logs across both C:\logs and c:\resources
  4. Review custom logs in the Volume E: or F: for any custom role startup logging

  5. Run AzureTools and attach to startup processes (download WinDBG, use Utils->Attach Debugger, select process - WaWorkerHost/WaIISHost, etc), use G to continue and watch debugger output for assemblies failing to load.

    Installing Azure Debugging Tools via Powershell

    PS> md c:\tools; Import-Module bitstransfer; Start-BitsTransfer http://dsazure.blob.core.windows.net/azuretools/AzureTools.exe c:\tools\AzureTools.exe; c:\tools\AzureTools.exe

If all items above fail - try using other tools in the AzureTools treasure trove - such as fusion logging, etc, this approach above will work!

WinDBG Sample Output - Failing to Locate Assembly (WaIISHost)

enter image description here

Upvotes: 2

burton
burton

Reputation: 601

If the issue is caused by a startup batch file, I have stopped the loop by editing the batch file on the instance to include "exit /b 0" at the beginning. This will tell Azure that the startup was successful and you then have all the time you need to diagnose issues without the VM getting killed.

Upvotes: 0

Vijay Arumugam
Vijay Arumugam

Reputation: 136

Just a suggestion: Also Check the installable(if any)and any other references you use are 64bit.Azure VMs have 64bit OS. Once i was stuck up with this kind of problem due to 32/64 bit issues.

Upvotes: 1

AvkashChauhan
AvkashChauhan

Reputation: 20576

In case of missing Reference you can troubleshoot this issue by:

Unzip your CSPKG file and then again unzip .CSSX file (just rename CSSX to zip) and match that everything references and static content is all there.. This way you can match what is on VM. Also in 2 minute windows when you RDP, try to look for Application event log for exception and get it because that would be the key to find the root cause.

IF you could see the exception in event log and look for the exception, you sure can find where it was generated. You can also use Intellitrace which might require you to redeploy the app.

Also there are ways were copying WinDBG and locking to the specific process you can debug it. I am not sure how much you would want to try but just copy the WinDBG to VM and use it would be enough (not sure how much experience you have with WinDBG though and how much time you would want to spent.)

Upvotes: 5

RobVious
RobVious

Reputation: 12925

From Avkash above:

Yes. this mean some issue in your Worker Role code is causing your Worker Role Host Process to crash.. If you look your fault stack you must see the function or the link from your code which generate this fault. IF you need help open a free Azure Support incident to Windows Azure Support team and they will help you.

Upvotes: 1

Jason Coyne
Jason Coyne

Reputation: 6636

Are your worker roles exiting their work loop? A local recycle is very fast and you might not notice it, but spin-up time in the cloud can be long.

Upvotes: 0

BrentDaCodeMonkey
BrentDaCodeMonkey

Reputation: 5523

The most likely cause is that you have a missing assembly. One tactic to catch this is to wrap any startup processing in a master try/catch that manual logs the error to Azure storage.

If you added any referrences, check to make sure they're set to copylocal=true and that any external assets that were included in your service package were also set to be included.

Upvotes: 1

Related Questions