CODEWITHSUNDEEP
azure-blob-storageazure-webjobsazure-webjobssdkazure-storage-queues
donquijote
donquijote

Reputation: 1812

Long running webjob fail due to Azure Storage timeout

The issue is with a long running azure webjob on a daily schedule. Each run takes 2-4 hours doing data analytics. The only dependencies are with Azure SQL database via EF and with Azure Storage, just setting up AzureWebJobsDashboard and AzureWebJobsStorage connections on the App.Config, standard setup on VS with webjob SDK. Most of the time of the webjob is consumed with EF's SaveChanges(). I also do an important amount of logging to monitor progress with aprox 3000 lines of Console output. The web app is configured as Allways ON and the WEBJOBS_IDLE_TIMEOUT is set to a very high number.

The following is the log of the error:

[10/20/2016 07:48:17 > 492c46: ERR ] Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout.

[10/20/2016 07:48:17 > 492c46: ERR ] --- End of inner exception stack trace ---

[10/20/2016 07:48:17 > 492c46: ERR ] at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End()

[10/20/2016 07:48:17 > 492c46: ERR ] at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadText(IAsyncResult asyncResult)

[10/20/2016 07:48:17 > 492c46: ERR ] at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.b__3(IAsyncResult ar)

Upvotes: 2

Views: 1470

Answers (2)

JasonCoder
JasonCoder

Reputation: 1136

This is a few months old, but for those that come later...

You mention you do a lot of logging. There is an issue that was logged dealing with something similar. Apparently the WebJobs SDK does a periodic save of the log data to blob storage. If you are using a lot of bandwidth or otherwise consuming a lot of resources you may run into timeouts from the SDK trying to save to blob storage. Note the upload call in your stack trace.

I'm seeing this sporadically in a process that is punishing the wire pretty good so I am disabling logging via the WebJobs logging facility.

Upvotes: 1

iikkoo
iikkoo

Reputation: 2856

A triggered webjob is aborted if it is idle, has no cpu time or output for a certain amount of time. Try and increase it by setting configuration WEBJOBS_IDLE_TIMEOUT to a large number, for instance 3600.

It could also be aborted if your instances hasn't configured Always on.

If that doesn't help you should try handle the amount of logging. Could it be that you try and write to many messages to fast? Have a look at this answer to see if that coould be the case.

Upvotes: 0

Related Questions