CODEWITHSUNDEEP
c#azure-webjobssdk
tillerstarr
tillerstarr

Reputation: 2646

Azure Webjob HTTP 409 Conflict errors

I had a C# WebJob that was working nicely with the alpha WebJob api. I just updated it to the beta release, and after fixing connection strings and namespaces I get a HTTP 409 (Conflict) error when the JobHost tries to connect.

Here is the call stack:

Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.Protocol.TableOperationHttpResponseParsers.TableOperationPreProcess(Microsoft.WindowsAzure.Storage.Table.TableResult result, Microsoft.WindowsAzure.Storage.Table.TableOperation operation, System.Net.HttpWebResponse resp, System.Exception ex)   Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.TableOperation.InsertImpl.AnonymousMethod__2(Microsoft.WindowsAzure.Storage.Core.Executor.RESTCommand<Microsoft.WindowsAzure.Storage.Table.TableResult> cmd, System.Net.HttpWebResponse resp, System.Exception ex, Microsoft.WindowsAzure.Storage.OperationContext ctx) Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync<Microsoft.WindowsAzure.Storage.Table.TableResult>(Microsoft.WindowsAzure.Storage.Core.Executor.RESTCommand<Microsoft.WindowsAzure.Storage.Table.TableResult> cmd, Microsoft.WindowsAzure.Storage.RetryPolicies.IRetryPolicy policy, Microsoft.WindowsAzure.Storage.OperationContext operationContext)  Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.TableOperation.Execute(Microsoft.WindowsAzure.Storage.Table.CloudTableClient client, Microsoft.WindowsAzure.Storage.Table.CloudTable table, Microsoft.WindowsAzure.Storage.Table.TableRequestOptions requestOptions, Microsoft.WindowsAzure.Storage.OperationContext operationContext)  Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.CloudTable.Execute(Microsoft.WindowsAzure.Storage.Table.TableOperation operation, Microsoft.WindowsAzure.Storage.Table.TableRequestOptions requestOptions, Microsoft.WindowsAzure.Storage.OperationContext operationContext)    Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.Host.Storage.SdkCloudStorageAccount.Table.GetOrInsert<Microsoft.Azure.Jobs.Host.Runners.HostEntity>(Microsoft.Azure.Jobs.Host.Runners.HostEntity entity) Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.Host.Runners.HostTable.GetOrCreateHostId(string hostName)    Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHostContext.JobHostContext(string dashboardConnectionString, string storageConnectionString, string serviceBusConnectionString, Microsoft.Azure.Jobs.ITypeLocator typeLocator, Microsoft.Azure.Jobs.INameResolver nameResolver)   Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.GetHostContext(Microsoft.Azure.Jobs.ITypeLocator typesLocator, Microsoft.Azure.Jobs.INameResolver nameResolver)  Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.JobHost(System.IServiceProvider serviceProvider) Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.JobHost(Microsoft.Azure.Jobs.JobHostConfiguration configuration) Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.JobHost()    Unknown

Exception text:

[Microsoft.WindowsAzure.Storage.StorageException] {"The remote server returned an error: (409) Conflict."} Microsoft.WindowsAzure.Storage.StorageException

packages.config:

package id="Microsoft.Azure.Jobs" version="0.3.1-beta" targetFramework="net45" package id="Microsoft.Azure.Jobs.Core" version="0.3.1-beta" targetFramework="net45" package id="Microsoft.Azure.Jobs.ServiceBus" version="0.3.1-beta" targetFramework="net45" package id="Microsoft.Bcl" version="1.1.7" targetFramework="net45" package id="Microsoft.Bcl.Build" version="1.0.14" targetFramework="net45" package id="Microsoft.Data.Edm" version="5.6.0" targetFramework="net45"
package id="Microsoft.Data.OData" version="5.6.0" targetFramework="net45" package id="Microsoft.Data.Services.Client" version="5.6.0" targetFramework="net45" package id="Microsoft.Net.Http" version="2.2.19" targetFramework="net45"
package id="Microsoft.WindowsAzure.ConfigurationManager" version="2.0.3" targetFramework="net45" package id="Newtonsoft.Json" version="6.0.4" targetFramework="net45" package id="System.Spatial" version="5.6.0" targetFramework="net45" package id="WindowsAzure.MobileServices" version="1.3.0-alpha4" targetFramework="net45" package id="WindowsAzure.ServiceBus" version="2.4.2.0" targetFramework="net45" package id="WindowsAzure.Storage" version="4.2.0" targetFramework="net45"

Any ideas?

Upvotes: 6

Views: 5796

Answers (5)

twasbrillig
twasbrillig

Reputation: 18921

When I was getting this error, I realized that the Jenkins job I was calling didn't have the parameters I was passing to it. Once I added the parameters, it worked fine.

Upvotes: 0

James Cuthbert
James Cuthbert

Reputation: 79

We started receiving a 409 Conflict exception from Azure Web Jobs shortly after a release thrown from Microsoft.WindowsAzure.Storage. Not long before this release we had implemented archiving of older blobs. It turns out having readonly blobs in our storage (Archived) was the reason behind this exception being thrown in our case. It was not immediately evident that this was the culprit, I suspect it took time before the webjob started hitting the archived blobs. Once we removed the archived messages from our blob storage, the webjob was able to resume.

Upvotes: 1

zmaten
zmaten

Reputation: 477

I was getting the same error code

2018-08-18T23:06:02.2102822Z ##[error]Conflict
2018-08-18T23:06:02.2119417Z ##[error]Unable to retrieve connection details for Azure App Service : KaktusWatch. Status Code: 409 (Conflict)
2018-08-18T23:06:02.2150200Z ##[section]Finishing: Deploy Azure App Service

In my case the webjob-publish-settings.json was obsolete.

I had to fill in interval and jobRecurrenceFrequency settings parameters.

Upvotes: 1

OOZ
OOZ

Reputation: 113

Changing the jobRecurrenceFrequency in webjob-publish-settings.json to "Hour" worked for my https://error404.atomseo.com project!

I had the same problem and it turned out that publish process failed because I set it to recur every 10 minutes while the app was meant to run in free tier. As MS describes here:

https://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-deploy-webjobs/

one can deploy with all frequencies other than those defined in minutes.

Upvotes: 4

Joy George Kunjikkuru
Joy George Kunjikkuru

Reputation: 1527

I did some more research and could see that this can happen if there are concurrent requests to create queue/table with same name. Azure seems rejecting those requests with custom 409 http code which is meant for applications.

To reproduce even in the latest Azure SDK, just have a webjob to process queues and start the JobHost only after having many requests in queue. In the Job function Get the blob as TextWriter and write into it. Makes sure the blob 'logs/webjob1' is not present when the JobHost starts to see the issue.

public static void ProcessQueueMessage([QueueTrigger("testqueue1")] string inputText,
       [Blob("logs/webjob1")]TextWriter writer)
        {
            writer.WriteLine(inputText);
        }

I just had 4 items in the queue when I started JobHost and was able to see the issue from Azure. Used free hosting.

Upvotes: 2

Related Questions