Microsoft Dynamics CRM Asynchronous Processing service not working (probably)

Question

Multiple problems are arising right now on our development CRM server. At first i could not import a solution due to some obscure error (failed to parse XAML). A solution that was unchanged and worked in the past.

Next i couldnt connect to the server anymore using the plugin registration tool. No to any organization on the server.

Next thing i noticed we could not open any workflows anymore.

Based on the points above we concluded it was a server problem, and not an organization problem. This was proven by the fact that creating a blank organization immediately had the same problem.

We think we have narrowed it down to the CRM Asynchronous Processing service not working properly. It's started but it doesn't seem to do anything anymore. Rebooting server and restarting IIS didn't do anything.

in the eventlog i find following entry after i try to edit a workflow:

Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 28/11/2014 11:23:13 
Event time (UTC): 28/11/2014 10:23:13 
Event ID: 281d2875de2f43cf95ca6820a6d58f5c 
Event sequence: 1801 
Event occurrence: 6 
Event detail code: 0 

Application information: 
Application domain: /LM/W3SVC/1/ROOT-1-130616421223647071 
Trust level: Full 
Application Virtual Path: / 
Application Path: d:\Program Files\Microsoft Dynamics CRM\CRMWeb\ 
Machine name: ************

Process information: 
Process ID: 5348 
Process name: w3wp.exe 
Account name: ************* 

Exception information: 
Exception type: CrmException 
Exception message: Error generating UiData
at Microsoft.Crm.Workflow.ObjectModel.UIDataGenerator.GenerateUIData()
at Microsoft.Crm.Application.Platform.WorkflowLibrary.WorkflowAdapter.CreateWorkflowObjectModel(Guid workflowEntityId, Boolean useSystemUserContext)
at Microsoft.Crm.Web.Sfa.WorkflowDetailPage.GetWorkflowStep()
at Microsoft.Crm.Web.Sfa.WorkflowDetailPage.ConfigureForm()
at Microsoft.Crm.Application.Controls.PageManager.OnPreRender(EventArgs e)
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Error deserializing XAML
at Microsoft.Crm.Workflow.ObjectModel.UIDataGenerator.DeserializeXaml(String xaml)
at Microsoft.Crm.Workflow.ObjectModel.UIDataGenerator.GenerateUIData()

Failed to create a 'Settings' from the text 'Assembly references and imported namespaces for internal implementation'.
at System.Xaml.XamlObjectWriter.Logic_CreateFromValue(ObjectWriterContext ctx, XamlValueConverter`1 typeConverter, Object value, XamlMember property, String targetName, IAddLineInfo lineInfo)
at System.Xaml.XamlObjectWriter.Logic_CreateFromValue(ObjectWriterContext ctx, XamlValueConverter`1 typeConverter, Object value, XamlMember property, String targetName)
at System.Xaml.XamlObjectWriter.Logic_CreatePropertyValueFromValue(ObjectWriterContext ctx)
at System.Xaml.XamlObjectWriter.WriteEndMember()
at System.Xaml.XamlServices.Transform(XamlReader xamlReader, XamlWriter xamlWriter, Boolean closeWriter)
at System.Xaml.XamlServices.Load(XamlReader xamlReader)
at System.Activities.XamlIntegration.ActivityXamlServices.Load(XamlReader xamlReader)
at Microsoft.Crm.Workflow.ObjectModel.UIDataGenerator.DeserializeXaml(String xaml)

Could not load type 'System.Runtime.CompilerServices.ExtensionAttribute' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
at System.ModuleHandle.ResolveType(RuntimeModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
at System.ModuleHandle.ResolveTypeHandleInternal(RuntimeModule module, Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
at System.ModuleHandle.ResolveTypeHandle(Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(CustomAttributeRecord caRecord, MetadataImport scope, Assembly& lastAptcaOkAssembly, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, Object[] attributes, IList derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctor, Boolean& ctorHasParameters, Boolean& isVarArg)
at System.Reflection.CustomAttribute.IsCustomAttributeDefined(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable)
at System.Reflection.CustomAttribute.IsDefined(RuntimeAssembly assembly, RuntimeType caType)
at Microsoft.VisualBasic.Activities.XamlIntegration.VisualBasicExpressionConverter.AssemblyCache.EnsureInitialized()
at Microsoft.VisualBasic.Activities.XamlIntegration.VisualBasicExpressionConverter.WrapCachedMapping(NamespaceDeclaration prefix, ReadOnlyXmlnsMapping& readOnlyMapping)
at Microsoft.VisualBasic.Activities.XamlIntegration.VisualBasicExpressionConverter.CollectXmlNamespacesAndAssemblies(ITypeDescriptorContext context)
at Microsoft.VisualBasic.Activities.XamlIntegration.VisualBasicSettingsConverter.ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, Object value)
at MS.Internal.Xaml.Runtime.ClrObjectRuntime.CreateObjectWithTypeConverter(ServiceProviderContext serviceContext, XamlValueConverter`1 ts, Object value)
at System.Xaml.XamlObjectWriter.Logic_CreateFromValue(ObjectWriterContext ctx, XamlValueConverter`1 typeConverter, Object value, XamlMember property, String targetName, IAddLineInfo lineInfo)



Request information: 
Request URL: https://*************/*********Development/sfa/workflow/edit.aspx?id={936C2198-4B0B-4A2A-A667-5F9B5D1B9845} 
Request path: /*********Development/sfa/workflow/edit.aspx 
User host address: 10.32.100.12 
User: *************
Is authenticated: True 
Authentication Type: NTLM 
Thread account name: ****************

Thread information: 
Thread ID: 53 
Thread account name: **************
Is impersonating: False 
Stack trace: at Microsoft.Crm.Workflow.ObjectModel.UIDataGenerator.GenerateUIData()
at Microsoft.Crm.Application.Platform.WorkflowLibrary.WorkflowAdapter.CreateWorkflowObjectModel(Guid workflowEntityId, Boolean useSystemUserContext)
at Microsoft.Crm.Web.Sfa.WorkflowDetailPage.GetWorkflowStep()
at Microsoft.Crm.Web.Sfa.WorkflowDetailPage.ConfigureForm()
at Microsoft.Crm.Application.Controls.PageManager.OnPreRender(EventArgs e)
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)


Custom event details: 

EDIT: We have tried to reinstall the backend services as well as the front end. Neither provided a solution

Answer 1

We've seen this in our production environment at work, after endless debugging sessions and trying everything in the end the fix was just to re-install Dynamics CRM (and, for good measure, we also re-installed the .NET framework, no idea if this was needed, but there you go).

An hour later our account managers and salesreps could work again and we could push our stuff to production via the plugin registration tool like we did before.

We never found out what caused the corruption in our installation though :(, luckily it didn't re-occur.

HTH, bovako