Handling WorkerErrorEvent for runtime:dotnet-isolated, workerId:dotnet-isolated. Failed with: System.TimeoutException: The operation has timed out

Question

I have a Function App targeting .NET 6.0 that logs this error every 10-20 minutes:

22:29:45Z [Verbose] Handling WorkerErrorEvent for runtime:dotnet-isolated, workerId:dotnet-isolated. Failed with: System.TimeoutException: The operation has timed out. at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.PendingItem.OnTimeout() in /_/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs:line 1767

22:29:45Z [Verbose] Attempting to dispose webhost or jobhost channel for workerId: '########-####-####-#################', runtime: 'dotnet-isolated'

22:29:45Z [Verbose] Disposing language worker channel with id:########-####-####-#################

22:29:45Z [Verbose] Disposed language worker channel with id:########-####-####-#################

22:29:45Z [Verbose] No initialized worker channels for runtime 'dotnet-isolated'. Delaying future invocations

22:29:45Z [Verbose] Restarting worker channel for runtime: 'dotnet-isolated'

22:29:45Z [Verbose] Adding jobhost language worker channel for runtime: dotnet-isolated. workerId:########-####-####-#################

22:29:45Z [Information] Worker process started and initialized.

Instead of the Restarting worker channel for runtime: 'dotnet-isolated' step, this error is occasionally logged instead:

[Error] Exceeded language worker restart retry count for runtime:dotnet-isolated. Shutting down and proactively recycling the Functions Host to recover

[Verbose] Hosting stopping

[Verbose] Stopping file watchers.

[Verbose] Waiting for RpcFunctionInvocationDispatcher to shutdown

[Verbose] Draining invocations from language worker channel completed. Shutting down 'RpcFunctionInvocationDispatcher'

[Information] Stopping JobHost

[Verbose] Stopping ScriptHost instance '########-####-####-#################'.

[Information] Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.Http.HttpTriggerAttributeBindingProvider+HttpTriggerBinding+NullListener' for function '######'

[Information] Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.Http.HttpTriggerAttributeBindingProvider+HttpTriggerBinding+NullListener' for function '######'

[Information] Job host stopped

[Verbose] Stopped ScriptHost instance '########-####-####-#################'.

[Verbose] Hosting stopped

[Verbose] Disposing ScriptHost.

[Verbose] Initiating background SyncTriggers operation

From the Azure Portal > Diagnose and solve problems > Availability and Performance > Functions that are not triggering > Last 24 hours:

Exception Type	Total Occurrences	Exception Message
System.TimeoutException	123	Loading function failed. The operation has timed out.
Exceeded language worker restart retry count for runtime:dotnet-	41	Exceeded language worker restart retry count for runtime:dotnet-isolated. Shutting down and proactively recycling the Functions Host to recover
System.Net.Sockets.SocketException	19	Exception encountered while listening to EventStream. An existing connection was forcibly closed by the remote host.

The first request after the Function App idles for 20 minutes or more takes around 5 - 10 seconds longer to process, even with the Function App set to Always On. Any requests directly after that is processed within a second.

Environment Variables:

FUNCTIONS_WORKER_RUNTIME: dotnet-isolated

FUNCTIONS_EXTENSION_VERSION: ~4

Answer 1

The errors stopped and the initial requests are no longer delayed after changing Settings > Environment variables > FUNCTIONS_WORKER_RUNTIME from dotnet-isolated to dotnet in all Function Apps in the App Service Environment.