CODEWITHSUNDEEP
c#.netsocketssharepointcsom
Chuck Norris
Chuck Norris

Reputation: 15190

SharePoint CSOM request hangs

This is something rare, but it came up several times already. This time a request to download a file got stuck and our software is hanging for almost 17 hours already. We took a ProcDump and it shows that the stalling happened during "Socket.Receive" call which is a result of the Microsoft.SharePoint.Client.ClientContext.ExecuteQuery call.

Here is the ProcDump stacktrace:

ChildEBP RetAddr  Caller, Callee
08a5de44 6f45a7b5 mswsock!SockWaitForSingleObject+0x125, calling ntdll_77a80000!NtWaitForSingleObject
08a5dea0 6f46c7a8 mswsock!WSPRecv+0x2e8, calling mswsock!SockWaitForSingleObject
08a5df14 75681560 ws2_32!recv+0x100
08a5df68 72bccbff (MethodDesc 729f4eb4 +0x3b DomainBoundILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags))
08a5df90 72bccbff (MethodDesc 729f4eb4 +0x3b DomainBoundILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags))
08a5dfac 72b6c3dd (MethodDesc 729e2b04 +0xbd System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)), calling 72aed3fc
08a5dfd4 72b6c2ce (MethodDesc 729e2af8 +0x1e System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags)), calling (MethodDesc 729e2b04 +0 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef))
08a5dffc 72b6c203 (MethodDesc 729e0590 +0x83 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)), calling (MethodDesc 729e2af8 +0 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags))
08a5e030 72b5335d (MethodDesc 729e0ad0 +0x21 System.Net.FixedSizeReader.ReadPacket(Byte[], Int32, Int32))
08a5e04c 72b9b65e (MethodDesc 72a3fcb8 +0xa2 System.Net.Security._SslStream.StartFrameBody(Int32, Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)), calling (MethodDesc 729e0ad0 +0 System.Net.FixedSizeReader.ReadPacket(Byte[], Int32, Int32))
08a5e06c 72b9b3ba (MethodDesc 72a3fcac +0x92 System.Net.Security._SslStream.StartFrameHeader(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)), calling (MethodDesc 72a3fcb8 +0 System.Net.Security._SslStream.StartFrameBody(Int32, Byte[], Int32, Int32, System.Net.AsyncProtocolRequest))
08a5e094 72b9b087 (MethodDesc 72a3fca0 +0x77 System.Net.Security._SslStream.StartReading(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)), calling (MethodDesc 72a3fcac +0 System.Net.Security._SslStream.StartFrameHeader(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest))
08a5e0bc 72b9af60 (MethodDesc 729e0374 +0xc8 System.Net.Security._SslStream.ProcessRead(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)), calling (MethodDesc 72a3fca0 +0 System.Net.Security._SslStream.StartReading(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest))
08a5e0fc 72b532f3 (MethodDesc 72a4170c +0x4f System.Net.TlsStream.Read(Byte[], Int32, Int32)), calling (MethodDesc 729e0374 +0 System.Net.Security._SslStream.ProcessRead(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest))
08a5e12c 72b6c0d0 (MethodDesc 72a3d738 +0x68 System.Net.PooledStream.Read(Byte[], Int32, Int32))
08a5e164 73216b6c (MethodDesc 72a3aa78 System.Net.ChunkParser.HandlePayload())
08a5e188 72b8ec41 (MethodDesc 729db7ec +0x3d System.Net.ChunkParser.ProcessResponse()), calling (MethodDesc 72a3aa78 +0 System.Net.ChunkParser.HandlePayload())
08a5e19c 730428a7 (MethodDesc 72a3aa24 +0x2f System.Net.ChunkParser.Read(Byte[], Int32, Int32)), calling (MethodDesc 729db7ec +0 System.Net.ChunkParser.ProcessResponse())
08a5e1c0 7320b446 (MethodDesc 72a3ee4c System.Net.ConnectStream.ReadWithoutValidation(Byte[], Int32, Int32, Boolean)), calling (MethodDesc 72a3aa24 +0 System.Net.ChunkParser.Read(Byte[], Int32, Int32))
08a5e1f4 72b6d885 (MethodDesc 729dff68 +0xe5 System.Net.ConnectStream.Read(Byte[], Int32, Int32)), calling (MethodDesc 72a3ee4c +0 System.Net.ConnectStream.ReadWithoutValidation(Byte[], Int32, Int32, Boolean))
08a5e230 013b8087 (MethodDesc 013849a0 +0xb7 Microsoft.SharePoint.Client.Mime.BufferedReadStream.Read(Byte[], Int32, Int32))
08a5e250 013b7eb2 (MethodDesc 0826e348 +0x2a Microsoft.SharePoint.Client.Mime.DelimitedStreamReader.Read(DelimitedReadStream, Byte[], Int32, Int32))
08a5e26c 013b7d08 (MethodDesc 0826e6a0 +0x48 Microsoft.SharePoint.Client.Mime.DelimitedStreamReader+DelimitedReadStream.Read(Byte[], Int32, Int32)), calling (MethodDesc 0826e348 +0 Microsoft.SharePoint.Client.Mime.DelimitedStreamReader.Read(DelimitedReadStream, Byte[], Int32, Int32))
08a5e28c 013b9c95 (MethodDesc 08d5ef3c +0x45 Microsoft.SharePoint.Client.ChunkStreamBuilder.CopyFrom(System.IO.Stream))
08a5e2a8 096da950 (MethodDesc 08d5abf8 +0x2a0 Microsoft.SharePoint.Client.ClientRequest.ProcessResponse()), calling (MethodDesc 08d5ef3c +0 Microsoft.SharePoint.Client.ChunkStreamBuilder.CopyFrom(System.IO.Stream))
08a5e2f8 096da228 (MethodDesc 08d5ac1c +0xc8 Microsoft.SharePoint.Client.ClientRequest.ExecuteQueryToServer(Microsoft.SharePoint.Client.ChunkStringBuilder)), calling (MethodDesc 08d5abf8 +0 Microsoft.SharePoint.Client.ClientRequest.ProcessResponse())
08a5e324 08d4cf9d (MethodDesc 08d5956c +0x18d Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()), calling (MethodDesc 08d5ac1c +0 Microsoft.SharePoint.Client.ClientRequest.ExecuteQueryToServer(Microsoft.SharePoint.Client.ChunkStringBuilder))
...our code which calls Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()

Is there something we can do with this? Or we should rely on Microsoft to fix this issue in their CSOM library?

NOTE: we use version 16.1.19927.12000 of the Microsoft.SharePointOnline.CSOM library (https://www.nuget.org/packages/Microsoft.SharePointOnline.CSOM/16.1.19927.12000).

RequestTimeout is not set explicitly, so it should use the default value (3 minutes). We usually see timeouts from time to time, which is normal. But we do not expect to see a request which hangs forever.

EDIT:

I have some updates regarding this question. Last time it happened, we did not stop the process and decided to wait. After 32.5 hours of waiting, the code finally got an IOException thrown and we were able to finish the process.

IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

We are using .NET 4.6.1.

Upvotes: 0

Views: 1449

Answers (3)

Antonio Leonardo
Antonio Leonardo

Reputation: 1862

Recently, I have this issue from a query a lo of files throw folders and subfolders recursively (more than 9000 files) in SharePoint by CSOM Approach and, after long research, I encountered this Microsoft Documentation with explanation and how to resolve this: Avoid getting throttled or blocked in SharePoint Online

And I extract this code bellow with a Extension method suggested by this Microsoft Docs article.

public static class CsomExtensions
{
    public static void ExecuteQueryWithIncrementalRetry(this ClientContext clientContext, int retryCount, int delay)
    {
        int retryAttempts = 0;
        int backoffInterval = delay;
        int retryAfterInterval = 0;
        bool retry = false;
        ClientRequestWrapper wrapper = null;
        if (retryCount <= 0)
            throw new ArgumentException("Provide a retry count greater than zero.");
        if (delay <= 0)
            throw new ArgumentException("Provide a delay greater than zero.");

        // Do while retry attempt is less than retry count
        while (retryAttempts < retryCount)
        {
            try
            {
                if (!retry)
                {
                    clientContext.ExecuteQuery();
                    return;
                }
                else
                {
                    //increment the retry count
                    retryAttempts++;

                    // retry the previous request using wrapper
                    if (wrapper != null && wrapper.Value != null)
                    {
                        clientContext.RetryQuery(wrapper.Value);
                        return;
                    }
                    // retry the previous request as normal
                    else
                    {
                        clientContext.ExecuteQuery();
                        return;
                    }
                }
            }
            catch (WebException ex)
            {
                var response = ex.Response as HttpWebResponse;
                // Check if request was throttled - http status code 429
                // Check is request failed due to server unavailable - http status code 503
                if (response != null && (response.StatusCode == (HttpStatusCode)429 || response.StatusCode == (HttpStatusCode)503))
                {
                    wrapper = (ClientRequestWrapper)ex.Data["ClientRequest"];
                    retry = true;

                    // Determine the retry after value - use the `Retry-After` header when available
                    string retryAfterHeader = response.GetResponseHeader("Retry-After");
                    if (!string.IsNullOrEmpty(retryAfterHeader))
                    {
                        if (!Int32.TryParse(retryAfterHeader, out retryAfterInterval))
                        {
                            retryAfterInterval = backoffInterval;
                        }
                    }
                    else
                    {
                        retryAfterInterval = backoffInterval;
                    }

                    // Delay for the requested seconds
                    Thread.Sleep(retryAfterInterval * 1000);

                    // Increase counters
                    backoffInterval = backoffInterval * 2;
                }
                else
                {
                    throw;
                }
            }
        }
        throw new MaximumRetryAttemptedException($"Maximum retry attempts {retryCount}, has be attempted.");
    }
}

[Serializable]
public class MaximumRetryAttemptedException : Exception
{
    public MaximumRetryAttemptedException(string message) : base(message) { }
}

Upvotes: 0

Vineet Desai
Vineet Desai

Reputation: 932

I can think of couple of reasons:-

  1. Connection is being made over TLS 1.0. Try installing OS updates, updating .NET framework, making the connection over TLS 1.2 by explicitly setting. Ref: https://learn.microsoft.com/en-us/dotnet/framework/network-programming/tls

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

  1. Check the Firewall rules to ensure they are not blocking the connection

Upvotes: 1

Ben L
Ben L

Reputation: 744

Without any code example I’m going to have to make some usage assumptions. If you are using it in a similar way to this, we can at least break out of the stall using a cancellation token. Then in the catch you can handle it as you see fit. It may be as simple as retrying, but without debugging the behavior it is hard to say.

    try
    {
        var cancelToken = new CancellationTokenSource();
        var ms = 600_000;
        cancelToken.CancelAfter(ms);
        await ExecuteQueryWithTimeout(cancelToken);
    }
    catch (TaskCanceledException)
    {
        Console.WriteLine("Tasks cancelled: timed out");
    }
    finally
    {
        cancelToken.Dispose();
    }

Async method for ExecuteQuery

private static async Task ExecuteQueryWithTimeout(CancellationToken token)
{
    await Task.Run(()=> {
      Uri uri = new Uri(folderUrl);
      SP.Folder folder = web.GetFolderByServerRelativeUrl(uri.AbsolutePath);
      context.Load(folder);
      context.ExecuteQuery();
    });
}

Upvotes: 1

Related Questions