ASP.NET Core 9 hangs in docker container while running sub-processes

Question

I have an ASP.NET Core 9 application running in a linux docker container.

Made using this dockerfile:

FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS base
WORKDIR /app

# Install base components
RUN apt update && \
    apt install -y curl htop nano
RUN curl -sSL https://get.docker.com/ | sh

# Set up the build env
FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build
WORKDIR /source

# Install NuGet Packages
COPY "src/Dictus.AsrEvaluator.Backend/Dictus.AsrEvaluator.Backend.csproj" "src/Dictus.AsrEvaluator.Backend/Dictus.AsrEvaluator.Backend.csproj"
COPY "src/SunShared/Dictus.Sun.Shared/Dictus.Sun.Shared.csproj" "src/SunShared/Dictus.Sun.Shared/Dictus.Sun.Shared.csproj"
RUN dotnet restore "src/SunShared/Dictus.Sun.Shared/Dictus.Sun.Shared.csproj"
RUN dotnet restore "src/Dictus.AsrEvaluator.Backend/Dictus.AsrEvaluator.Backend.csproj"

# Build the app
COPY "src/" "src/"
ARG VersionSuffix=0
RUN dotnet publish "src/Dictus.AsrEvaluator.Backend/Dictus.AsrEvaluator.Backend.csproj" -c Release -o /app /p:VersionSuffix=$VersionSuffix

# Copy the app to the final build image
FROM base AS final
WORKDIR /app
COPY --from=build /app .

# Setup defaults
HEALTHCHECK CMD curl --max-time 10 --fail http://localhost:80/health || exit 1
ENV ASPNETCORE_HTTPS_PORTS=80
ENTRYPOINT ["dotnet", "Dictus.AsrEvaluator.Backend.dll"]

But once in a while it completely hang.

I detect the hang by:

When trying to connect to the server like https://my-url.com it never produces an response. I have tried adding an endpoint which just returns a const number, and this endpoint never returns.
The docker container is running, I can docker exec into the docker container and see the dotnet process is still running. The dotnet process does not use any CPU and have normal much RAM usages. It is like the process is just idle.
top inside the docker container gives the following output:

PID USER    PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1 root      20   0  263.8g 641220 196992 S   0.0   2.0 242:33.04 dotnet

I have tried to get some information out of my application. Inside the docker container I have tried to install and run:

dotnet-counters monitor --process-id 1
dotnet-trace collect -p 1
dotnet-dump collect --process-id 1

None of them can get any information out of my application.

dotnet-trace collect and dotnet-counters monitor both completly hangs, and I cannot exit them using CTRL+C, I have to close the terminal. dotnet-dump never creates or writes anything to its output file.

The application logs just stops when it hangs, not providing any additional information.

When does it hang?

Once in a while my application gets a new job sent to it. To handle this job it prepares a number of files and folder, and then starts 2 sub-processes which runs in parallel to complete the job.

It is while the 2 sub-processes are running the main ASP.NET Core 9 application hangs. It is very random if the application hangs or not. I can run for weeks without any problems and then suddenly it just hangs, or maybe just a few days before hanging.

The ASP.NET Core 9 application starts the 2 sub-processes and listens to their STDOUT and STDERR

ProcessStartInfo startInfo = new ProcessStartInfo
{
     WindowStyle = ProcessWindowStyle.Hidden,
     RedirectStandardOutput = true,
     RedirectStandardError = true,
     UseShellExecute = false,
     FileName = "docker -v /job_data:/job_data run my_other_docker",
     Arguments = arguments,
};

using Process process = new Process
{
     StartInfo = startInfo,
     EnableRaisingEvents = true
};

// Outputs
StringBuilder stdOut = new StringBuilder();
StringBuilder stdErr = new StringBuilder();
 
// Runs the command
process.Start();

ReadStream(process.StandardOutput, stdOut, progress, true);
ReadStream(process.StandardError, stdErr, progress, false);
 
while (!process.HasExited || !isReadStdOutDone || !isReadErrOutDone)
{
     if (token.IsCancellationRequested)
     {
         process.Kill();

         token.ThrowIfCancellationRequested();
     }

     await Task.Delay(100);
}

// Reads the output
string stdout = stdOut.ToString();
string stderr = stdErr.ToString();
 
private void ReadStream(
     StreamReader stream,
     StringBuilder read,
     IProgress? progress,
     bool isStdOut)
{
     Task.Run(async () =>
     {
         var line = CleanLogLine(await stream.ReadLineAsync());
         while (line != null)
         {
             line = $"{name} - {line}";
             read.AppendLine(line);
             if (progress != null)
                 progress.Report(line);
             line = CleanLogLine(await stream.ReadLineAsync());
         }
         if (isStdOut)
             isReadStdOutDone = true;
         else
             isReadErrOutDone = true;
     });
}

private string? CleanLogLine(string? line)
{
     if (line != null &&
         line.Count(x => x == '\b') > 1000)
     {
         line = line[0..1000];
     }
     return line;
}

My application hangs WHILE the 2 sub processes are running. I can see both of the two sub processes completes successfully. So I doubt the problem is with the sub processes.

Thoughts and ideas

As far as I can see it is not a memory leak (out of memory exception).
I do not know if it is thread starvation. As I mentioned dotnet-counters and alike is not able to provide any outputs.
I do not know if it is how I handle the STDOUT and STDERR which results in a problem

How can I continue to debug and figure out why it hangs?

ASP.NET Core 9 hangs in docker container while running sub-processes

Answers (1)

Related Questions