I am not able to display all my log information or error messages(if any) on browser in my ASP.NET Web with Worker service project using Serilog

Question

Project description: I am using .NET Core7 which I used to built an web application with background service using asp.net web api template and worker service template. I am using Serilog provider to use the logger functionality in a more structured way and ILogger interface from Microsoft for logging information, errors messages and etc. Problem statement: I tried to configure the Serilog in Program.cs (did not use Startup.cs) using appsetting.json file. I want to log all the log information (LoggerExtensions.Information) that I implemented in my Background service and also in the Rest API Service(Controller and other files) in the Browser when the client request for data and also on Console in real time. Here I customize the Middleware to collect all the request and response logs of my application which will hel to to provide all the logs on browser when requested. But I can see the logs on the console but not on the web browser using the GET endpoint that I created in the Controller class of the MVC pattern. I will provide the code below for each of the file that I created

This is Program.cs file where I did the configuration of Serilog

    using Serilog;
    using TraceanalyzeServiceWithApi_project;
    
    public class Program
    {
        // this function helps us to configure the services and to run those services 
        public static async Task Main(string[] args)
        {
            var configuration = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile("appsettings.json")
                .Build();
            Log.Logger = new LoggerConfiguration()
                .ReadFrom.Configuration(configuration)
                .CreateLogger();
            try
            {
                var app = CreateHostBuilder(args).Build();
                app.Urls.Add("http://localhost:4242/");
                //Log.Information($"My Url {http://localhost:4242/}");
                Log.Information($"Application is running at: http://localhost:4242/");
    
    
                /*// Retrieve the logger service
                var logger_program = app.Services.GetRequiredService<ILogger<Program>>();*/
                
    #if DEBUG
                app.UseDeveloperExceptionPage();
    #endif
                app.UseRouting();
                
                app.UseMiddleware<LogToResponseMiddleware>();
                
                app.UseEndpoints(endpoints => { endpoints.MapControllers(); });
                Task.Run(() =>
                {
                    app.Services.GetRequiredService<BackgroundWorkerService>().StartAsync(CancellationToken.None);
                });
                app.Run();
    
            }
            catch (Exception ex)
            {
                Log.Fatal(ex, "The application failed to start correctly");
            }
            finally
            {
                await Log.CloseAndFlushAsync();
            }
        }
        // this function create a Host builder and configuring it
        public static WebApplicationBuilder CreateHostBuilder(string[] args)
        {
            WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
            builder.Services.AddControllers();
            builder.Services.AddSingleton<BackgroundWorkerService>();
            builder.Services.AddLogging();
            
            builder.Logging.ClearProviders();
            builder.Logging.AddSerilog();
            builder.Services.AddTransient<LogToResponseMiddleware>();
            return builder;
        }
    }

Here is my appsettings.json

    {
      "Serilog": {
        "Using": ["Serilog.Sinks.Console"],
        "MinimumLevel": {
          "Default": "Information",
          "Override": {
            "Microsoft": "Warning"
          }
        },
        "WriteTo": [
          {
            "Name": "Console"
          }
        ]
      }
    }

Here is my customize Middleware to get all the log information, please do check this file (new in this customization).

    using Microsoft.AspNetCore.Http;
    
    namespace TraceanalyzeServiceWithApi_project
    {
        public class LogToResponseMiddleware : IMiddleware
        {
            private readonly ILogger<LogToResponseMiddleware> _logger;
    
            public LogToResponseMiddleware(ILogger<LogToResponseMiddleware> logger)
            {
                _logger = logger;
            }
    
            public async Task InvokeAsync(HttpContext context, RequestDelegate next)
            {
                // Capture log messages
                var logMessages = new List<string>();
    
                // Intercept the response stream
                var originalResponseBody = context.Response.Body;
                var responseBodyStream = new MemoryStream();
                context.Response.Body = responseBodyStream;
    
                context.Response.OnStarting(() =>
                {
                    responseBodyStream.Seek(0, SeekOrigin.Begin);
                    using var reader = new StreamReader(responseBodyStream);
                    var responseBody = reader.ReadToEnd();
                    logMessages.Add(responseBody); // Add log messages to the list
                    _logger.LogInformation("Response sent: {Response}", responseBody);
                    return Task.CompletedTask;
                });
    
                await next(context);
    
                // Send log messages to the browser
                var logs = string.Join("\n", logMessages);
                await context.Response.WriteAsync(logs);
                context.Response.Body = originalResponseBody;
            }
        }
    }

This is my Controller, which handles the POST request and also provide GET endpoint route.

    using System.Text;
    using domainFiles;
    using Microsoft.AspNetCore.Mvc;
    using Newtonsoft.Json.Linq;
    using OfficeOpenXml.Export.ToDataTable;
    using TraceanalyzeServiceWithApi_project.Utilities;
    //using  
    
    namespace TraceanalyzeServiceWithApi_project.Controllers
    {
        [ApiController]
        [Route("api/Trace")]
        public class TraceController : ControllerBase
        {
            public static int CounterTraceFileInQueue;
           // public static Queue<string> traceDataQueue = new Queue<string>();
            public static Queue<(string jsonContent, string TracePath, string testname)> exam_json = new Queue<(string, string, string)>();
            
            public static string? jsonDataExam;
            public static string test_name;
            private readonly ILogger<TraceController> _loggerTraceController;
            public TraceController(ILogger<TraceController> logger)
            {
                _loggerTraceController = logger;
            }
            
            [HttpPost("process")]
            public async Task<ActionResult<string>> Post()
            {
                using (StreamReader reader = new StreamReader(Request.Body, Encoding.UTF8))
                {
    
                    try
                    {
                        jsonDataExam = await reader.ReadToEndAsync();
                        var jsonContent = jsonDataExam;
                        if (jsonContent!=null)
                        {
                            _loggerTraceController.LogInformation("Data Received from EXAM");
                            
                            //exam_json.Enqueue(jsonContent);
                            var loggerWrapper = new LoggerWrapper<JsonMetaDataExtractor>(_loggerTraceController);
                            JObject jsonObject= JsonMetaDataExtractor.JsonObject_Creator_for_ExamData(jsonContent);
                            string trace_path = JsonMetaDataExtractor.Extract_tracePath_from_metaData(jsonObject, loggerWrapper);
                            test_name= DirectoryInitialization.jsonDump(jsonContent);
                            exam_json.Enqueue((jsonContent,trace_path,test_name));
                            int traceCount = exam_json.Count;
                            if (trace_path != null)
                            {
                                CounterTraceFileInQueue += 1;
                            }
                            _loggerTraceController.LogInformation($"Number of tarce files to process {traceCount}");
                            jsonDataExam = null;
                        }
                        _loggerTraceController.LogInformation("POST request accepted.");
                    }
                    catch (Exception ex)
                    {
                        _loggerTraceController.LogError(ex, "Error writing to file");
                        return StatusCode(500, "Internal Server Error"); // You may want to return a more appropriate response based on your application's needs.
                    }
    
                    
                    return Ok(" Your data is receieved");
                }
            }
            
            [HttpGet ("log")]
            public IActionResult Get()
            {
                
                return Ok("Service is Alive hurray!!!");
            }
        }
    }

And here is my Background Worker service which start process the message (trace-file). The Background worker service is running continuouls which helps for processing the message that we receive from POST request.

    using domainFiles;
    using Microsoft.OpenApi.Models;
    using T___________________________________File;
    using TTT________________________________Options;
    using TraceanalyzeServiceWithApi_project.Utilities;
    using TraceanalyzeServiceWithApi_project.AnalyzeFrame;
    using TraceanalyzeServiceWithApi_project.Controllers;
    using Newtonsoft.Json.Linq;
    
    using TraceanalyzeServiceWithApi_project.Controllers;
    
    
    namespace TraceanalyzeServiceWithApi_project
    {
        public class BackgroundWorkerService : IHostedService
        {
            public static string traceRequest;
            public static string processed_testName;
            
            private readonly ILogger<BackgroundWorkerService> _logger;
    
            public BackgroundWorkerService(ILogger<BackgroundWorkerService> logger)
            {
                _logger = logger;
            }
            public Task StartAsync(CancellationToken stoppingToken)
            {
                _logger.LogInformation("Backgroung service entry point");
                while (!stoppingToken.IsCancellationRequested) // background service implementation
                {
                    while (TraceController.exam_json.Count > 0)
                    {
                        _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
                        var foo = TraceController.exam_json.Dequeue();
                        traceRequest = foo.TracePath;
                        processed_testName= foo.testname;
                        string jsonContent = foo.jsonContent;
                        _logger.LogInformation($"the trace file named {traceRequest} is under processing");
                        ProcessTraceRequest(traceRequest, processed_testName, jsonContent); 
    
                    }
                }
                return Task.CompletedTask;
            }
    
            public Task StopAsync(CancellationToken cancellationToken)
            {
                _logger.LogInformation("Stopping the BackgorundWorkerService at: {time}", DateTimeOffset.Now);
                return Task.CompletedTask;
            }
    
            // main logic of our background service
            private void ProcessTraceRequest(string traceRequest, string testname, string jsonContent)  
            {
                try
                {
                    _logger.LogInformation("Post Analysis started" );
                    DirectoryInitialization.folder_init(testname, jsonContent );
                    FileWriter.InitilialiseColumns();
                    //var processor = new AnalyzeClass();
                    var processor = new AnalyzeClass(new LoggerWrapper<AnalyzeClass>(_logger));
                    var options = new FrameProcessingOptions(true, null);
                    TraceProcessingChain chain = TraceProcessingChainFactory.GetCustomChain(new GeneralOptions(traceRequest,
                        true, false), options, null, processor);
                    chain.StartInParallel();
    
                    while (!chain.Done)
                    {
                        Thread.Sleep(30);
                    }
    
                }
                catch (Exception ex)
                {
                    _logger.LogError($"Problem found {ex}");
                }
            }
            
        }
    }

I have other class as well which are part of Background worker service which help to process different data of the message(trace-file) that we receive.

Output : When I run the project/application it build without error ans also run perfectly fine. I receive all the log information messages which I implemented in all the above files as well as in other file on Console but not in Browser.

    "C:\Program Files\JetBrains\JetBrains Rider 2023.2.2\plugins\dpa\DotFiles\JetBrains.DPA.Runner.exe" --handle=8956 --backend-pid=23920 --etw-collect-flags=3 --detach-event-name=dpa.detach.8956 C:/Users/10195/Documents/testhaus_postanalysis/TraceanalyzeServiceWithApi_project/bin/Debug/net7.0/TraceanalyzeServiceWithApi_project.exe
    [17:58:15 INF] Application is running at: http://localhost:4242/
    [17:59:20 INF] Post Analysis started
    [17:59:20 INF] Response sent:  Your data is receieved
    [17:59:26 INF] Data Received from EXAM
    [17:59:26 INF] Number of tarce files to process 1
    [17:59:26 INF] POST request accepted.
    [17:59:26 INF] Response sent:  Your data is receieved
    [17:59:41 INF] Data written to C:\Users\Public\PostAnalysis\small_city\CommunicationSessions.csv
    [17:59:41 INF] Processing of C:\Users\10195\Desktop\SII_Technology\New_project\tll_files\newTtlFile\012_20191206_131211_PM200_PPE_PPC_Passive_2.10.5.ttl is Complet
    ed
    [17:59:41 INF] 1 out of 2 file Completed
    [17:59:41 INF] Done                     
    [17:59:41 INF] Worker running at: 04/08/2024 17:59:41 +02:00
    [17:59:41 INF] the trace file named C:\Users\10195\Desktop\SII_Technology\New_project\tll_files\newTtlFile\012_20191206_131211_PM200_PPE_PPC_Passive_2.10.5.ttl is 
    under processing
    [17:59:41 INF] Post Analysis started
    [17:59:57 INF] Data written to C:\Users\Public\PostAnalysis\heavy_book\CommunicationSessions.csv
    [17:59:57 INF] Processing of C:\Users\10195\Desktop\SII_Technology\New_project\tll_files\newTtlFile\012_20191206_131211_PM200_PPE_PPC_Passive_2.10.5.ttl is Complet
    ed
    [17:59:57 INF] 2 out of 2 file Completed
    [17:59:57 INF] Done
    [18:00:05 INF] Response sent: 
    [18:01:29 INF] Response sent: 
    [18:01:37 INF] Response sent: 
    [18:01:45 INF] Response sent: Service is Alive hurray!!!

When I use the localhost which I provided in the Pogram.cs file with http://localhost:4242/ /api/Trace/log i.e. the GET request on my browser I does not print anything it only print on the console.

Please let me know how to show all the log messages in real time on browser when the client request the current status of the application?