pbuf Reference Error in Async Web Server on ESP32

Question

I'm trying to build an asynchronous web server for file serving (mostly for upload) on an ESP32 dev board. Basically, I'm merging the official examples of file serving and async handlers such that, for example, multiple clients can upload files concurrently.

Unfortunately, I'm facing a weird issue about the pbuf struct which underlies the low-level socket implementation of LWiP, which is what the ESP32 uses under the hood for a TCP connection as I understand. Precisely, I get this error:

assert failed: pbuf_free /IDF/components/lwip/lwip/src/core/pbuf.c:753 (pbuf_free: p->ref > 0)

It happens at apparently random times during the upload of a file, and it seems that something frees the pbuf while is still in use or that it got freed twice. I have two workers (handled by FreeRTOS), and the upload function is executed by a worker to make it be async: without workers/tasks the code works just fine.

I even tried to put a mutex around the httpd_req_recv() (in the upload_post_handler function, defined below), which receives the file data in chunks using an underlying socket to handle the request.

Here is my code (simplified):

#include <stdlib.h>
#include <string.h>
#include <sys/param.h>
#include <sys/unistd.h>
#include <sys/stat.h>
#include <dirent.h>

#include <esp_err.h>
#include <esp_log.h>
#include <esp_event.h>
#include <esp_system.h>

#include <esp_vfs.h>
#include <esp_http_server.h>
#include "esp_tls_crypto.h"

#include "freertos/FreeRTOS.h"
#include "freertos/semphr.h"
#include "freertos/task.h"

// async helpers as defined in 
https://github.com/espressif/esp-idf/blob/master/examples/protocols/http_server/async_handlers/main/main.c#L41-L222

static bool is_on_async_worker_thread(void);
static esp_err_t submit_async_req(httpd_req_t *req, httpd_req_handler_t handler);
static void async_req_worker_task(void *p);
esp_err_t start_async_req_workers(void);

// the file upload handler
static esp_err_t upload_post_handler(httpd_req_t *req)
{
    char file_path[FILE_PATH_MAX];
    FILE *fd = NULL;
    struct stat file_stat;

    /* checks on file and URI skipped */
    
    fd = fopen(file_path, "w");
    if (!fd) {
        return ESP_FAIL;
    }

    /* Retrieve the pointer of temporary storage */
    char *buf = ((file_server_data_t *) req->user_ctx)->scratch;
    int received;

    /* Content length of the request gives
     * the size of the file being uploaded */
    int remaining = req->content_len;

    while (remaining > 0) {
        /* Receive the file part by part into a buffer */
        // SCRATCH_BUFSIZE is 1024
        if ((received = httpd_req_recv(req, buf, MIN(remaining, SCRATCH_BUFSIZE))) <= 0) {
            if (received == HTTPD_SOCK_ERR_TIMEOUT) {
                /* Retry if timeout occurred */
                continue;
            }

            /* In case of unrecoverable error,
            * close and delete the unfinished file*/
            fclose(fd);
            unlink(file_path);

            return ESP_FAIL;
        }
       
        /* Write buffer content to file on storage (SD Card) */
        if (received && (received != fwrite(buf, 1, received, fd))) {
            /* Couldn't write everything to file!
            * Storage may be full? */
            fclose(fd);
            unlink(file_path);

            return ESP_FAIL;
        }

        /* Keep track of remaining size of
         * the file left to be uploaded */
        remaining -= received;
    }

    /* Close file upon upload completion */
    fclose(fd);

    /* Redirect to root */
    httpd_resp_set_status(req, "303 See Other");
    httpd_resp_set_hdr(req, "Location", "/");
    httpd_resp_sendstr_chunk(req, NULL);

    return ESP_OK;
}

// my async handler for upload
static esp_err_t async_upload_handler(httpd_req_t *req) {
    // This handler is first invoked on the httpd thread.
    // In order to free the httpd thread to handle other requests,
    // we must resubmit our request to be handled on an async worker thread.
    if (is_on_async_worker_thread() == false) {
        // submit
        if (submit_async_req(req, upload_post_handler) == ESP_OK) {
            return ESP_OK;
        } else {
            httpd_resp_set_status(req, "503 Busy");
            httpd_resp_sendstr(req, "<div> no workers available. server busy.</div>");
            return ESP_OK;
        }
    }

    return upload_post_handler(req);
}

This code is run by the web server, and the upload part is triggered by a form inside an HTML page. On form submit the following (simplified) JS code is executed:

    function upload() {
            var filePath = document.getElementById("filepath").value;
            var upload_path = "/upload/" + current_loc() + filePath;
            var file = document.getElementById("newfile").files[0];

            var xhttp = new XMLHttpRequest();
                
            xhttp.onreadystatechange = () => {
                // if DONE
                if (xhttp.readyState == 4) {
                    const status = xhttp.status;
                    
                    if (status >= 200 && status < 400) {
                        // request completed successfully
                        document.open();
                        document.write(xhttp.responseText);
                        document.close();
                    } else {
                        // error happened in the request, redirect to error page
                        location.replace("/error.html");
                    }
                }
            }
                
            xhttp.open("POST", upload_path, true);
            xhttp.send(file);         
        }

When I try to upload a file, the server manages to receive some chunks but then the error happens at random points during the reception of the file. The issue still happens for files that are small enough to be received in a single chunk.

Backtrace:

assert failed: pbuf_free /IDF/components/lwip/lwip/src/core/pbuf.c:753 (pbuf_free: p->ref > 0)

0x400819c2: panic_abort at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_system/panic.c:452

0x400890a9: esp_system_abort at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_system/port/esp_system_chip.c:84

0x400903d5: __assert_func at C:/Espressif/frameworks/esp-idf-v5.1.2/components/newlib/assert.c:81

0x4011f833: pbuf_free at C:/Espressif/frameworks/esp-idf-v5.1.2/components/lwip/lwip/src/core/pbuf.c:753 (discriminator 1)

0x4011bdc6: lwip_recv_tcp at C:/Espressif/frameworks/esp-idf-v5.1.2/components/lwip/lwip/src/api/sockets.c:1010

0x4011d2bb: lwip_recvfrom at C:/Espressif/frameworks/esp-idf-v5.1.2/components/lwip/lwip/src/api/sockets.c:1218

0x4011d3ba: lwip_recv at C:/Espressif/frameworks/esp-idf-v5.1.2/components/lwip/lwip/src/api/sockets.c:1283

0x400ec0a0: recv at C:/Espressif/frameworks/esp-idf-v5.1.2/components/lwip/include/lwip/sockets.h:38
 (inlined by) httpd_default_recv at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_http_server/src/httpd_txrx.c:660

0x400eb9e3: httpd_recv_with_opt at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_http_server/src/httpd_txrx.c:121

0x400eba16: httpd_recv at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_http_server/src/httpd_txrx.c:142

0x400ebfa5: httpd_req_recv at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_http_server/src/httpd_txrx.c:542

0x400ecd09: httpd_req_delete at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_http_server/src/httpd_parse.c:815

0x400eb82f: httpd_sess_process at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_http_server/src/httpd_sess.c:430 (discriminator 15)

0x400eab30: httpd_process_session at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_http_server/src/httpd_main.c:255 (discriminator 15)

0x401640ab: httpd_sess_enum at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_http_server/src/httpd_sess.c:50 (discriminator 1)

0x400eb212: httpd_server at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_http_server/src/httpd_main.c:305

0x400eb27a: httpd_thread at C:/Espressif/frameworks/esp-idf-v5.1.2/components/esp_http_server/src/httpd_main.c:327 (discriminator 15)

0x4008bb89: vPortTaskWrapper at C:/Espressif/frameworks/esp-idf-v5.1.2/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:162

Any help or suggestion is greatly appreciated. Also tell me if you need more info about the issue.