WebServer vs esp_http_server performance

Question

I'm trying to make a small web server for my ESP32 using the Arduino environment. I would like to switch from Arduino's WebServer to ESP-IDF's esp_http_server because it has useful features like websockets and connection keep-alive. However I noticed that esp_http_server takes longer to respond to requests, which surprised me as it is the standard server in the ESP-IDF.

So I investigated using the following program

#include <WiFi.h>
#include <WebServer.h>
extern "C" {
#include "esp_http_server.h"
}

WebServer arduinoWebServer(81); // Arduino web server
httpd_handle_t espWebServer; // ESP-IDF web server

void setup() {
  Serial.begin(115200);
  WiFi.begin("ssid", "password");
    while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); }
  Serial.print("Connected, IP address: "); Serial.println(WiFi.localIP());
  // WiFi.softAPConfig(IPAddress(192, 168, 4, 7), IPAddress(192, 168, 4, 254), IPAddress(255, 255, 255, 0));
  // WiFi.softAP("esptest", "12345678");

  // Arduino web server
  arduinoWebServer.on("/hello", HTTP_GET, arduinoHandler);
  arduinoWebServer.begin();

  // ESP-IDF web server
  httpd_config_t config = HTTPD_DEFAULT_CONFIG();
  config.server_port = 82;
  httpd_start(&espWebServer, &config);
  httpd_uri_t helloUri = {
    .uri = "/hello",
    .method = HTTP_GET,
    .handler = espHandler,
    .user_ctx = NULL
  };
  httpd_register_uri_handler(espWebServer, &helloUri);
}

void loop() {
  arduinoWebServer.handleClient();
}

void arduinoHandler() {
  arduinoWebServer.send(200, "text/html", "<h1>Hello</h1>");
}

esp_err_t espHandler(httpd_req_t *req) {
  httpd_resp_set_type(req, "text/html");
  httpd_resp_send(req, "<h1>Hello</h1>", HTTPD_RESP_USE_STRLEN);
  return ESP_OK;
}

And here are the results

Wifi mode	Library	Average response time in chrome
Access point	Arduino WebServer	19.3 ms
Access point	ESP-IDF esp_http_server	61.2 ms
Station	Arduino WebServer	197 ms
Station	ESP-IDF esp_http_server	493.6 ms

I tested it using Wifi in both access point and station mode, on multiple ESP32, installed ESP-IDF to try esp_http_server without Arduino, and always got similar results. As you can see it's not unusable, but even for an ESP32, a 500 ms response time for HTTP seems like a lot to me. Where this slowness comes from? Can I do something about it? I probably can, since Arduino's library is faster?