ESP HTTPS OTA Begin failed

Question

I hope you're doing well. I'm currently working on an ESP-IDF project where I'm attempting to perform OTA (Over-The-Air) firmware updates using HTTPS. Specifically, I'm using the advanced HTTPS OTA example code provided by ESP-IDF.

My goal is to update the firmware on an ESP32 device by downloading the firmware binary file from an S3 bucket hosted on AWS. I have ensured that the S3 bucket has public access, and I've placed my helloworld.bin file in it.

However, when I run the advanced HTTPS OTA example code and replace the URL with the URL of my S3 bucket, I encounter the following error:

I (19042) advanced_https_ota_example: Starting Advanced OTA example
E (19052) esp_https_ota: No option for server verification is enabled in esp_http_client config.
E (19062) advanced_https_ota_example: ESP HTTPS OTA Begin failed

It seems that the ESP HTTPS OTA process is failing at the beginning stage. I've tried to troubleshoot this issue, but I'm unsure how to resolve it.

Could someone kindly guide me on how to address this error? Additionally, if there's a better or alternative way to implement OTA updates on ESP32 devices, I would greatly appreciate any suggestions or insights.Thank you very much for your time and assistance.

Best regards, Rahul B.

Answer 1

For better diagnostics you should enable some more traces. To do that add the following section to your platformio.ini:

build_flags =
    -DCORE_DEBUG_LEVEL=5 ; 4=DEBUG, 5=VERBOSE

Additionally you need to enable that log level in runtime, please call esp_log_level_set at the beginning of your ota task or earlier:

void advanced_ota_example_task(void *pvParameter)
{
    esp_log_level_set("*", ESP_LOG_VERBOSE);

When it works correctly your output should look like this

[  7439][V][ssl_client.cpp:68] start_ssl_client(): Starting socket
[  7494][V][ssl_client.cpp:146] start_ssl_client(): Seeding the random number generator
[  7503][V][ssl_client.cpp:155] start_ssl_client(): Setting up the SSL/TLS structure...
[  7511][V][ssl_client.cpp:190] start_ssl_client(): Attaching root CA cert bundle
[  7524][V][ssl_client.cpp:254] start_ssl_client(): Setting hostname for TLS session...
[  7532][V][ssl_client.cpp:269] start_ssl_client(): Performing the SSL/TLS handshake...
[  7559][D][esp_crt_bundle.c:108] esp_crt_verify_callback(): 147 certificates in bundle
[  7587][I][esp_crt_bundle.c:142] esp_crt_verify_callback(): Certificate validated
[  8043][D][esp_crt_bundle.c:108] esp_crt_verify_callback(): 147 certificates in bundle
[  8217][I][esp_crt_bundle.c:142] esp_crt_verify_callback(): Certificate validated

esp_crt_verify_callback() is called 2 times because there is one intermediate CA in my case and the function is called for each intermediate and the root CA.

You can lower the log level at the end of the task or better remove it in production version.

To make the certificate bundle work you also need to enable it in the config struct and set the member .crt_bundle_attach:

    esp_http_client_config_t config = {
        .url = CONFIG_EXAMPLE_FIRMWARE_UPGRADE_URL,
        .auth_type = HTTP_AUTH_TYPE_NONE,
        .timeout_ms = CONFIG_EXAMPLE_OTA_RECV_TIMEOUT,
        .max_redirection_count = 5,
        .buffer_size = CONFIG_EXAMPLE_HTTP_REQUEST_SIZE, // without setting this, we get an error "esp_https_ota: Reached max_authorization_retries (401)"
        .buffer_size_tx = 2048, // TODO: adjust to request length. The github download url is alone 600 bytes long
        .crt_bundle_attach = arduino_esp_crt_bundle_attach,
        .keep_alive_enable = true,
        .keep_alive_interval = 30,
        .keep_alive_count = 300,
    };

You can see that I use the Arduino Framework, in IDF your need to use esp_crt_bundle_attach instead. Please also check here.

For ArduinoSDK it is additionally needed to set that bundle for a WifiClientSecure instance:

  wifiClient.setCACertBundle(tt_x509_crt_bundle);

In IDF Framework there is a fallback implemented which is missing in Arduino Framework, therefore there you don't need the setCACertBundle call:

esp_err_t esp_crt_bundle_attach(void *conf)
{
    // If no bundle has been set by the user then use the bundle embedded in the binary
    if (s_crt_bundle.crts == NULL) {
        ret = esp_crt_bundle_init(x509_crt_imported_bundle_bin_start, x509_crt_imported_bundle_bin_end - x509_crt_imported_bundle_bin_start);

here you can find instructions how to add the certificate bundle in VS Code+PlatformIO environment.
Finally you need to add such a line in platformio.ini.

board_build.embed_files = data/cert/x509_crt_bundle.bin

Additionally I figured out that in my environment there is a bug in the file esp_https_ota/src/esp_https_ota.c in esp_https_ota_get_img_desc(). The calculated offset is wrong. This might depend on your environment and version, e.g. I'm using VSCode+PlatformIO+Arduino Framework.

Here the fix for my environment:

    // this line is wrong:
    //const int app_desc_offset = sizeof(esp_image_header_t) + sizeof(esp_image_segment_header_t);

    // this line works for me
    const int app_desc_offset = 32;

Michael