mbedTLS on ESP8266 SDK - memory issue?

abomin3v3l · April 3, 2020, 2:10am

I don’t know if I can create new topics here since there is a new support method. If the support now is only via email, please just inform me about.

I am on ESP8266 SDK, and in my test application I have only one task/thread, which is related to mbedTLS, where I use it to write to Google’s Firebase database. After the write is finished, there is a 4 seconds delay, and then it writes again, and this is the loop, the task keep writing on the database.

On main(), I am monitoring the current heap memory available, and keeping in a variable the minimum value that was reached. Every 500ms I send both values (minimum and current values) via UART.

void min_heap (void)
{
    static uint32_t min_free = UINT32_MAX;

    for (;;) 
    {
        uint32_t current = esp_get_free_heap_size();

        if (current < min_free) 
        {
            min_free = current;
        }  

        ESP_LOGI(TAG, "HEAP. MIN=%d, NOW=%d...\r\n", min_free, current);
        vTaskDelay(500/portTICK_PERIOD_MS);
    }
}

As stated I have only one thread, and I have observed that the minimum value of free heap memory is reaching very low values like 2444 bytes (a little more than 2KB), whereas the ESP8266 has 64KB of RAM. If I just don’t create this task, the minimum heap value observed is 61KB (after some minutes running).

Why is my free heap reaching values that are so low? It seems there is a problem when freeing the memory after ssl usage for each loop iteration. This compromises the system I’m trying to develop.

Here is the code of my task:

static void https_write_task (void *pvParameters)
{
    char writeREQUEST[2048];
    char buf[512];
    int ret, flags, len;

    /////////////////////////////////////////////////////////////////////////////
    //HTTP CONTENT
    /////////////////////////////////////////////////////////////////////////////
    char msg[128];
    int msg_len;
    static uint16_t number = 1;
    char content_length_string[128];
    sprintf (msg, "{\"value\":\"%d\"}", number++);
    msg_len = strlen(msg);
    sprintf (content_length_string, "Content-Length: %d\r\n", msg_len);
    strcpy (writeREQUEST, "PUT "WEB_URL" HTTP/1.0\r\n");
    strcat (writeREQUEST, "Host: "WEB_SERVER"\r\n");
    strcat (writeREQUEST, "User-Agent: esp-idf/1.0 esp32\r\n");
    strcat (writeREQUEST, "Content-Type: application/json\r\n");
    strcat (writeREQUEST, content_length_string);
    strcat (writeREQUEST, "\r\n");
    strcat (writeREQUEST, msg);
    /////////////////////////////////////////////////////////////////////////////

    mbedtls_ssl_context ssl;
    mbedtls_net_context server_fd;
    mbedtls_ssl_session ssl_session;

    while (1)
    {
        mbedtls_ssl_init(&ssl);

        /* Hostname set here should match CN in server certificate */
        if((ret = mbedtls_ssl_set_hostname(&ssl, WEB_SERVER)) != 0)
        {
            ESP_LOGE(TAG, "mbedtls_ssl_set_hostname returned -0x%x", -ret);
            abort();
        }

        if ((ret = mbedtls_ssl_setup(&ssl, &conf)) != 0)
        {
            ESP_LOGE(TAG, "mbedtls_ssl_setup returned -0x%x\n\n", -ret);
            goto exit;
        }

        mbedtls_net_init(&server_fd);

        //ESP_LOGI(TAG, "Connecting to %s:%s...", WEB_SERVER, WEB_PORT);

        if ((ret = mbedtls_net_connect(&server_fd, WEB_SERVER, WEB_PORT, MBEDTLS_NET_PROTO_TCP)) != 0)
        {
            //ESP_LOGE(TAG, "mbedtls_net_connect returned -%x", -ret);
            goto exit;
        }

        //ESP_LOGI(TAG, "Connected.");

        mbedtls_ssl_set_bio(&ssl, &server_fd, mbedtls_net_send, mbedtls_net_recv, NULL);
        mbedtls_net_set_nonblock(&server_fd);

        //ESP_LOGI(TAG, "Performing the SSL/TLS handshake...");

        while ((ret = mbedtls_ssl_handshake(&ssl)) != 0)
        {
            if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE)
            {
                //ESP_LOGE(TAG, "mbedtls_ssl_handshake returned -0x%x", -ret);
                goto exit;
            }

            vTaskDelay(50/portTICK_PERIOD_MS);
        }

        //ESP_LOGI(TAG, "Verifying peer X.509 certificate...");

        if ((flags = mbedtls_ssl_get_verify_result(&ssl)) != 0)
        {
            /* In real life, we probably want to close connection if ret != 0 */
            //ESP_LOGW(TAG, "Failed to verify peer certificate!");
            bzero(buf, sizeof(buf));
            mbedtls_x509_crt_verify_info(buf, sizeof(buf), "  ! ", flags);
            //ESP_LOGW(TAG, "verification info: %s", buf);
        }

        else 
        {
            //ESP_LOGI(TAG, "Certificate verified.");
        }

        mbedtls_ssl_get_session(&ssl, &ssl_session);

        //ESP_LOGI(TAG, "Cipher suite is %s", mbedtls_ssl_get_ciphersuite(&ssl));

        //ESP_LOGI(TAG, "Writing HTTP request...");

        size_t written_bytes = 0;
        do 
        {
            vTaskDelay(50/portTICK_PERIOD_MS);
            ret = mbedtls_ssl_write(&ssl, (const unsigned char *)writeREQUEST + written_bytes, strlen(writeREQUEST) - written_bytes);
            
            if (ret >= 0) 
            {
                //ESP_LOGI(TAG, "%d bytes written", ret);
                written_bytes += ret;
            } 
            
            else if (ret != MBEDTLS_ERR_SSL_WANT_WRITE && ret != MBEDTLS_ERR_SSL_WANT_READ) 
            {
                //ESP_LOGE(TAG, "mbedtls_ssl_write returned -0x%x", -ret);
                goto exit;
            }

        } while(written_bytes < strlen(writeREQUEST));

        //ESP_LOGI(TAG, "Reading HTTP response...");

        do
        {
            vTaskDelay(50/portTICK_PERIOD_MS);
            len = sizeof(buf) - 1;
            bzero(buf, sizeof(buf));
            ret = mbedtls_ssl_read(&ssl, (unsigned char *)buf, len);

            if(ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE)
                continue;

            if(ret == MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY) 
            {
                //ESP_LOGI(TAG, "#### PEER CLOSE NOTIFY ####");
                ret = 0;
                break;
            }

            if(ret < 0)
            {
                //ESP_LOGE(TAG, "mbedtls_ssl_read returned -0x%x", -ret);
                break;
            }

            if(ret == 0)
            {
                //ESP_LOGI(TAG, "connection closed");
                break;
            }

            len = ret;
            //ESP_LOGD(TAG, "%d bytes read", len);

            /* Print response directly to stdout as it is read */
            for(int i = 0; i < len; i++) 
            {
                //putchar(buf[i]);
            }

        } while(1);

        mbedtls_ssl_close_notify(&ssl);

    exit:
        //mbedtls_ssl_session_reset(&ssl);
        mbedtls_ssl_session_free(&ssl_session);
        mbedtls_ssl_free(&ssl);
        mbedtls_net_free(&server_fd);

        //4 seconds delay, the will write again
        vTaskDelay(4000 / portTICK_PERIOD_MS);

        /////////////////////////////////////////////////////////////////////////////
        // UPDATE HTTP CONTENT
        /////////////////////////////////////////////////////////////////////////////
        sprintf (msg, "{\"value\":\"%d\"}", number++);
        msg_len = strlen(msg);
        sprintf (content_length_string, "Content-Length: %d\r\n", msg_len); 
        strcpy (writeREQUEST, "PUT " WEB_URL " HTTP/1.0\r\n");
        strcat (writeREQUEST, "Host: "WEB_SERVER"\r\n");
        strcat (writeREQUEST, "User-Agent: esp-idf/1.0 esp32\r\n");
        strcat (writeREQUEST, "Content-Type: application/json\r\n");
        strcat (writeREQUEST, content_length_string);
        strcat (writeREQUEST, "\r\n");
        strcat (writeREQUEST, msg);
        /////////////////////////////////////////////////////////////////////////////
    }
}

If needed, here is my setup:

void mbedTlsInit()
{
    int ret;

    mbedtls_x509_crt_init(&cacert);
    mbedtls_ctr_drbg_init(&ctr_drbg);
    mbedtls_ssl_config_init(&conf);

    mbedtls_entropy_init(&entropy);
    if((ret = mbedtls_ctr_drbg_seed(&ctr_drbg, mbedtls_entropy_func, &entropy, NULL, 0)) != 0)
    {
        ESP_LOGE(TAG, "mbedtls_ctr_drbg_seed returned %d", ret);
        abort();
    }

    ret = mbedtls_x509_crt_parse(&cacert, server_root_cert_pem_start, server_root_cert_pem_end-server_root_cert_pem_start);
    if(ret < 0)
    {
        ESP_LOGE(TAG, "mbedtls_x509_crt_parse returned -0x%x\n\n", -ret);
        abort();
    }

    if((ret = mbedtls_ssl_config_defaults(&conf, MBEDTLS_SSL_IS_CLIENT, MBEDTLS_SSL_TRANSPORT_STREAM, MBEDTLS_SSL_PRESET_DEFAULT)) != 0)
    {
        ESP_LOGE(TAG, "mbedtls_ssl_config_defaults returned %d", ret);
        abort();
    }

    mbedtls_ssl_conf_authmode(&conf, MBEDTLS_SSL_VERIFY_OPTIONAL);
    mbedtls_ssl_conf_ca_chain(&conf, &cacert, NULL);
    mbedtls_ssl_conf_rng(&conf, mbedtls_ctr_drbg_random, &ctr_drbg);
}

And here is the result of my log, the strings sent via UART every 500ms.
I was not able to put the log directly in this post due to limit of characters, but it can be seen here:

github.com

jefersonpehls/esp8266_freeRTOS_firebase/blob/master/log_firebase_v1.txt

<20200402 22:34.53:447 RX>
<LF><ESC>[0m
<20200402 22:34.53:447 RX>
<LF><ESC>[0;32mI (50453) debug: HEAP. MIN=35980, NOW=35980...
<20200402 22:34.53:947 RX>

<20200402 22:34.53:947 RX>
<LF><ESC>[0m
<20200402 22:34.53:948 RX>
<LF><ESC>[0;32mI (50953) debug: HEAP. MIN=35696, NOW=35696...
<20200402 22:34.54:445 RX>

<20200402 22:34.54:445 RX>
<LF><ESC>[0m
<20200402 22:34.54:445 RX>
<LF><ESC>[0;32mI (51453) debug: HEAP. MIN=34120, NOW=34120...
<20200402 22:34.54:946 RX>

<20200402 22:34.54:946 RX>
<LF><ESC>[0m

This file has been truncated. show original

At menuconfig of the example, the maximum fragment lenght for incoming and outgoing messages are both set to 2.5KB.

EDIT

Another test I did was to place a semaphore before the start of the while loop of “https_write_task”, and give the semaphore via UART, so then the loop execute one time completely. This was my results in terms of free heap memory after the N iterations.

before iteration 1: 51.8 (KB)
after iteration 1: 32.8
after iteration 2: 31.3
after iteration 3: 28.2
after iteration 4: 24.6
after iteration 5: 24.6
after iteration 6: 20.9
after iteration 7: 20.9
after iteration 8: 19.7
after iteration 9: 17.2
after iteration 10: 17.2
after iteration 11: 17.2
after iteration 12: 17.2 
after iteration 13: 17.2
after iteration 14: 15.2
after iteration 21: 15  
after iteration 25: 14.1
after iteration 26: 13.4
after iteration 32: 11.3
after iteration 38: 10.3
after iteration 39: 9.4
after iteration 41: 7.5 
after iteration 47: 6
after iteration 61: 5.7
after iteration 65: 4.6

On the menuconfig of mbedTLS component we have these options:

Memory allocation strategies available are:
options2

Changing to “Default alloc mode” gives the same results, minium heap reaching very low values like 2.2KB.

Regards.

roneld01 · April 5, 2020, 6:51am

Hi @abomin3v3l

Yes, as mentioned in the anouncement, Mbed TLS project is now maintained under the governace of http://trustedfirmware.org/. As such, its non Pelion related support has moved as well, to the mailing list stated in the post.
Since you are not working on Pelion, a more suitable place for you rquestion would be there.

Glancing at your issue, since adding the semaphore before the while loop shows you a higher free memory, I woudl say you are trying to do multi TLS sessions at a time, which means you are trying to allocate 5KB per session.
However, the log also shows that your minimal heap is decreasing by each iteration, which means you have some memory leak, and you aren’t freeing a resource somewhere
Regards,
Mbed Support
Ron

abomin3v3l · April 7, 2020, 9:00pm

Hi @roneld01 . Do you know how much days can they take to reply? Now has gone around 24 hours after I sent email to them.

About the topic, basically I have this:

while (1)
{
	mbedtls_ssl_init(&ssl);
	//handshake
	after handshake: mbedtls_ssl_get_session(&ssl, &ssl_session);
	//other steps, like write and read.
	exit:
	//mbedtls_ssl_session_reset(&ssl);
	mbedtls_ssl_session_free(&ssl_session);
	mbedtls_ssl_free(&ssl);
	mbedtls_net_free(&server_fd);
}

but I’m not figuring out what is wrong.

Regards.

roneld01 · April 12, 2020, 7:16am

Hi @abomin3v3l

Do you know how much days can they take to reply? Now has gone around 24 hours after I sent email to them.

I’m sorry, but I can’t answer this question, as I am not part of the project.

About the topic, basically I have this:

Do you also call :


    mbedtls_x509_crt_free( &cacert );
  
    mbedtls_ssl_config_free( &conf );
    mbedtls_ctr_drbg_free( &ctr_drbg );
    mbedtls_entropy_free( &entropy );

At the end? Although, I don’t think this would explain why every iteration reduces free Heap.
I don’t see at the moment any outstanding cause for the heap. Do you have a different implementation for mbedtls_calloc() and mbedtls_free()?
Have you tried a static analyzer on your code?
regards

Topic		Replies	Views
Mbedtls_ssl_handshake gives 0x6180 error (failed to allocate memory) after 2 hours of operation Mbed TLS	1	747	December 2, 2019
Static memory instead of the heap Mbed TLS	2	2264	July 21, 2021
Multiple HTTPS instances with mbedTLS on multithread/multitask system - ESP8266 Platform specific questions mbed_client	10	3659	April 1, 2020
Could memory leak or heap fragmentation be the issue? Crypto and SSL questions	0	671	July 29, 2021
Mbedtls_ecp_check_pubkey leaks memory on first call Mbed TLS mbed_tls	0	504	January 17, 2022

mbedTLS on ESP8266 SDK - memory issue?

Related topics