MbedTLS RSA gen key calloc failing and hardfault

Hi Everyone,
I’m trying to integrate the mbedTLS library into my existing project and facing some issues while trying to create the RSA keys.
Before I get into the issue, I would like to give some insight about the hardware and the software:

  1. Kinetis K24 ARM Cortex-M4
  2. Using MQXLite (RTOS)
  3. Compiling and running through Kinetis Design Studio (based on Eclipse)
  4. using the latest from the github branch of mbedtls

I initially tested the mbedTLS lib on Ubuntu and it worked perfectly. I even created my own custom example of rsa init, encrypt and decrypt based on the 3 rsa example files in programs/pkey so I can easily integrate it in my exisiting firmware.

While trying to integrate the library in my firmware, I provided a custom random generator function as well by using the mbedtls_hardware_poll(). Tested that out and works okay!

Now coming to the issue:
The calloc() call inside the mbedtls_rsa_gen_key() is always failing specifically from line 559 from rsa.c which is MBEDTLS_MPI_CHK( mbedtls_mpi_lset( &ctx->E, exponent ) );
This eventually is leading to a hard fault as well (maybe not related to calloc() failing but I’m not sure)

When I put a break point where it’s failing on the calloc(), I see it’s trying to grow one limb of size 4 bytes. I tried calling calloc() and free() in my init function and allocated 10 bytes of uint32_t and it worked fine.

This is my init code:

 * \brief          This function initializes the RSA.
 * \note           The function creates all the necessary keys (public, private)
 *                 and other equation values required for RSA encryption/decryption
 * \param rsa       The RSA context to initialize. This must not be \c NULL.
 * \param entropy   The entropy context to initialize. This must not be \c NULL.
 * \param ctr_drbg  The CTR_DRBG context to initialize. This also must not be \c NULL.
 * \param pers      Personalization data, that is device-specific identifiers. Can be NULL.
 * \return          \c 1 on success
 * \return          \c -1 on failure
int rsa_init(mbedtls_rsa_context *rsa, mbedtls_entropy_context *entropy,
             mbedtls_ctr_drbg_context *ctr_drbg, const char *pers) {

    int ret = 1;

    // init rnga module

    // init rsa contexts
    mbedtls_rsa_init(rsa, MBEDTLS_RSA_PKCS_V15, 0);

    if( ( ret = mbedtls_ctr_drbg_seed(ctr_drbg, mbedtls_entropy_func, entropy,
                               (const unsigned char *) pers,
                               strlen( pers ) ) ) != 0 )
        //print_nonblocking(TIME_STAMP, DEBUG, " failed\n  ! mbedtls_ctr_drbg_seed returned %d\n", ret );
        return -1;

    if( ( ret = mbedtls_rsa_gen_key(rsa, mbedtls_ctr_drbg_random, ctr_drbg, KEY_SIZE,
                                     EXPONENT ) ) != 0 )
        //print_nonblocking(TIME_STAMP, DEBUG,  " failed\n  ! mbedtls_rsa_gen_key returned %d\n\n", ret );
        return -1;
    return 1;

The parent function (defined in another file) that calls the rsa init:

static mbedtls_rsa_context rsa;
static mbedtls_entropy_context entropy;
static mbedtls_ctr_drbg_context ctr_drbg;
static bool is_key_generated = false;

 * \brief   API to create the RSA keys
 * \param   none
 * \return  \c SUCCESS on success
 * \return  \c FAILURE on failure
int generate_rsa_keys() {
    // if key is already generated, we need to de-init the contexts first
    if (is_key_generated) {
        rsa_deinit(&rsa, &entropy, &ctr_drbg);
        is_key_generated = false;

    // init rsa module
    int ret = rsa_init(&rsa, &entropy, &ctr_drbg, "rsa_keygen");
    if (ret == ERR) {
        print_nonblocking(TIME_STAMP, ERROR, "RSA INIT ERROR!");
        return ERR;

    // print the keys
    print_nonblocking(TIME_STAMP, DEBUG, "\n Public key (N - %d): ", rsa.N.n);
    for (uint16_t i=0; i<rsa.N.n; i++) {
        print_nonblocking(NO_TIME_STAMP, CLI_RESPONSE, "%02lX ", rsa.N.p[i]);

    print_nonblocking(TIME_STAMP, DEBUG, "\n Public exponent (E - %d): ", rsa.E.n);
    for (uint16_t i=0; i<rsa.E.n; i++) {
        print_nonblocking(NO_TIME_STAMP, CLI_RESPONSE, "%02lX", rsa.E.p[i]);

    print_nonblocking(TIME_STAMP, DEBUG, "\n Private key (D - %d): ", rsa.D.n);
    for (uint16_t i=0; i<rsa.D.n; i++) {
        print_nonblocking(NO_TIME_STAMP, CLI_RESPONSE, "%02lX", rsa.D.p[i]);
    print_nonblocking(NO_TIME_STAMP, CLI_RESPONSE, "\n");

    return ret;

Flags currently enabled (changed by me) in config.h:


Any ideas or suggestion why this might be happening?
I would really appreciate any help!

Thank you!

Hi All,
Any suggestions on what could be wrong and what else I should be trying?
Any help will be really appreciated.

Thank you :slight_smile:

Hi @dhavalhparikh26
Thank you for your question!

It seems to me that you are running out of memory on your platform
I would like to know the following information:

  • What is the value of KEY_SIZE?
  • Where do you set is_key_generated to true?

These could cause a repeated allocation of memory, and if KEY_SIZE is too big, then also you can run out of memory quick.
Mbed TLS Team member

Hi @roneld01
Thanks for answering my question!
To answer your questions:

  1. Value of KEY_SIZE is set to 2048
  2. Good catch! I wasn’t setting the is_key_generated flag to true anywhere yet as I was still debugging the cause of the hard fault. But now I’m setting it right after I call the rsa_init()


What should I set the KEY_SIZE to instead?


Debugging the cause of the hardfault again, it’s now crashing from mbedtls_sha512_finish_ret() getting called from mbedtls_ctr_drbg_seed()

Thank you,



I changed the KEY_SIZE to 512 instead of 2048
and disabled the MBEDTLS_SHA512_C and enabled the MBEDTLS_ENTROPY_FORCE_SHA256 and now it’s not hard-faulting any more but it’s still failing at the calloc() call inside the mbedtls_rsa_gen_key() is again from line 559 from rsa.c which is MBEDTLS_MPI_CHK( mbedtls_mpi_lset( &ctx->E, exponent ) );

Note that key size of 512 is very unsecure.
Have you considered using ECC keys instead?

It seems you have low memory issuers.
Is it only heap problem, or RAM?

Could this post help you?

In addition, You could read this article to see ways to reduce RAM usage.

Hi @roneld01
I changed the KEY_SIZE back to 2048 and it’s not crashing anymore as it’s using SHA256 now instead of SHA512. But it’s still failing on calloc() call.

How do I use ECC keys for encryption and decryption? Can you point me to how to use them?

Also, from my debugging, it’s been always failing on calloc() so looks like it’s a heap issue…

Thanks for sending the links to the articles. I’ll follow the articles and try to reduce the RAM usage.

Thank you,

Hi @dhavalhparikh26
If you need for encryption, then ECC wil not work, as it is used for sugning, and for key exchange.
However, you may consider using ECDH to exchange secret between remote peer, and the secret will be used as key material for generating symmetric keys ot be used for encryption. Is this something you may consider?

Also, from my debugging, it’s been always failing on calloc() so looks like it’s a heap issue…

My question is whether it’s only heap, or the whoel RAM is not enough. Could you use a static buffer instead of heap, as described in this article?


Hi @roneld01
I’m definitely open to consider using ECDH.
Is there an example like rsa_init, rsa_encrypt and rsa_decrypt for ecdh that inits the ecdh, encrypts the message and also decrypts the message? I’m pretty new to this and haven’t used ECDH before…

Also, I’m not sure at the moment whether the whole RAM is not enough.
Is there a way to know how much RAM would RSA require?

Using a static buffer is definitely better for us. In that case, what is the minimum size of the static buffer we should declare?

Thank you,

Hi @dhavalhparikh26

ECDH is a key exchange algorithm, the is used to exchange secrets between two peers, using the Diffie-Hellman algorithm, with Elliptic curves Cryptography.
You can look at the [tests code] (https://github.com/ARMmbed/mbedtls/blob/development/tests/suites/test_suite_ecdh.function#L302) to see how a client and server exchange a secret.
The derived symmetric key, for example, and AES 256 bit key, should be used for the encryption and decryption.

Is there a way to know how much RAM would RSA require?

It mostly depends on the value of MBEDTLS_MPI_MAX_SIZE, as this is the bottle neck in most functions. The RAM should be higher than this value, as it’s not the only value used in the RAM.

Using a static buffer is definitely better for us. In that case, what is the minimum size of the static buffer we should declare?

Again, it depends on how much heap your application uses. I suggest you run some memory analyzer tool on your PC, to understand max heap usage of your application.

Hi @roneld01
I was able to make some progress since yesterday and the library is not crashing anymore. You were right that this was a memory issue. I’m still using RSA as I thought I should try out the static memory option before moving to ECDH.

I defined a static array of 3K bytes and passed it to the mbedtls_memory_buffer_alloc_init() so that mbedtls doesn’t use calloc/malloc.

Summary of all the changes made in the config.h file to make this work on our ARM Cortex M4 system:

  1. From the config.h file of mbedtls:
    - Enabled MBEDTLS_PLATFORM_MEMORY so that mbedtls doesn’t use calloc and malloc (heap memory).
    - Enabled MBEDTLS_ENTROPY_HARDWARE_ALT to use an alternate entropy generator.
    - Enabled MBEDTLS_AES_ROM_TABLES so that mbedtls uses ROM tables for AES and not generate tables in RAM on startup.
    - Disabled MBEDTLS_FS_IO as we don’t have support for standard file system functions.
    - Enabled MBEDTLS_ENTROPY_FORCE_SHA256 so we use SHA256 instead of SHA512. The mbedtls mentions that it’s better to use SHA256 on 32-bit systems.
    - Enabled MBEDTLS_MEMORY_BUFFER_ALLOC_C to use the buffer allocator layer which is needed if we don’t wish to use calloc/malloc.
    - Disabled MBEDTLS_SHA512_C as we are using SHA256.
    - Set MBEDTLS_MPI_WINDOW_SIZE to 4. The value 6 was causing a hard fault most likely because we were running out of memory.
    - Enabled the MBEDTLS_MPI_MAX_SIZE which is default set to 1024.

With these settings the mbedtls was getting initialized and was generating the required N,E,D keys.

With the same settings I proceeded to encrypt and decrypt a message and it hard-faulted again. So I increased the static array size from 3K -> 10K bytes and also reduced the MBEDTLS_MPI_WINDOW_SIZE from 4 to 1 and now it seems to be decrypting the message but after some time it hard faults.

I’m trying to figure out what’s going on and I’ll keep you updated :slight_smile:

Thank you,

Have u solved this error? Im Having the same