Segmentation fault when using mbedtls to generates RSA key pair

Mbed TLS Generic
1

Hi,

I am a new tried to make some test with mbedtls; I want to call some basic function from library as a result I wrote a script to call mbedtls_gen_key functions from library.
Now I am trying to generate an RSA key pair :

mbedtls_rsa_context ctx;
mbedtls_ctr_drbg_context ctr_drbg;
mbedtls_ctr_drbg_init( &ctr_drbg );
mbedtls_rsa_init(&ctx, MBEDTLS_RSA_PKCS_V15, 0);
mbedtls_rsa_gen_key(&ctx, mbedtls_ctr_drbg_random, &ctr_drbg,2048, 65537);

I could compile the code however when I try to run the code it will said
“Segmentation fault (core dumped)”
in the command line.

Is something wrong with the pointer when I tried to called the function mbedtls_rsa_gen_key?

Best Wishes,
Klaus

2

Hi Klaus,
Is this the full flow that you are doing? If so, you are missing seeding the ctr_drbg with entropy. This probably causes your segmentation fault.
Please look at the sample application for rsa key generation and follow the article
Regards,
Mbed TLS Team member
Ron

3

HI Ron,
Yes you are right after seeding the ctr_drbg with entropy this problem is gone.
Now the code is like:
mbedtls_ctr_drbg_init( &ctr_drbg );
mbedtls_entropy_init( &entropy );
mbedtls_ctr_drbg_seed( &ctr_drbg, mbedtls_entropy_func, &entropy,
(const unsigned char *) pers,
strlen( pers ) );
mbedtls_rsa_init(&ctx, MBEDTLS_RSA_PKCS_V15, 0);
ret = mbedtls_rsa_gen_key(&ctx, mbedtls_ctr_drbg_random, &ctr_drbg,2048, 65537);
if(ret == 0)
printf( " RSA key generate successful\n ");

Now looks like I am successful generate an RSA key pair.
One question about here, I think I am successful since the function mbedtls_rsa_gen_key value is 0, however where should I found the key at this moment?
Thanks for your useful information!

Best Wishes,
Klaus

4

Hi Klaus,
Side note: ‘mbedtls_ctr_drbg_seed()’ can also fail, and you should check its return code as well.

The ctx you are giving as parameter to mbedtls_rsa_gen_key() is of type mbedtls_rsa_context which contains all the key pair information. As shown in the sample application, you can call mbedtls_rsa_export() to export the components.
Note that the gen_key application also writes the generated key into a PEM \ DER formatted file, so you can look at it to see how to write the components into a standard file.
Regards,
Mbed TLS Team member
Ron

5

Hi Ron,
Yes I will also check seeding function return code as well.
Thanks for your help again!
I have another question which is also related with mbedtls and Segmention fault.

I implemented hash256 function with intel sgx. I could compile the code however when I run the code it will say:
*** stack smashing detected ***
I defined a function call hash256 function inside enclave:

int enc_mbedtls_sha256_ret(const unsigned char *str, size_t ilen, unsigned char digest[32], int is256)
{
mbedtls_sha256((unsigned char *) str, 10, digest, 0 );
return 0;
}
I called this function from untrust zone:

enc_mbedtls_sha256_ret(enclave_id,&ret,(unsigned char *) str, 10, digest, 0 );

I have already init the init_sha256, is somewhere wrong that leading this problem?
Thanks again with your time and help!

Best Wishes,
Klaus

6

Hi Klaus,
I am less familiar with enclave, so I will ask you guiding questions.

I am assuming you have implemented your alternative Sha256 as shown in this article.

  • Is the memory region you are sending as the str accessible by enclave?
  • Can you specify whether the segmentation fault occurs within enclave or not?

side notes:

  • Why are you sending 10 as the input length instead of ilen?
  • It is better you call mbedtls_sha256_ret() instead of mbedtls_sha256(), as the latter is deprecated and the former returns an error code.
  • It is better you send is224 ( !is256 ) instead of 0;

Regards,
Ron

7

Hi RON,

I am using Mbedtls-SGX instead of Mbedtls, I noticed that it only have mbedtls_sha256() defined in sha256.h instead of mbedtls_sha256_ret().
sha256.h has the mbedtls_sha256() defined here:

void mbedtls_sha256( const unsigned char *input, size_t ilen,
unsigned char output[32], int is224 );

I rechecked my code and from your suggestion, changed calling function in the enclave.c to :
int enc_mbedtls_sha256(const unsigned char *str, size_t ilen, unsigned char digest[32], int is224)
{

mbedtls_sha256((unsigned char *) str, ilen, digest, is224 );
return 0 ;

}

The stack smashing detected is exactly happened when I tried to calling the function from enclave:
unsigned char digest[32];
char str = “helloworld”;
enc_mbedtls_sha256(enclave_id, &ret,(unsigned char *) str, 10, digest, 0 );

So looks like here is something wrong with this function I defined and calling mbedtls_sha256 function from encalve that leading to stack smashing detected.
And for the Alternative cryptography engines implementation file you send me, I think I should uncomment with
//#define MBEDTLS_SHA256_ALT
//#define MBEDTLS_SHA256_PROCESS_ALT
Then what should I do for the next? Should I created sha256_alt.h somewhere and redefine something inside?

Thanks for your time and suggestion!

Best Wishes,
Klaus

8

Hi Klaus,

I am using Mbedtls-SGX instead of Mbedtls, I noticed that it only have mbedtls_sha256() defined in sha256.h instead of mbedtls_sha256_ret().

This probably means the Mbed TLS version is not so current. According to the repository, it is version 2.6.0.
It is not an LTS version, but it shouldn’t be problematic for you, at the time being.
I can only assume that the input \ output isn’t accessible by enclave.

As mentioned in the article I referenced, If you want to use an alternative implementation of SHA256, you will need to define your context and API in aes_alt.h and add the file in your include path.
You should define MBEDTLS_SHA256_ALT in your configuration, and implement your SHA256 API as it fits in your application. For example, do you need some IPC between normal world and trusted world?
If you only want to implement an alternative for the actual SHA256 operation, you will only need to define MBEDTLS_SHA256_PROCESS_ALT and implement mbedtls_sha256_process() in a c file that is compiled in your library.
As mentioned, I am not so familiar with enclave, but according to the description of the mbedtls-SGX (note this is not distributed by us) it should be able to run in enclave.
Please check that the parameters are accessible, and that casting the const str to a non const variable should work in enclave.

9

Hi Ron,

Thanks for the time and useful suggestion, I will try it to see whether it is work or not. Like you said it should work with enclave.

Best Wishes,
Klaus

10

Hi Ron,

I changed define mbedtls_sha256_context* ctx; to mbedtls_sha256_context ctx;
And this problem is gone. It looks like worked.

I think I could move to the next, really thanks for your help again!\

Best Wishes,
Klaus

11

Hi Klaus,
Yes, if your context was a pointer on tnhe stack, and you sent that pointer to the SHA256 funciton, then yes, that would cause a segmentation fault, as the pointer doesn’t really point to a real mbedtls_sha256_context context.
I am glad you solved your problem.