Why can't a mbedtls_ssl_config be used simultaneously from multiple threads?

Mbed TLS Generic
1

Why is there any contention in the data structure when used from multiple threads? I feel like it should be read only since (I think) the idea is that you use one config for multiple connections. If not, well I guess I have to pre-allocate one config per possible connection, which can take a long time.

2

Hi Nick,
Thank you for your query!

As you can see in the multi threaded example for a TLS server, the mbedtls_ssl_config structure is set only once, and sent to each thread to get configuration information. The intention is that the configuration is set once for all threads. Of course, you can change this design, if you need different configurations for different connections \ threads.
Why do you think this structure cannot be used from multiple threads simultaneously? Could you elaborate?
Regards,
Mbed TLS Team member
Ron

3

Hi Roneld

In my application I have several cores dedicated to handling network encryption and decryption because it is designed to handle a large amount of connections simultaneously. Each core has two threads assigned to it that work on IO jobs posted through a proactor. So in total there’s only a handful of threads to handle all the connections and not one thread per connection.

If I use a mutex to synchronize the config (as I’m guessing the example you posted does), then I can’t scale across multiple threads because they’re all constantly waiting on the mutex. If I to use the code without synchronization, I get a crash. I feel like the correct behavior is that I can use the config to send and receive data simultaneously from multiple threads without synchronization and without crashing. So my question is - if the config is essentially just a bunch of data used to initialize a connection, shouldn’t I be able to do this? Why is there contention on this data that is ostensibly read-only?

4

Hi Nick,

(as I’m guessing the example you posted does)

The example I posted doesn’t do that, and the config is not protected with a mutex, because, as you said, the config is essentially read only data.

My question is, do you set this in a single location? As shown in the example I posted, the config is set in the beginning, in the main thread, and then sent to each thread, to be set to the connection. In a multithreaded environment, you will need a mutex to protect against crashes, but to protect other resources.
regards

5

Yes, the config is set up in the main thread and then send to the IO threads in my code. Then all connection instances are create using that config.

What other resources need to be protected by mutexes? Are those resources referenced internally through the config?

I’m asking because if I run a test where I pre-allocate a bunch of configs and then assign one per connection, I don’t crash. If I do it the normal way, I crash almost immediately under load.

6

If you use mbedtls_ssl_ticket_context, mbedtls_ssl_cookie_ctx or mbedtls_ssl_cache_context then these have mutex protection within theeir contexts. The callbacks are defined in the configuration, but the callbacks themselves use mutexes.

7

I’m not using any of those explicitly, but maybe they are enabled internally? As far as I can tell, I’m not doing anything out of the ordinary.

Here is what my config initialization code looks like:

8

You could try disabling MBEDTLS_SSL_SESSION_TICKETS, MBEDTLS_SSL_DTLS_HELLO_VERIFY ( for not using cookie) in your Mbed TLS configuration, and you could try setting NULL callbacks for your cache, to disable the session cache:
mbedtls_ssl_conf_session_cache(&ssl_data.m_SSLConfig, NULL, NULL )

Of course, you will lose some functionality.

Since yout application is a TLS server, MBEDTLS_SSL_DTLS_HELLO_VERIFY can easily be undefined. (It is probably not used, and not causing your problem. )

I would try understanding which of the two ( session ticket and session cache) is causing your problem, and perhaps write your own callback that will fit your needs, in case you need that functionality.
Regards

9

I had some time to look into this more and the problem is related to the mbedtls_pk_sign being run in two threads at the same time with the same context. Since the context is retrieved out of the mbedtls_ssl_config, and is being written to by this function, it causes contention and either a crash or the MBEDTLS_ERR_RSA_PRIVATE_FAILED error. If you enable threading, then it will lock the mutex in mbedtls_rsa_context and there’s no problem but - couldn’t that locking be avoided by having the mbedtls_rsa_context not shared across connections?

The mbedtls_rsa_context structure isn’t that big, maybe it could just exist on the stack? If not, maybe have a some structure that could be allocated per-thread that has memory to store all the intermediate data used during handshaking?