Mbedtls-ssl read/write concurrently from multiple ssl connections

I have a requirement, where my mbedtls based ssl_server needs to have maintain multiple client connections.
Lets say ssl_server has 3 client connections.
Server needs to broadcast, message1 to all 3 clients.
Wait for reply.
If there is a quorum(2 out of 3 clients) replied, server can move on to next step.
Server will send message2 to all clients, that replied back.
Wait for reply.
If there is a quorum(2 out of 3 clients) replied, server can move on to next step.

For above usage, I guess one way to do is to spawn multiple threads, one thread per client connection.
In this case, will need 3 threads for 3 client connection,
Also need 1 extra master thread.
This master thread, will monitor whether quorum has been received from clients(i.e 2 out of 3 replied), based on global variables, that are shared across threads to check status of client communication.

ALTERNATE APPROACH:
But instead of using threads like explained above, was wondering, if it is possible to have a mbedtls_net_context, that has multiple client_fds.
Currently mbedtls_receive_timeout does select on 1 client_fd.
In this case, customized mbedtls_receive_timeout, needs to do select on those multiple client_fds.
So when one of the client replies, then select would unblock, and server can call mbedtls_net_receive, to get the client’s reply.

This may be a less memory intensive compared to threads, and maybe more efficient.

An example below, using select, without using threads:

Please let me know, if this alternate approach above is feasible.

Hi @krshnkmr
Thank you for your information and for your suggestion!

Mbed TLS is designed as a stack to protect your transport layer. It should n’t be aware of your application needs. Your use case suggests it should be handled in the application layer, and your suggestion should probably be done in that layer. however, I can see why in embedded system, your approach has its advantages, but this requires further investigation.
From brief view, I am not sure this approach is feasible without a complete redesign of the stack, as in current implementation for every TLS session, you l need a separate TLS context, to handle all the TLS data. Every TLS session is on a different fd, so having the TLS context handle multiple fd’s doesn’t have it’s value, IMHO.
You can, however, have on your server, a mechanism to hold several TLS sessions and FD, and in your server you can probably add a select mechanism on all the connected fds. You will need to modify your bio callbacks to support this, nonetheless.
Does this make sense?
Regards,
Mbed TLs team member
Ron

Hi Ron Eldor,

Thanks for your reply.
One more additional point, I have this application, that does below:

  1. works as both server and client.
  2. runs as server(on the front-end).
  3. for incoming client requests, this server makes a new ssl connections to 1 other server. So in this case, it
    acts like a client, and communicates with 1 other server.
    The above steps works fine using mbedtls.
    But now I am trying to add changes so that it in Step 3), it can create multiple client connections.
    So using threads in Step3) is quite expensive, since I may use threads on the front-end(where it acts as server) for incoming requests.

I did think about a quick resolution, like how you have described, but there could be some issues with that.

mbedtls_net_context server_fd, stores 1 FD.
mbedtls_ssl_context ssl, seems to be associated with 1 FD.

Now, yes, I can create multiple connections, and have a group of FDs, and for each connection, I can have a SSL_CONTEXT.

And in theory, I can have my own my_mbedtls_net_recv_timeout, which can then do a select on this group of FDs.

But the call to bio function, seems to pair 1 SSL_CONTEXT with 1 FD. Essentially, those bio_callbacks are dependant on traffic on 1 FD.
mbedtls_ssl_set_bio( &ssl, &server_fd, mbedtls_net_send, NULL, my_mbedtls_net_recv_timeout);
I dont know, if there is a way to specify multiple FDs, when calling mbedts_ssl_set_bio function.

So, are you suggesting that I call mbedtls_ssl_set_bio, for each connection(SSL_CONTEXT and FD).
So if there are 5 ssl connections, I would have to call, mbedtls_ssl_set_bio 5 times.
But within my_mbedtls_net_recv, I would have to do select on those 5 FDs.
In that case, I was not sure, of re-entrancy aspects, and whether I need some mutex to handle it.
Like if if one of the FDs has traffic, and my_*recv_timeout gets called…and while it is processing, if another FD has traffic, will my_*recv_timeout get called back by mbedtls code, in which case, will the current execution get interrupted, do we need to consider re-entrancy issues, within my_*recv_timeout.

Please let me know.

Hi Sudha,
I apologize for not explaining my intention correct.

Yes, every ssl context should have one fd, as a context is a single TLS session, for a single connection. As I see it, you will need to call mbedtls_ssl_set_bio() for every connection you have, with a different fd, but with same callback. In your application, you will have an array of ssl contexts ( limited with the number of allowed connection on your system). It will be some sort of hash table, that associates ssl contexts with their fd’s You will have your own function that will do the following:

  1. call select() (outside the TLS stack)
  2. call mbedtls_ssl_write() and mbedtls_ssl_read() with the TLS context the selected fd is associated with.

I don’t think this suggestion requires multi threading, nor does it require changing Mbed TLS code, nor setting your own proprietary bio callback, after thinking a bit more on this suggestion…

Please update with your thoughts on this suggestion
Regards,
Mbed TLS Team member
Ron

Hi Ron,

Thanks for clarifying.

I have a few questions.
So it is clear that for each connection, I need to call mbedtls_ssl_set_bio, that will take that connection’s SSL_CONTEXT and FD. But the callback function, passed will be the same everytime mbedtls_ssl_set_bio gets called.
My question is around this callback function. My understanding is that it needs to be a custom callback function, for example, custom_mbedtls_net_rcv_timeout, that can do below:

  1. call select(on the group of FDs).
  2. call mbedtls_ssl_write() or mbedtls_ssl_read using the SSL_CONTEXT associated with the selected FD.

Questions below:

  1. Are the assumptions above correct.

  2. In your last sentence, you have mentioned this will NOT require setting your own proprietary bio callback, that part was NOT clear to me. My interpretation is that this custom_mbedtls_net_rcv_timeout will be a custom(proprietary) callback function that will be passed, when calling mbedtls_ssl_set_bio.

  3. Also when you say "call select() (outside the TLS stack), my assumption is that it will be outside the TLS stack, when select gets invoked from this custom_mbedtls_net_rcv_timeout callback function.

  4. And finally, can we assume, that there wont be any code re-entrancy issues to worry about, like if this custom_mbedtls_rcv_timeout were to get interrupted by mbedtls invoking the callback function, due to traffic on another FD, while we are in the middle of executing the callback function(custom_mbedtls_net_rcv_timeout).

Please let me know, regarding my assumptions/questions.

thanks,

Sudha

Hi Sudha,

Following are my answers to your questions

Are the assumptions above correct.

No, This is not what I meant. What I mean is, that your send \ recv bio callbacks will call your platform’s network send receive. for custom_mbedtls_net_rcv_timeout you will call select on one fd. Please look at the example mbedtls_net_rcv_timeout(), mbedtls_net_rcv() and mbedtls_net_send() for reference. These example work on BSD sockets, and if your platform supports BSD socket API, you can use these callbacks with minimal change. The select on the group of FDs will happen on your application layer, in your main thread, after a TLS session has been established already. After select finishes, you will call in your main thread ( or a different thread, as you choose), mbedtls_ssl_write() and mbedtls_ssl_read() on each selected FD, separately. Note that mbedtls_ssl_write() and mbedtls_ssl_read() call internally the bio callbacks, so you can’t call these function within the callbacks.

In your last sentence, you have mentioned this will NOT require setting your own proprietary bio callback, that part was NOT clear to me. My interpretation is that this custom_mbedtls_net_rcv_timeout will be a custom(proprietary) callback function that will be passed, when calling mbedtls_ssl_set_bio.

This was assuming your platform supports the BSD sockets. If your platform has differnt functions for send \ receive to socket, you will have to have your own custom callbacks. These callbacks are called within the TLS handshake, and from the mbedtls_ssl_write() and mbedtls_ssl_read() functions.

Also when you say "call select() (outside the TLS stack), my assumption is that it will be outside the TLS stack, when select gets invoked from this custom_mbedtls_net_rcv_timeout callback function.

No, your functions should look something like the following:

int custom_mbedtls_net_send( void *ctx, const unsigned char *buf, size_t len )
{
       call platform socket write with the FD that is associated with the context
       
      if  would block return MBEDTLS_ERR_SSL_WANT_WRITE
     handle errors.
    return error_code or number of bytes written if succeeded
}

int custom_mbedtls_net_recv( void *ctx, unsigned char *buf, size_t len )
{
     cal platform socket read that is associated with the context
     if  would block return MBEDTLS_ERR_SSL_WANT_WRITE
     handle errors.
    return error_code or number of bytes readif succeeded
}

int custom_mbedtls_net_recv_timeout( void *ctx, unsigned char *buf,
                              size_t len, uint32_t timeout )
{
    call platform select with the given timeout on the FD associated with the context
   if timeout return MBEDTLS_ERR_SSL_TIMEOUT
   
   return  call from custom_mbedtls_net_recv    
}

Your main thread:
For every tls connection you are about to negotiate:
call mbedtls_ssl_set_bio with the relevant paramters and callbacks

Once TLS handshake occurs ( probably in different thread, otherwise you will wait for previous handshake to finish until you start a new negotiation). 
once you have TLS handshake, you can return to main thread, 
You will then call select on all FDs you have that the handshake has finished
After that, call `mbedtls_ssl_write()` and `mbedtls_ssl_read()` , according to the needed functionality

If you can’t have multithread environment, you can consider doing the handshake as part of the select as well, but call mbedtls_ssl_handshake_step() for every selected handshake ( until handshake is over ), with the associated SSL context, instead of calling mbedtls_ssl_handshake() which does the whole handshake.

Another approach, for the handshake, is if you use non blocking sockets, and then the handshake will always return with MBEDTLS_ERR_SSL_WANT_WRITE or MBEDTLS_ERR_SSL_WANT_READ, and then whenever you get one of these errors, you return to select all the connected FDs, or to next FD with handshake

Note this wasn’t tested, and it might have some glitches, but I hope explained my main idea better.
Regards,
Ron

Hi Ron,

Thanks again, for your reply. I can see that you have explained to the best extent possible.

I am using mbedtls on Ubuntu platform, and BSD sockets are supported.
So, all 3 of the mbedtls bio callback functions are working fine( mbedtls_net_rcv_timeout() , mbedtls_net_rcv() and mbedtls_net_send()).

To be honest, I am not 100% clear on how to implement your suggestion. Also need to look through the code on what you have said on the whole ssl_handshake related changes, to understand your suggestions for implementation.

I am likely to implement these changes, later this year.
It will probably become more clear, once I start to implement these changes.
If I have any further questions, will ping back on this thread at that time.