Handshake behavior with asynchronous socket

I would like to understand better the behavior of MbedTLS library when doing handshake over non-blocking sockets. Answers are appreciated :slight_smile:

I am making call to mbedtls_ssl_handshake and retrying it if I got MBEDTLS_ERR_SSL_WANT_READ or MBEDTLS_ERR_SSL_WANT_WRITE. In other cases I assume the return value is the actual result of the negotiation.

When running this code I noticed that mbedtls_ssl_handshake returns -1 result which does not seem to be a documented return value.

I understood that I get this return value due to the fact that in the underlying bio I simply call into libc’s send and recv over non-blocking socket which returns -1 when reading from a socket which has an empty incoming buffer. As a test, I have made retries to mbedtls_ssl_handshake while the return value was -1 (busy loop) and as a result the TLS handshake was eventually completed successfully.

I have to tell that the behavior is strange for me and I wanted to clarify it.

In case that the application calls into mbedtls_ssl_handshake and it expect incoming data, but gets -1 from recv (with errno set to EAGAIN) shouldn’t it return MBEDTLS_ERR_SSL_WANT_READ ? This would mean that caller can call select/poll/epoll on the socket and wait until data can be accessed and mbedtls_ssl_handshake can be called again.

I don’t see anywhere that mbedtls_ssl_handshake can return -1. Is it an expected behavior?

Hi @benhirschberg
-1 is not an Mbed TLS error, and it is probably returned by the underlying bio callback.
However, as you probably see in the implementation of the reference for BSD like recv, mbedtls_net_recv() , if the socket would block, then MBEDTLS_ERR_SSL_WANT_READ is returned.
Have you called mbedtls_net_set_nonblock() on your socket context?
Mbed TLS Support