We have an application where we are using the mbedtls SSL libraries to run a server that communicates via SSL sockets. This code has been working for a while now, but has recently started crashing.
MbedTLS version 2.7.0
Operating system is Ubuntu 16.04 LTS
This is the SERVER side of the SSL connection. We are running an SSL server.
As the title says, the call to mbedtls_ssl_handlshake crashes. First the SSL context is initialized and set up with code like the following:
Blockquote
mbedtls_ctr_drbg_init(&cx->ctr_drbg);
mbedtls_ctr_drbg_seed(&cx->ctr_drbg,mbedtls_entropy_func,&sd->entropy,“ABC”,3);
mbedtls_ssl_init(&cx->tls);
mbedtls_ssl_config_init(&cx->config);
mbedtls_ssl_config_defaults(&cx->config, MBEDTLS_SSL_IS_SERVER,MBEDTLS_SSL_TRANSPORT_STREAM,0);
mbedtls_ssl_conf_authmode(&cx->config,MBEDTLS_SSL_VERIFY_NONE);
mbedtls_ssl_conf_ca_chain(&cx->config,&sd->ca_list,NULL);
mbedtls_ssl_conf_own_cert(&cx->config,&sd->srv_cert_chain,&sd->cert_priv_key);
mbedtls_ssl_conf_min_version(&cx->config, MBEDTLS_SSL_MAJOR_VERSION_3,MBEDTLS_SSL_MINOR_VERSION_0);
mbedtls_ssl_conf_rng(&cx->config,mbedtls_ctr_drbg_random,&cx->ctr_drbg);
mbedtls_ssl_conf_dbg(&cx->config,tls_wrdb,NULL);
mbedtls_ssl_conf_session_cache(&cx->config,&sd->cache, mbedtls_ssl_cache_get,mbedtls_ssl_cache_set);
(“Bind TLS setup configuration to context\n”);
mbedtls_ssl_setup(&cx->tls,&cx->config);
mbedtls_ssl_set_hostname(&cx->tls,port->name);
mbedtls_ssl_session_reset(&cx->tls);
mbedtls_ssl_set_bio(&cx->tls,ns,tls_netsend,tls_netrecv,NULL);
mbedtls_ssl_handshake(&cx->tls);
I’ve cut out all the error checking for clarity, but this is the gist of the code. The vast majority of the time, the above code works, and the SSL communications are all successful. Every once in a while, the last line of code (mbedtls_ssl_handshake) crashes. The problem is how to troubleshoot this.
As you can see, I’ve set up a callback to read and write to the physical socket (with mbedtls_ssl_set_bio). I’ve put debug/log statements into those callback functions to see what is being sent and received during the handshake. Basically the following happens:
- Read 5 bytes from the client.
- Read another 192 bytes from the client.
- Write 96 bytes to the client.
- Write another 1743 bytes to the client.
- Application CRASHES at this time, before returning from mbedtls_ssl_handshake.
So the question is, any suggestions on troubleshooting this? I was thinking about running it in Valgrind to look for memory errors, but the problem only occurs on our production server where there are lots of SSL connections being established, and I don’t want Valgrind to slow down our production server. I can’t reproduce it in a test environment.
One interesting sidenote, is that all the crashes that I’ve cataloged appear to be hackers from China/Russia. We wouldn’t have any legitimate users from those areas. But the fact that this is a public server listening on port 443 allows hackers everywhere to try and have a crack at our server. Perhaps the hackers are sending malformed SSL packets? I’ve never seen a crash with a legitimate user…
Thanks for any suggestions!
-Eric