Microsoft Azure rejecting encrypted MQTT payloads - 0x7880

Mbed TLS Crypto and SSL questions
1

I’m using MbedTLS that is built into the NodeMCU firmware for ESP8266 and I am trying to connect to Microsoft Azure IoT Hub and publish sensor data to it via the MQTT protocol.

I have successfully connected securely using a SAS token and Baltimore Root certificate which all works very well.

However, I have run into problems when publishing an MQTT message to Azure. Please see here for an issue I have opened on the GitHub repository that explains this in detail. The general consensus there is that this is a TLS issue and not a NodeMCU issue.

I have noticed that a certain IF statement within the ssl_tls.c file (line 1338) returns true or false intermittently, meaning that unpredictably, the cipher selected is incorrect and therefore badly encrypts the payload to be sent to Azure, I’m assuming. This is then rejected by Azure and my device is kicked offline.

Here is the IF statement that is sometimes true and sometimes false…

if( mode == MBEDTLS_MODE_STREAM ||
        ( mode == MBEDTLS_MODE_CBC
#if defined(MBEDTLS_SSL_ENCRYPT_THEN_MAC)
          && ssl->session_out->encrypt_then_mac == MBEDTLS_SSL_ETM_DISABLED
#endif
        ) )

It appears that, on an unsuccessful publish/keepalive, I am getting:

MBEDTLS_ERR_SSL_INVALID_MAC - 0x7180 - Verification of the message MAC failed.

…followed by…

client’s data invalid protocol

…then…

MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY - 0x7880 - The peer notified us that the connection is going to be closed.

This falls in line with the fact that the ssl_encrypt_buf function is to blame. Sometimes the MAC is computed, sometimes not. I cannot see why this would be skipped occasionally.

This occurs randomly, sometimes affecting the MQTT keep-alive (ping) packet too which is basically a mini publish. I need to identify what variable is changing here, causing two different outcomes to exactly the same publish event. Anyone have any ideas?

Many thanks, George

2

Hi George,
If this if statement sometimes returns true and sometimes return false, it probably means that either mode or ssl->session_out->encrypt_then_mac is changed between every run. This shouldn’t happen, as this is depending on the negotiated ciphersuite (the mode), or whether encrypt_ten_mac was enabled or not.
If this has changed during your session, this may hint on some memory corruption in your system.
Regards,
Mbed TLS Support
Ron

3

Thank you Ron.

Please can you provide me with some code to print this to debug so I can find which of the two variables is changing?

I’m not very good with C!

I’ve tried this, with no luck:

MBEDTLS_SSL_DEBUG_MSG( 1, ( "mode = " + mode ) );
MBEDTLS_SSL_DEBUG_MSG( 1, ( "encrypt_then_mac = " + ssl->session_out->encrypt_then_mac ) );

As a workaround, is it possible to force this to compute the mac every time (i.e. bypass the IF statement)? As I will not need to use any other cipher suite, ever, for my application.

4

Hi George,
In order to print the information, you will need to use the printf string format specifiers, as shown in this page.

Please can you provide me with some code to print this to debug so I can find which of the two variables is changing?

In the logs you showed in in the github page, you will need to do the following, according to the parameter being printed:

MBEDTLS_SSL_DEBUG_MSG( 1, ( "MODE = %u", mode ) );
MBEDTLS_SSL_DEBUG_MSG( 1, ( "MBEDTLS_MODE_STREAM = %u ", MBEDTLS_MODE_STREAM ) );
MBEDTLS_SSL_DEBUG_MSG( 1, ( "MBEDTLS_MODE_CBC = %u", MBEDTLS_MODE_CBC ) );
MBEDTLS_SSL_DEBUG_MSG( 1, ( "ssl->session_out->encrypt_then_mac = ", ssl->session_out->encrypt_then_mac ) );
MBEDTLS_SSL_DEBUG_MSG( 1, ( "MBEDTLS_SSL_ETM_DISABLED = %d", MBEDTLS_SSL_ETM_DISABLED ) );

Assuming you have set the debug function in your application using mbedtls_ssl_conf_dbg() you should see the correct logs.

As a workaround, is it possible to force this to compute the mac every time (i.e. bypass the IF statement)?

Note that mac should always be calulated (according to negotiated ciphersuite). If it is not used when needed, then as mentioned, there is probably memory corruption. Considering you mentioned your knowledge in c, then this could be the cause.
Please try building your library without MBEDTLS_SSL_ENCRYPT_THEN_MAC defined in your configuration file. The mode would still be used, but this will narrow down the checks.

As I will not need to use any other cipher suite, ever, for my application.

It is still not clear to me what the negotiated ciphersuite was. This will at least show what should be the mode etc…

Regards

5

You’re a great help Ron, thanks again :slight_smile:

Right, so, by using this debug info you’ve given me, I have determined the following:

Constants (that we already knew):

MBEDTLS_MODE_STREAM = 7
MBEDTLS_MODE_CBC = 2
MBEDTLS_SSL_ETM_DISABLED = 0

On a successful MQTT publish:

mode = 2
ssl->session_out->encrypt_then_mac = 0

On an unsuccessful MQTT publish:

mode = 2
ssl->session_out->encrypt_then_mac = 1701013878

So that’s the discrepancy.

mode appears to always be 2 but the number returned by encrypt_then_mac appears to be a random integer of length 10, when using this debug output, as suggested:

MBEDTLS_SSL_DEBUG_MSG( 1, ( "ssl->session_out->encrypt_then_mac = %d", ssl->session_out->encrypt_then_mac ) );

(Is using %d for ssl->session_out->encrypt_then_mac correct? You didn’t specify in your last post.)

#define MBEDTLS_SSL_ENCRYPT_THEN_MAC exists in both the config.h file and the user_mbedtls.h file.

Which of them need to be commented out? Or both?

6

Hi @georeb

(Is using* %d for ssl->session_out->encrypt_then_mac *correct? You didn’t specify in your last post.)

Actually, since encrypt_then_mac is a char type, it should have been %c. Sorry I missed that. However, it doesn’t matter, it still shouldn’t have been 1701013878 This strengthens my assumption that there is some memory corruption in your system, that over wrote your session context.
In your application, have freed all the variables that you have allocated, after use?

#define MBEDTLS_SSL_ENCRYPT_THEN_MAC exists in both the config.h file and the user_mbedtls.h file.
Which of them need to be commented out? Or both?

Since you are using MBEDTLS_USER_CONFIG_FILE, which adds to the existing configuration, and doesn’t replace it, you should try vcalling #undef MBEDTLS_SSL_ENCRYPT_THEN_MAC in your user_mbedtls.h.
Of course, this would probably only work around the issue, however it will not fix your memory corruption or memory leak in your system.
Regards,
Ron

7

Thank you for your support Ron, it has helped me to identify the problem.

In your application, have freed all the variables that you have allocated, after use?

This is beyond my C capability and will be one for the NodeMCU team I’m afraid! Thanks again.