What does RSA - The PKCS#1 verification failed error means?

Mbed TLS
1

Hi guys,

I have mbedtls integrated in our iOS and Android app for more than 2 years and everything is (was) working fine and well.

Recently we started randomly seeing the following errors during the handshake but after the certificate validation from iOS devices only. I can not reproduce it locally, but as far as I understand from the logs, the certificate validation phase went fine, but the steps that follow fail with the following two errors:

sslError(error: -17280, description: “RSA - The PKCS#1 verification failed”)
sslError(error: -17028, description: “RSA - The public key operation failed : BIGNUM - Bad input parameters to function”)

Could anybody please help me understand where to look for the root cause of these errors? I assume right now the TLS client might be sending in bad or corrupted data at some point in time, which would cause the handshake fail in the final stages.

Both sides of the communication are various versions of mbedlts stable library. Android is not having this problem and iOS is. Both implementations are the same by design.

Thanks for any pointers, I’m not familiar enough with the internals of TLS/DTLS to understand where to look.

Thanks for your help,
Martin

2

Hi @mman
Thank you for your question!
The error means, as it states, that there was a failure during rsa verification, probably when verifying the certificate.
I suggest you look further at the logs to understand failure. Usually certificate verification happens when the public key used for verification is not the key pair of the private key used for signing the certificate, however, from the description and error code, the cause for this failure is probably different.
Since you say that certificate verification was successful, this failure probably originated from the key exchange verification

I believe that the failure originated from here.

Since the failure you are getting are randomly happening, it could suggest you have some sort of memory corruption, probably due to memory leak.
regards,
Mbed TLS Team member
Ron

3

Thanks Ron for your reply, you are absolutely correct. By investigating I have actually found out that under certain circumstances our key and certificate loaded on iOS in the background from iOS keychain get corrupted and key and cert do not match. Interestingly enough, the certificate validation actually proceeds fine, since the cert was apparently fine, we just missed the private key. I have implemented check pair before starting up and better handle the iOS keychain failures and all seems so far to be good again.

Thanks for your kick in the right direction,
Martin