I have mbedtls integrated in our iOS and Android app for more than 2 years and everything is (was) working fine and well.
Recently we started randomly seeing the following errors during the handshake but after the certificate validation from iOS devices only. I can not reproduce it locally, but as far as I understand from the logs, the certificate validation phase went fine, but the steps that follow fail with the following two errors:
sslError(error: -17280, description: “RSA - The PKCS#1 verification failed”)
sslError(error: -17028, description: “RSA - The public key operation failed : BIGNUM - Bad input parameters to function”)
Could anybody please help me understand where to look for the root cause of these errors? I assume right now the TLS client might be sending in bad or corrupted data at some point in time, which would cause the handshake fail in the final stages.
Both sides of the communication are various versions of mbedlts stable library. Android is not having this problem and iOS is. Both implementations are the same by design.
Thanks for any pointers, I’m not familiar enough with the internals of TLS/DTLS to understand where to look.
Thanks for your help,