I am working on some old code that created signatures and performed signature verification with RSA keys. We have upgraded to mbedtls 3.2.1, and the RSA signature/verification no longer works.
The old code to sign a document hash looked like this:
mbedtls_rsa_pkcs1_encrypt(private_rsa_key, mbedtls_ctr_drbg_random, &ctr_drbg, MBEDTLS_RSA_PRIVATE, hash_length, hash_buf, signature_buf);
The old code to verify a signature looked like this:
mbedtls_rsa_pkcs1_decrypt(public_rsa_key, mbedtls_ctr_drbg_random, &ctr_drbg, MBEDTLS_RSA_PUBLIC, signature_length, signature_buf, hash_buf);
Essentially the signing code takes the hash of the document (in hash_buf) and does an encryption operation with the private RSA key, creating the signature (in outbuf). The verification code takes the signaure (in signature_buf) and does a decryption operation with the public RSA key, recreating the original document hash (in has_buf). This worked great up until this upgrade to 3.2.1.
Now the 4th parameter of the above function (where we pass MBEDTLS_RSA_PRIVATE) is gone. So the library apparently assumes that when I’m encrypting, I “must” be encrypting with the public key, and when I’m decrypting, I “must” be decrypting with the private key. Except that’s not what the code wants. It wants to encrypt with the private key to create a signature, and it wants to decrypt with the public key to verify a signature.
How can I perform this operation in the latest version of mbedtls 3.2.1?
I notice that mbedtls_rsa_pkcs1_sign and mbedtls_rsa_pkcs1_verify are provided to generate signatures and verify signatures respectively. But the question is as follows:
Suppose a signature is generated with mbedtls_rsa_pkcs1_sign, can I then verify that signature by recovering the original hash with the above old code making an RSA decryption call using an old version of mbedtls 2.x library?
And vica-versa: Suppose a signature is generated with the above old code to encrypt-with-private-key using an old version of mbedtls 2.x library, can I verify that signature by recovering the original hash with mbedtls_rsa_pkcs1_verify?