Invalid RSA signature format;

Mbed TLS Crypto and SSL questions
1

Hi

When i am creating a signature using openssl and verifying using rsa_verify

I am getting Invalid RSA signature format.

Please find the steps followed.

Create a file containing all lower case alphabets

echo abcdefghijklmnopqrstuvwxyz > myfile.txt

Generate 1024 bit Private key

openssl genrsa -out myprivate.pem 1024

Separate the public part from the Private key file.

openssl rsa -in myprivate.pem -pubout > mypublic.pem

Sign the file using sha256 digest and PKCS1 padding scheme

openssl dgst -sha256 -sign myprivate.pem -out myfile.txt.sig myfile.txt

Verify the signature of file

openssl dgst -sha256 -verify mypublic.pem -signature myfile.txt.sig myfile.txt

#output
#Verified OK

check the sha256 hash value

openssl dgst -sha256 myfile.txt

<<COMMENT1
SHA256(myfile.txt)= 1010a7e761610980ac591359c871f724de150f23440ebb5959ac4c0724c91d91
COMMENT1

check the signature

openssl rsautl -verify -inkey myprivate.pem -in myfile.txt.sig -raw -hexdump

: ’
0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff …
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff …
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff …
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff …
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff 00 30 31 30 …010
0050 - 0d 06 09 60 86 48 01 65-03 04 02 01 05 00 04 20 …`.H.e…
0060 - 10 10 a7 e7 61 61 09 80-ac 59 13 59 c8 71 f7 24 …aa…Y.Y.q.$
0070 - de 15 0f 23 44 0e bb 59-59 ac 4c 07 24 c9 1d 91 …#D…YY.L.$…
’

#Generating the N and E
./key_app mode=private filename=myprivate.pem > rsa_priv.txt

#remove first 3 lines in the rsa_priv.txt
sed -i ‘1,3d’ rsa_priv.txt

#dos2unix
dos2unix rsa_priv.txt

#creating the rsa_pub.txt, deleting the 3 to 8 lines in the rsa_private.txt and redirecting to rsa_pub.txt
sed ‘3,8d’ rsa_priv.txt>temp.txt; mv temp.txt rsa_pub.txt

#Verfiying the with rsa_verify
./rsa_verify myfile.txt

. Reading public key from rsa_pub.txt
! Invalid RSA signature format

But i am able to verify the signature when i am creating the signature using rsa_sign.
I am not sure what is problem, when i am creating the signature using openssl.

2

Hi @raj_p
Thank you fore your question!
The applications rsa_sign and rsa_verify are reference applications, and do not interoperate with open SSL.
For example, the files rsa_priv.txt and rsa_pub.txt are not in some standard, but written in human friendly format.
The way you convert to rsa_priv.txt is not entirely the proper way, as in addition to removing the first three lines, you will need to change the : part to =.

In addition, openssl generates the signature in binary format, but the expected signature file ( myfile.txt.sig) should be in human readable hex string.
I tried your steps, once by creating the signature with open ssl (myfile.txt.sig.os) and once by using rsa_sign (myfile.txt.sig) using the private file and got the same signature:

./cat myfile.txt.sig
7E CF 6C 3A F8 3B 8A 86 87 2D 97 A0 38 F1 15 D0
D7 B1 22 E9 FB 21 AA EA F4 85 BC 37 ED 30 75 A0
73 6B 6A 6B 57 16 D3 EB 67 93 61 45 5B C0 3A EF
A1 ED 51 B0 D5 37 E4 25 C6 7D C2 29 6F A3 65 73
07 68 9E F7 7A A5 A0 E1 32 7F 3D 50 23 2D DE 5E
41 15 33 7E 29 09 BC 29 CD D5 6B EA 5D 49 35 74
34 6E FC 93 D2 13 02 C1 E7 5B 94 3D 3E 45 9C C1
4F 19 09 C3 92 8F 96 AC DE AF 5D 6C 61 9C 6F 94

hexdump myfile.txt.sig.os
0000000 cf7e 3a6c 3bf8 868a 2d87 a097 f138 d015
0000010 b1d7 e922 21fb eaaa 85f4 37bc 30ed a075
0000020 6b73 6b6a 1657 ebd3 9367 4561 c05b ef3a
0000030 eda1 b051 37d5 25e4 7dc6 29c2 a36f 7365
0000040 6807 f79e a57a e1a0 7f32 503d 2d23 5ede
0000050 1541 7e33 0929 29bc d5cd ea6b 495d 7435
0000060 6e34 93fc 13d2 c102 5be7 3d94 453e c19c
0000070 194f c309 8f92 ac96 afde 6c5d 9c61 946f
0000080

One comment on the way you are creating the public key file. A better way, should be IMHO:

./key_app_writer mode=private filename=myprivate.pem output_mode=public output_file=public.pem

If you want to test interoperability, I suggest you use the pk_verify application.

Regards,
Mbed TLS Team member
Ron