I’m using mbedTLS to do ECDH using the Curve25519. The private and public keys I want to create using the openssl (1.1.0) cmdline tool. Starting with the private key, I’m experience problems that I think originate from different encoding/format in the corresponding octet strings.
openssl genpkey -algorithm X25519 -outform DER -out privKey1.der
openssl pkey –in privKey1.der –outform DER –pubout –out key1.pub
To use the keys in mbedTLS, I extract the octet string as bytes from the certificate and use mbedtls_mpi_read_binary to load it into mbedtls_pk_ec->d. After that the mbedtls_exp_check_privkey fails as the key is not a multiple of 8.
After some debugging of the openssl app I found that it seems to put the key into the file in little endian format. Therefore I inverted the bytes of the private key. Now the privkey check succeeds but ECDH fails. I assume the public and private keys don’t match any more.
I was of the impression that any integer in a ASN.1 file is in big-endian format and can be read with mbedTLS as is.
Does anyone know, what happens here and how the openSSL genpkey feature can be used alongside mbedTLS ECDH?