RSA SHA-256 Encrypt String on ESP32

Hey there! I’m trying to base64 encode an encrypted string of data on an ESP-32 using a private key for use a signature on an HTTP request to an API. I feel like I’m almost there, but, the encrypted output doesn’t look right to me (looks like binary?) and the base64 encoded output looks even worse. Can someone advise on what I’m missing here? Here is the Arduino code:

void encrypt(unsigned char to_encrypt[]) {
  //openssl rsa -inform PEM -text -noout < ~/.oci/oci_api_key_no_passphrase.pem 
  mbedtls_rsa_context rsa;
  
  mbedtls_mpi N, P, Q, D, E, DP, DQ, QP;
  const char *n = "00dc...[redacted]";
  const char *e = "10001";
  const char *d = "00b62...[redacted]";
  const char *p = "00ef8d...[redacted]";
  const char *q = "00eb631...[redacted]";
  const char *dp = "1d6629...[redacted]";
  const char *dq = "7e33507...[redacted]";
  const char *qp = "4013c959...[redacted]";
  
  mbedtls_rsa_init( &rsa, MBEDTLS_RSA_PKCS_V15, 0 );
  
  mbedtls_mpi_init( &N ); mbedtls_mpi_init( &P ); mbedtls_mpi_init( &Q );
  mbedtls_mpi_init( &D ); mbedtls_mpi_init( &E ); 
  mbedtls_mpi_init( &DP );mbedtls_mpi_init( &DQ );mbedtls_mpi_init( &QP );
  
  mbedtls_mpi_read_string(&rsa.N, 16, n);
  mbedtls_mpi_read_string(&rsa.E, 16, e);
  mbedtls_mpi_read_string(&rsa.D, 16, d);
  mbedtls_mpi_read_string(&rsa.P, 16, p);
  mbedtls_mpi_read_string(&rsa.Q, 16, q);
  mbedtls_mpi_read_string(&rsa.DP, 16, dp);
  mbedtls_mpi_read_string(&rsa.DQ, 16, dq);
  mbedtls_mpi_read_string(&rsa.QP, 16, qp);
  
  int b = mbedtls_rsa_complete(&rsa);
  rsa.len = ( mbedtls_mpi_bitlen( &rsa.N ) + 7 ) >> 3;
  int keyValid = mbedtls_rsa_check_privkey(&rsa);
  unsigned char buf[512];

  if( keyValid == 0 ) {
    Serial.print("Encrypting: ");
    Serial.println((char*) to_encrypt);
    int success = mbedtls_rsa_pkcs1_sign( &rsa, NULL, NULL, MBEDTLS_RSA_PRIVATE, MBEDTLS_MD_NONE, strlen((char *) to_encrypt), to_encrypt, buf ); // rng: mbedtls_ctr_drbg_random
    Serial.print("Encrypted? ");
    Serial.println(success);
    Serial.print("Buffer: ");
    //Serial.println((char*) buf);
    size_t i;
    char bufHex[1024];
    for( i = 0; i < rsa.len; i++ ) {
      Serial.print( buf[i] );
      Serial.print( " " );
      bufHex[i] = buf[i];
    }
    Serial.println(bufHex);
    unsigned char encoded[64];
    size_t encodedOutLen;
    unsigned char test[] = "foo";
    mbedtls_base64_encode( encoded, sizeof(encoded), &encodedOutLen, buf, sizeof(buf) );
    Serial.print("Base64 Encoded: ");
    Serial.println((char*) encoded);
    
  }

And here is an example of the output:

10:41:59.208 -> Encrypting: foo
10:41:59.743 -> Encrypted? 0
10:41:59.743 -> Buffer: 10 95 76 78 36 186 235 237 91 82 42 134 58 30 119 218 188 95 85 128 63 23 138 225 84 229 11 52 250 209 97 145 12 210 164 162 99 254 212 154 85 242 166 30 92 17 10 197 82 76 71 246 79 164 96 147 141 244 229 215 149 182 64 121 196 199 56 239 6 226 180 132 31 43 124 0 218 133 167 199 108 125 147 31 80 144 156 58 157 203 249 98 173 234 102 4 193 194 143 64 89 129 242 41 100 191 64 67 92 144 107 135 79 200 134 191 165 202 119 107 178 182 26 66 96 142 193 204 151 106 161 154 157 170 130 87 173 108 31 246 217 137 129 121 1 133 101 154 201 234 166 229 236 25 39 117 107 227 155 69 44 231 95 27 245 146 76 194 129 12 119 142 10 148 148 201 235 97 104 252 122 191 179 71 191 147 183 95 129 153 174 104 53 132 101 122 129 63 103 231 178 147 71 123 222 253 122 212 226 143 149 2 199 235 193 212 20 108 22 201 232 53 31 134 186 60 173 189 71 61 90 134 162 47 228 119 45 99 64 6 61 77 254 110 48 153 86 202 247 4 154 160 197 80 235 160 
10:41:59.846 -> _LN$⸮⸮⸮[R*⸮:wڼ_U⸮?⸮⸮T⸮4⸮⸮a⸮Ҥ⸮c⸮ԚU⸮\
10:41:59.846 -> ⸮RLG⸮O⸮`⸮⸮⸮⸮ו⸮@y⸮⸮8⸮ⴄ+|
10:41:59.846 -> Base64 Encoded: ⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮
10:41:59.846 -> _LN$⸮⸮⸮[R*⸮:wڼ_U⸮?⸮⸮T⸮4⸮⸮a⸮Ҥ⸮c⸮ԚU⸮\
10:41:59.846 -> ⸮RLG⸮O⸮`⸮⸮⸮⸮ו⸮@y⸮⸮8⸮ⴄ+|

To answer my own question:

mbedtls_pk_context pk;
  mbedtls_pk_init(&pk);
  char* pwd = NULL;
  int pwdLen = 0;
  if(privateKeyPassphrase){
    pwd = privateKeyPassphrase;
    pwdLen = strlen(pwd);
  }
  mbedtls_pk_parse_key(&pk, (const unsigned char*) privateKey, strlen(privateKey)+1, (const unsigned char*) pwd, pwdLen);
  mbedtls_rsa_context *rsa = mbedtls_pk_rsa(pk);

  int keyValid = mbedtls_rsa_check_privkey(rsa);
  unsigned char encryptBuffer[512];
  if( keyValid == 0 ) {
    byte hashResult[32];
    mbedtls_sha256(toEncrypt, strlen((char*) toEncrypt), hashResult, 0);
    int success = mbedtls_rsa_rsassa_pkcs1_v15_sign( rsa, NULL, NULL, MBEDTLS_RSA_PRIVATE, MBEDTLS_MD_SHA256,  strlen((char*) hashResult), hashResult, encryptBuffer ); 
    size_t encodedOutLen;
    mbedtls_base64_encode( encoded, sizeof(encoded), &encodedOutLen, (const unsigned char*) encryptBuffer, sizeof(encryptBuffer) / 2 );
  }

Hi, i am facing simular problem, could you help me? i need to do PKCS1_OAEP decryption wiht sha256 algorithm. i was trying to get your code to work(and then edit it to decrypt), but i have some problems, could you help me?