Padding issue mutual TLS in 802.1x, EAP-TLS

Hi,

I’m working on an 802.1x supplicant which has to perform mutual TLS with the Radius server. That radius server is giving me the following error on my handshake:

(9) eap_tls: (TLS) Handshake state - Server SSLv3/TLS read client certificate
(9) eap_tls: (TLS) recv TLS 1.2 Handshake, ClientKeyExchange
(9) eap_tls: (TLS) Handshake state - Server SSLv3/TLS read client key exchange
(9) eap_tls: (TLS) recv TLS 1.2 Handshake, CertificateVerify
(9) eap_tls: (TLS) send TLS 1.2 Alert, fatal decrypt_error
(9) eap_tls: ERROR: (TLS) Alert write:fatal:decrypt error
(9) eap_tls: ERROR: (TLS) Server : Error in error
(9) eap_tls: ERROR: (TLS) Failed reading from OpenSSL
(9) eap_tls: ERROR: (TLS) error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding
(9) eap_tls: ERROR: (TLS) error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed
(9) eap_tls: ERROR: (TLS) error:1417B07B:SSL routines:tls_process_cert_verify:bad signature
(9) eap_tls: ERROR: (TLS) System call (I/O) error (-1)
(9) eap_tls: ERROR: (TLS) EAP Receive handshake failed during operation
(9) eap_tls: ERROR: [eaptls process] = fail
(9) eap: ERROR: Failed continuing EAP TLS (13) session.  EAP sub-module failed
(9) eap: Sending EAP Failure (code 4) ID 10 length 4
(9) eap: Failed in EAP select
(9)     [eap] = invalid
(9)   } # authenticate = invalid
(9) Failed to authenticate the user

The certificate isn’t the issue, since it works fine with the Windows supplicant. Which also rules out the RADIUS server as an issue. The certificate and key files are input using \n line endings.

The embedded supplicant isn’t checking the server certificate chain, and with a much smaller, simpler certificate I’ve already got this working.

mbedtls_ssl_handshake isn’t returning an error.

Does anyone have some pointers as to where to look?

My setup code is as below. There is a little wrapper around mbedtls for unit testing purposes, but when in normal use it’s passing everything through to the underlying library without change.

void EapTls::start(Eapol* eapol)
{
  if (EapTlsState == STARTED)
  {
    return;
  }

  if (mbedtls == nullptr)
  {
    DEBUG_PRINT_(
      TraceLevel::LEVEL_ERROR, 
      "Tried to start without a valid MbedTls wrapper\n");
    return;
  }

  EapTlsState = STARTED;

  receiveBuffer->Clear();
  sendBuffer->Clear();
  handshaking = false;

  DEBUG_PRINT_(TraceLevel::LEVEL_INFO, "Alloc/free balance: %i\n", ++allocFree);
  DEBUG_PRINT_(TraceLevel::LEVEL_INFO, "start\n");
  int status = 0;
  mbedtls->mbedtls_entropy_init(&tlsEntropyContext);
  mbedtls->mbedtls_ctr_drbg_init(&tlsCtrDrbgContext);
  mbedtls->mbedtls_ssl_init(&tlsContext);
  mbedtls->mbedtls_ssl_config_init(&tlsConfiguration);
  mbedtls->mbedtls_x509_crt_init(&tlsIdentityCertificate);
  mbedtls->mbedtls_x509_crt_init(&tlsCaCertificate);
  mbedtls->mbedtls_pk_init(&tlsPrivateKey);

  status = mbedtls->mbedtls_ctr_drbg_seed(
    &tlsCtrDrbgContext,
    mbedtls_entropy_func,
    &tlsEntropyContext,
    (const unsigned char*)initialRandomSeed, 
    strlen(initialRandomSeed));

  if (status != 0) 
  {
    DEBUG_PRINT_(TraceLevel::LEVEL_ERROR, "Failed to initialise MbedTls random source: 0x%X\n", status);
    return;
  }


  mbedtls->mbedtls_ssl_config_defaults(
    &tlsConfiguration, 
    MBEDTLS_SSL_IS_CLIENT, 
    MBEDTLS_SSL_TRANSPORT_STREAM,
    MBEDTLS_SSL_PRESET_DEFAULT);

  mbedtls->mbedtls_ssl_conf_rng(
    &tlsConfiguration, 
    mbedtls_ctr_drbg_random, 
    &tlsCtrDrbgContext);

  mbedtls->mbedtls_ssl_setup(&tlsContext, &tlsConfiguration);
  mbedtls->mbedtls_ssl_set_bio(
    &tlsContext, 
    this, 
    cbEapTlsSend, // Return MBEDTLS_ERR_SSL_WANT_WRITE if otherwise blocking
    cbEapTlsReceive, // Returns MBEDTLS_ERR_SSL_WANT_READ if otherwise blocking
    nullptr); // No blocking receive function provided. Library selects non-blocking one

  mbedtls->mbedtls_ssl_conf_dbg(&tlsConfiguration, ::cbMbedTlsPrint, nullptr);
  mbedtls->mbedtls_debug_set_threshold(4);

  // Don't bother checking the server certificate
  mbedtls->mbedtls_ssl_conf_authmode(
    &tlsConfiguration,
    MBEDTLS_SSL_VERIFY_NONE);

  const size_t pkSize = eapol->getConfiguration()->getPrivateKeyLength() + 1;
  if (pkSize == 1)
  {
    DEBUG_PRINT_(
      TraceLevel::LEVEL_ERROR,
      "Unable to load the private key from the configuration!\n");
    this->stop();
    eapol->markEapFailure("Private key unavailable!");
    return;
  }
  unsigned char pkBuf[pkSize];
  memset(pkBuf, 0, pkSize);
  eapol->getConfiguration()->getPrivateKey((char*)&pkBuf, pkSize);

  status = mbedtls->mbedtls_pk_parse_key(&tlsPrivateKey, pkBuf, pkSize, nullptr, 0);
  if (status)
  {
    DEBUG_PRINT_(
      TraceLevel::LEVEL_ERROR, 
      "Unable to parse the private key! %X\n",
      -status);

    this->stop();
    eapol->markEapFailure("Unable to parse private key!");
    return;
  }

  const size_t idSize = eapol->getConfiguration()->getCertificateLength() + 1;
  if (idSize == 1) {
   DEBUG_PRINT_(
      TraceLevel::LEVEL_ERROR,
      "Unable to load the identity certificate from the configuration!\n");
    this->stop();
    eapol->markEapFailure("Unable to load the identity certificate!");
    return;
  }

  unsigned char idBuf[idSize];
  memset(idBuf, 0, idSize);
  eapol->getConfiguration()->getCertificate((char*)&idBuf, idSize);

  status = mbedtls->mbedtls_x509_crt_parse(&tlsIdentityCertificate, idBuf, idSize);
  if (status) 
  {
    DEBUG_PRINT_(
      TraceLevel::LEVEL_ERROR, 
      "Unable to parse the identity certificate! %X\n",
      -status);
    this->stop();
    eapol->markEapFailure("Unable to parse the identity certificate!");
    return;
  }

  mbedtls->mbedtls_ssl_conf_own_cert(
    &tlsConfiguration,
    &tlsIdentityCertificate,
    &tlsPrivateKey);

  mbedtls->hardenSocket(&tlsConfiguration);

  DEBUG_PRINT_(TraceLevel::LEVEL_INFO, "start success!\n");
}

The Harden socket call sets the minimum version to be TLS 1.2 and selects the following ciphers:

    MBEDTLS_TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
    MBEDTLS_TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
    MBEDTLS_TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
    MBEDTLS_TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
    MBEDTLS_TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
    MBEDTLS_TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
    MBEDTLS_TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
    MBEDTLS_TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
    MBEDTLS_TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
    MBEDTLS_TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
    MBEDTLS_TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
    MBEDTLS_TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
    0,

The cipher in use is: TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256

The logs from MbedTLS (2.28.0) are:

{delta in ms}[file:line nr] Debug message

{     4}[library\ssl_msg.c:2853] <= write record
{     3}[library\ssl_msg.c:2661] <= write handshake message
{     5}[library\ssl_tls.c:2318] <= write certificate
{     4}[library\ssl_cli.c:4406] client state: 8
{     4}[library\ssl_msg.c:2087] => flush output
{     4}[library\ssl_msg.c:2099] <= flush output
{     3}[library\ssl_cli.c:3675] => write client key exchange
{   235}[library\ssl_cli.c:3840] value of 'ECDH: Q(X)' (255 bits) is:
{     0}[library\ssl_cli.c:3840]  49 dc ba e0 4c 21 83 9b 7a e1 32 3d b2 36 6c fa
{     0}[library\ssl_cli.c:3840]  9f 42 f5 45 81 1a 40 ed 44 05 f9 ce dc 05 1a 4f
{     0}[library\ssl_cli.c:3840] value of 'ECDH: Q(Y)' (255 bits) is:
{     0}[library\ssl_cli.c:3840]  69 9b f6 d3 1a 46 73 a7 c9 f0 71 25 8d 5a 81 41
{     0}[library\ssl_cli.c:3840]  08 ee 9e f5 ae 6c ef a2 d2 a7 6a 9b 2b ab 4b c7
{   222}[library\ssl_cli.c:3868] value of 'ECDH: z' (256 bits) is:
{     0}[library\ssl_cli.c:3868]  cf 2d 26 01 b9 e9 16 ce f3 fc 63 2e 36 a8 d7 69
{     0}[library\ssl_cli.c:3868]  18 d6 71 63 97 fd 0f 95 c2 c8 e4 32 e2 73 fc 26
{     0}[library\ssl_msg.c:2525] => write handshake message
{     0}[library\ssl_msg.c:2684] => write record
{     0}[library\ssl_msg.c:2797] output record: msgtype = 22, version = [3:3], msglen = 70
{     0}[library\ssl_msg.c:2802] dumping 'output record sent to network' (75 bytes)
{     0}[library\ssl_msg.c:2802] 0000:  16 03 03 00 46 10 00 00 42 41 04 49 dc ba e0 4c  ....F...BA.I...L
{     0}[library\ssl_msg.c:2802] 0010:  21 83 9b 7a e1 32 3d b2 36 6c fa 9f 42 f5 45 81  !..z.2=.6l..B.E.
{     0}[library\ssl_msg.c:2802] 0020:  1a 40 ed 44 05 f9 ce dc 05 1a 4f 69 9b f6 d3 1a  .@.D......Oi....
{     0}[library\ssl_msg.c:2802] 0030:  46 73 a7 c9 f0 71 25 8d 5a 81 41 08 ee 9e f5 ae  Fs...q%.Z.A.....
{     0}[library\ssl_msg.c:2802] 0040:  6c ef a2 d2 a7 6a 9b 2b ab 4b c7                 l....j.+.K.
{     0}[library\ssl_msg.c:2087] => flush output
{     1}[library\ssl_msg.c:2105] message length: 75, out_left: 75
{     6}[library\ssl_msg.c:2112] ssl->f_send() returned 75 (-0xffffffb5)
{     6}[library\ssl_msg.c:2140] <= flush output
{     3}[library\ssl_msg.c:2853] <= write record
{     4}[library\ssl_msg.c:2661] <= write handshake message
{     5}[library\ssl_cli.c:4077] <= write client key exchange
{     5}[library\ssl_cli.c:4406] client state: 9
{     3}[library\ssl_msg.c:2087] => flush output
{     4}[library\ssl_msg.c:2099] <= flush output
{     4}[library\ssl_cli.c:4120] => write certificate verify
{     5}[library\ssl_tls.c:1716] => derive keys
{     3}[library\ssl_tls.c:1918] => calc verify sha256
{     5}[library\ssl_tls.c:1925] dumping 'calculated verify result' (32 bytes)
{     6}[library\ssl_tls.c:1925] 0000:  8f fc cb c6 f7 be fa c7 18 14 48 ce 16 bf a4 4a  ..........H....J
{     9}[library\ssl_tls.c:1925] 0010:  0d 2f 06 f7 4a 9c 6e 24 a6 e0 78 c8 59 06 23 31  ./..J.n$..x.Y.#1
{     9}[library\ssl_tls.c:1926] <= calc verify
{     4}[library\ssl_tls.c:1634] dumping 'session hash for extended master secret' (32 bytes)
{     8}[library\ssl_tls.c:1634] 0000:  8f fc cb c6 f7 be fa c7 18 14 48 ce 16 bf a4 4a  ..........H....J
{     9}[library\ssl_tls.c:1634] 0010:  0d 2f 06 f7 4a 9c 6e 24 a6 e0 78 c8 59 06 23 31  ./..J.n$..x.Y.#1
{    10}[library\ssl_tls.c:1699] dumping 'premaster secret' (32 bytes)
{     5}[library\ssl_tls.c:1699] 0000:  cf 2d 26 01 b9 e9 16 ce f3 fc 63 2e 36 a8 d7 69  .-&.......c.6..i
{     9}[library\ssl_tls.c:1699] 0010:  18 d6 71 63 97 fd 0f 95 c2 c8 e4 32 e2 73 fc 26  ..qc.......2.s.&
{    13}[library\ssl_tls.c:1068] ciphersuite = TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256
{     3}[library\ssl_tls.c:1070] dumping 'master secret' (48 bytes)
{     5}[library\ssl_tls.c:1070] 0000:  c2 52 fb bd e6 64 3c b1 88 d4 1d 90 be 5a 36 29  .R...d<......Z6)
{    10}[library\ssl_tls.c:1070] 0010:  6b 84 19 0e 11 a9 63 d7 65 9d e9 c3 5a bc dd b3  k.....c.e...Z...
{     9}[library\ssl_tls.c:1070] 0020:  25 44 c6 50 f4 fa dd e3 03 17 d4 d3 44 56 3e fc  %D.P........DV>.
{     9}[library\ssl_tls.c:1071] dumping 'random bytes' (64 bytes)
{     5}[library\ssl_tls.c:1071] 0000:  33 dd 32 a4 82 51 ef d7 bb fa 6d f5 32 ad 4b f6  3.2..Q....m.2.K.
{     9}[library\ssl_tls.c:1071] 0010:  44 7c c5 86 64 77 c0 84 44 4f 57 4e 47 52 44 01  D|..dw..DOWNGRD.
{     9}[library\ssl_tls.c:1071] 0020:  64 bf 93 1b 02 22 73 f0 4e 40 cf e0 c6 50 80 86  d...."s.N@...P..
{     9}[library\ssl_tls.c:1071] 0030:  d7 ed ae 48 a4 e8 0e 40 86 63 24 63 6c 69 d3 fd  ...H...@.c$cli..
{     9}[library\ssl_tls.c:1072] dumping 'key block' (256 bytes)
{     6}[library\ssl_tls.c:1072] 0000:  b7 3b 86 dc f3 17 9f 69 f3 0a b6 cc ee 70 68 17  .;.....i.....ph.
{     9}[library\ssl_tls.c:1072] 0010:  f1 f1 53 1a 62 8d ef cb bb f9 c7 2f 40 91 dc 5b  ..S.b....../@..[
{     9}[library\ssl_tls.c:1072] 0020:  a4 bc 9d 37 f7 90 67 66 08 df c4 32 fd 63 62 12  ...7..gf...2.cb.
{     9}[library\ssl_tls.c:1072] 0030:  78 54 93 3b 38 37 c4 c4 7b 76 b5 dc 94 b4 72 f7  xT.;87..{v....r.
{     9}[library\ssl_tls.c:1072] 0040:  1e 20 77 90 a5 84 e6 9a fb 38 5f b7 3f 3c 19 fa  . w......8_.?<..
{     9}[library\ssl_tls.c:1072] 0050:  49 34 96 54 17 70 9d b6 f4 62 8a c4 76 f8 86 4b  I4.T.p...b..v..K
{     9}[library\ssl_tls.c:1072] 0060:  be 8b 34 07 55 e9 b9 51 e0 d8 ef 44 ef 55 08 7e  ..4.U..Q...D.U.~
{     9}[library\ssl_tls.c:1072] 0070:  3e fe 71 00 b1 b1 09 d2 f6 bf b0 ec 42 8a 68 3f  >.q.........B.h?
{     9}[library\ssl_tls.c:1072] 0080:  4c 64 dd 5c 0e 5c 31 e0 8d 24 cf 52 e8 73 1b 23  Ld.\.\1..$.R.s.#
{     9}[library\ssl_tls.c:1072] 0090:  54 57 a6 4a 99 1b 56 80 9f ab 4f 07 d8 78 53 90  TW.J..V...O..xS.
{     9}[library\ssl_tls.c:1072] 00a0:  e6 49 e5 da 92 82 3b 4f 90 fb a8 d3 a3 40 ba 40  .I....;O.....@.@
{     9}[library\ssl_tls.c:1072] 00b0:  20 0b 98 5b c6 aa c7 01 04 11 81 b6 6b ae 8a d8   ..[........k...
{     9}[library\ssl_tls.c:1072] 00c0:  4f 40 5f 5d 4d ab 17 6b 70 a8 1c fe 8c 86 0e 63  O@_]M..kp......c
{     9}[library\ssl_tls.c:1072] 00d0:  db 3c 39 50 f9 eb 27 3d f4 f4 fd a9 18 df d3 10  .<9P..'=........
{     9}[library\ssl_tls.c:1072] 00e0:  2a 8d f0 b7 75 ee 39 f4 23 64 af 1d 64 bf fd 97  *...u.9.#d..d...
{    10}[library\ssl_tls.c:1072] 00f0:  66 1c 22 03 57 3c b3 f6 81 95 63 da 89 03 a6 49  f.".W<....c....I
{     9}[library\ssl_tls.c:1214] keylen: 16, minlen: 24, ivlen: 12, maclen: 0
{     6}[library\ssl_tls.c:1795] <= derive keys
{     4}[library\ssl_tls.c:1918] => calc verify sha256
{     4}[library\ssl_tls.c:1925] dumping 'calculated verify result' (32 bytes)
{     6}[library\ssl_tls.c:1925] 0000:  8f fc cb c6 f7 be fa c7 18 14 48 ce 16 bf a4 4a  ..........H....J
{    10}[library\ssl_tls.c:1925] 0010:  0d 2f 06 f7 4a 9c 6e 24 a6 e0 78 c8 59 06 23 31  ./..J.n$..x.Y.#1
{     8}[library\ssl_tls.c:1926] <= calc verify
{  2352}[library\ssl_msg.c:2525] => write handshake message
{     0}[library\ssl_msg.c:2684] => write record
{     0}[library\ssl_msg.c:2797] output record: msgtype = 22, version = [3:3], msglen = 264
{     0}[library\ssl_msg.c:2802] dumping 'output record sent to network' (269 bytes)
{     0}[library\ssl_msg.c:2802] 0000:  16 03 03 01 08 0f 00 01 04 04 01 01 00 85 f1 41  ...............A
{     0}[library\ssl_msg.c:2802] 0010:  b3 09 6c bb e0 bf 53 f9 93 67 bc e0 b2 23 9b 83  ..l...S..g...#..
{     0}[library\ssl_msg.c:2802] 0020:  da f0 b1 0d bf 09 76 62 a4 88 5a 5f 28 88 bc 35  ......vb..Z_(..5
{     0}[library\ssl_msg.c:2802] 0030:  67 e1 fe af 82 e1 ef 36 ba e6 ac e5 de 41 82 78  g......6.....A.x
{     0}[library\ssl_msg.c:2802] 0040:  71 09 e8 41 e6 7c 95 54 c0 45 fa 5f 60 fa 0b 24  q..A.|.T.E._`..$
{     0}[library\ssl_msg.c:2802] 0050:  ca 56 11 fa 0f 03 32 70 df a0 c5 8c e3 c8 3b 1d  .V....2p......;.
{     0}[library\ssl_msg.c:2802] 0060:  f7 9e 30 48 d5 f0 34 45 f4 24 be a0 ca fe 58 b2  ..0H..4E.$....X.
{     0}[library\ssl_msg.c:2802] 0070:  3c 2b 8f 95 d2 69 4e a8 37 cd 0f 1c 61 ec d2 b6  <+...iN.7...a...
{     5}[library\ssl_msg.c:2802] 0080:  b7 7e 5d 64 37 22 e7 c6 5e 6a a5 b0 42 2c 44 b2  .~]d7"..^j..B,D.
{     9}[library\ssl_msg.c:2802] 0090:  f7 37 2e 8d be 0f df dd f3 e7 32 dc f1 74 c3 26  .7........2..t.&
{     9}[library\ssl_msg.c:2802] 00a0:  4b 5d c6 ab 70 bb 03 30 1c 67 22 50 dd e7 57 d4  K]..p..0.g"P..W.
{     9}[library\ssl_msg.c:2802] 00b0:  99 07 f3 ee 50 5b 0d 9f f6 f3 23 ce 8e 4a 78 92  ....P[....#..Jx.
{    10}[library\ssl_msg.c:2802] 00c0:  cd 8c fe 6a 32 da 46 b2 0b 3b ef 07 a5 d1 a7 7d  ...j2.F..;.....}
{     9}[library\ssl_msg.c:2802] 00d0:  58 6b 89 fb 51 17 4b 8d 34 82 36 c1 5c 31 b6 61  Xk..Q.K.4.6.\1.a
{     9}[library\ssl_msg.c:2802] 00e0:  62 10 88 d5 04 76 3c 66 6a 38 db 9b f7 16 54 a7  b....v<fj8....T.
{     9}[library\ssl_msg.c:2802] 00f0:  71 67 fe 6e 11 aa 43 78 61 cc 8d ab ec d5 76 b5  qg.n..Cxa.....v.
{     9}[library\ssl_msg.c:2802] 0100:  9f 11 7d 51 06 53 b1 ab 0e 0f 9e 73 f6           ..}Q.S.....s.
{     8}[library\ssl_msg.c:2087] => flush output
{     4}[library\ssl_msg.c:2105] message length: 269, out_left: 269
{     6}[library\ssl_msg.c:2112] ssl->f_send() returned 269 (-0xfffffef3)
{     6}[library\ssl_msg.c:2140] <= flush output
{     3}[library\ssl_msg.c:2853] <= write record
{     4}[library\ssl_msg.c:2661] <= write handshake message
{     5}[library\ssl_cli.c:4273] <= write certificate verify
{     5}[library\ssl_cli.c:4406] client state: 10
{     3}[library\ssl_msg.c:2087] => flush output
{     4}[library\ssl_msg.c:2099] <= flush output
{     4}[library\ssl_msg.c:4943] => write change cipher spec
{     5}[library\ssl_msg.c:2525] => write handshake message
{     4}[library\ssl_msg.c:2684] => write record
{     4}[library\ssl_msg.c:2797] output record: msgtype = 20, version = [3:3], msglen = 1
{     8}[library\ssl_msg.c:2802] dumping 'output record sent to network' (6 bytes)
{     7}[library\ssl_msg.c:2802] 0000:  14 03 03 00 01 01                                ......
{     8}[library\ssl_msg.c:2087] => flush output
{     3}[library\ssl_msg.c:2105] message length: 6, out_left: 6
{     6}[library\ssl_msg.c:2112] ssl->f_send() returned 6 (-0xfffffffa)
{     5}[library\ssl_msg.c:2140] <= flush output
{     4}[library\ssl_msg.c:2853] <= write record
{     4}[library\ssl_msg.c:2661] <= write handshake message
{     5}[library\ssl_msg.c:4957] <= write change cipher spec
{     4}[library\ssl_cli.c:4406] client state: 11
{     4}[library\ssl_msg.c:2087] => flush output
{     4}[library\ssl_msg.c:2099] <= flush output
{     4}[library\ssl_tls.c:3510] => write finished
{     4}[library\ssl_tls.c:3300] => calc  finished tls sha256
{     5}[library\ssl_tls.c:3322] dumping 'calc finished result' (12 bytes)
{     6}[library\ssl_tls.c:3322] 0000:  c4 86 4c be 83 26 54 54 b5 a7 3c e3              ..L..&TT..<.
{     8}[library\ssl_tls.c:3326] <= calc  finished
{     4}[library\ssl_tls.c:3555] switching to new transform spec for outbound data
{     7}[library\ssl_msg.c:2525] => write handshake message
{     5}[library\ssl_msg.c:2684] => write record
{     4}[library\ssl_msg.c:619] => encrypt buf
{     3}[library\ssl_msg.c:641] dumping 'before encrypt: output payload' (16 bytes)
{     7}[library\ssl_msg.c:641] 0000:  14 00 00 0c c4 86 4c be 83 26 54 54 b5 a7 3c e3  ......L..&TT..<.
{     9}[library\ssl_msg.c:886] dumping 'IV used (internal)' (12 bytes)
{     6}[library\ssl_msg.c:886] 0000:  a4 bc 9d 37 00 00 00 00 00 00 00 00              ...7........
{     9}[library\ssl_msg.c:888] dumping 'IV used (transmitted)' (8 bytes)
{     6}[library\ssl_msg.c:888] 0000:  00 00 00 00 00 00 00 00                          ........
{     8}[library\ssl_msg.c:891] dumping 'additional data used for AEAD' (13 bytes)
{     7}[library\ssl_msg.c:891] 0000:  00 00 00 00 00 00 00 00 16 03 03 00 10           .............
{     8}[library\ssl_msg.c:893] before encrypt: msglen = 16, including 0 bytes of padding
{     8}[library\ssl_msg.c:912] dumping 'after encrypt: tag' (16 bytes)
{     6}[library\ssl_msg.c:912] 0000:  29 6d cf cc 19 ba 10 6a 59 41 7b ba 5a da a7 d4  )m.....jYA{.Z...
{     9}[library\ssl_msg.c:1107] <= encrypt buf
{     3}[library\ssl_msg.c:2797] output record: msgtype = 22, version = [3:3], msglen = 40
{     8}[library\ssl_msg.c:2802] dumping 'output record sent to network' (45 bytes)
{     7}[library\ssl_msg.c:2802] 0000:  16 03 03 00 28 00 00 00 00 00 00 00 00 97 bb 05  ....(...........
{     9}[library\ssl_msg.c:2802] 0010:  81 6a 8a 56 43 73 0c 53 59 77 c0 c8 af 29 6d cf  .j.VCs.SYw...)m.
{     9}[library\ssl_msg.c:2802] 0020:  cc 19 ba 10 6a 59 41 7b ba 5a da a7 d4           ....jYA{.Z...
{     9}[library\ssl_msg.c:2087] => flush output
{     4}[library\ssl_msg.c:2105] message length: 45, out_left: 45
{     5}[library\ssl_msg.c:2112] ssl->f_send() returned 45 (-0xffffffd3)
{     6}[library\ssl_msg.c:2140] <= flush output
{     4}[library\ssl_msg.c:2853] <= write record
{     3}[library\ssl_msg.c:2661] <= write handshake message
{     5}[library\ssl_tls.c:3619] <= write finished

P.S. These are test certificates from a test PKI which have 0 consequence if any private key data is leaked.

Just now found a new lead.

The broken handshake happens when generating the key and certificate from the following context:

  • Hashi Corp vault 1.14.0
  • OpenSSL 3.0.9
  • Fedora 38
  • Virtual box 7.0

When generating the certificate from a CSR made using the following context:

  • OpenSSL 1.1.1n build
  • Debian 11
  • WSL 1
  • Windows 11

The CSR is still signed with the vault environment.

Could it be there is some kind of difference in the notation of the private keys between the two environments? How would I check that?

It also works fine when generating the CSR in the virtual box context, just using OpenSSL without vault, then signing the certificate using vault. Both when generating the private key using OpenSSL in the command line and when generating it using Vault.

Could it be that some of the fields may not be recognised by mbedTLS and therefore not included in the calculation of the signature?