How to download a file using mbedTLS

Mbed TLS
1

How do I download a file using mbedTLS? I created a request that looks like this:

"GET /filename HTTP/1.1\r\n" \
"User-Agent: mbed-TLS-2.13.1\r\n" \
"Accept: */*\r\n" \
"Cache-Control: no-cache\r\n" \
"Host: project.hostname.com\r\n" \
"Accept-Encoding: gzip, deflate, br\r\n" \
"Connection: keep-alive\r\n\r\n"

What I got back was:

< Read from server: 555 bytes read
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 174553
Cache-Control: max-age=3600
Content-Encoding: gzip
Content-Type: application/octet-stream
Etag: b78b3fbb0cd2e0f915087f22dc6b55bbf0dbbe2084dc4d3063f662c951812427
Last-Modified: Fri, 24 Jan 2020 18:36:32 GMT
Strict-Transport-Security: max-age=31556926; includeSubDomains; preload
Accept-Ranges: bytes
Date: Tue, 11 Feb 2020 19:39:17 GMT
X-Served-By: cache-pdk17845-PDK
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1581449957.464767,VS0,VE1
Vary: x-fh-requested-host, accept-encoding

My data buffer is 1 MB and I was expecting a file about 260 KB in size. It looks like I all I got was a header, no binary file. Using the Chrome web browser, I can get the file from

https://project.hostname.com/filename

Chrome does ask where I want to put the file. Postman also works just fine with the GET request above and immediately returns the binary file. What am I missing?

2

HI @jeffthompsoninvue
The question is more related to HTTP protocol than TLS, however I will do my best to answer it.

Could you confirm that this request looks exactly like the request you do in chrome? (except maybe the User-agent)

Are you using mbedtls_ssl_write() to write you message?
Regards,
Mbed TLS Support
Ron

3

Ron,

Yes to both. Someone else here speculated that perhaps there was a separate response object that I should have downloaded. Chrome did ask where I wanted to save the file, maybe that had some bearing here?

Thanks!

Jeff

4

HI Jeff,
I would suggest you capture the HTTP requests that you send to the server using Chrome ( Wireshark? F12 to record the network in Chrome?), and see what you are missing
Regards

5

Ron,

I think I may have found the problem. The example code I started with has this:

int read_request()

{

/*

  • Read the HTTPS response

*/

int ret = 0;

int len = 0;

PRINTF(" < Read from server:");

do

{

len = sizeof(https_buf) - 1;

memset(https_buf, 0, sizeof(https_buf));

ret = mbedtls_ssl_read(&(tlsDataParams.ssl), https_buf, len);

if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE)

continue;

if (ret == MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY)

break;

if (ret < 0)

{

PRINTF(“failed\n ! mbedtls_ssl_read returned %d\n\n”, ret);

goto exit;

}

if (ret == 0)

{

PRINTF(“\n\nEOF\n\n”);

break;

}

len = ret;

PRINTF(" %d bytes read\n\n%s", len, (char *)https_buf);

} while (1);

return ret;

exit:

https_client_tls_release();

return -1;

}

I have modified it to this:

int read_request()

{

/*

  • Read the HTTPS response

*/

int ret = 0;

uint32_t len = 0;

PRINTF(" < Read from server:");

memset(https_buf, 0, sizeof(https_buf));

do

{

len = sizeof(https_buf);

ret = mbedtls_ssl_read(&(tlsDataParams.ssl), https_buf + ret, (len >= UINT_MAX) ? UINT_MAX : len);

if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE)

continue;

if (ret == MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY)

break;

if (ret < 0)

{

PRINTF(“failed\n ! mbedtls_ssl_read returned %d\n\n”, ret);

goto exit;

}

if (ret == 0)

{

PRINTF(“\n\nEOF\n\n”);

break;

}

len -= ret;

// verbose debug, not suitable when getting binary data

// PRINTF(" %d bytes read\r\n%s\r\n", len, https_buf);

PRINTF(" %d bytes read\r\n", ret);

} while ( ret > 0 );

return ret;

exit:

https_client_tls_release();

return -1;

}

What I get back is:

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 174553

Cache-Control: max-age=3600

Content-Encoding: gzip

Content-Type: application/octet-stream

Etag: b78b3fbb0cd2e0f915087f22dc6b55bbf0dbbe2084dc4d3063f662c951812427

Last-Modified: Fri, 24 Jan 2020 18:36:32 GMT

Strict-Transport-Security: max-age=31556926; includeSubDomains; preload

Accept-Ranges: bytes

X-Served-By: cache-pdk17851-PDK

X-Cache: HIT

X-Cache-Hits: 1

X-Timer: S1581510695.851197,VS0,VE1

Vary: x-fh-requested-host, accept-encoding

followed by binary data. It’s not the data I expected, but going by what the response header has it could be gzip compressed, and 7zip actually does unzip the binary data to match the first part of the file I downloaded.

However, I’m not getting the entire file, only about 2394 bytes. So there must something I need to do in order to get the rest. Any suggestions?

Thanks!

image001.gif

6

Hi Jeff,
I have some comments:

  • Change len to uint32 is better (I would change it to size_t for portability )but I doubt this was the cause for previous error, is even MAX_INT is ~2GB
  • in the begining of your do\while loop, you inisitiate len to sizeof(https_buf), so updating it at the end to len -= ret doesn’t have effect. I would set len to sizeof(https_buf) at definition \ before the loop, but again, I doubt that this is the root cause, as eventually, only what is ready is returned.

Could you specify how many iterations the loop ran?
What is the output of every PRINTF(" %d bytes read\r\n", ret); ?

I am guessing that you are getting the error MBEDTLS_ERR_SSL_WANT_READ , which your continue statement jumps to the loop condition (while ( ret > 0 )), which fails and stops the loop because this eror has a negative value.
I would try changing the while statement to:

while ( ret > 0  ||
        ret == MBEDTLS_ERR_SSL_WANT_READ ||
        ret == MBEDTLS_ERR_SSL_WANT_WRITE )

Regards

7

Hi,
I have another minor comment,
Since https_buf is the buffer which you want to read to, you don’t really need to change the position of where to read to, and always read into the begining of https_buf len bytes. However, you will probably need another buffer to store the binary data to.

8

Ron,

I’ve changed the function to

int read_request()

{

/*

  • Read the HTTPS response

*/

int ret = 0;

uint32_t bufLen = sizeof(https_buf);

uint32_t bytesRead = 0;

PRINTF(" < Read from server:");

memset(https_buf, 0, sizeof(https_buf));

do

{

ret = mbedtls_ssl_read(&(tlsDataParams.ssl), https_buf + bytesRead, (bufLen >= UINT_MAX) ? UINT_MAX : bufLen);

if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE)

continue;

if (ret == MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY)

break;

if (ret < 0)

{

PRINTF(“failed\n ! mbedtls_ssl_read returned 0x%04x\n\n”, ret);

goto exit;

}

if (ret == 0)

{

PRINTF(“\n\nEOF\n\n”);

break;

}

bytesRead += ret;

bufLen -= bytesRead;

// verbose debug, not suitable when getting binary data

// PRINTF(" %d bytes read\r\n%s\r\n", len, https_buf);

PRINTF(" %d bytes read\r\n", ret);

} while ( ret > 0 );

return ret;

exit:

https_client_tls_release();

return -1;

}

and its now downloading my entire file. I just need to figure out where the response header ends and the file begins. I can see it is; I just need to find it programmatically.

image001.gif

image002.jpg

9

Ron,

I now read until there is no more to read, or there is an error, and everything goes into the same buffer. Figuring out where the header ends and the binary data begins is the last task.

Thanks for all the help!

image001.gif

10

HI @jeffthompsoninvue
I believe that the line:

if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE)

continue;

is equivalent to

if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE)

break;

Since if the returned error code is one of the two the latter will break the loop, and the former will go to the while condition, and then exit the loop because the returned error code has a negative value. I am not sure this is what you intended. I would recomend you also check in the while condition for these errors as well, if this is what you intended.
As for finding where the header ends and where the binary data begins, as mentioned in https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages:

HTTP requests, and responses, share similar structure and are composed of:

  1. A start-line describing the requests to be implemented, or its status of whether successful or a failure. This start-line is always a single line.
  2. An optional set of HTTP headers specifying the request, or describing the body included in the message.
  3. A blank line indicating all meta-information for the request have been sent.
  4. An optional body containing data associated with the request (like content of an HTML form), or the document associated with a response. The presence of the body and its size is specified by the start-line and HTTP headers.

So, I believe that in the response you are getting there is a blank line between the header and the content.
Regards

11

Ron,

Thanks again! Below is my finished read_request(). I think the code I started with may have been written by an intern. It’s what NXP provides as their lwip_httpscli_mbedTLS_freertos example for MCUXpresso.

BTW, the link you supplied didn’t work for me. I’ve been referring to HTTP 1.1
here.

int read_request()

{

/*

  • Read the HTTPS response

*/

int ret = 0;

size_t bufLen = sizeof(https_buf);

size_t remaining = bufLen;

bool headerRead = false;

size_t bytesReadTotal = 0;

PRINTF(" < Read from server:");

memset(https_buf, 0, sizeof(https_buf));

do

{

ret = mbedtls_ssl_read( &(tlsDataParams.ssl), https_buf + bytesReadTotal, remaining );

if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE)

{

#ifdef HTTPS_DEBUG_VERBOSE

PRINTF( “%s %d ret %d\r\n”, FUNCTION, LINE, ret );

#endif

break;

}

if (ret == MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY)

{

#ifdef HTTPS_DEBUG_VERBOSE

PRINTF( “%s %d ret %d\r\n”, FUNCTION, LINE, ret );

#endif

break;

}

if (ret < 0)

{

PRINTF(“failed\n ! mbedtls_ssl_read returned 0x%04x\n\n”, ret);

ret = -1;

#ifdef HTTPS_DEBUG_VERBOSE

PRINTF( “%s %d ret %d\r\n”, FUNCTION, LINE, ret );

#endif

break;

}

if (ret == 0)

{

PRINTF(“\n\nEOF\n\n”);

#ifdef HTTPS_DEBUG_VERBOSE

PRINTF( “%s %d ret %d\r\n”, FUNCTION, LINE, ret );

#endif

break;

}

bytesReadTotal += ret;

if( !headerRead )

{

char* endOfHeaderLocation = strstr( (char*) https_buf, “\r\n\r\n” );

// be sure the end of the header is in sight before trying to find the content length

if( NULL != endOfHeaderLocation )

{

#ifdef HTTPS_DEBUG_NORMAL

PRINTF( “Header:\r\n%s\r\n”, https_buf );

#endif

char* contentLengthLocation = strstr( (char*) https_buf, "Content-Length: " );

if( NULL != contentLengthLocation )

{

if( 1 == sscanf(contentLengthLocation, “Content-Length: %u\r\n”, &remaining ) )

{

headerRead = true;

}

else

{

// this should not happen

remaining -= ret;

}

}

else

{

// this should not happen

remaining -= ret;

}

}

}

else

{

remaining -= ret;

}

// verbose debug, not suitable when getting binary data

#ifdef HTTPS_DEBUG_VERBOSE

PRINTF( " bytes read %d remaining %d\r\n", ret, remaining );

#endif

} while ( remaining > 0 );

#ifdef HTTPS_DEBUG_VERBOSE

PRINTF( “%s %d ret %d\r\n”, FUNCTION, LINE, ret );

#endif

return ret > 0 ? 1 : ret;

}

image001.gif

12

HI @jeffthompsoninvue
Thanks for supplying the working solution.
I think there could be some optimization, by reducing number of variables and code.
For example:

f (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE)
{
#ifdef HTTPS_DEBUG_VERBOSE
PRINTF( “%s %d ret %d\r\n”, FUNCTION, LINE, ret );
#endif
break;
}

if (ret == MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY)
{
#ifdef HTTPS_DEBUG_VERBOSE
PRINTF( “%s %d ret %d\r\n”, FUNCTION, LINE, ret );
#endif
break;
}
if (ret < 0)
{
PRINTF(“failed\n ! mbedtls_ssl_read returned 0x%04x\n\n”, ret);
ret = -1;
#ifdef HTTPS_DEBUG_VERBOSE
PRINTF( “%s %d ret %d\r\n”, FUNCTION, LINE, ret );
#endif
break;
}

Are conditions that do the same thing. Unless you have special behavior for MBEDTLS_ERR_SSL_WANT_READ , MBEDTLS_ERR_SSL_WANT_WRITE or MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY, I would just combine them with if (ret < 0)
Regards

13

Thanks, Ron. I just wanted to clearly identify the function and line number where the error was generated. The PRINTF goes to a debug port that only developers have access to using a board populated with the connector; it won’t be turned
on for production code.

image001.gif

14

Thanks for your ideas and solutions. I have same code but most of the time I am able to download 16 384 bytes only. At maximum, once it was 98 304 bytes. Real file size is 1.7MB. Same file can be sent using POST http form (http only, without SSL), but not as client download. Buffer for MbedSSL is in internal SRAM, Storage location for file download is placed in external SDRAM.

Client runs on STM32 MCU (HAL + FreeRTOS + LwIP + TouchGFX).
Code is taken from mbedtls/ssl_client1.c at development · Mbed-TLS/mbedtls · GitHub

Log says:

<= fetch input input record: msgtype = 133, version = [169:203], msglen = 34693 unknown record type

Wrong message type (133) causes read fail at ssl_tls.c : 4064:

    /* Check record type */
    if( ssl->in_msgtype != MBEDTLS_SSL_MSG_HANDSHAKE &&
        ssl->in_msgtype != MBEDTLS_SSL_MSG_ALERT &&
        ssl->in_msgtype != MBEDTLS_SSL_MSG_CHANGE_CIPHER_SPEC &&
        ssl->in_msgtype != MBEDTLS_SSL_MSG_APPLICATION_DATA )
    {
        MBEDTLS_SSL_DEBUG_MSG( 1, ( "unknown record type" ) );
        ...
        return( MBEDTLS_ERR_SSL_INVALID_RECORD );
    }

Any ideas how to make file read reliable?

Thanks!

Full logs:

  < Read from server:

=> read
=> read record
=> fetch input in_left: 0, nb_want: 5in_left: 0, nb_want: 5
<= fetch input input record: msgtype = 23, version = [3:3], msglen = 16408
=> fetch input in_left: 5, nb_want: 16413in_left: 5, nb_want: 16413in_left: 5235, nb_want: 16413in_left: 6695, nb_want: 16413in_left: 11075, nb_want: 16413in_left: 12535, nb_want: 16413in_left: 15455, nb_want: 16413
<= fetch input
=> decrypt buf
<= decrypt buf
<= read record
<= read
=> read
=> read record
=> fetch input in_left: 0, nb_want: 5in_left: 0, nb_want: 5
<= fetch input input record: msgtype = 23, version = [3:3], msglen = 16408
=> fetch input in_left: 5, nb_want: 16413in_left: 5, nb_want: 16413in_left: 4882, nb_want: 16413in_left: 6342, nb_want: 16413in_left: 10722, nb_want: 16413in_left: 12182, nb_want: 16413
<= fetch input
=> decrypt buf
<= decrypt buf
<= read record
<= read
=> read
=> read record
=> fetch input in_left: 0, nb_want: 5in_left: 0, nb_want: 5
<= fetch input input record: msgtype = 23, version = [3:3], msglen = 16408
=> fetch input in_left: 5, nb_want: 16413in_left: 5, nb_want: 16413in_left: 4529, nb_want: 16413in_left: 5989, nb_want: 16413in_left: 10369, nb_want: 16413in_left: 11829, nb_want: 16413in_left: 16209, nb_want: 16413
<= fetch input
=> decrypt buf
<= decrypt buf
<= read record
<= read
=> read
=> read record
=> fetch input in_left: 0, nb_want: 5in_left: 0, nb_want: 5
<= fetch input input record: msgtype = 23, version = [3:3], msglen = 16408
=> fetch input in_left: 5, nb_want: 16413in_left: 5, nb_want: 16413in_left: 5636, nb_want: 16413in_left: 10016, nb_want: 16413in_left: 11476, nb_want: 16413in_left: 12936, nb_want: 16413
<= fetch input
=> decrypt buf
<= decrypt buf
<= read record
<= read
=> read
=> read record
=> fetch inputin_left: 0, nb_want: 5in_left: 0, nb_want: 5
<= fetch input input record: msgtype = 133, version = [169:203], msglen = 34693 unknown record type
=> send alert message send alert level=2 message=10
=> write record
=> encrypt buf before encrypt: msglen = 10, including 0 bytes of padding
<= encrypt buf output record: msgtype = 21, version = [3:3], msglen = 26
=> flush output message length: 31, out_left: 31
<= flush output
<= write record
<= send alert message failed
  ! mbedtls_ssl_read returned FFFF8E00

 65536 bytes read

=> write close notify
=> send alert message send alert level=1 message=0
=> write record
=> encrypt buf before encrypt: msglen = 10, including 0 bytes of padding
<= encrypt bufoutput record: msgtype = 21, version = [3:3], msglen = 26
=> flush outputmessage length: 31, out_left: 31
<= flush output
<= write record
<= send alert message
<= write close notify
=> free
<= free

SSL - An invalid SSL record was received