K64F retrieving file over HTTPS triggers multiple read interrupts

Mbed OS
,
1

(also posted this on the old forum, this one appears to be more active so I’m also posting this here.)

I’m in the process of adapting mbed-tls-sockets to be able to retrieve .bin files of a webserver over https, I’m currently building the files using a yotta docker image.

The binary file that I’m trying to retrieve is 20.4 kb.

I’m initiating a handler according to the example.

s->setOnReadable(Socket::ReadableHandler_t(this, &HelloHTTPS::onReceive));

And after setting up a connection with the server, and sending my data, onReceive triggers multiple times.

I tried to increase the size of the _buffer in main.cpp, and the size of the IO buffer in ssl.h.

main.cpp

 char _buffer[RECV_BUFFER_SIZE]; /**< The response buffer */

ssl.h

#define MBEDTLS_SSL_MAX_CONTENT_LEN         16384   /**< Size of the input / output buffer */

This did change the behaviour, where onReceive triggered less often when the buffers were bigger. Unfortunately it seems that the processor is too fast where it thinks a receive is over, or that some buffer is getting filled before the actual receiving of data is completed. I spend a huge chunk of my day altering settings and adjusting configurations, only the end result was the same.

I also tried to adjust the size of pbuf, or the HEAP allocation size:

/**
 * MEM_SIZE: the size of the heap memory. If the application will send
 * a lot of data that needs to be copied, this should be set high.
 */
#if !defined MEM_SIZE || defined __DOXYGEN__
#define MEM_SIZE                        1600
#endif

Whatever I do, besides the slightly changing behaviour, onReceive in main.cpp keeps triggering multiple times.

I was wondering if anyone has some tips for me, I might be missing something.

UPDATE:
I found out that the maximum body chunk size is dependant on the server which you connect to, in my case it is nginx, which has a max body of 16k (16384 bytes).

This results in that even though I have a buffer size of 20000 bytes, onReceive triggers 2 times, when it has to send something along the lines of 21000 bytes. This is something I can enforce by removing s->recv(_buffer_rx, &_bpos);

Now I’m setting the buffer size to 16k, so I can hold the maximum chunk a server can send.

So the function becomes

/**
 * On Receive handler
 * Parses the response from the server, to check for the HTTPS 200 status code and the expected response ("Hello World!")
 */
void onReceive(Socket *s) {
    if(_flushing == false){
        _flushing=true;
        printf("HTTPS Response received.\r\n");
        _bpos = sizeof(_buffer_rx);
        /* Read data out of the socket */
        s->recv(_buffer_rx, &_bpos);

        /* Print status messages */
        printf("HTTPS: Received %d chars from server\r\n", _bpos);
        printf("%s", _buffer_rx);

        memset(&_buffer_rx[0], 0, sizeof(_buffer_rx));
        _flushing = false;
    }else{
        printf("Cant you see I'm busy?!!\r\n");
    }
}

I find that onReceive triggers 4 times.

Connected to XXXXX:443
Sending HTTPS Get Request...

HTTPS Response received.
HTTPS: Received 16384 chars from server
HTTP/1.1 200 OK
Server: nginx/1.11.4
Date: Tue, 04 Oct 2016 12:12:17 GMT
Content-Type: application/octet-stream
Content-Length: 20980
Last-Modified: Thu, 29 Sep 2016 10:07:03 GMT
Connection: keep-alive
ETag: "57ece7c7-51f4"
Accept-Ranges: bytes

HTTPS Response received.
HTTPS: Received 16384 chars from server

HTTPS Response received.
HTTPS: Received 4852 chars from server

HTTPS Response received.
HTTPS: Received 16384 chars from server

{{success}}
{{end}}
MBED: Socket Error: Connection reset by peer (21)
{{failure}}
{{end}}

Now by removing s->recv from this function I get 2 responses.

Connected to XXXXX:443
Sending HTTPS Get Request...

HTTPS Response received.
HTTPS: Received 16384 chars from server

HTTPS Response received.
HTTPS: Received 16384 chars from server

Where _bpos is not altered to show the correct amount of chars received from the server. Yet it triggers 2 times which theoretically could hold the received data from the server, as the buffer is filled fully, and then partially.

UPDATE 2:

I’m now just circumventing the false onReceive triggers by making the receiving buffer a little bit too big for max chunk size coming in through the socket. The size of _buffer_rx is now 16385, and when s->recv returns a value lower than this, I know there is data in the socket which can be extracted.

When s->recv returns 16385, it hasnt done anything with _bpos, which means that there is no change in the data contained by the socket, and nothing to be done.

        _bpos = sizeof(_buffer_rx)
        s->recv(_buffer_rx, &_bpos);
        if(_bpos == sizeof(_buffer_rx)){
            //printf("This does not seem right.");
            _flushing=false;
            return;
        }
        printf("HTTPS Response received.\r\n");

This method results in 4 triggers, where 2 triggers ‘return’ and this way I can extract the data that I need.

The question remains, what triggers onReceive?