Arm Mbed and Pelion Device Management support forum

POST is sent twice

I am currently developing an IoT device which must connect to an API-server.
Since this is my first project with mbedtls on freertos I am a bit lost and I hope I can learn fast.

System summary:
Controller: STM32F746IGT
OS: Freertos
TCP/IP-Stack: LWIP
And of course mbedTLS

So bascially what the programm needs to do:

  • Start a TLS connection
  • Get an OAuth2 token from the server
  • Do something with it (not yet clear because I am missing a LOT of information)
  • TBD

So I managed with the help of others to start up the TLS connection. This works so far. It correctly encrypts and decrypts messages with the server. I checked that with Wireshark and I extracted the Master Secret from it. (I have made an HTML/JS application to extract the ‘client hello random’ and ‘master secret’ key from the debug output of mbedtls if anyone wants it)

The problem
So after analyzing the whole datastream which is sent to the serve I noticed that the POST header is sent twice for no apparant reason.
I only send it once in the code. It looks like this:

sprintf((char *) buf, POST_REQUEST);
len = strlen((char *) buf);
while((ret = mbedtls_ssl_write(&ssl, buf, len)) <= 0) { ...

But on Wireshark I see this after I decrypted the whole transaction:
Sent to server: (one package)

POST /uaa/oauth/token HTTP/1.1
Host: my.server.com
Content-type: application/x-www-form-urlencoded

POST /uaa/oauth/token HTTP/1.1
Host: my.server.com
Content-type: application/x-www-form-urlencoded

grant_type=password&username=some.email@address.org&password=password1234&client_id=iiot-gateway&client_secret=xxxxxxxxx

Received from server:

HTTP/1.1 401
Server: nginx/1.14.0 (Ubuntu)
Date: Tue, 28 Jan 2020 10:38:15 GMT
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-store
Pragma: no-cache
WWW-Authenticate: Basic realm="UAA/client", error="unauthorized", error_description="Bad credentials"
X-XSS-Protection: 1; mode=block
X-Frame-Options: DENY
X-Content-Type-Options: nosniff

{"error":"unauthorized","error_description":"Bad credentials"}HTTP/1.1 400 Bad Request
Server: nginx/1.14.0 (Ubuntu)
Date: Tue, 28 Jan 2020 10:38:15 GMT
Content-Type: text/html
Content-Length: 182
Connection: close
<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.14.0 (Ubuntu)</center>
</body>
</html>

apperantly the server also receives two POST requests.

I also have the entire debuglog of this session stored so if more informations are needed I can provide.
All login data and server addresses have been changed.

Hi Adrian,
I’m not very good at this, however I’m using Mbed TLS connection for use on Firebase. Below is my POST function that works, or at least it appears to be working, I cant’ see any double posting issues and runs for days. I’m using the Socket reuse method but have found that after precisely 8 minutes I get

HttpRequest failed (error code -3012)

where I detect this and close the socket and open a new one, other than that no issues. The single non reuse function works with no issues.

I see you have a ‘while’ in your write line, perhaps try a different way?? but I know noting about this its only a tool for me to send stuff to Firebase.

bool postFirebase(char* FirebaseUrl, char *stringToProcess)
{  
 // HttpsRequest* post_req = new HttpsRequest(network, SSL_CA_PEM, HTTP_POST, FirebaseUrl);   // non socket reuse function
HttpsRequest* post_req = new HttpsRequest(socket, HTTP_POST, FirebaseUrl);  // socket reuse function   
post_req->set_header("Content-Type", "application/json");
HttpResponse* post_res = post_req->send(stringToProcess, strlen(stringToProcess));    
if (!post_res) {
    time_t seconds = time(NULL);                                        
    printf("Https POST failed (error code %d), %s", post_req->get_error(), ctime(&seconds));
    socket->close();
    delete socket;
    delete post_req;
    TLSSocket* socket = new TLSSocket();
    startTLSreusesocket((char*)FirebaseID); // restart TLS reuse socket if failure 
    return 0;
    }
    else{
        //dump_response(post_res);    
        delete post_req;     
        return post_res;
        }           

}

My library is here, you could pick the bones out it and if you find out why it trips out after 8 minutes, let me know.

https://os.mbed.com/users/star297/code/Firebase-Example/

BR

Paul

Hello Paul
Thanks for your input but that doesn’t help me at all. I am using the C version of mbedtls not C++.

Also we managed to find something out about that behaviour.

If the POST request is declared like:

const char POST_REQUEST[] = "POST /uaa/oauth/token HTTP/1.1\r\n"
                       "Host: uaa.test.liquidtool.com\r\n"
                       "Content-type: application/x-www-form-urlencoded\r\n"
                       "\r\n"
                       "grant_type=password&etc...";

We get the POST sent twice (like I showed in the orignal post) but if we declare it like this:

const char POST_REQUEST[] = "POST /uaa/oauth/token HTTP/1.1\r\n"
                       "Host: uaa.test.liquidtool.com\r\n"
                       "Content-type: application/x-www-form-urlencoded\r\n"
                       "grant_type=password&etc...";

The POST-Request is only sent once with the data directly attached to it. After a while the server closes the connection since the request has the wrong format and it doesn’t know how to handle it.
The POST-Request will look something like this:

POST /uaa/oauth/token HTTP/1.1
Host: my.server.com
Content-type: application/x-www-form-urlencoded
grant_type=password&username=some.email@address.org&password=password1234&client_id=iiot-gateway&client_secret=xxxxxxxxx

The server gives a correct reply but of course terminates the connection.

Hi @daybreakerflint

As mentioned by @star297, you are sending your POST message in a while loop.
I am guessing that the first ‘write’ returned some error or 0 ( please check the error code), and the second write returned a positive number. Sinceyou are not modifying the positionin thebufbetween everty iteration, according to how many bytes were written, you are trying to write samebuf, with same original len. I would first check what the return code from mbedtls_ssl_write()` is and change the code to the following:

sprintf((char *) buf, POST_REQUEST);
len = strlen((char *) buf);
ret = 0;
while((ret = mbedtls_ssl_write(&ssl, buf + ret, len - ret)) <= 0) { ...

Regards,
Mbed TLS Support
Ron

1 Like

Hey @roneld01
Thanks for the tip but I don’t think this is the problem but I will try tomorrow. Hopefully I’ll be at least one step further. I keep you posted on what the result will be. I will have the tools then again to analyze the network traffic.
All the best
Adrian

Hi Adrian,
Are you sure that your data is correct? Your server response looks just like I had before I got the authorization and .json data format correct.
Perhaps to save you pulling the rest of your hair out, could I suggest to try my example.
It may give you a starting point. Then you can work back on your C version.

My server response, although a PUT in this case, not POST, looks like this:

Status: 200 - OK
Headers:
Server: nginx
Date: Mon, 03 Feb 2020 17:44:27 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 107
Connection: keep-alive
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Strict-Transport-Security: max-age=31556926; includeSubDomains; preload

Body (107 bytes):

{“Humidity”:“96.00”,“Pressure”:“1081.00”,“TempIn”:“73.85”,“TempOut”:“108.48”,“Time”:"18:44:10 02/03/2020 "}
PUT Current okay, time: 18:44:10 02/03/2020

Paul

1 Like

Hey Paul
I would not know where to start to be frank… I am currently answering you outside of my working hours trying to find a solution for this problem. (Frankly, I pulled to many hairs out already and I am wondering if I am even smart enough to solve this.)

First of all it is a custom board with a PHY and certain pinout I don’t know how easy this would be to port. I don’t know if that porting work is worth the time.
Second, I don’t have the tools yet to make this work and I would need to study those too which will take up another day or so.
I am honestly lost. I can try and make it work here at home but then I would need to port it at work.

Adrian, your custom board is basically this with a different pin count ?

https://eu.mouser.com/ProductDetail/STMicroelectronics/NUCLEO-F746ZG?qs=mKNKSX85ZJdK2WTD64xf8A==&vip=1&gclid=CjwKCAiAyeTxBRBvEiwAuM8dnfjyT7yVSLWXmQPdbubUqGz801bWeWByr-I1ed3RVkrQidm74rfPYRoCwzkQAvD_BwE

You could use Mbed’s RTOS and this board to develop your application. once you get it working all you would need to do is change the pin defines for the PHY in the “target_overrides” in the mbed_app.json file to suit your custom board. It may save you a lot of time?

Hey Paul
If I would have the board and if we didn’t already had a bigger application running on it, I would change the OS but for now I will not do that. Since I have no idea on how I can explain this to my boss. I already stepped outside my boundaries as I ordered the managed switch. I was barely able to start this programm up let alone now starting up a whole new OS.
At least it would be the same PHY.

@roneld01 I tried what you told me but it still seems to send out the POST twice. I am lost on what to do next.

Regards Adrian

@daybreakerflint
Thank you for trying my suggestion. Have you checked what the return code from the write function was?
Regards

It returns the lenght of the string and is not called more than 1 time.

I also started fiddling around with the POST format.
It seemed to have a big inpact on how the server interprates the data depending on the formation.

const char POST_REQUEST[] = "POST /uaa/oauth/token HTTP/1.1\r\n"
                       "Host: uaa.test.liquidtool.com\r\n"
                       "Content-type: application/x-www-form-urlencoded\r\n"
                       "\r\n"
                       "grant_type=password&etc...";
const char POST_REQUEST[] = "POST /uaa/oauth/token HTTP/1.1\r\n"
                       "Host: uaa.test.liquidtool.com\r\n"
                       "Content-type: application/x-www-form-urlencoded\r\n"
                       "\r\n"
                       "grant_type=password&etc...\r\n";

If the last newline/cariage return is left out, only the POST seems to be sent by wireshark.
But on closer inspection the whole package is sent out correctly.
I am starting to think this is an problem because of the formating and the interpretation of the server.

Is there a norm how to format the POST-Request? Because it seems to be highly dependent on this.

So @roneld01 and @star297
This matter is closed because our customer provided us with more information.

The tag Content-Lenght in the header must be provided.
so it looks like this:

POST /uaa/oauth/token HTTP/1.1
Host: my.server.com
Contet-Lenght: 150
Content-type: application/x-www-form-urlencoded

grant_type=password&username=some.email@address.org&password=password1234&client_id=iiot-gateway&client_secret=xxxxxxxxx

So yeah. This seems to work…
I got worked up for nothing. Thanks for your help and support! Hope you have a good day!