I’m debugging an issue with Engine.IO over Websocket over Mbed-TLS 3.5.0 over LwIP-TCP, where my client’s EngineIO PONG packets are sometimes delayed, causing an Engine.IO timeout and connection loss. I’ve managed to catch this in WireShark, and I suspect that what’s happening is the client calls mbedtls_ssl_write(), but the PONG is stuck in a buffer in Mbed-TLS somewhere. Eventually when the server timeout expires and it issues a WebSocket close, the client response actually contains the delayed PONG.
Is there something like fflush() for Mbed-TLS? Or something like TCP_NOPUSH /TCP_CORK, but for Mbed-TLS? I’ve come to the conclusion that after sending a PONG I need to wait for the TLS-encrypted PONG packet to make it through Mbed-TLS and at least as far as the TCP send buffer.
I did find mbedtls_ssl_flush_output(), but it looks like this is an internal function. Is there a right way to trigger it after calling mbedtls_ssl_write()?
Thanks for taking a look,
-Richard
Link to the chip-vendor ticket.
Below here I attach the packet log that led me to this conclusion:
- PktNo 905396: Server sends EIO PING(“2”) in single WebSocket frame in single TLS frame in single TCP frame
- PktNo 905397: Client sends only header of EIO PONG in single TLS frame, single TCP frame
- PktNo 905398-905420: network hiccup, TCP recovers
- PktNo 905421: Server, having not received complete EIO PONG(“3”) sends WebSocket Close. On ESP32-S3, this wakes WebSocket client thread in esp_websocket_client_recv() from esp_transport_read() to receive WebSocket close message.
- PktNo 905429: Client responds to close-message, sending final TLS/TCP frame, containing two parts: first, it contains the “3” payload that belongs to the header in PktNo 905397. Next, client sends WebSocket OpCode8, meaning closure.
- PktNo 905428-905431: TLS and TCP teardown
