Mbedtls handshake process

I need help with my mbedtls server
So my system is a modbus server together with the mbedtls server running in diffrent thread on FreeRtos
Both the threads are at same priority and working with time slicing between them
I need my main thread which is modbus to not get impacted with the handshake process which takes almost 6 sec in my system
Is it possible to have least impact of handshake on the modbus TCP communication
Because the some handshake steps related to the calculation of curves blocking the main thread for long time