We have did a custom secure protocol which is similar to ssl for low end devices. Basically it uses hmac, &x.509 (only rsa2048)certificate for authentication and AES GCM for cipher. We try to keep the in-house designed protocol simple. TLS/DTLS might not be feasible as the channel is serial with 9600 baud rate, with ROM cap 150 KB.
Currently, the code size takes around 170 KB, for the program. We do handshake similar to ssl to authenticate device, symmetric key, & network time using certificate and hmac combined. THen we use AES GCM for encryption after handshake. Tried to disable unwanted modules, like chacha20, sha512, md5, ecdh, dsa, etc by removing them from build directory. However checkconfig is throwing many define errors, although they were never called in any of my program.
What is the best way to optimize the output code size