How to shorten compilation time up to six times

hudakz · December 2, 2020, 12:33pm

Hello folks,

I saw several posts on the forum advising to use the .mbedignore configuration file to reduce the compilation time and binary size. However, I haven’t seen specific examples. Here, I’d like share a .mbedignore file I use in my projects. It tries to group Mbed main features enabling to add them one by one as needed. If one adds this file to the root directory of a project it can significantly shorten the compilation time. For example, in case of building a simple blinky project for an mbed LPC1768 using Mbed OS 6.5.0 it reduces the compilation time about six times.

/* Bootloader */
mbed-os/features/FEATURE_BOOTLOADER/*

/* BLE */
mbed-os/connectivity/drivers/ble/*
mbed-os/connectivity/FEATURE_BLE/*

/* Cellular */
mbed-os/connectivity/cellular/*
mbed-os/connectivity/drivers/cellular/*
mbed-os/connectivity/netsocket/source/Cellular*.*

/* Device Key */
mbed-os/drivers/device_key/*

/* Experimental /
mbed-os/platform/FEATURE_EXPERIMENTAL_API/*

/* FPGA */
mbed-os/features/frameworks/COMPONENT_FPGA_CI_TEST_SHIELD/*

/* Greentea client */
mbed-os/features/frameworks/greentea-client/*

/* LORAWAN */
mbed-os/connectivity/drivers/lora/*
mbed-os/connectivity/lorawan/*

/* LWIP */
mbed-os/connectivity/drivers/emac/*
mbed-os/connectivity/lwipstack/*

/* Mbed-client-cli */
mbed-os/features/frameworks/mbed-client-cli/*

/* MBED TLS */
mbed-os/connectivity/drivers/mbedtls/*
mbed-os/connectivity/mbedtls/*

/* Nanostack */
mbed-os/connectivity/drivers/emac/*
mbed-os/connectivity/libraries/mbed-coap/*
mbed-os/connectivity/libraries/nanostack-libservice/*
mbed-os/connectivity/libraries/ppp/*
mbed-os/connectivity/nanostack/*

/* Netsocket */
mbed-os/connectivity/drivers/emac/*
mbed-os/connectivity/netsocket/*
mbed-os/libraries/mbed-coap/*
mbed-os/libraries/ppp/*

/* NFC */
mbed-os/connectivity/drivers/nfc/*
mbed-os/connectivity/nfc/*

/* RF */
mbed-os/connectivity/drivers/802.15.4_RF/*

/* Storage */
mbed-os/storage/filesystem/*
mbed-os/storage/kvstore/*
mbed-os/storage/platform/*

/* Tests */
mbed-os/platform/test/*
mbed-os/TEST_APPS/*
mbed-os/TESTS/*
mbed-os/UNITTESTS/*

/* Unity */
mbed-os/features/frameworks/unity/*

/* Utest */
mbed-os/features/frameworks/utest/*

/* USB */
mbed-os/drivers/usb/source/*
mbed-os/hal/usb/source/*
mbed-os/hal/usb/TARGET_Templates/*

/* WiFi */
mbed-os/connectivity/drivers/wifi/*

To include (add) some additional features to the project just comment out the selected entry by adding // in front of the associated line. For example, the following is needed to include Ethernet for the mbed NUCLEO_F767ZI target:

/* Bootloader */
mbed-os/features/FEATURE_BOOTLOADER/*

/* BLE */
mbed-os/connectivity/drivers/ble/*
mbed-os/connectivity/FEATURE_BLE/*

/* Cellular */
mbed-os/connectivity/cellular/*
mbed-os/connectivity/drivers/cellular/*
mbed-os/connectivity/netsocket/source/Cellular*.*

/* Device Key */
mbed-os/drivers/device_key/*

/* Experimental /
mbed-os/platform/FEATURE_EXPERIMENTAL_API/*

/* FPGA */
mbed-os/features/frameworks/COMPONENT_FPGA_CI_TEST_SHIELD/*

/* Greentea client */
mbed-os/features/frameworks/greentea-client/*

/* LORAWAN */
mbed-os/connectivity/drivers/lora/*
mbed-os/connectivity/lorawan/*

/* LWIP */
//mbed-os/connectivity/drivers/emac/*
//mbed-os/connectivity/lwipstack/*

/* Mbed-client-cli */
mbed-os/features/frameworks/mbed-client-cli/*

/* MBED TLS */
//mbed-os/connectivity/drivers/mbedtls/*
//mbed-os/connectivity/mbedtls/*

/* Nanostack */
//mbed-os/connectivity/drivers/emac/*
//mbed-os/connectivity/libraries/mbed-coap/*
//mbed-os/connectivity/libraries/nanostack-libservice/*
//mbed-os/connectivity/libraries/ppp/*
//mbed-os/connectivity/nanostack/*

/* Netsocket */
//mbed-os/connectivity/drivers/emac/*
//mbed-os/connectivity/netsocket/*
mbed-os/libraries/mbed-coap/*
mbed-os/libraries/ppp/*

/* NFC */
mbed-os/connectivity/drivers/nfc/*
mbed-os/connectivity/nfc/*

/* RF */
mbed-os/connectivity/drivers/802.15.4_RF/*

/* Storage */
mbed-os/storage/filesystem/*
mbed-os/storage/kvstore/*
mbed-os/storage/platform/*

/* Tests */
mbed-os/platform/test/*
mbed-os/TEST_APPS/*
mbed-os/TESTS/*
mbed-os/UNITTESTS/*

/* Unity */
mbed-os/features/frameworks/unity/*

/* Utest */
mbed-os/features/frameworks/utest/*

/* USB */
mbed-os/drivers/usb/source/*
mbed-os/hal/usb/source/*
mbed-os/hal/usb/TARGET_Templates/*

/* WiFi */
mbed-os/connectivity/drivers/wifi/*

I hope someone will find this useful.

Best regards, Zoltan

ladislas · December 3, 2020, 10:42am

Thanks @hudakz for sharing that!

I’ll compare it to our current .mbedignore and make the appropriate changes

dudmuck · December 3, 2020, 6:04pm

i think that only speeds up compile the first time building. Once the object files are there, those arent re-built unless source changes.

But i have noticed that windows compile time is perhaps 3 to 4 times slower compared to linux. I have a dual boot machine, so its apples to apples. It probably has to do something with anti-malware or windows security, because compiler object files are such a threat to windows.

hudakz · December 3, 2020, 7:34pm

Thank you for the feedback. However, Mbed project is rebuilt also after modifying the mbed_app.json file even when the source files have no been modified. So if one fiddles with the project configuration it can save valuable time.

In addition to shorter compilation time the size of the BUILD directory is also significantly reduced. So it saves disk space too. For example in case of the already mentioned blinky project the size is reduced from 205.8 MB to 44.3 MB (about 4.6 times).

The main reason why I switched from Windows to Linux was faster compilation. However, it’s still far from beating Arduino compilation time.

hudakz · December 11, 2020, 1:57pm

To improve it a bit I added:

/* Device Key */
mbed-os/drivers/device_key/*

to the file in my first post. Such .mbedignore file seems to work better with STM targets.

sstaub · January 13, 2021, 12:11pm

Tried also with NUCLEO_F767ZI and Ethernet, I have some fails with:

    /* Cellular */
    mbed-os/connectivity/cellular/*
    mbed-os/connectivity/drivers/cellular/*
    mbed-os/connectivity/netsocket/source/Cellular*.*

I don’t understand what Ethernet have to do with cellular.

hudakz · January 13, 2021, 2:16pm

It’s hard to say without having the source code, but It seems you have some portions in your project related to the Cellular. It could be a leftover library or test code.

The following example compiles and runs fine on a NUCLEO-F767ZI when built with Mbed OS 6.6.0 along with the second .mbedingnore file shown in my first post:

#include "mbed.h"
#include "EthernetInterface.h"
#include "TCPSocket.h"
#include <stdio.h>
#include <string>

using namespace std;

#define IP      "192.168.1.181"
#define GATEWAY "192.168.1.1"
#define NETMASK "255.255.255.0"
#define PORT    80

EthernetInterface*  net;
TCPSocket           server;
TCPSocket*          clientSocket;
SocketAddress       clientAddress;
char                receiveBuf[1024];
const int           OFF = 0;
const int           ON = 1;
DigitalOut          sw(LED1);               // A digital output to be switched on/off
float               roomTemp = 21.8;        // A temperature sensor output
const string        PASSWORD = "secret";    // Change as you like
const string        HTTP_OK = "HTTP/1.0 200 OK";
const string        MOVED_PERM = "HTTP/1.0 301 Moved Permanently\r\nLocation: ";
const string        UNAUTHORIZED = "HTTP/1.0 401 Unauthorized";
string              httpHeader;             // HTTP header
string              httpContent;            // HTTP content

/**
 * @brief   Analyses the received URL
 * @note    The string passed to this function will look like this:
 *          GET /HTTP/1.....
 *          GET /password HTTP/1.....
 *          GET /password/ HTTP/1.....
 *          GET /password/?sw=1 HTTP/1.....
 *          GET /password/?sw=0 HTTP/1.....
 * @param   url URL string
 * @retval -3 just refresh page
 *         -2 no command given but password valid
 *         -1 invalid password
 *          0 switch off
 *          1 switch on
 */
int8_t analyseURL(string& url)
{
    if (url.length() < 5 + PASSWORD.size() + 1)
        return(-1);

    if (url.substr(5, PASSWORD.size()) != PASSWORD)
        return(-1);

    uint8_t pos = 5 + PASSWORD.size();

    if (url.substr(pos, 1) != "/")
        return(-1);

    if (url.substr(pos++, 1) == " ")
        return(-2);

    string  cmd(url.substr(pos, 5));

    if (cmd == "?sw=0")
        return(0);

    if (cmd == "?sw=1")
        return(1);

    return(-3);
}

/**
 * @brief
 * @note
 * @param
 * @retval
 */
string& movedPermanently(uint8_t flag)
{
    if (flag == 1)
        httpContent = "/" + PASSWORD + "/";
    else
        httpContent = "";

    httpContent += "<h1>301 Moved Permanently</h1>\r\n";

    return(httpContent);
}

/**
 * @brief
 * @note
 * @param
 * @retval
 */
string& showWebPage(int status)
{
    char    roomTempStr[5];

    //roomTemp = ds1820.read();
    sprintf(roomTempStr, "%3.1f", roomTemp);

    /*$off*/
    httpContent  =
        "<head>"
        "<meta charset=\"utf-8\">"
        "<meta name=\"viewport\" content=\" initial-scale=1.0; maximum-scale=1.0; minimum-scale=1.0; user-scalable=0;\"/>"
        "<title>Smart Home</title>"
        "<link href='http://fonts.googleapis.com/css?family=Droid+Sans&v1' rel='stylesheet' type='text/css'>"
        "<style>"
        ".switch {"
            "position: relative;"
            "display: inline-block;"
            "width: 60px;"
            "height: 34px;"
        "}"
        ".switch input {display:none;}"
        ".slider {"
            "position: absolute;"
            "cursor: pointer;"
            "top: 0;"
            "left: 0;"
            "right: 0;"
            "bottom: 0;"
            "border-radius: 34px;"
            "background-color: #ccc;"
            "-webkit-transition: .4s;"
            "transition: .4s;"
        "}"
        ".slider:before {"
            "position: absolute;"
            "content: \"\";"
            "height: 26px;"
            "width: 26px;"
            "left: 4px;"
            "bottom: 4px;"
            "border-radius: 50%;"
            "background-color: white;"
            "-webkit-transition: .4s;"
            "transition: .4s;"
        "}"
        "input:checked + .slider {"
            "background-color: #8ce196;"
        "}"
        "input:focus + .slider {"
            "box-shadow: 0 0 1px #8ce196;"
        "}"
        "input:checked + .slider:before {"
            "-webkit-transform: translateX(26px);"
            "-ms-transform: translateX(26px);"
            "transform: translateX(26px);"
        "}"
        "</style>"
        "</head>"

        "<body>"
        "<h2><a href=\".\" title=\"Click to refresh the page\">Smart Home</a></h2>"
        "<pre>Temperature:\t" + string(roomTempStr) + "&deg;C</pre>"
        "<pre>Heating:\t";

    if(status == ON) {
        httpContent +=
            "<a href=\"./?sw=0\" class=\"switch\"> "
            "<input type=\"checkbox\" checked>";
    }
    else {
        httpContent +=
            "<a href=\"./?sw=1\" class=\"switch\"> "
            "<input type=\"checkbox\">";
    }

    httpContent +=
        "<div class=\"slider\"></div>"
        "</a>"
        "</pre>"
        "<hr>"
        "<pre>2017 ARMmbed</pre>"
        "</body>";

    return httpContent;
    /*$on*/
}

/**
 * @brief
 * @note
 * @param
 * @retval
 */
void sendHTTP(TCPSocket& client, string& header, string& content)
{
    /*$off*/
    char    content_length[5] = { };

    header +=
        "\r\nContent-Type: text/html\r\n"
        "Content-Length: ";
    sprintf(content_length, "%d", content.length());
    header +=
        string(content_length) + "\r\n"
        "Pragma: no-cache\r\n"
        "Connection: About to close\r\n\r\n";

    string  webpage = header + content;
    client.send((char*)webpage.c_str(), webpage.length());
    printf("HTTP message sent to client.\n\r");
    /*$on*/
}

/**
 * @brief
 * @note
 * @param
 * @retval
 */
int main(void)
{
    printf("Starting.. \r\n\r\n");

    net = new EthernetInterface();

    if (!net) {
        printf("Error! No network inteface found.\n");
        return 0;
    }

    //net->set_network (IP, NETMASK, GATEWAY);  // include this for using static IP address
    nsapi_size_or_error_t   r = net->connect();

    if (r != 0) {
        printf("Error! net->connect() returned: %d\n", r);
        return r;
    }

    // Show the network address
    SocketAddress ip;
    SocketAddress netmask;
    SocketAddress gateway;

    net->get_ip_address(&ip);
    net->get_netmask(&netmask);
    net->get_gateway(&gateway);

    ip.set_port(PORT);

    const char*     ipAddr = ip.get_ip_address();
    const char*     netmaskAddr = netmask.get_ip_address();
    const char*     gatewayAddr = gateway.get_ip_address();

    printf("IP address: %s\r\n", ipAddr ? ipAddr : "None");
    printf("Netmask: %s\r\n", netmaskAddr ? netmaskAddr : "None");
    printf("Gateway: %s\r\n\r\n", gatewayAddr ? gatewayAddr : "None");
    printf("Usage: Type %s/%s/ into your web browser and hit ENTER\r\n\r\n", ipAddr, PASSWORD.c_str());

    /* Open the server on ethernet stack */
    server.open(net);

    /* Bind the HTTP port (TCP 80) to the server */
    server.bind(ip);

    /* Can handle 5 simultaneous connections */
    server.listen(5);

    //listening for http GET request
    while (true) {
        printf("=========================================\r\n");

        nsapi_error_t   error;

        clientSocket = server.accept(&error);
        if (error != 0) {
            printf("Connection failed!\r\n");
            clientSocket->close();
            continue;
        }

        clientSocket->getpeername(&clientAddress);
        printf("Client with IP address %s connected.\r\n\r\n", clientAddress.get_ip_address());
        clientSocket->recv(receiveBuf, 1023);
        printf("Data received:\r\n%s\n\r", receiveBuf);

        string  received(receiveBuf);

        if (received.substr(0, 3) != "GET") {
            httpHeader = HTTP_OK;
            httpContent = "<h1>200 OK</h1>";
            sendHTTP(*clientSocket, httpHeader, httpContent);
            clientSocket->close();
            continue;
        }

        if (received.substr(0, 6) == "GET / ") {
            httpHeader = HTTP_OK;
            httpContent = "<p>Usage: Type http://ip_address/password/ into your web browser and hit ENTER</p>\r\n";
            sendHTTP(*clientSocket, httpHeader, httpContent);
            clientSocket->close();
            continue;
        }

        int cmd = analyseURL(received);

        switch (cmd) {
            case -3:
                // update webpage
                httpHeader = HTTP_OK;
                sendHTTP(*clientSocket, httpHeader, showWebPage(sw));
                break;

            case -2:
                // redirect to the right base url
                httpHeader = MOVED_PERM;
                sendHTTP(*clientSocket, httpHeader, movedPermanently(1));
                break;

            case -1:
                httpHeader = UNAUTHORIZED;
                httpContent = "<h1>401 Unauthorized</h1>";
                sendHTTP(*clientSocket, httpHeader, httpContent);
                break;

            case 0:
                sw = OFF;   // turn the switch off
                httpHeader = HTTP_OK;
                sendHTTP(*clientSocket, httpHeader, showWebPage(sw));
                break;

            case 1:
                sw = ON;    // turn the switch on
                httpHeader = HTTP_OK;
                sendHTTP(*clientSocket, httpHeader, showWebPage(sw));
                break;
        }

        clientSocket->close();
    }
}

sstaub · January 13, 2021, 3:27pm

Interesting, my code is also simple using UDP and TCP, no special thing. I’m using Mbed Studio.
Also mbed-os/drivers/device_key/*gives an error.

hudakz · January 13, 2021, 3:58pm

Could you share the error messages reported by the compiler?

sstaub · January 13, 2021, 4:02pm

This happens with cellular and device key, I made clean build. All tools are latest version.

sstaub · January 13, 2021, 4:27pm

In Output I get following Errors:

[Warning] @0,0: L3912W: Option 'legacyalign' is deprecated.

[Error] @0,0: L6218E: Undefined symbol CellularInterface::get_target_default_instance() (referred from BUILD/NUCLEO_F767ZI/ARMC6/mbed-os/connectivity/netsocket/source/NetworkInterfaceDefaults.o).

Warning: L3912W: Option 'legacyalign' is deprecated.

Error: L6218E: Undefined symbol CellularInterface::get_target_default_instance() (referred from BUILD/NUCLEO_F767ZI/ARMC6/mbed-os/connectivity/netsocket/source/NetworkInterfaceDefaults.o).

Finished: 0 information, 1 warning and 1 error messages.

hudakz · January 13, 2021, 4:56pm

[Error] @0,0: L6218E: Undefined symbol CellularInterface::get_target_default_instance() (referred from BUILD/NUCLEO_F767ZI/ARMC6/mbed-os/connectivity/netsocket/source/NetworkInterfaceDefaults.o).

When you open the mbed-os/connectivity/netsocket/source/NetworkInterfaceDefaults.cpp file are the following lines

#if MBED_CONF_CELLULAR_PRESENT
MBED_WEAK CellularInterface *CellularInterface::get_default_instance()
{
    return get_target_default_instance();
}
#endif // MBED_CONF_CELLULAR_PRESENT

disabled (displayed in different background color than the other lines)?

Also check whether the mbed_config.h file located in the root directory of your project includes a line like:

#define MBED_CONF_CELLULAR_PRESENT                                       1

What is the content of your mbed_app.json file?

sstaub · January 13, 2021, 7:44pm

hudakz:

#if MBED_CONF_CELLULAR_PRESENT
MBED_WEAK CellularInterface *CellularInterface::get_default_instance()
{
    return get_target_default_instance();
}
#endif // MBED_CONF_CELLULAR_PRESENT

This lines are not present (v.6.6) in the file.
There is no file mbed_config.h in the projects root
mbed_app.json contains only printf support for float

{
	"target_overrides": {
		"*": {
			"target.printf_lib": "std"
		}
	}
}

hudakz · January 13, 2021, 8:23pm

It seems you are not using Mbed OS 6.6.0. Check lines 46 to 51 in the repository.
Also check the history indicated at the top on that site. It says the CELLULAR dependency was removed by Jerome Coutant on Oct 23, 2020 (click on the commit link in the history to see the details).

To check which version of Mbed is used in your project open the file

mbed-os/platform/include/platform/mbed_version.h

In case of Mbed OS 6.6.0:

...
/** MBED_MAJOR_VERSION
  * Mbed OS major version
  *
  * @note 99 is default value for development version (master branch)
  */
#define MBED_MAJOR_VERSION 6

/** MBED_MINOR_VERSION
  * Mbed OS minor version
  *
  * @note 99 is default value for development version (master branch)
  */
#define MBED_MINOR_VERSION 6

/** MBED_PATCH_VERSION
  * Mbed OS patch version
  *
  * @note 99 is default value for development version (master branch)
  */
#define MBED_PATCH_VERSION 0
...

sstaub · January 13, 2021, 8:52pm

The version is the right one:

sstaub · January 13, 2021, 9:07pm

OK, i found it, I replaced

mbed-os/connectivity/cellular/*
mbed-os/connectivity/drivers/cellular/*
mbed-os/connectivity/netsocket/source/Cellular*.*

with

connectivity/cellular*
connectivity/drivers/cellular*

now it can build.
I think the problem is when .mbedignore is in the project path the ```mbed-os/*`` path is not necessary.

hudakz · January 13, 2021, 10:01pm

I think the problem is when .mbedignore is in the project path the mbed-os/* path is not necessary.

I don’t use the Mbed Studio yet but rather mbed-cli. However I think the path is necessary. If you remove it the project compiles because those lines won’t be found and hence won’t be ignored.

It’s weird that you don’t have the same NetworkInterfaceDefaults.cpp in your mbed-os as published in the repository.

sstaub · January 13, 2021, 10:28pm

I will go deeper inside tomorrow. Thank you for support.

sstaub · January 14, 2021, 7:37am

OK, it seems when I use the options without ``mbed-os/``` I can compile but nothing changes.

CopperBot · February 1, 2021, 3:53pm

Thank you very much @hudakz for this example !
It reduces the build time of the official mbed blinky from 12min to 1min on my side, using MBED 6.6.

About Netsocket part, there is no librairies folder in mbed-os root folder, did you mean:
mbed-os/connectivity/libraries/mbed-coap/*
mbed-os/connectivity/libraries/ppp/*
instead ?

And I think there is a missing s for test in the line:
mbed-os/platform/test/*

Thanks again, this is the best .mbedignore file I’ve seen so far

Topic		Replies	Views
Mbed compiles and links a lot of unnecessary files Mbed OS mbed_os	10	2027	October 14, 2020
.mbedignore with Mbed Studio Mbed Studio	4	1071	January 8, 2023
Reduce program size on mbed os 6 / lorawan Mbed OS stmicroelectronics , lora	3	654	December 13, 2020
Reducing Size of the release version of the bin file that is generated using MBED CLI Mbed CLI	0	383	October 29, 2020
Mbed build vs clean build Bugs	4	688	February 26, 2021

How to shorten compilation time up to six times

Related topics