[MBed CLI Alternative] mbed-cmake

Mbed OS
1

At USC Rocket Propulsion Lab, we wanted to take MBed OS to space. However, to do that, we needed a more feature-rich build system, one which could reliably handle a variety of different build targets and options. So, we wrote one! Based on the build system used on our Traveler 4 rocket, mbed-cmake is a build system template that lets you use the descriptive power of CMake to create build systems that use MBed OS.

MBed-cmake was designed to optimize the common case of embedded development: where a program and an environment are configured once for a specific processor, and then left in that configuration forever. Normally, MBed OS’s build system forces you to reconfigure things like the target, the toolchain, and the ignore files each time a project is set up, as MBed OS and all of its dependencies are downloaded by the build system on your computer at build time. In contrast, MBed-cmake allows you to check the entire OS and configuration into version control. This makes builds easy, repeatable, and dependable as there are no external dependencies.

Ordinarily, MBed OS can’t be built easily using CMake. The source files to build vary by processor and are generated automatically according to a complex set of rules, and additional flags and compile definitions are scattered in JSON files all around MBed OS’s source tree.

A stopgap solution is to use MBed’s “export” functionality to create a Makefile or CMake build system and then develop that into a full build system. However, this has major disadvantages: MBed itself is exported as a compiled lib, so you can’t debug it or view its source code as easily. Even worse, you’re stuck using a generated build system, so if you need to switch to a different processor or change MBed options you’ll have to re-export a new build system and make all of your modifications again.

MBed-cmake fixes this properly. We’ve created an automatic scraper script that runs MBed OS’s Python build scripts once to collect all of the source files and flags needed for a specific processor. It then writes these into config files which are read by the CMake build system. CMake can then build MBed OS (and your applications that use it) using this information. This lets you build your MBed application easily using the full power of CMake scripts.

Features

  • Completely expandable: You can add any and all custom build options, commands, flags, and configurations in your CMake scripts, and generated source files can be easily added using a variety of first- and third-party modules.
  • Multiple targets: You can create multiple main programs in a single project, and selecting them is as easy as make <program>.
  • Custom flash logic: MBed-cmake includes an automatic code uploader similar to MBed CLI. However, since the logic is in CMake scripts inside the project, you can easily change it or add new upload methods to suit your project.
  • IDE support: MBed-cmake lets you use any and all IDEs supported by CMake to develop your project. This provides improved code-assist functionality and many other conveniences (and prevents wars between people with different IDE preferences).
  • Automatic debugging: Certain upload methods also provide debugging capability. MBed-cmake provides convenient build system targets to automatically start a GDB server and run GDB for a specific program. This functionality gets even more powerful when paired with an IDE that provides a graphical debugger.
  • Easy configuration switching: MBed-cmake maps MBed’s debug, develop, and release configurations to CMake’s Debug, RelWithDebInfo, and Release configurations respectively. Switching is as easy as a single CMake flag or menu option.
  • Use standard MBed OS configurations: MBed OS is configured exactly like normal, using mbed_app.json and .mbedignore files. CMake will load the final MBed OS data after these files have been processed.
  • Completely reliable: All code for your target is stored under a single Git repository, so there’s no danger of code in external repositories failing to download, being modified, or causing problems.
  • Fewer dependencies: MBed OS’s build system has a zillion Python dependencies that can be difficult to install correctly. Though you will need to install these for the initial configuration, they aren’t needed after that point, so you only need to install them on one system.

Instructions and Download

See our GitHub repository here.

Questions or Bugs

Post in this thread, or file an issue on our repo.

4 Likes
2

this is amazing! I’ve wanted to do the same for months now but never got the time! I’ll try it out asap! :slight_smile:

3

There is a lot of action in Mbed about directory restructuring and adding cmake. Have you checked the feature-cmake branch? Looks like some double work, or are you involved in the feature-cmake?
Is Mbed going to switch to cmake?

4

No, we haven’t heard of this branch before. Hmm, from a quick look it looks like they’re doing something somewhat similar (relying on config files generated from some other sort of script). Well, if it ever gets a stable release, it might be worth using here.

5

Hi @MultipleMonomials,

thanks for sharing your work ! Glad to see this. I’ll checkout mbed-cmake in more detail and I will get back to you. I would like to understand how we can help each other, to make updated mbed tools with CMake working for everyone (including your project).
Any feedback/contributions welcome (we are in early stages, a review could influence the next stages) ! You can just reply here or find my email via github.

As mentioned earlier, we are working on updating tools around CMake. You can checkout feature-cmake branch (rebased almost daily, worth fetching often). We got couple of examples building, but the recent changes broke some so only blinky is currently in green state (it will be fixed in a day or so). All examples should have the same named feature-cmake branch.

We will work on updating https://github.com/ARMmbed/mbed-os/blob/feature-cmake/docs/design-documents/tools/cmake.md this week. I’ll do some edits tomorrow to bring this up to date.

3 Likes
6

@Kojto I’m really happy to see this coming! I’ll give it a try as well when the other examples are fixed.

1 Like
7

I have to say, looking over that repository, I have some concerns about how the cmake feature seems to be structured. It looks like you’re taking pains to get rid of the historical system of recursively finding source files, and replace that with encoding source files directly in CMakeLists. This seems like it’s probably a good change to make in isolation, but it also seems like it’s a ton of work, and it’s one of the reasons why your project seems like it’s taking a lot of work to create and keep compatible with all targets. And mbed-cmake is able to get functionally the same result as yours (editing and building mbed-os and libraries) with a lot less effort by just integrating with the existing Python build system.
Also, looking over the top level script I didn’t see anything that would set the toolchain and compiler flags, so I’m assuming that you still need to use an external script/program to generate the toolchain file (and compiler flags?). If you use a script for that, then why not use my script for the source files? (I came up with a cool trick to avoid having to pass a toolchain file in mbed-cmake, I can give you some tips on toolchain file hacking if you want).
And last but not least, unless you’re planning on parsing json in CMake (and risk invoking ZA̡͊͠͝LGΌ), then this build system seems like it would not be compatible with the existing mbedignore and json configuration system for mbed and its libraries.
Now before I say this, I want to make it clear that I really love using MBed OS. The C++ API is really a pleasure to use and it’s a very clean and convenient wrapper over all of the more complex underlying C code. However, MBed development has had a really bad track record with build systems, and it seems like this is going to make it worse. First you had the online compiler only, then you had Yotta, then you introduced CLI with its nest of custom Python, and now lots of people still use the online compiler but there’s certain things you can’t do with it like mbedignores and mbed_app.json, so people who need those have to use CLI… it’s already kind of a mess. And this seems like the cmake feature would break compatibility with nearly all existing libraries and programs, again. As someone who has released a couple libraries, I can say that developers will only deal with so much of this kind of thing before they start becoming really suspicious of new things, or give up entirely. If you change the build system to something incompatible again, or even just fragment the market by introducing another official option, then you guys risk losing a lot of users.

8

I don’t know the strategy, but I guess using the fixed CMakeLists is neccessary to increase speed. The python scanning for source files takes a long time already. I’m using a different project structure where mbed-os, libs, custom_targets and apps are on the same level. so in app/program1 I add all directories as --source …/…/xxx. And scanning my growing libs takes more and more time. So I’m also excited to see the results of using cmake.

9

I don’t know the strategy, but I guess using the fixed CMakeLists is neccessary to increase speed. The python scanning for source files takes a long time already.

Yeah, but with mbed-cmake you only need to do this once when you initially configure the project, and never again unless you change targets or mbed configurations (the configs get checked into version control). So it shouldn’t be that noticeable over time!

10

ok, that should work also. But when I check out mbed-os, the makelists are known for mbed-os and could be fix.
I would like some configuration system where I can add or remove features to shorten build time. Would this be possible with the new cmake? I know .mbedignore, but this a teddious way because of the unknown dependencies. And changing the .mbedignore always causes a build all.

11

Great analysis and work.

I have always assumed that the new build system was preparing for Mbed Studio to replace the online IDE. But things are so secretive within ARM that I’ve never been sure about my guesses for the Mbed project.

The other MAJOR weakness that needs addressed however is the existing tool’s inability to properly manage dependencies and packages. I can not fault ARM forn this weakness as its a problem that should have been solved by the C/C++ standards community years ago.

The new tools are supposed to somewhat address this problem.

I have seen a couple of posts now similar this where a community member does great work but in a slightly different direction than ARM is moving.

Though Mbed development is all mostly done in a way that is accessible, the direction the project moves seems to vary with the wind. All stakeholders are inadequately represented.

@andypowers @Kojto @MarceloSalazar @AnnaBridge @JoeA @donatien @willlordarm @sam_taylor_arm

I’d like to call for a more transparent Mbed steering process. One that will benefit developers, integrators, architects, silicon vendors, and ARM.

Also for those no longer with us… @sam_grove, @mbedAustin, #SimonFord

12

I would like some configuration system where I can add or remove features to shorten build time. Would this be possible with the new cmake? I know .mbedignore, but this a teddious way because of the unknown dependencies. And changing the .mbedignore always causes a build all.

With mbed-cmake you still have to use mbedignores, however I took steps to make this less painful. I created and submitted a patch to allow un-ignore rules, and mbed-cmake comes with the patch applied by default. We also ship a default mbedignore that turns off all optional features, keeping build time down.

Also, as a fun extra, enjoy (?) my MBed 6.1 Ethernet support mbedignore (but nothing extra like USB, wireless, or cellular):

# Ignore all extra features (cellular, encryption, storage) by default
components/*
features/*

# disable USB
drivers/source/usb/*

# Unignore block device library since it's a common utility (and is needed for USB)
!features/storage/blockdevice/*

# enable encryption (needed for networking)
!features/mbedtls/*
!components/TARGET_PSA/*
!features/storage/*
features/storage/kvstore/securestore/*
!components/storage/blockdevice/COMPONENT_FLASHIAP/*

# enable mbed-trace (needed for networking... even though it does nothing by default)
!features/frameworks/mbed-trace/*

# enable mbed randlib (needed by lwip)
!features/frameworks/mbed-client-randlib/*
!features/nanostack/nanostack-hal-mbed-cmsis-rtos/arm_hal_random.c

# enable wired networking libraries
!features/frameworks/nanostack-libservice/*
!features/netsocket/*
features/netsocket/Cellular*

# uncomment to enable nanostack library
#!features/frameworks/mbed-coap/*
#!features/nanostack/*

# enable lwip
!features/lwipstack/*

You know, it might be neat to keep a repository of all these mbedignores somewhere. Though it would be even better if the mbed directory structure could be shifted around a bit to make them easier to figure out. For example, why why does MBed TLS depend on blockdevice/COMPONENT_FLASHIAP?

2 Likes
13

Regarding this… my feeling has always been that any sort of “online” dependency manager / package manager is not a good fit for embedded systems in general. On embedded devices the cost of introducing bugs by upgrading dependencies is much higher, so you generally want to carefully think through when to upgrade libraries. Also, you generally want to understand the libraries that you are adding to your project, so it’s always good to give them at least a cursory read-through before using them. Finally, you need to make sure that your project will be able to build just the same way years in the future, without possibility of it not working.
Because of this, I’ve always been down for just copying dependency libraries directly into my project sources, or maybe, at most, using a submodule pointing to a specific commit of the library. This is slightly more work to set up, but is much more reliable over time. So, I’m actually 100% fine with the existing MBed library repository system as it is, even if you end up having to manually add and update dependencies a lot of the time.
This is completely my opinion though! I can definitely understand if you call me old-style. I prefer “reliability focused” though – working on rockets tends to make you paranoid.

2 Likes
14

thanks. I have not worked with cmake yet, I want to wait until some new release is using it. Its already hard to follow all the API changes and I’m happy that I have a reliable way now for building the projects :slight_smile:
Before, I used also some exporters, but it is a nightmare to keep them up to date. The mbed-cli without changes is much more easy to use, and integration to VSCode is simple with its tasks.json.

15

Goal for me with dependency and package managment is 100% reproducability, fat finger prevention, and very fast onboarding of new coders and or employees.

Not looking for NPM like ease of adding broken packages.

Because of Mbed’s test infrastructure I would not be hopping on to the latest of anything until it passes all automated tests. (Rocket included)

16

@MultipleMonomials I’ve spent yesterday setting up mbed-cmake with my project and I really like it so far. Had to tweak it a little bit to make it work.

Are you open to questions, suggestions, improvements, etc., in your project via issues or PR? Do you have a public roadmap of where you want to go?

17

Glad to hear! For sure, feel free to submit issues on the repository. And I’m interested to hear any suggestions you have!

At the moment, mbed-cmake is basically in maintenance mode – We’re just trying to maintain it as an alternative build system for projects that aren’t a good fit for the MBed CLI build system. We’re not planning to submit it as a PR to mbed-os, since it’s not (and isn’t trying to be) a build system that works for all mbed os users.

The only large feature I’m thinking about adding is support for IAR and/or Arm Compiler, but I’d have to figure out how to get them installed first. Does anyone know how I can get a license? I have a university student email if that helps.

18

So I got access to a machine with Arm Compiler on it, and I’m happy to report that mbed-cmake now supports Arm Compiler 6 in addition to GCC Arm! Just pass the -t ARMC6 argument to configure_for_target.py to select Arm Compiler. I’ve tested it and it seems to be working fine for a couple different programs, though it isn’t as well tested with mbed-cmake as GCC Arm. Also note that mbedignore files for GCC Arm don’t necessarily work with Arm Compiler… that was a fun issue to figure out.

1 Like
19

We just started using this on MIT Rocket Team and it’s really incredible! Especially helps with IDE (CLion) integration.

20

So I had a chance to speak with Martin at ARM yesterday, and I’m happy to report that I have some updates regarding the state of mbed-cmake and Mbed’s own move to CMake.

Many of you have probably heard that ARM is working on its own first-party CMake build system for Mbed OS, and that at some point it will become the new official build system. While ARM’s build system isn’t quite ready yet, clearly this is duplicating the functionality provided by mbed-cmake, and one wonders if there is a good reason to keep both projects around in the long term.

Well, the good news I have today is that the ARM CMake build system will provide functionality extremely similar to mbed-cmake’s current feature set. While buildfiles will be auto-generated for existing apps by the Python mbed-tools program, the build system also allows you to create your own top-level buildscript to handle building your own code in the way you need. What this means is that switching from mbed-cmake to ARM’s build system, once it’s stable, will be extremely easy – you just need to copy some files around and replace the boilerplate code in the top-level CMakeLists.txt with different boilerplate.

Currently, besides the limited set of supported processors, ARM’s build system is missing two key features: .mbedignore support (or another way to prevent unneeded parts of the OS from building), and automatic uploading (and debugging!) of code. In my opinion, these omissions make ARM’s CMake build system less usable than mbed-cmake at present – however this will change! To address the first issue, Martin explained that the long term plan is to attempt to split Mbed OS up into several smaller libraries (e.g. core functionality, USB, RTOS, Ethernet, etc.) which you can choose to link with or not in your code. This actually sounds way better than the mbedignore system as you can remove specific parts of the OS without worrying about accidentally breaking dependencies. And for the uploading, ARM agrees that it would be useful and is actually considering adopting mbed-cmake’s UploadMethods.cmake script to provide this feature.

So what does this mean for mbed-cmake? Well the plan is that once ARM’s CMake system is feature complete and matches mbed-cmake’s functionality (which is likely to be some time 6-12 months from now), I’ll stop actively maintaining mbed-cmake and encourage users to switch to the official system. But as I said, this should be an extremely easy migration and likely all you’ll need to do is change a few lines in the top-level CMakeLists.txt. And mbed-cmake will keep working fine for people who want to use it, it just won’t work for newer Mbed OS releases once the old build system is eventually removed. So, bottom line: you should definitely switch to ARM’s build system once it comes out, but there’s no reason to wait – you can start using CMake right now and switch over easily later!

4 Likes