High Level Mbed OS testing

Mbed OS Best practice
,
1

Hi everyone,

I am working with Mbed OS to develop high-level applications and high-level libraries. Some of my libraries depends on the Mbed OS Connectivity API. I wanted to implement Test in order to test this libraries easily.

As an example, you can imagine you want to test the AWS client without having an available AWS account and an available AWS cloud (See GitHub - ARMmbed/mbed-os-example-for-aws: Mbed OS example to connect to AWS IoT Core). Could anyone recommend a way?

I can imagine some ways of doing this:

  1. Using greentea testing: you can implement a server mock using host_tests and perform all the tests but this highly depends on the hardware you use and it also depends on your infrastructure (you cannot test a Cellular device in your LAN).
  2. Using Unit tests. For me, it seems to be the best way to test the software. But on the other hand, I really can’t imagine a good solution to mock the mbed OS APIs, for example: you have to mock NetworkInterface, TCPSocket, etc. and it can be a really hard work.

What’s your opinion about this?

Related thread:

2

You test your high level code, not AWS’ code. The libs/sdk they provide are usually very well tested. Coming from them, I would expect they provide you with the mock/stubs/fakes that you’d need.

If not you can either create your own for their code, but it can quickly become a nightmare.

What I’ve found that works great is to add an interface between their code and my code. For example, instead of using mbed::PwmOut, I’ll create a (in our case) leka::interface::PwmOut and create a Google mock of it wherever I need to use it.

We also did the same with all the HAL and LL functions from STMCube.

The interface layer you create only does exactly what you need, nothing more. Then you’ll have to make a concret implementation to run on your target, using the real mbed libs.

For example with the Pwm:

// Leka - LekaOS
// Copyright 2021 APF France handicap
// SPDX-License-Identifier: Apache-2.0

#ifndef _LEKA_OS_DRIVER_LK_CORE_PWM_OUT_H_
#define _LEKA_OS_DRIVER_LK_CORE_PWM_OUT_H_

#include "drivers/PwmOut.h"

#include "interface/drivers/PwmOut.h"

namespace leka {

class CorePwm : public interface::PwmOut
{
  public:
	explicit CorePwm(mbed::PwmOut &pwm) : _pwm {pwm} {};

	auto read() -> float final;
	void write(float value) final;

  private:
	mbed::PwmOut &_pwm;
};

}	// namespace leka

#endif	 //_LEKA_OS_DRIVER_LK_CORE_PWM_OUT_H_

Of course it means your code will use vtables but in most case, especially high level code, it should not make any difference. If you find that the bottle neck is there, you can optimize later down the road when you hit the roadblock.

3

Of course I test my high level code, it was just an example about a kind of library I want to test. I have my own clients and other high-level libraries which I want to test.

If I have understood you, to test high-level code you have like a C++ API above Mbed OS API which you really mock. I think (correct me if I am wrong) it’s similar to mock the Mbed OS API directly.

I think it’s not a bad point, maybe more work than I would like but as you said it can be optimized later.

Anyway, I continue viewing some blocking points such as RTOS API or async operations. How do you mock for example Event Queues or Event Flags if you want to run the unit tests in a Linux/Windows environment? Do you use general C++ libraries or just design the tests to avoid async operations?

Thanks for your response!

4

Yes, an API that perfectly fits my needs that I completely control.

It would be, but most mbed os API don’t provide virtual interface that you can mock with Google Mock. So I take it out of the equation. It also allows me, if needed, to replace the mbed API with my custom low level drivers, for the Pwm example, not use mbed::PwmOut but ST’s LL_PwmOut.

As I control the layer, I can then implement it however I want.

Testing is work. That’s why TDD is so great: you write tests first, so you never have to write tests again :slight_smile:

You don’t, unit tests should not have async features, that’s why you mock stuff, so you don’t have to wait for the database to respond.

Unit tests are for units of code. If a driver needs async stuff, then you’re not testing the driver, you’re testing the driver + the async stuff, so IMHO it’s not unit tests anymore.

You move to functional tests and then greentea can be your friend. But keep in mind that functional tests are harder to run, harder to debug, and cost time and money.

In our project, we made an arbitrary division between drivers and libs:

  • drivers: talk directly to the hardware or use API that talk directly to the hardware with no RTOS except for ThisThread::sleep_for()
  • libs: use drivers and RTOS API to create higher functionalities

Even in libs, the RTOS things can be stubbed away for unit tests and mbed provides a lot of stubs that you can use directly to do that.

I would not do functional tests on the host computer, it doesn’t make any sense. We have a very few functional tests, we manually run them before merging to test key components of the robot.

Every time we write a lib or driver, we also must provide a spike, a very simple program to show if and how it works.

1 Like
5

This is very interesting, it’s true that mbed API does not provide a flexible API in some cases. It’s a good point, I will think about it.

I also agree with you in this, we have functional tests but are harder to run due to the hardware dependency.

Thank you again for your response, it was very useful for me :slightly_smiling_face:

6

For integration/functional/system tests (name it however you want) on the host I want to evaluate Renode, which creates a virtual target for your binary to run on. Seems interesting to test applications with Mbed without hardware. IMHO the ideal way would be have Mbed implement a Posix API and be able to run the apps on Linux, like Zephyr or NuttX do.

For stubbing/mocking Mbed or other libraries, you can either go Ladislas’ way and create an interface layer for a pure C++ framework (IMHO too much effort) or you can use a test framework with support for C-style mocks, such as CppUTest/CppUMock. I did a small PoC with CppUTest and got it working, but I still don’t have it integrated into our development yet, therefore I can’t give a thorough feedback.

7

I found Renode some months ago and I found it very interesting too, but I don’t see too much information about how to integrate with Mbed OS so I can eat too many hours. Anyway, I would be happy to contribute if you want to do a POC.

About mocking Mbed, I would like to implement a solution like yours. I mean, IMHO the more layers you have to maintain, the more work and, in some cases, if you don’t have a high-automated development it can lead to many hours of refactoring and coding.

Anyway, I’ll try to implement a Unit Test system in a medium term, I can keep you up to date. But before that I need to learn the best practices. As I read, Modern C++ Programming with Test-Driven Development is a good option, I will take a look.

8

I looked into it but I did not really like the coding style/documentation. For our project it was too much work compared to having interface and “simply” using google test/mock.

We made the choice of embracing C++ as much as possible and we love coding against interface instead of implementation.

It’s actually less work for unit testing:

  • define an interface
  • create a mock implementation (it’s just copy/paste)
  • start writing your code TDD style

The goal is not to have a lot of layers, it’s to decouple what can change often and what won’t and have a way to deal with that.

Unit testing with mocks is closely related to dependency injections and for that your need interface :slight_smile:

I highly recommend this talk: https://www.youtube.com/watch?v=yVogS4NbL6U

9

That’s true, but it is an important design decision in a big project. However, it is interesting to read different opinions :slightly_smiling_face:

I will take a look!

10

Currently working on Renode, if you are interested you can follow this issue

1 Like
11

I’ll follow that closely!

Quick feedback: your sample code uses the macro BLINKING_RATE but there is no #define – I’m sure you have it in your program but you should update the code so that people won’t lose time saying “you forgot the macro!” :wink:

12

Done! Thanks you for the comment!

13

I have just started a new public repository from my GitLab account to correctly track the PoC. You can see them here. Please, feel free to contribute :slightly_smiling_face:

1 Like