Unit tests

[toxygen]

Hi,
I wonder if question of unit tests ever arose during development. While going through the forum recently, there seems to be a lot of small bugs/typos in the code that make the code work 99% of the time but some corner cases are not always ok.

I reckon that making unit tests is not the most interesting part of coding but I was thinking that maybe ChibiOS could try to apply as a mentor to either GSoC or ESA's SOCIS (Summer of Code in Space).

What are your thoughts about this?

[Giovanni]

Hi,

The ChibiOS part that is formally tested is the kernel (the RTOS): unit tests, code coverage, static analysis, performance, size.

HAL and device drivers are maintained "best effort" and are not part of the commercial offering. There is a small test application for each driver on each device family but it is not supposed to be an exhaustive test. The problem is that testing device driver is an hard problem, the idea is that device drivers converge to zero defects with use and continuous bug fixing.

I am planning to introduce HAL testing but limited to the high level using stubs as low level drivers.

Consider that even in the current scenario the HAL and device drivers take like 90% of my development time (and it does not directly generate revenue). Any idea to improve the situation is welcome.

Giovanni

[steved]

(and it does not directly generate revenue).
Giovanni

As you say, not directly. But I think the HAL is a key reason for people to use Chibios in the first place. Certainly when I looked at options for an RTOS Chibios had the best HAL support over the type of chips I was interested in.
(And maybe you should ask ST for some commission - I for one ended up using ST chips thanks to the excellent support in ChibiOS)

[Giovanni]

This why I specified "directly" and the reason why I put like 90% of my bandwidth on its development ChibiOS/RT is an excellent kernel in itself but probably it would have gone unnoticed without the HAL.

About the defects, mainly this happens in the low level drivers (almost never in the RT or high level HAL), my analysis is that it is caused by the continuous reworks dictated by the introduction of newer STM32 sub-families and contributed parts.

Going forward, the community repository will be a stage for new code, I am thinking to contribute myself, a necessary step before allowing new code in the main repository which should always represent a reliable core of well tested functionalities.

The version 3.0 is also an attempt at stabilization, the clean-up of certain internal interfaces should guarantee less regressions in the existing code when new platforms are added. Layers are much better defined and this should greatly help isolating the new code from the stabilized one.

I for one ended up using ST chips thanks to the excellent support in ChibiOS

I know they don't care. I am supporting STM32 because:
1) Discovery kits.
2) I developed experience on it.
3) I need a preferential platform for HAL development.
4) It is "generic" enough, I saw much more weirdness in other families.
5) We built momentum on it.

[toxygen]

Ok, are there no plans for more rigid testing? I don't mean for Giovanni to do it, but preferably have someone do it within GSoC or SOCIS, where some bright students could contribute this (I think) important thing and get paid at the same time.

What I mean is having unit or complex tests for low-level drivers. Either in terms of SIL simulation (Labview someone?) or maybe creating testing image of ChibiOS for some reference BSP. Idea is to create test scenarios including extreme cases for stuff like I2C, CAN, SPI, SDIO, ... not only some bugs would be caught during development of this test image, but any new port could be tested against this reference image.

[Giovanni]

Testing drivers is hard and probably also not cheap because boards would have to be instrumented.

Anyway, lets assume I am interested, how would we have to proceed? what would be required? do you have references?

Giovanni

[steved]

Testing drivers is hard and probably also not cheap because boards would have to be instrumented.
Giovanni

If 'standard' boards aren't available with the necessary hardware configurations, possibly some of the community members have something they could provide. Certainly I've got boards with ST devices which might be useful, primarily with serial ports and I2C implemented. It might not be difficult to tweak a design to add other devices. I'd be happy to contribute hardware to this kind of exercise.

[toxygen]

I have experience from SOCIS.
The process is pretty straightforward. You just fill out some forms and setup a wiki page with proposal and detailed description.
Then you need to provide a mentor - you have a choice, it might be you or someone from community.

My suggestion would be following:

1) Create wiki page for idea proposals.
2) Come up with more ideas. Rigid testing being only one of them.
3) Let members of community throw in some ideas as well.
4) Decide which ideas are most desirable and invest some time in providing detailed description what would need to be done.
5) Fill out the forms.

that's it.

Regarding the space stuff in connection with ChibiOS, there is no doubt there will be cubesats utilizing ChibiOS as opposed to RTEMS and others.
I.e. Lunarsail, AMSAT FOX-1 and others...

As a sidenote, the rigid driver testing might be a big incentive for space use of ChibiOS.

Regarding the GSoC, I don't have any experience with that, but I would expect the process to be similar, alas it is much more competitive, so chances are slimmer.