Interesting topics about electronics, product development and more.

Testing firmware and software

by Timo Stehmann20. November 2014 18:42

This is a quick blog about testing your software. How to test your firmware and software remains an important topic. There are many ways to do this and often people can be very religious about the approach to take. I have seen some of these approaches, including unit testing. My opinion on this matter is: It depends. Use testing methods based on their merit!


Unit testing

I, personally, am not very fond of unit testing. It is a very heavy-weight approach. Writing tests before you write your code is not very practical in most cases. Be realistic! Unit tests can also increase viscosity for refactoring code, because now you have even more code to refactor. I understand the principle and the fact that unit tests are a safety net. It all sounds very logic until you start refactoring and changing APIs. What? You shouldn't dramatically change APIs? Tell this to the customer or the developer that has to work with a bad design. Unit tests are closely linked to this API even if you try your best to decouple things. Refactoring will end up in fixing the business end of the code and the unit tests. This will introduce bugs on both ends.

Unit tests do have their place. In complex systems I prefer limiting their use to utilities. Those parts of the code with algorithms, which can be easily abstracted. These are also the examples you often see presented in conjunction with unit tests. I am still waiting to see examples of unit tests being shown in complex UIs or software which needs to communicate with USB devices, etc.


Black box testing

This approach is useful for several reasons. It provides the possibility for a separate team of testers to write tests that interact with the UI. The main prerequisite is that the UI must provide the capability to interact with an external test script. Luckily, most UI frameworks have support for this. A separate team also provides additional benefits: objectivity and separation of concerns. The price for this is additional salaries, but at the end of the day you have a test suite with minimal impact on the development team.

It is a problem, however, to get adequate code coverage by this type of tests and this would require additional tools to measure code coverage by the test scripts. It is all about balance and the combination of unit tests on low-level APIs with black box testing can be a very powerful approach.


Self-checking code

Self-checking code is a must! It is really simple to implement and in the C/C++ world this boils down to adding asserts to your code. Some product managers will never agree, but it is important to keep asserts even in released code. Asserts test the programmer's assumptions. If you assume "a=b" and this not the case, how can the code continue? In my opinion, to simply continue and try to ignore the problem is a bad approach. The program has already been compromised. How can any decision after and assert be a valid one?

Rather implement a process to deal with asserts in releases code in an efficient way. Believe it or not: if a customer reports an assert and you can provide a fix for the problem within 5 days (even better, the same day), your rating as a good vendor will increase dramatically. So much so, I almost advise to build in asserts you can remove when it is triggered by a customer. Only joking, but a personal and fast response is key to success. Fast response times will provide you access to more customers! However, as already stated, you will need the right process to deal with this. I am not a supporter of Kanban when it comes to the normal development cycles, but Kanban is a powerful tool when it comes to increase responsiveness to bug reports.



When it comes to writing software for interfacing with external hardware (e.g. USB devices) all of the above approaches are difficult to implement. In order to test the PC software every tester and developer needs to have access to a hardware device. To make things worse, there is the never ending finger-pointing between software and firmware developers. For me it is easy, I am both, but finding the cause of a bug can still be a huge task.

Emulating your hardware device is one approach to solve the problem. This is implemented on the software side. By implication your software requires enough abstraction to "plug in" virtual devices. This should not have a big impact on your design, assuming you have a software architecture. The impact on the development should be low. In one project I supervised the software development time was 53 hours, the firmware development time was 34 hours and adding an emulator framework took 8 hours. This is way less than using unit tests. After adding an emulator framework it was possible to simulate external connected USB devices. Whole systems could be simulated. In addition it was possible to double-check the implementation of the the firmware implementation. Unfortunately, when implementing emulators, parts of the firmware needs to be duplicated in the software environment. It would be nice to emulate the firmware (often written in C) on a VM, which interfaces with  application software. However, this requires a huge implementation effort.

In the end you have decide which approach suites you best. It is better to have some form of testing than no testing!



Permalink | Comments (0)

Add comment

  Country flag

  • Comment
  • Preview