Due to increasing connectivity and dependencies, modern embedded applications in many industries including automotive, aviation, and even automated cow brushes (no joke) are constantly growing more complex. This complexity comes with implications for embedded testing tools and requires plenty of manual effort, depending on the toolchain. From an operational perspective, many embedded industries are tightly staffed and work in long cycles with strict deadlines. In addition, embedded software systems often have hard dependencies on third-party applications and integrations that introduce extra complexity for testing. When things get tight, this can cause time-critical matters to be prioritized over software testing.
Common Issues in C/C++-Based Software
When talking about embedded programming languages, the first one that comes to most developers' minds is usually C/C++, as it’s one of the largest embedded ecosystems, mainly because C/C++ are low-level languages that can access hardware components directly, e.g., microcontrollers and system memory. This low-level control is both C/C++’s strength and its weakness, as this means there is a large class of issues with memory management and resource overflows. From a security perspective, testing an embedded application is vital since the device running the application is often part of a critical system.
Embedded Testing Tools and Approaches
Broadly, there are two main types of embedded testing tools: static analysis and dynamic analysis. Static analysis approaches examine and test source code before it is executed, whereas dynamic analysis approaches examine code while executing it. Both approaches complement each other and are not mutually exclusive.
Static Analysis Approaches and Tools
Static analysis approaches test the source code without executing it, making the process faster and focused on specific units. This is particularly helpful for embedded C/C++ code since the application does not need to be compiled or deployed into production. Typically these embedded systems are more constrained in terms of computational resources (memory, processing, etc) and are not as developer-friendly as a development environment. This makes static analysis approaches particularly helpful for finding issues with embedded software.
Advantages of static analysis include:
- Being fast since there is no compilation step for analyzing code
- Providing compliance with code styles and guidelines
- Finding known code smells and issues by parsing the code itself
- Providing code coverage data by parsing the code
- Allowing for code testing and analysis in the development environment instead of the production environment
Linting
Linting is the process of parsing source code (in C/C++ or other programming languages) to find bugs, incorrect code, suspicious formatting or defects such as unused libraries from include statements. In the case of C++ codebases, linting can find errors when updating compiler versions, such as upgrading a code base from C++03 to C++11.
Linting is highly effective not just for finding issues in code but also for making sure style guidelines are followed which can also help prevent issues with miscommunication on larger software projects. For example, the Motor Industry Software Reliability Association (MISRA), publishes a C programming style guide for best practices with embedded C programs. Linting is one approach for finding violations of such conventions.
Some linters that are designed for C++ are clang-tidy, which is a tool created and maintained by the Clang team that not only finds issues in C++, but can automatically fix them as well. Another great linting tool is the Intellisense Code Linter for C++ developed by the Visual Studio team at Microsoft for Visual C++.
Using Compiler Tools and Flags
An essential approach is to use the compiler directly to find significant errors during development. While compilers can be used simply to produce an executable program, they can also be used to explore a code base from a static analysis perspective. Most mainstream C/C++ compilers will flag lines of code with a warning (the code can compile but there could be an issue) and errors (the code cannot compile and there is an issue). It is possible to make all warnings into errors that require fixing before compilation or to treat specific warnings as errors.
While using compiler tools and flags is very helpful for securing and testing C/C++ code, some options can be confusing or applied incorrectly. For example, compiler tools can be configured to treat only specific warnings as errors. Tooling options may be slightly different between different compilers as well. Using compiler tools may be necessary but not sufficient to secure C/C++ applications.
Major compilers for C/C++ include clang, GCC, G++, and Visual C++. Each of these compilers includes features for treating warnings as errors, ignoring certain warnings, and allowing for strict C/C++ compatibility when compiling code against particular versions of C/C++.
Dynamic Analysis Approaches and Tools
Also called DAST, Dynamic Application Security Testing tests for vulnerabilities in software by executing it using randomized or predefined inputs with tools such as AFL. The goal of the inputs is to trigger a program state in which the program does not behave as it should. Advantages of dynamic analysis are
- Black-box approaches do not require the source code for testing
- Allowing for code execution on both production and development environments for comparisons
- Testing the application with “real” integrations and system dependencies
- Detecting runtime issues
Two common approaches for undertaking dynamic code analysis with C++ are unit testing and fuzz testing.
Unit Testing
Unit testing is a process in which developers write automated tests for a given unit (such as a function or a class) to examine whether the code within the unit works as intended. There are different approaches to using unit testing such as test-driven development where tests are written first, to help guide developers write application code. Another method is using unit tests as regression tests to find a “regression”, or bug introduced into previously working code. Unit testing is particularly useful for C/C++ developers working on large projects since the entire project does not need to be compiled in order to test portions of the codebase.
Unit testing frameworks for C++ include CppUnit and Boost. Visual Studio also comes with C++ testing tools built-in.
Fuzz Testing Approaches
Fuzz testing is another dynamic analysis approach, where invalid or random data (“fuzzy” data) is passed into the application under test and then the result is analyzed. Similar to unit testing the application under test is exercised under defined scenarios. However there is a key difference: Unit testing is deterministic, meaning that unit tests check whether a piece of code provides the expected result. Fuzz testing however is non-deterministic or exploratory, meaning that the expected result of a fuzz test is not known ahead of time. This is helpful since fuzz testing can find bugs and security issues - including critical ones - that developers may not have even thought of.
Simulating dependencies is a crucial part of embedded security testing to mimic real-world usage. Fuzzing can help in various ways; you can fuzz the hardware-dependent functions to get static values or use the fuzz engine dynamically to get the return values of the functions you mocked.
In the first case, you are not likely to increase code coverage as you do not account for all possible behavior. However, in the second instance, feedback-based fuzzing can help simulate the behavior of external sources in a realistic environment and provide better coverage. It is recommended to combine fuzzing with other static and dynamic approaches to get the best result.
Where Are Embedded Testing Tools Heading?
In the past, embedded systems usually were shipped and therefore couldn’t be updated. Now, many embedded systems can be connected to the cloud, meaning that updates and bug fixes can be deployed even to running systems. This means that modern software development approaches such as Continuous Integration and Continuous Delivery can be applied to embedded software. While this new form of software development poses many challenges for embedded testing tools, it also opens some doors, as it enables embedded software teams to integrate automated software tests into their CI/CD for continuous testing.
IoT devices also increase the need for security testing since now there are new security concerns with network connectivity on these systems. Fuzz testing can be a great technique to find and assess security vulnerabilities before shipping a connected device.
In light of the above, a great approach for securing embedded C/C++ applications is to combine the approaches of code linting, using compiler flags, and fuzz testing. One of the best code linters for C/C++ is clang-tidy (open source) and we recommend using CI Fuzz for an easy start to fuzzing C/C++ code.