Integration and Testing

Why Integrate and Test

When building a vehicle, OEMs aren’t mining materials, making paint, raising cattle for leather, and designing their components. Automakers are building a network of suppliers to make each of these components and combining them to make a vehicle. While the suppliers may be designing and evaluating the cybersecurity of their products, it’s impossible to know the exact environment the automaker plans to integrate them into, along with all the products from other suppliers. Think of this process as building a brick house. The suppliers provide the bricks, but only the automaker has the mortar to assemble them. Here, you can find weaknesses in the seams that hold them together

A vehicle-level cybersecurity feature such as secure network communication will require multiple components to mitigate external threats. Suppliers testing the developed component cannot ensure the network is secure. Therefore, it eventually falls on the automakers to evaluate the combined system with a more holistic point of view. In addition, components could be reused from platform to platform with minor variations, which could cause misalignment in the system’s security function. The combination of hardware cost-saving and ever-changing software puts unique pressure on automakers to integrate the technologies safely and securely.

This process of breaking down systems, specifying interfaces, developing components, and then integrating them back together is performed at every level of development. Suppliers of individual ECUs also perform a similar process to break down an ECU into components, going as far as individual software modules and integrating them back together. Therefore, the product varies even as the integration and testing process remains unchanged.

This lesson will discuss the various techniques to integrate and test vehicle components to reduce cybersecurity risks.

Cybersecurity Verification

Testing is generality composed of Verification and Validation and usually follows design and development activities. Other approaches, such as Test-Driven Development, move verification activities to run alongside development.

Verification is focused on ensuring the product meets its intended design. Were the requirements from standards and specifications from the design phase applied to the product and implemented correctly? Verification ensures the product meets the expected design criteria by analyzing and testing against known threats and vulnerabilities.

Cybersecurity Verification is required as a part of ISO/SAE 21434 Clause 9: Clause 10: Product development. Cybersecurity verification aims to provide evidence that the results of the implementation and integration of components conform to cybersecurity specifications. Hence, document the proof that the product meets requirements.

Depending on the phases of development and types of cybersecurity requirements, the objectives of verification and strategies are different. ISO/SAE 21434 request verification for:

• Cybersecurity goals
• Cybersecurity concept
• Cybersecurity design phase
• Integration and testing phase (this phase could be closely coupled with the design phase using Agile methodologies)

During the Concept Phase, verification should be performed by technical reviews to confirm the completeness, correctness, and consistency of the outcome of various concept phase activities (goals, claims, concept/requirements).

During the Design Phase, verification should be performed by reviews of requirements and detailed designs. Note that the Design and Testing phases usually occur cyclically, as testing could reveal design flaws and therefore require an update to the specifications.

During the Integration and Testing phase, verification is usually performed by leveraging the following test methodologies:

• Verification against Cybersecurity Design Specifications
• Cybersecurity functional testing
• Vulnerability scanning
• Fuzz testing
• Penetration testing

Cybersecurity Validation

Cybersecurity Validation focuses on ensuring the vehicle’s design is secure (i.e., Are we building the right product?). This is a difficult task, as we could do our absolute best today to defend against every known attack, but tomorrow someone comes up with a new attack, and the vehicle could no longer be secure. Therefore, Cybersecurity Validation, in practice, ensures that the confidence level is high and that the residual risk is reasonable.

Note that cybersecurity validation requires a holistic understanding of the system to be able to determine whether a risk is at a reasonable level. Suppose a supplier develops a component assuming the product is deeply embedded in the vehicle and not connected to external interfaces. In that case, they may consider the risk of being physically tampered with is low. Therefore, it is the OEM’s responsibility to ensure this is true for the vehicle on which this component is installed. Cybersecurity validation is considered at the vehicle level. However, suppliers may need to provide information to support the activity. Moreover, some suppliers who build entire systems may be held responsible for the cybersecurity assurance of the system.

Per ISO/SAE 21434:2021, validation activities are mainly composed of two tasks:

The first validation step is confirming the adequacy of cybersecurity goals and the validity of cybersecurity claims/requirements within the vehicle system. This part is mainly done through a review of documentation. For example, in the earlier example, the OEM should review and confirm the validity of the supplier’s cybersecurity claim that physical tampering is of negligible risk because of the location and exposure of the component.

The second step is conducting a penetration test to confirm the achievement of cybersecurity goals within the vehicle. Pentesting will also provide confidence on the risk level of weaknesses not addressed by the goals, as good test engineers not only test against cybersecurity goals but will also use attack methods known to the general cybersecurity industry. The effectiveness of validation via penetration tests depends highly on test scope, time and effort spent, and the tester’s domain expertise.

Of course, to demonstrate that the residual risk is reasonable, any vulnerabilities identified by the validation activities should be fixed/managed accordingly.

Test Methods

The following section will focus on the test methods instead of the design review, as reviews are covered in the Concept and Design lesson. Earlier, we learned that both UN R155 and ISO/SAE 21434:2021 requires tests to be performed to demonstrate the effectiveness of cybersecurity mitigations and the acceptable level of residual risks. Let’s recap the types of tests mentioned:

• Cybersecurity Functional Testing
• Vulnerability Scanning
• Fuzz Testing
• Penetration Testing

Documentation of testing activities will be necessary to prove compliance with ISO/SAE 21434:2021 and UN R155 regulation auditing during vehicle type approval.

Let’s take a look at each of these test methods.

Cybersecurity Functional Testing

CS requirements-based testing evaluates the correctness of implemented cybersecurity features, ensuring that the controls operate as intended. To determine whether or not the system is working as it should, teams will need to understand how the security feature should behave when it receives incorrect input. To do this, they should define misuse cases describing how functional and security features may be abused. With these misuse cases in mind, the testing teams can test the system’s functionality with clear parameters and expectations.

For example, Secure Boot feature is supposed to protect software from being tampered with through unofficial modification methods such as abusing incorrect flashing sequences, using unlocked debug ports, or leveraging existing unofficial programming/calibration tools with weak protection. So, in this case, we have already listed a few misuse cases for normal functions to test with. In addition, ways to abuse secure boot include power glitching, bypassing secure boot through external bootloader configurations, or modifying code/data early in the boot process. In general, when testing security features, it is critical to experiment with all kinds of non-standard and unexpected operational parameters, the type that someone trying to hack the system would try to use to get the target to behave not according to spec.

Cybersecurity requirements-based testing is a relatively immature area of testing for the automotive industry. A good approach is to have the CS engineers who authored the target requirements collaborate with the current test teams. This is the typical approach most companies have taken to address the lack of cybersecurity knowledge among test engineers. Note that the engineer who develops the product is not recommended to perform verification activities, as per ISO/SAE 21434 Annex E.

An often overlooked part of cybersecurity testing is the verification activities already being performed as a regular part of quality control for potentially safety-critical systems such as a vehicle. Typical product functional testing already ensures the product will behave as specified. If done correctly, good functional testing should minimize unknown behavior, which may compromise Functional Safety and Cybersecurity. Other tests for reliability/robustness are also required to demonstrate the sufficient capability of hardware to support cybersecurity features and high-quality code. These tests include resource usage evaluation, interface tests, control and data flow verification, dynamic analysis, and static analysis. These test methods are commonly found as a part of software integration tests required by ASPICE and ISO 26262 for Functional Safety.

In summary, the automotive industry faces many challenges in cybersecurity requirements-based testing. Although some automakers have comprehensive cybersecurity requirements, the methods for testing their implementation are not clearly defined or understood. Traditional functional testing teams may lack the necessary cybersecurity expertise to understand the test goal, so they usually do not test enough against unexpected inputs and operational parameters. The industry often relies on penetration testing to catch any problems and meet compliance standards, but this approach is ineffective. Penetration testing requires much more specialized knowledge and can be very time-consuming. Additionally, it is more expensive and takes more time to have penetration testers identify failures to implement requirements correctly than for internal test teams, as external testers are unfamiliar with the product and associated requirements.

Static and Dynamic Code Analysis

One of the verification methods discussed in the previous section was static and dynamic code analysis. It is usually performed as a part of software integration testing and can be used as a quality assurance method to minimize bugs and potential weaknesses in software. Static code analysis checks source or compiled code for compliance with coding standards and potential security vulnerabilities. In contrast, dynamic code analysis runs the code and monitors its behavior, detecting runtime errors and measuring performance.

Commercial static analysis tools in the automotive industry usually cover commonly used embedded coding standards such as MISRA-C and CERT C. Some popular FREE static code analysis tools include SonarQube, FindBugs, and Pylint.

On the other hand, dynamic code analysis involves running the code and monitoring its behavior while it executes. This can include tracking the code’s memory usage, performance, and input/output. This type of analysis is often used to identify runtime errors, such as memory leaks and race conditions, and to measure performance. Some popular dynamic code analysis tools include Valgrind, GDB, and JProfiler.

Because these scans typically require one to have the source code, placing this responsibility on suppliers is recommended. Automakers should review the results with suppliers to plan for appropriate remediation and fixes during the later stages of the software development process when the software becomes available for testing. Many scan tools can be integrated into IDEs to be run continuously. Still, it should be done at least periodically on every major release to allow awareness and tracking of issues that need to be fixed.

Vulnerability Scanning

Vulnerability scanning is performed by running source or binary code through analysis tools to compile a list of known software. Then, this list is checked against the CVE database to identify if there are known vulnerabilities in the code. These vulnerability scanners work very similarly to a traditional virus scan, where a list of virus definitions (hashes of the code) are stored in a database and compared against the files on a target system. More sophisticated scanners look for patterns of code behavior to identify the software and can be used to detect even snippets of code copy-pasted from the internet.

Vulnerability scanning is crucial in ensuring the security of connected and autonomous vehicles, given many open-source libraries in central processors and connected components. This starkly contrasts most deeply embedded ECUs, which usually run on proprietary software with almost no open-source components. In the latter case, vulnerability scans usually return no findings.

In conjunction with other testing techniques, the scan report can be used to comply with the verification requirement from ISO/SAE 21434 to confirm that unknown vulnerabilities are minimized. Given the list of identified vulnerabilities in the code, developers can patch them and reduce the risk of these vulnerabilities being exploited for an attack. It is essential to review and triage the scan findings to ensure they are addressed reasonably.

As mentioned earlier in the course, vulnerability management is a continuous activity. Along with monitoring for cybersecurity incidents, vulnerability scanning should be performed regularly after the product enters production and throughout its useful lifecycle. This activity is crucial to stay updated with the latest vulnerabilities and to ensure the system remains secure. The results of vulnerability scans can change over time as new vulnerabilities are documented, and development continues. The growth of known vulnerabilities in the CVE database over time illustrates this point.

There are unique challenges presented in the automotive industry due to the integration of many custom systems, and vulnerability scanning cannot be relied upon to give a perfectly reliable summary of all vulnerabilities. However, this testing technique will save time by being able to run these scans as the design evolves consistently. As new vulnerabilities are discovered, and the vulnerability databases are updated, these previously undiscovered weaknesses will be revealed.

Fuzz Testing

Fuzz testing or fuzzing is a method of testing software where large amounts of invalid, malformed, or random data inputs are sent to the target under test to see how it responds to unexpected data. This can be useful in discovering bugs or other vulnerabilities. Fuzz testing will attempt to get a device or software to crash, stall, or behave unexpectedly. Analysts can perform root cause analysis of specific fuzz testing crashes or misbehaviors in security research and can discover weaknesses that might be exploitable.

Fuzz testing could be performed in a software environment against source code. Still, at the automaker, this will likely involve developing hardware connected to a testing computer over attack surfaces such as CAN, Ethernet, Bluetooth, and WiFi. Fuzz tools will need to be configured to communicate with the target. Commercial fuzz tools will include test suites for various attack surfaces. Custom fuzz test suites can be designed. The simplest would be randomly generated inputs. More complex fuzz test suites might consider the protocol in more depth and be more targeted in its data modifications.

Once it runs, the fuzz test suite is mainly automated, going through thousands or millions of test messages. Some manual intervention could be necessary to keep the hardware running, or self-automated development can do this for you. For example, if fuzz testing an ECU over CAN, the module might crash and not reset. In such a case, a hard power reset might be necessary to restore the device under test. The amount of fuzz test cases you want to run can be specific to the target and the attack surface you are testing. For example, if testing on classic CAN, you might consider 100,000 to 200,000 an adequate amount of test cases. A security expert can help decide the appropriate volume of test cases.

Penetration Testing at the OEM

Penetration testing is a more open-ended and creative testing method where highly motivated and skilled testers will play the role of malicious actors to test if the system has any unknown weaknesses and if those weaknesses can be exploited. TARA is a good input for penetration testing, and a pentest can help validate if the assumptions made in the TARA were accurate. Still, penetration testing will go beyond just testing hypotheses about threats in TARA.

The industry leans most heavily on penetration testing, sometimes as the single testing method, but all four methods should be considered and employed in many cases. If other testing methods are skipped, the pentest results will not address those gaps efficiently. The creative efforts of a penetration testing team will be more effectively spent trying to identify novel vulnerabilities rather than low-hanging fruit that could be discovered more quickly through any of the other, quicker methods. 

A pentest on a product that has not undergone functional testing might waste time conducting testing on security features that are not present or configured correctly in the target under test.

The responsibility of penetration testing is often pushed toward suppliers with a limited understanding of how their system will be implemented in the vehicle.

OEMs should focus on large-scope penetration tests for vehicle systems and full vehicles, whereas suppliers should be focused on component and feature penetration testing.

It makes the most sense to conduct penetration testing on a more extensive scope to be closer to the environment real adversaries will be working in once the product is released. 

Successful penetration testing teams need advanced skills. A team with broad knowledge and depth in various domains will be successful at identifying potential weaknesses and attempting to exploit them for different attack surfaces. Detailed knowledge of how multiple protocols work, understanding how the applications work, and how the individual components operate as a system. Typical backgrounds for penetration testers are computer engineering, computer science, electrical engineering, and software development. Sometimes they are self-taught security researchers. Various certifications outside of academic degrees, independent research or contributions, and publication of open-source code can be good credentials to evaluate the skills of penetration testing teams.

• Communication testing
• CAN
• Ethernet
• Wi-Fi
• Cellular
• GPS
• Bluetooth
• Bluetooth low energy
• Testing of radio frequency communications in specific frequency bands
• TPMS
• Key Fobs
• Cellular: 2G, 3G, 4G LTE, 5G
• Hardware reverse engineering
• Software reverse engineering

Summary

Verification and Validation require many different types of testing. They can be used in tandem to identify weaknesses and vulnerabilities to help engineers develop more secure products.

Key Types of Testing:

• Functional testing confirms the systems work as intended. If the system does not perform as designed, other testing methods may be invalidated if the system needs to be redesigned.
• Vulnerability scanning scans packages installed on a system and compares them to a database of vulnerabilities to determine whether the system is susceptible to known attacks.
• Code scanning comes in two types, Static and Dynamic. Static code analysis is the process of examining code to look for potential weaknesses or deficiencies. Dynamic code analysis is performed on running code to look for problems such as memory usage, performance problems, or crashes.
• Fuzz testing injects random, invalid, or malformed data to evaluate a program’s ability to handle errors and unexpected conditions.
• Security experts conduct penetration testing with many techniques to break into the target. Pentesting is typically used to perform a final validation against cybersecurity goals and catch anything the design team might have missed.