Increasing Efficiency of ISO 26262 Verification and Validation
by Combining Fault Injection and Mutation Testing with Model
based Development
Rakesh Rana1, Miroslaw Staron1, Christian Berger1, Jörgen Hansson2,
Martin Nilsson3 and Fredrik Törner3
1
Computer Science & Engineering, University of Gothenburg, Gothenburg, Sweden
2
Computer Science & Engineering, Chalmers University of Technology, Gothenburg, Sweden
3
Volvo Car Corporation, Gothenburg, Sweden
Keywords: Fault Injection, Mutation Testing, ISO 26262, Simulink, Model based Development, Automotive Domain,
Safety Critical Software.
Abstract: The rapid growth of software intensive active safety functions in modern cars resulted in adoption of new
safety development standards like ISO 26262 by the automotive industry. Hazard analysis, safety
assessment and adequate verification and validation methods for software and car electronics require effort
but in the long run save lives. We argue that in the face of complex software development set-up with
distributed functionality, Model-Based Development (MBD) and safety-criticality of software embedded in
modern cars, there is a need for evolving existing methods of MBD and complementing them with methods
already used in the development of other systems (Fault Injection and Mutation Testing). Our position is
that significant effectiveness and efficiency improvements can be made by applying fault injection
techniques combined with mutation testing approach for verification and validation of automotive software
at the model level. The improvements include such aspects as identification of safety related defects early in
the development process thus providing enough time to remove the defects. The argument is based on our
industrial case studies, the studies of ISO 26262 standard and academic experiments with new verification
and validation methods applied to models.
1 INTRODUCTION Software for custom functionality in modern cars is
usually developed by multiple suppliers although it
Nowadays, a typical premium car has up to 70 ECUs is largely designed by a single OEM (Original
which are connected by several system buses to Equipment Manufacturer) like Volvo Cars. The
realize over 2000 functions (Broy, 2006). As around distributed development and use of standards like
90% of all innovations today are driven by AUTOSAR aims to facilitate reuse of software and
electronics and software, the complexity of car’s hardware components between different vehicle
embedded software is already high and expected to platforms, OEMs and suppliers (Fennel et al., 2006).
grow further. The growth is fuelled by cars However, testing of such systems is more complex
beginning to act more proactively and provide more and even today testing of software generally
assistance to its drivers, which requires software to accounts for almost 50% of overall development
interact with hardware more efficiently and making costs (Boehm and Basili, 2001).
more decisions automatically (e.g. collision ISO-26262 in automotive domain poses stringent
avoidance by braking, brake-by-wire or similar requirements for development of safety critical
functions). In total with about 100 million lines of applications and in particular on the testing
code (SLOC), premium segment vehicles carry more processes for this software. These requirements are
software code than in modern fighter jets and intended to increase the safety of modern cars,
airliners (Charette, 2009). although they also increase the cost of modern cars.
Rana R., Staron M., Berger C., Hansson J., Nilsson M. and Törner F..
Increasing Efficiency of ISO 26262 Verification and Validation by Combining Fault Injection and Mutation Testing with Model based Development.
251
DOI: 10.5220/0004592002510257
In Proceedings of the 8th International Joint Conference on Software Technologies (ICSOFT-EA-2013), pages 251-257
ISBN: 978-989-8565-68-6
Copyright c 2013 SCITEPRESS (Science and Technology Publications, Lda.)
ICSOFT2013-8thInternationalJointConferenceonSoftwareTechnologies
The position for which we argue in this paper is that brief discussion on related work in section 3 and our
efficient verification and validation of safety position is presented and discussed in section 4.
functions requires combining Model Based Section 5 concludes our work.
Development (MBD) with fault injection into models
with mutation testing. This position is based on the
studies of the ISO 26262 standard (mainly chapter 6 2 BACKGROUND
that describes requirements on software
development but also chapter 4, which poses In this section we take a brief overview on the
requirements on product development (ISO 26262 -
current state of automotive software development
2011, 2011)). It is also based on previous case process and environment, how safety is important in
studies of the impact of late defects on the software safety critical applications and overview of
development practices in the automotive domain
theoretical background on fault injection techniques
(e.g. (Mellegård et al., 2013)) and mutation testing.
The requirements from the ISO 26262 standard
on using fault injection techniques is challenging
since it relates to the development of complete
2.1 Automotive Software
functions rather than components or sub-components Development & ISO 26262
of software. The current situation in the automotive
sector is that fault injection is used, but it is used at Various software functions/applications developed
the level of one electronic component (ECU) or one within the automotive industry today are classed as
software system and rarely at the function level safety critical, for example Volvo’s City Safety
(Hillenbrand et al., 2010; Schätz, 2010). feature consists of components that are safety
The current state of art testing is not enough for critical.
detecting safety defects early in the automotive
software development process since fault injection is
done late in the development (when ECUs are being
developed), which usually makes the detection of
specification-related defects difficult and costly
(Mellegård et al., 2013). As much possible this
detection should be done at the model level when the
ECUs’ functionality is still under design and thus, it
is relatively cheap to redesign/reconfigure. The
evidence from literature on successful use of fault
injection shows that the techniques are indeed
efficient in finding dependability problems of Figure 1: Volvo Cars city safety function, image provided
hardware and software systems when applied to by Volvo Car Corporation.
compute (Hsueh et al., 1997). Finally, to be able to
increase the effectiveness of the fault injection (Broy, 2006) gives examples of functions/areas
strategies and identify whether the faults should be within automotive domain with recent development
injected at the model, software or ECU level - which includes crash prevention, crash safety,
Mutation testing should be applied to verify the advanced energy management, adaptable man-
adequacy of test cases and finally how the machine interface, advanced driver assistance,
combination of these approaches when applied at the programmable car, car networking etc., much of
model level will enhance the detection of safety these also fall within the safety critical functionality
defects right at the design stage. and thus demands high quality and reliability. Also a
In this paper, we provide a roadmap, which number of on-going projects are directed towards the
shows how to introduce fault injection and mutation goal of self-driving cars.
testing to modelling of automotive software in order Software development in automotive sector in
to avoid costly late defects and increase the safety of general follows the ‘V’ process, where OEMs take
modern and future cars. the responsibility of requirement specification,
The remaining of the paper is structured as system design, and integration/acceptance testing.
follows: In the next section (2) we provide an This is followed by the supplier, which develops the
overview of software development in automotive actual code that runs on ECUs. Although the code is
domain and associated concepts. This is followed by tested at the supplier level (mainly unit testing), the
252
IncreasingEfficiencyofISO26262VerificationandValidationbyCombiningFaultInjectionandMutationTestingwith
ModelbasedDevelopment
OEMs are responsible for the final integration, detected during testing do not depend on actual
system and acceptance testing to ensure that the implementation of code, about 50% of defects
given implementation of a software (SW) meets its detected during testing in the study by (Megen and
intended functional and safety goals/demands. Meyerhoff, 1995), were found during the test
preparation, an activity independent of the
executable code. And since automotive sector has
already widely adopted MBD for the software
development of embedded systems, a high potential
exists for using the behavioural modes developed at
the early stages of software development for
performing some of the V&V (Verification &
Validation). Early V&V by helping to detect defects
early will potentially save significant amount of cost
for the projects and reduce the cycle time.
Figure 2: The V-model in the automotive industry with
distinction between the OEM and supplier contributions. 2.2 ISO 26262
In this model of software/product development ISO/IEC 26262 is a standard describing safety
(see Figure 2) testing is usually concentrated in the requirements. It is applied to safety-related systems
late stages of development, which also implies that that include one or more electrical and/or electronic
most of the defects are discovered late in the (E/E) systems. The overview of safety case and
development process. In a recent study using real argumentation is represented in Figure 4, based on
defect data from an automotive software project (ISO 26262 - 2011, 2011).
from the industry (Mellegård et al., 2012) showed
•The item representing a system or a function is
that late detection of defects is still a relevant Item defined.
problem and challenge yet to overcome. The defect
inflow profile presented in this study is reproduced •A Preliminary Hazard Analysis & Risk Assessment is
in Figure 3 for reference, which exhibits a clear peak PHA done to assign an appropriate ASIL level.
in number of open defects in the late stages of •Safety Goals are derived from the Hazard Analysis
function development/testing. SG and they inherit the assigned ASIL level.
•Functional Safety Requirements are drawn such that
FSR the set Safety Goals are met.
•The Technical Safety Requirements are formulated
TSR describing how to implement FSR.
•Further development includes implementation,
Doc integration and documentation of safety cases.
Figure 4: Overview of ISO-26262 safety case &
argumentation process.
Figure 3: Defect inflow profile for automotive software Written specifically for automotive
project, as given in (Mellegård et al., 2012). domain/sector, the ISO-26262 standard is adapted
for the V-model of product development
Testing the software is an important tool of corresponding to the current practice in the industry.
ensuring correct functionality and reliability of The guidelines are laid out for system design,
systems but it is also a very resource intensive hardware and software design & development and
activity accounting for up to 50% of total software integration of components to realize the full product.
development costs (Jones, 2001) and even more for ISO-26262 includes specifications for MBD and
safety/mission critical software systems. Thus provides recommendations for using fault injection
having a good testing strategy is critical for any techniques for hardware integration and testing,
industry with high software development costs. It software unit testing, software integration testing,
has also been shown that most of the defects hardware-software integration testing, system
253
ICSOFT2013-8thInternationalJointConferenceonSoftwareTechnologies
integration testing and vehicle integration testing, for level in automotive industry is currently at its
overview on fault injection recommendations in infancy. Figure 5 shows a mind map of classification
ISO-26262 see (Rana et al., 2013). Although the of fault injection techniques based on how the
functional safety standard specifies clearly the technique is implemented; some of the tools which
recommendations for using fault injection during are developed based on given approach are also
various stages of testing but it does not recommend listed for reference. For a good overview of fault
anything with respect to using mutation testing. This injection techniques readers are referred to (Hsueh et
also reflects the current standard practice within the al., 1997; Ziade et al., 2004).
automotive industry where mutation testing is not
widely adopted yet. 2.4 Mutation Testing
2.3 Fault Injection Mutation testing is technique for assessing the
adequacy of given test suite. Mutation testing
Fault injection techniques are widely used for includes injection of systematic, repeatable seeding
experimental dependability evaluation. Although of faults in large number thus generating number of
these techniques have been used more widely for copies of original software artefacts with artificial
assessing the hardware/prototypes, the techniques fault infestation (called a mutant). Percentage of
are now about to be applied at behavioural models of mutations detected by the given test cases/suite is a
software systems (Svenningsson et al., 2010) - thus metrics (called “mutation adequacy score” (Jia and
enabling early verification of intended functionality Harman, 2011)) used for measuring effectiveness of
as well as enhancing communication between the given test suite. The variants of code (faults) can
different stakeholders. Fault injection techniques be introduced by hand or auto-generated using tools
applied at models level offer distinct advantages like Insure++, Plextest, Certitude, ESPT for C/C++
especially in an industry using MBD for its software codes. It has been shown that the use of mutants
development, but use of these techniques at model
Figure 5: Common classification of fault injection techniques and implementation tools, description available in (Ziade et
al., 2004; Hsueh et al., 1997).
254
IncreasingEfficiencyofISO26262VerificationandValidationbyCombiningFaultInjectionandMutationTestingwith
ModelbasedDevelopment
yields trustworthy results (Andrews et al., 2005), i.e. Another work (Brillout et al., 2010) with its root in
mutants do reflect characteristics of real faults. the European CESAR (Cost-efficient methods and
Mutation theory is based on two fundamental processes for safety relevant embedded systems)
hypotheses namely Competent Programmer project provides a good theoretical overview of how
Hypothesis and the Coupling Effect, both introduced fault and mutation based test coverage can be used
by (DeMillo et al., 1978). The Competent for automated test case generation for Simulink
Programmer hypothesis reflects the assumption that models. We provide a practical framework on how
programmers are competent in their job and thus fault injection combined with mutation testing
would develop programme close to correct version within an MDB environment can be used in the
(although making a number of mistakes) while the industry. And how will this practice enhance the
Coupling Effect hypothesis means that complex verification and validation of software under
mutants are coupled to simple mutants in such a way development, its functional validation that would
that a test data that detects large percent of simple generates statistics for the effective argumentation of
faults is also effective in detecting high percentage ISO 26262 compliance.
of the complex defects” (Offutt, 1992).
4 ROAD MAP FOR EARLY
3 RELATED WORK DEFECT DETECTION
A number of European Union sponsored projects, We contend that fault injection can be effectively
within the area of embedded software development used at the model level to verify and validate the
and safety critical systems have looked at and attainment or violation of safety goals. We also
developed techniques to effectively use fault propose that it should be complemented with
injection for safe and reliable software development. mutation testing approach at the model level to
The examples include the ESACS (Enhanced Safety provide enough statistical evidence for argumenting
Assessment for Complex Systems) (Joshi and the fulfilment of safety goals as per the ISO-26262
Heimdahl, 2005) and the ISAAC (Kakade et al., safety standard requirements.
2010)(Improvement of Safety Activities on A major challenge in successful argumentation
Aeronautical Complex systems). These projects of ISO-26262 compliance is to provide statistical
have used the SCADE (Safety-Critical Application evidence that safety goals (SGs) would not be
Development Environment) modelling environment violated during operation and collecting the
to simulate hardware failure scenarios to identify evidence for this argumentation within reasonable
fault combinations that lead to safety case violations. testing efforts.
A model-implemented fault injection plug-in to If we are able to differentiate early between
SCADE called FISCADE is introduced in (Vinter et defects that can cause the violation of SGs and those
al., 2007). The plug-in tool utilizes approach similar that cannot cause the violation, the amount of testing
to mutation based testing, where it replaces the required will be manageable. With MBD the
original model operators by their equivalent fault functional testing could be done using fault injection
injection nodes. The derived models are then used to techniques and this can be complemented with later
inject the fault during execution and log the results system testing of the actual code using the mutation
which are analysed later. Dependability evaluation testing approach.
of automotive functions using model based software The framework on how this could be achieved in
implemented fault injection techniques have also practice is as follows:
been studied in (Plummer, 2006).
A generic tool capable of injecting various types
of faults on the behavioural/functional Simulink
models is also developed and introduced in
(Svenningsson et al., 2010). The tool called MODIFI
(or MODel-Implemented Fault Injection tool) can be
used to inject single or multiple point faults on
behavioural models, which can be used to study the
effectiveness/properties of fault tolerant system and
identify the faults leading to failure by studying the
Figure 6: MBD based representation of a general system
fault propagation properties of the models. with inputs, outputs and dependencies.
255
ICSOFT2013-8thInternationalJointConferenceonSoftwareTechnologies
As illustrated in Figure 6, a given system/function Further to make this framework/approach more
generally have following common features (in effective in industrial practice we identify a number
context of model based development): firstly it will of best practices that will have positive impact on
have x inputs (i1,2…x); it would have dependencies to detecting defects early in the development process
other y components/ functions (d1,2…y); it will have z and thus have effective V&V of ISO-26262:
outputs (o1,2…z); and it will have a number of sub- a. The best practice is to build and maintain
units/modules within it that implement the intended models corresponding to each abstraction layer
functionality, let us assume that this part contains n of software architecture.
basic blocks in the modelling environment b. The next best practice is to specify and test
corresponding to n statements for a hand written these models for FSRs and TSR at the
code. To verify and validate the correct functionality appropriate abstraction level.
and ISO-26262 compliance of this generic function c. Also identification of different types of
using fault and mutation testing approach we can defects/faults and at what stage they could be
follow the steps as: modelled/injected in the behavioural models
a. Assign or define the technical safety would ensure that models are tested for these
requirements (TSRs) corresponding to the faults at the earliest - leading to models being
functional safety requirements (FSRs) for the build that are robust right from the start instead
given system/function to its z outputs. of adding fault tolerance properties in the later
b. Use fault injection techniques to inject faults stages of development.
which are similar to commonly occurring
defects and other possible fault conditions at
the x inputs of the function. 5 CONCLUSIONS
c. Fault scenarios that leads to violation of
TSRs/FSRs are identified, statistics are built on
The development of software in the automotive
what percentage of total faults lead to such domain has widely adopted the paradigm of model
failures and fault propagation properties of
based development to allow for easier integration of
such cases are studied to build the fault functionality usually developed by multiple
tolerance within the system for given fault suppliers. By the nature of the domain much of the
conditions.
functionality developed and implemented in cars is
d. Repeat steps (b) & (c) to test, correct and safety critical; the criticality that requires
validate the given system/function for its observation of stringent quality assessment and
dependencies on other functions/components.
adherence to functional safety standards such as ISO
e. Cause mutations to the n basic blocks of given 26262.
functional model and asses the detection Development of behavioural models in MBD
effectiveness of test suite/cases for possible
offers significant opportunity to do functional testing
implementation bugs. early in the development process. Fault injection and
f. Examine the mutants which are not killed by mutation testing approach in combination can be
given set of test cases/suits for their effect on
used to effectively verify and validate the functional
FSRs. If a given mutation violates the FSRs properties of a software system/function. The
then a suitable test case is created to detect/kill approach also provides required statistics for the
such mutants, i.e. detect such bugs in actual
argumentation of safety standards compliance. In
code. this paper the need for such validation and a
By following the above mentioned steps we not only
framework on how this could be achieved in practice
ensure that the given function holds the FSRs and
is discussed. The results are a roadmap for further
TSRs under faulty inputs, but we can also prevent research and tool support to bring this approach into
potential implementation defects and ensure that we
wider industrial adoption.
have test cases ready to catch such faults that can By detecting defects early and being able to do
potentially violate the FSRs/TSRs already at the much of verification and validation of intended
design (model) level.
functionality, robustness and compliance to safety
It is also worthwhile to note here that steps (a) to (e) standards on the models – the quality and reliability
can be easily automated using the currently available of software in automotive domain can be
testing methodologies, which makes the usability
significantly enhanced. Effective approaches and
and industrial viability much higher that testing tools support reduce the V&V costs and lead to
frameworks requiring high manual interventions. shorter development times.
256
IncreasingEfficiencyofISO26262VerificationandValidationbyCombiningFaultInjectionandMutationTestingwith
ModelbasedDevelopment
ACKNOWLEDGEMENTS Megen, R., Meyerhoff, D. B., 1995. Costs and benefits of
early defect detection: experiences from developing
client server and host applications. Softw. Qual. J. 4,
The work has been funded by Vinnova and Volvo 247–256.
Cars jointly under the FFI programme (VISEE, Mellegård, N., Staron, M., Tö, rner, F., 2012. A light-
Project No: DIARIENR: 2011-04438). weight defect classification scheme for embedded
automotive software and its initial evaluation.
Mellegård, N., Staron, M., Törner, F., 2013. A Light-
Weight Defect Classification Scheme for Embedded
REFERENCES Automotive Software Development.
Offutt, A. J., 1992. Investigations of the software testing
Andrews, J. H., Briand, L. C., Labiche, Y., 2005. Is coupling effect. Acm Trans. Softw. Eng. Methodol.
mutation an appropriate tool for testing experiments? Tosem 1, 5–20.
[software testing], in: Software Engineering, 2005. Plummer, A., 2006. Model-in-the-loop testing. Proc. Inst.
ICSE 2005. Proceedings. 27th International Mech. Eng. Part J. Syst. Control Eng. 220, 183–199.
Conference On. pp. 402–411. Rana, R., Staron, M., Berger, C., Hansson, J., Nilsson, M.,
Boehm, B., Basili, V. R., 2001. Defect Reduction Top 10 Törner, F., 2013. Improving Fault Injection in
List. Computer 135–137. Automotive Model Based Development using Fault
Brillout, A., He, N., Mazzucchi, M., Kroening, D., Bypass Modeling. Submitted To: 2nd Workshop on
Purandare, M., Rümmer, P., Weissenbacher, G., 2010. Software-Based Methods for Robust Embedded
Mutation-based test case generation for simulink Systems, Informatik 2013, Koblenz, Germany.
models, in: Formal Methods for Components and Schätz, B., 2010. Certification of Embedded Software–
Objects. pp. 208–227. Impact of ISO DIS 26262 in the Automotive Domain,
Broy, M., 2006. Challenges in automotive software in: Leveraging Applications of Formal Methods,
engineering, in: Proceedings of the 28th International Verification, and Validation. Springer, pp. 3–3.
Conference on Software Engineering. pp. 33–42. Svenningsson, R., Vinter, J., Eriksson, H., Törngren, M.,
Charette, R. N., 2009. This Car Runs on Code. 2010. MODIFI: a MODel-implemented fault injection
[Link] tool. Comput. Saf. Reliab. Secur. 210–222.
car-runs-on-code. Vinter, J., Bromander, L., Raistrick, P., Edler, H., n.d.
DeMillo, R. A., Lipton, R. J., Sayward, F.G., 1978. Hints FISCADE - A Fault Injection Tool for SCADE
on test data selection: Help for the practicing Models, in: Automotive Electronics, 2007 3rd
programmer. Computer 11, 34–41. Institution of Engineering and Technology Conference
Fennel, H., Bunzel, S., Heinecke, H., Bielefeld, Jü, rgen, On. pp. 1–9.
Fü, rst, S., Schnelle, K.-P., Grote, W., Maldener, N., Ziade, H., Ayoubi, R. A., Velazco, R., others, 2004. A
Weber, T., Wohlgemuth, F., others, 2006. survey on fault injection techniques. Int Arab J Inf
Achievements and exploitation of the AUTOSAR Technol 1, 171–186.
development partnership. Convergence 2006, 10.
Hillenbrand, M., Heinz, M., Adler, N., Müller-Glaser,
K.D., Matheis, J., Reichmann, C., 2010. ISO/DIS
26262 in the context of electric and electronic
architecture modeling, in: Architecting Critical
Systems. Springer, pp. 179–192.
Hsueh, M. C., Tsai, T. K., Iyer, R. K., 1997. Fault
injection techniques and tools. Computer 30, 75–82.
ISO 26262 - 2011, 2011. Road vehicles -- Functional
safety -- Part 1-10.
Jia, Y., Harman, M., 2011. An analysis and survey of the
development of mutation testing. Softw. Eng. IEEE
Trans. 37, 649–678.
Jones, E. L., 2001. Integrating testing into the
curriculum—arsenic in small doses, in: ACM SIGCSE
Bulletin. pp. 337–341.
Joshi, A., Heimdahl, M. P. E., 2005. Model-based safety
analysis of simulink models using SCADE design
verifier.
Kakade, R., Murugesan, M., Perugu, B., Nair, M., 2010.
Model-Based Development of Automotive Electronic
Climate Control Software. Model. Found. Appl. 144–
155.
257