Model Based

Testing

Presenters
Ashish Mote
Amit Chigteri

© 2005 KPIT Cummins Infosystems Limited

We value our relationship

Prerequisites & Class Purpose

• For this session basic knowledge of Matlab-Simulink, Code

generation & S-function is required.

•To know about different types of testing used in software

industry, their objectives & when shall it performed??

• To learn different stages of testing & their use in model

based development(Simulink based development)

7-Mar-12 2 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Agenda

• Testing, SDLC’s, Objectives of testing and testing artifacts

• Classification of Testing Strategies based on different factors

• Model Based Testing – MIL, Plant Modeling, SIL, PIL & HIL

• Testing Automation, automation for MBT

7-Mar-12 3 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
 Software Testing

© 2005 KPIT Cummins Infosystems Limited

We value our relationship

What is testing?
• It is an activity aimed at evaluating an attribute or capability of a
program or system and determining that it meets its required results

• Software testing is an investigation conducted to provide

stakeholders with information about the quality of the product or
service under test.

• It is to be noted that testing can never completely identify all the

defects within software.

• When to perform testing?

7-Mar-12 5 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Software Development Life Cycles
Waterfall model Modified Waterfall model

V model Agile Model

7-Mar-12 6 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Testing Objective & Phases
• The objectives with which mostly testing is done software industries
are
1. Whether there are any bugs in implemented software.
2. Whether software/build is meeting requirement.
3. What is performance of system with this software.

• Traditionally testing was limited to debugging, but later when

separation was introduced, phases & goals in software testing
industry were classified as
• Until 1956 - Debugging oriented
• 1957–1978 - Demonstration oriented
• 1979–1982 - Destruction oriented
• 1983–1987 - Evaluation oriented
• 1988–2000 - Prevention oriented

7-Mar-12 7 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Testing Artifacts
1. Test Plan :- A test specification is called a test plan.

2. Traceability Matrix :- A traceability matrix is a table that

correlates requirements or design documents to test documents.

3. Test Vector :- It is a systematic collection in form of array/matrix

of values for a particular input or output for model under test.

4. Test Case & Test Suite :- Test plan is overall brief plan of
complete testing. A test case more detailed version for a small
part of test plan for a specific requirement. Collection of test
cases is known as Test suite.

5. Test Data :- All the values of test inputs, expected outputs, actual
outputs along with test environment related data are collected
together in file. This data is called Test data.

7-Mar-12 8 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
 Classification of testing

© 2005 KPIT Cummins Infosystems Limited

We value our relationship

Classification of testing

• Based on Scope
• Functional Testing
• Non-functional Testing(Performance, Scalability, Usability, Security
etc)

• Based on Method
• Static
• Dynamic

• Based On Knowledge tester has

• White Box Testing
• Black Box Testing
• Grey Box Testing

7-Mar-12 10 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Classification of testing

• Based on Level of testing

• Unit Testing/ Component Testing
• Integration Testing
• System Testing
• System Integration Testing

• Based on way test is executed

• Manual Testing
• Automated Testing

•Based on who is tester

• Alpha Testing
• Beta Testing

7-Mar-12 11 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
 Model Based Testing

© 2005 KPIT Cummins Infosystems Limited

We value our relationship

Model Based Testing

•The term ‘model-based testing’ is widely used today with

slightly different meanings. In the automotive industry it is
normally used to describe all testing activities in the context of
model-based development projects.

•One major benefit of model-based development is the

possibility to “do things as early as possible”. In terms of
testing this means to test the functionality in the model before
the software is implemented and integrated into the final ECU.

• Other advantages also include possibility of automation by use

of scripts, automatic test case generation, Specific
coding/modelling guideline checks using Model Advisor
7-Mar-12 13 Version 1.0
Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Levels of Model based Testing

• Between the initial modelling and the integration into the

ECU there are intermediate integration levels of testing.

• These integration levels are

• Model-In-Loop i.e. MIL Testing
• Software-In-Loop i.e. SIL Testing
• Processer-In-Loop i.e. PIL Testing
• Hardware-In-Loop i.e. HIL Testing

7-Mar-12 14 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
 Model In Loop(MIL)

7-Mar-12 15 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
MIL
• The first integration level is based on the model of the
system itself.
• Testing an embedded system on MIL level means that the
model and its environment are simulated (interpreted) in the
modelling framework without any physical hardware
components.
• Generally Objective of MIL level testing are
• Functionality/requirements testing
• Overflows/ Underflows detection
• Division by 0 and few other arithmetic errors

7-Mar-12 16 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
MIL Continued
• Model in Loop testing helps in finding bugs in implementation
in very early stage of software life cycle.
• For MIL/SIL in Matlab/Simulink Environment there are 2 ways
of testing
1. By using test vectors
2. By using Plant model(s)

MIL Using Test Vectors MIL Using Plant Modeling

7-Mar-12 17 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
MIL Continued
• For MIL/SIL testing wrapper model shall be created first from
design model.
• A design model is model which has all data defined & from
which code generated.
• Wrapper model is also knows as test model, Simulation model
or Feature/Component Simulation Model
• Design model is also known Code-gen Model, Controller Model
or Actual Model in very crude manner

7-Mar-12 18 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Testing using test vectors
• Test vector is systematic collection in form of array/matrix of
values for a particular input or output for model under test.
• Collection of such vectors together which can test 1 or more
requirement or even a complete single feature is known as Test case.
• Collection of all test cases together that can completely test model
under test in a data file(.mat file in Matlab/Simulink) is knows as Test
Data for model based testing.

• These vectors can be handwritten by tester or can be automatically

generated using Matlab based testing tools
• Sometimes these test vectors can also come from actual vehicle test
data.
• These test vectors shall be created in such way that they are re-
usable for SIL as well PIL & HIL.

7-Mar-12 19 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Testing Using Plant Model
• Plant Model is model made for simulating physical systems or
physical components

• Plant modelling shall be able to simulate physical

components that have sufficient fidelity to test the
various execution paths within the software.

• Testing becomes more effective when plant models with

right fidelity are used instead of simple test vectors.

• Plant models can be made to replicate behavior any

physical components such as sensors, actuators using
Simulink & Simscape which is also a tool by Mathworks

7-Mar-12 20 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Plant Model Example
• Suppose a Proportional controller for “controlling heater for
maintaining room temperature within given range” is designed
using Simulink.
• This controller is suppose provide Duty cycle input to heater
relay or percentage duty cycle to device driver of the heater

• To test this controller a plant model of room can be made as

in snapshots on next slides which can test how efficient is it’s
proportional control.

7-Mar-12 21 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Snapshot of top level of test model

7-Mar-12 22 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Inside Plant Model

7-Mar-12 23 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
 Software In Loop(SIL)

7-Mar-12 24 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
SIL
•Testing an embedded system on SIL level means that the
embedded software i.e. generated code is tested within
a simulated environment model but without any
hardware.
• Since the environments virtual, a real-time environment
is not necessary.

SIL Using Plant Modeling SIL Using Test Vectors

7-Mar-12 25 Version 1.0
Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
SIL Continued
• For SIL testing few objectives can be same as MIL, but
majorly following are objectives of SIL testing
• MIL v/s SIL behavior comparison to find differences if any
• Dead (or unreachable)Code finding
• Code Coverage testing to see how much part of code is really being
tested in test case you have designed.
• Out-of-bound array access
•If some legacy code has been mixed with Simulink-RTW
generated or Simulink-Targetlink code generation then Other
code errors that can show up, especially in a mix with hand-
generated code, include:
• Illegally de-referenced pointers
• Read-only access to non-initialized data
• Dangerous type conversions

• Usually SIL are performed on Windows/Linux based desktop.

7-Mar-12 26 Version 1.0
Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
SIL Continued

• For SIL testing in Matlab/Simulink Environment following

Steps shall be performed
• Define all data’s(Variables) correctly if not done at MIL stage
• From code generation model generate the code
• From this code make a separate S-function or for subsystem you
want to test directly make a S-function.
• In model used for MIL testing replace model with S-Function.
• Input/Import the test data to workspace & Run the model.
• Plot and analyze outputs. Compare with previously saved MIL
testing results if in scope of testing.

7-Mar-12 27 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
 Processor In Loop(PIL)

7-Mar-12 28 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
PIL

7 March 2012 29 29 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
PIL Conti
•Embedded controllers are integrated in embedded devices with proprietary hardware (ECU).
•Testing on PiL level is similar to SiL tests, but the embedded software runs on a target board
with the target processor or on a target processor emulator.
•Tests on PiL level are important because they can reveal faults that are caused by the target
compiler or by the processor architecture.
•It is the last integration level which allows debugging during tests in a cheap and
manageable way.
•Therefore the effort spent by PiL testing is worthwhile in almost all cases.

7 March 2012 30 30 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
 Hardware In Loop(HIL)

7-Mar-12 31 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
HIL
• When testing the embedded system on HIL level the
software runs on the final ECU.
• However the environment around the ECU is still a
simulated one. ECU and environment interact via the
digital and analog electrical connectors of the ECU.
•The objective of testing on HIL level is to reveal faults in
the low-level services of the ECU and in the I/O services.

7-Mar-12 32 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Kind of bugs that can be found at each level

7 March 2012 33 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Correct time in V model for each type of these testings

7 March 2012 34 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
 Testing Automation

© 2005 KPIT Cummins Infosystems Limited

We value our relationship

Test Automation

• Test automation is the use of software to automate testing procedures.

•Test Automation can be used

• To setup test environment
• To control the execution of tests
• To compare actual outcomes to predicted outcomes
• To set up of test preconditions
• To setup other test control and test reporting functions

• Test Automation objectives are

• To reduce efforts and save time in repetitive type manual testing
• To reduce human errors that can get introduced in manual testing
• To find certain classes of defects which cannot be effectively found out or can
be missed by manual testing

7-Mar-12 36 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Test Automation Approaches

• There are 2 general approaches to test automation are

1. Code driven testing

2. GUI driven testing

• Code driven testing(Scripting Based) :- The interfaces(ways of

inputting) to components, modules or libraries are tested with a
variety of input arguments to validate that the results that are
returned are correct.
• GUI driven testing :- A testing framework generates user interface
events such as keystrokes and mouse clicks, and observes the
changes that result in the user interface, to validate that the
observable behavior of the program is correct

7-Mar-12 37 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Test Automation in Model Based Development/Testing

• What Can be automated?

1. Wrapper Model Creation
2. Test vector creation
3. Test result display, comparison and recording
4. Test result report creation
5. Run time bug finding and reporting
6. Coverage tracking

• Reactis is one stand alone application that reads .mdl files

created in Matlab-Simulink/Stateflow environment

7-Mar-12 38 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Overview of Reactis

• A model-based design environment involving Reactis, Simulink and

Stateflow is depicted in Figure below

• Reactis contains three core components:

1. Tester:- Provides automated test generation from models.
2. Simulator:- Enables users to visualize model execution to debug
models and track coverage.
3. Validator:- Offers automated checks of Simulink models for
violations of user-specified requirements.
7-Mar-12 39 Version 1.0
Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Overview of Reactis Conti…

• The generated test suites provide comprehensive coverage of

different test coverage metrics - including the Modified
Condition/Decision Coverage (MC/DC) test coverage measure
• The automated test data later can be used for different purposes
like
1. Implementation conformance
2. Model testing and debugging
3. Reverse engineering of models from source

7-Mar-12 40 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Implementation conformance

7-Mar-12 41 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Model testing and debugging

7-Mar-12 42 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote
Reverse engineering of models from source

7-Mar-12 43 Version 1.0

Complied By: Amit Chigteri, Ashish Mote Presenter: Amit Chigteri, Ashish Mote