Basic Computer System Design

Testing

Muhammad Tariq Mahmood

tariq@[Link]
School of Computer Science and Engineering

At Sony, we assume all products of our competitors will have basically the same technology, price, performance, and features. Design is the one thing that
differentiates one product from another in the marketplace (–Norio Ohgo, Chairman and CEO, Sony)

Note: These notes are prepared from the following resources.

▶ (main text) Design for Electrical and Computer Engineers by Ralph Ford, Chris Coulston (McGraw-Hill Education 2008)
▶ Wiegers, Karl, and Joy Beatty. Software requirements. 3rd Edition, Pearson Education, 2013.
▶ Kendall, Kenneth E., and Julie E. Kendall. Systems analysis and design. Pearson Education, 2019 10Ed.
▶ https : //www .uml − [Link]/
▶ https : //www .[Link]/

1 / 29
 Contents

1 Testing Principles Learning Objectives

Types of Testing, Observability, and ▶ Understand the concepts of black box tests, white
Controllability box tests, observability, and controllability.
Stubs
▶ Understand the principles of debugging.
Test Case Properties
2 Constructing Tests ▶ Understand when a unit test is used and how it is
Debugging constructed.
Unit Testing
▶ Understand when an integration test is used and
Integration Testing
how it is constructed.
Integration Testing
3 Case Study: Security Robot Design ▶ Understand when an acceptance test is used and
4 Guidance how it is constructed.

2 / 29
 Testing

Testing
▶ systems should be tested to ensure that they meet the engineering requirements.

▶ The cost to correct problems increases exponentially with the lifetime of the project. Thus, testing should be
considered throughout system development.

▶ Testing means different things to different people. A field service technician, assembly line worker, and
designer will have their own definitions and requirements from a test.

▶ Testing is examined from the perspective of a systems designer intent on checking that the system meets
the engineering requirements.

▶ Along the way fundamental testing concepts like controllability and observability are explored. Approaches
to debugging systems are provided, followed by templates for building unit tests, integration tests, and
acceptance tests.

3 / 29
 Testing Principles
Testing Principles

▶ A common testing model, the "test vee," shown in

Figure .
▶ This model starts with the engineering requirements,
proceeds to the implementation, and then onto
testing.
▶ It emphasizes that every level of design has a
corresponding level of test.
▶ Each test performed in the right half of the test vee
must be carefully engineered during the development
of the system in the left side of the test vee.
▶ An acceptance test plan should be written with the
requirements specification, integration tests defined
and written during the system design Figure 1: The test vee. Design stages are on the left
and corresponding tests are on the right.

4 / 29
 Testing Principles (cont...)
▶ Clear tests need to be developed because:
1. The test cases define exactly what the module must do.

2. Testing prevents "feature creep, " since the development of a module is complete when its test is passed.

3. Test cases motivate developers by providing immediate feedback.

4. Test cases force designers to think about extreme cases.

5. Test cases are a form of documentation.

6. Test cases force the designer to consider the design of the module before building it.
▶ The test suite and its accompanying documentation contain important information about the behavior and
organization of a system and its module.
▶ The individual test cases show other engineers how to properly interface to a module, making that module
more reusable.
▶ Given enough time, a system could be tested by simply enumerating every conceivable input and observing
the outputs.
▶ Test documents can be used by other individuals in the organization such as technical writers, maintenance
technicians, and technical trainers
5 / 29
 Types of Testing, Observability, and Controllability

Types of Testing, Observability, and Controllability

▶ Tests fall into two general typesblack box and white box tests.
▶ Black box tests:
• Black box tests are those that are performed without any knowledge of the system’s internal organization.
• In a black box test, the testing is typically conducted by changing the inputs and comparing the system outputs
to their expected values.
• The input and output values can be classified as typical, boundary, extreme, and invalid.
• While these tests could be accomplished by enumerating every possible input to the system and observing the
output, this would take an unreasonable amount of time.
• Hence, the test writer must elect candidate inputs to represent the behavior of the system over a range of
possible inputs.
• An important goal of the test writer is to minimize the number of these equivalence classes while maximizing
coverage of the input domain.

6 / 29
 Types of Testing, Observability, and Controllability (cont...)

▶ White box tests

• White box tests are conducted with knowledge of the internal working of the system.
• The idea of white box testing is to build tests which target specific internal nodes of the system to check that
they are operating as expected.
• The tests should be written to check that node can handle typical, boundary, extreme and illegal situations.

▶ Testability: One of the many goals in designing a system is to increase its testability. A design is testable
when a failure of a component or subsystem can be quickly located. A testable design is easier to debug,
manufacture, and service in the field.
▶ Controllability: is the ability to set any node of the system to a prescribed value.
▶ Observability: is the ability to observe any node of a system
▶ In black box testing, both controllability and observability are low. In white box testňing, controllability and
observability may be higher, depending on the design.

7 / 29
 Types of Testing, Observability, and Controllability (cont...)
▶ The black box testing would consist of checking checked to see if they meet the expected design
supply and ground voltages, varying the input signal, values. This indicates a high degree of observability.
and observing the output signal. Again, this is a
low-controllability and low-observability situation.
▶ White box testing utilizes knowledge of the internal
workings of the design. In designing a transistor
amplifier, there are two major points to consider.
The DC bias voltňages in the circuit and its AC, or
time-varying, amplification behavior. The two
behaviors are related since the AC behavior depends
upon proper DC biasing of the circuit. During
detailed circuit design, the expected DC voltages for
different nodes in the circuit would be determined..
Thus, a white box test would consist of first checking
the power supply and ground voltages as was done in
the black box case. The next step would be different
in that the node voltages (VB, Vc, VE) would be Figure 2: Transistor amplifier design.

8 / 29
 Stubs

Stubs
▶ A stub is a device that is used to simulate a subcomponent of a system.

▶ Typically, stubs are used to simulate inputs or monitor outputs of the unit under test (UUT).

▶ Both hardware and software stubs can be used in designing a system.

▶ In software testing, stub routines are developed to either call other functions or act as those to be called by
the unit under test.

▶ Consider a hardware example, the transistor amplifier. Assume that the circuit is ultimately to be integrated
into a larger system. The input to this system is a time-varying source with certain resistive and capacitive
characteristics, while the output is connected to another system with a known input resistance range.

9 / 29
 Stubs (cont...)

Figure 3: The use of stubs for testing a transistor amplifier circuit. The function generator, resistor (R), and
capacitor (C) model the expected behavior of the input source in the final system implementation. The variable
resistance (RI) models the load that would be attached to the output.

10 / 29
 Test Case Properties

Test Case Properties

▶ Following properties for effective test cases:
1. Accurate : The test should check what it is supposed to and exercise an area of intent.

2. Economical: The test should be performed in a minimal number of steps.

3. Limited in complexity: Tests should consist of a moderate number (10-15) of steps.

4. Repeatable: The test should be able to be performed and repeated by another person.

5. Appropriate: The complexity of the test should be such that it can be performed by other individuals who are
assigned the testing task.

6. Traceable: The test should verify a specific requirement. The corresponding requirements for the different types
of test are derived from the associated development stages in the test vee model

7. Self-cleaning: The system should return to the pretest state after the test is complete.

11 / 29
 Debugging

Debugging
▶ Applying the functional decomposition paradigm should provide a clear idea of the inputs, outputs, and
behavior of the modules that are being built.

▶ Inevitably, there will come a point during the construction of a component when it will not function as
expected. This is commonly referred to as a bug.

▶ It requires the application of debugging skills to determine the root cause of the problem and correct it.

▶ Bohrbugs Bugs : Bohrbugs are named after the Bohr model of the atom that assumes that electrons have a
distinct position in space. Bohrbugs are reliable bugs, in which the error is always in the same place.

▶ Heisenbugs Bugs : Heisenbugs are named after the Heisenberg uncertainty principle, in which the position of
an electron is uncertain. Analogously, Heisenbugs may not always be reproducible with the same input.

12 / 29
 Debugging (cont...)

▶ In general, the debugging process is much the same as the scientific method.

▶ The steps of the debugging process are:

• Observe. Observe the problem under different operating conditions.

• Hypothesize. Form a hypothesis as to what the potential problem is.

• Experiment. Conduct experiments to confirm or eliminate the hypothesized source Of the problem.

• Repeat. Repeat until the problem is eliminated.

13 / 29
 Unit Testing

Unit Testing

▶ A unit test is a test of the functionality of a system the following pseudocode.

module in isolation

▶ Should be traceable to the detailed design.

▶ Consists of a set of test cases

▶ Each test case establish that a subsystem performs a

single unit of functionality to some specification.

▶ Test cases should be written with the express intent ▶ Clearly, we need to have at least two test cases, one
of uncovering undiscovered defects. for the "if’ clause and one for the "else" clause. In
addition, it would be a good idea to check the
▶ Let the operation of the module be represented by boundary conditions separating the if and else
clauses.

14 / 29
 Unit Testing (cont...)

▶ A processing path is a sequence of consecutive instructions or states encountered from the beginning to the
end of a computation process.

▶ Each processing path through the system represents a potential test case. The extent to which the test
cases cover all possible processing paths is called the test coverage.

▶ The ultimate in coverage is achieved by path complete coverage where every possible path has a test.

▶ Clearly documenting unit tests has added importance because the test cases are generally written by one
person or group and performed by a separate group.

▶ In order to organize the test cases they can be organized as matrices, step-by-step tests, or automated
scripts.

15 / 29
 Unit Testing (cont...)
Matrix Tests

▶ A matrix test is a test that is best suited to cases

where the inputs submitted are structurally the same
and differ only in their values.
▶ The test procedure is then "factored out," leaving a
list of inputs and their expected outputs.
▶ Let’s consider a test for the analog-to-digital
converter (ADC) that was used in the temperature
measuring system
▶ Assume that the ADC’s clock frequency is 10 kHz
and the input ranges from 0 to 5 V. The unit test
will consist of submitting different inputs to the
Figure 4: A matrix test for an analog-to-digitat
ADC and verifying the outputs. converter

▶ This test case exercises each bit of the ADC’s output independent of the other output bits. Other test cases
should examine extreme inputs as well as illegal inputs.
16 / 29
 Unit Testing (cont...)
Step-by-Step Tests

▶ A step-by-step test case is a prescription for

generating the test and checking the results.
▶ These descriptions are most effective when the test
consists of a complex sequence of steps.
▶ The test template for a step-by-step test has all the
information contained in the matrix test template.
▶ Recall the state diagram for the vending machine
that accepts nickels and dimes and dispenses candy
when a total of $0.25 (or more) is submitted.
▶ The state machine has different processing paths,
depending upon the combination and order of coins
deposited. Test cases can be written for each of the
processing paths through the system. Figure 5: A step-by-step test for a vending machine.

17 / 29
 Unit Testing (cont...)

Automated Test Scripts

▶ An automated test script is a sequence of commands provided to the Unit Test without user intervention.

▶ The outputs are usually automatically compared against the expected outputs to determine if the module
contains an error.

▶ Automated scripts are executed from a device referred to by many different names such as test harness, test
fixture, and test bench.

▶ While automated scripts carry a lot of up-front cost in terms of the time required putting them together,
they pay dividends when performing regression testing.

▶ Regression testing is the process of retesting a module after a modification in any related part of the system
to ensure that no errors were inadvertently introduced.

▶ Reducing the time spent on regression testing has a positive effect on the overall development time.

▶ The template for the matrix tests could be used to describe what an automated test script does.

18 / 29
 Integration Testing
Integration Testing
▶ After the individual subsystems have undergone their unit tests, they are then integrated into large
subcomponents leading eventually to the construction of the entire system.

▶ Integration testing checks that the major modules of the overall system operate correctly together.

▶ The test cases for integration testing must be traceable to the high-level design, and the test cases are
written on the basis of the characteristics of the design architecture.

▶ Test cases for integration can be derived from the following questions:
1. Have all the execution paths through the system been exercised?
2. Have all the modules been exercised at least once?
3. Have all the interface signals been tested?
4. Have all interface modes been exercised?
5. Does the system meet timing requirements?

▶ The integration tests themselves can be documented using either the matrix or step-by-step template
outlined for unit tests.

19 / 29
 Integration Testing

Integration Testing
▶ An acceptance test is a formal document stipulating the conditions under which the customer will accept
the system.

▶ It generally consists of a suite of test cases that exercise the systems according to the user’s environment.

▶ The test cases are constructed to ensure that the engineering requirements are met.

▶ The four attributes of a good requirement (abstract, unambiguous, traceable, and verifiable) are important
in building a good acceptance test.

▶ An unambiguous requirement will result in a test that everyone can agree on. A verifiable requirement sets
an objective pass/fail criterion on the acceptance test.

▶ Tests based on a traceable requirement imply they are directly assessing the needs of the project.

20 / 29
 Integration Testing (cont...)
▶ Integration Testing typically includes the following sections:
1. Testing Approach: The types, level, and methods employed to test the system.

2. Test Schedule: Start and end dates for the individual tests.

3. Problem Reporting: How the test results will be recorded.

4. Resource Requirements: The hardware, software, and people requirements needed to perňform the tests.

5. Test Environment: The set-up required to run the tests.

6. Test Equipment: Any special equipment or configurations required to run the test.

7. Post-delivery Tests: Tests performed on the deployed system.

8. Test Identification: Enumeration of test cases and their unique identifiers.

9. Corrective Action: What repairs must be made to the system in order to accept it.

21 / 29
 Case Study: Security Robot Design

Case Study: Security Robot Design

▶ Let’s consider the design of a security system that monitors an office complex looking for intruders.

▶ The team, along with the customer, developed a number of requirements, and from this we will focus on
two that address a fundamental navigational problem.
1. The robot’s center must stay within 12 to 18 centimeters of the wall over 90% of the course, while traveling
parallel to a wall over a 3 meter course.

2. The robot’s heading should never deviate more than 10 degrees from the wall’s axis while the robot travels
parallel to a straight wall over a 3 meter course.

▶ This case study explores test cases for the acceptance, integration, and unit testing related to these two
requirements.

22 / 29
 Case Study: Security Robot Design (cont...)
Acceptance Testing

▶ A number of tests would need to be built, and we

create a test only for the first engineering
requirement. A test could be performed by having
someone observe the robot moving along a wall and
mark (on the floor) whenever the robot strayed out
of bounds.
▶ Such a test would not easily be repeatable because
different people might judge what is meant by "out
of bounds" differently.
▶ A way to address these problems is to have the robot
monitor its own distance from the wall. This is done
in this case by a program written to monitor the Figure 6: A step-by-step acceptance test case for the
autonomous robot.
position of the robot over time and store these
values.

▶ It is clear that the test program must configure the robot to log the distance data while traversing the wall.

23 / 29
 Case Study: Security Robot Design (cont...)
Integration Testing
▶ A high-level design architecture that can meet the requirements.

Figure 7: Level 1 design architecture for the mobile robot

▶ The heart of the design is a microcontroller (MCU) that reads the sensor values, makes decisions, and
controls the speed of the two drive motors.

24 / 29
 Case Study: Security Robot Design (cont...)

▶ The robot moves and turns by adňjusting the relative speed of each motor, using a pulse-width-modulated
(PWM) signal from the MCU.

▶ Clearly, the interaction of the MCU with each of the compass, rangefinder, LCD, switches, and H-bridges
should be examined.

▶ There are many interactions between subsystems that could be tested during integration testing. A careful
examination of the system must be done to determine which combinations of subsystems are most likely to
create problems.
▶ Some Integration Test Possibilities
• MCU + motors + bridge + switches

• Chassis + digital compass + MCU + motors + bridge +LCD

• Chassis + range finder + MCU + motors + bridge

▶ A step-by-step integration test is created in Table

25 / 29
 Case Study: Security Robot Design (cont...)

Figure 8: A step-by-step integration test case for the compass and motors.

26 / 29
 Case Study: Security Robot Design (cont...)
Unit Testing

▶ Once the Level 1 architecture is developed and the

test cases written to ensure that the architecture is
capable of meeting the design requirements, the
design team moves on to selecting components to
use in the design.

▶ The design team must select the units so that the

resulting system can meet the engineering
requirements.

▶ A unit test for the digital compass component. The

functional design requirements for the unit are as Figure 9: The functional requirements for the digital
compass.
given

▶ The matrix test case in Table is constructed to configure the compass and then reads heading data from it.
This unit test looks for heading errors greater than 10 degrees.

27 / 29
Case Study: Security Robot Design (cont...)

Figure 10: Matrix unit test for the digital compass

28 / 29
 Guidance

Guidance
▶ Tests have a lifetime beyond the obvious need to check proper operation of the subsystems, their
integration, and the overall performance of the system.

▶ Reasons to develop and conduct tests

1. Testing reduces the number of bugs in existing and new features.
2. Tests are good documentation.
3. Tests improve design.
4. Tests allow you to refactor.
5. Tests constrain features.
6. Tests defend against other designers.
7. Testing is fun.
8. Testing forces you to slow down and think.
9. Testing makes development faster.
10. Tests reduce fear.

29 / 29