What is Testing?

Many people understand many definitions of testing:

1. Testing is the process of demonstrating that errors are not present.

2. The purpose of testing is to show that a program performs its intended

functions correctly.
3. Testing is the process of establishing confidence that a program does
what it is supposed to do.

These definitions are incorrect.

 A more appropriate definition is
“Testing is the process of executing a program
with the intent of finding errors.”
Why should We Test?
Although software testing is itself an expensive activity,
yet launching of software without testing may lead to cost
potentially much higher than that of testing, specially in
systems where human safety is involved.

In the software life cycle, the earlier the errors are

discovered and removed, the lower is the cost of their
removal.
Who should Do the Testing?
o Testing requires the developers to find errors from their
software.
o It is difficult for software developer to point out errors from
own creations.
o Many organizations have made a distinction between
development and testing phase by making different people
responsible for each phase.
 What should We Test?
We should test the program’s responses to every possible input.

It means, we should test for all valid and invalid inputs.

Suppose a program requires two 8 bit integers as input. Total

possible combinations are 28x28. If only one second is required to
execute one set of inputs, it may take 18 hours to test all
combinations. Practically, inputs are more than two and size is
also more than 8 bits. We have also not considered invalid inputs
where so many combinations are possible. Hence, complete
testing is just not possible, although, we may wish to do so.
Software Testing
(example)

Fig. 1: Control flow graph

7
Software Testing (example)
The number of paths in the example of Fig. 1 are 1014 or
100 trillions. It is computed from 520 + 519 + 518 + …… + 51;
where 5 is the number of paths through the loop body. If
only 5 minutes are required to test one test path, it may
take approximately one billion years to execute every path.

8
 Some Terminologies
➢ Error, Mistake, Bug, Fault and Failure
• People make errors. A good synonym is mistake. This may be a syntax
error or misunderstanding of specifications. Sometimes, there are logical
errors.

• When developers make mistakes while coding, we call these mistakes “bugs”.

• A fault is the representation of an error, where representation is the mode of

expression, such as narrative text, data flow diagrams, ER diagrams,
source code etc. Defect is a good synonym for fault.

• A failure occurs when a fault executes. A particular fault may cause different
failures, depending on how it has been exercised. 9
 Test, Test Case and Test Suite
Test and Test case terms are used interchangeably. In practice,
both are same and are treated as synonyms. Test case describes an
input description and an expected output description.
Test Case ID

Section-I (Before Execution) Section-II (After Execution)

Purpose : Execution History:
Pre condition: (If any) Result:
Inputs: If fails, any possible reason (Optional);
Expected Outputs: Any other observation:
Post conditions: Any suggestion:
Written by: Run by:
Date: Date:
Fig. 2: Test case template 10
The set of test cases is called a test suite. Hence any combination of test cases may generate a test suite.
 Verification and Validation
Verification (Static) is the process of evaluating a system or
component to determine whether the products of a given
development phase satisfy the conditions imposed at the start of
that phase.
 Are we building the product right

Validation (Dynamic) is the process of evaluating a system or

component during or at the end of development process to
determine whether it satisfies the specified requirements .
 Are we building the right product
Testing= Verification+Validation 12
 Alpha, Beta and Acceptance Testing
The terms alpha and beta testing are used when the software is developed as a
product for anonymous customers.
Alpha Tests are conducted at the developer’s site by some potential customers.
These tests are conducted in a controlled environment. Alpha testing may be
started when formal testing process is near completion.

Beta Tests are conducted by the customers / end users at their sites. Unlike alpha
testing, developer is not present here. Beta testing is conducted in a real
environment that cannot be controlled by the developer.

The term Acceptance Testing is used when the software is developed for a
specific customer. A series of tests are conducted to enable the customer to
validate all requirements. These tests are conducted by the end user / customer
and may range from adhoc tests to well planned systematic series of tests.
Functional Testing (Black Box)

Output domain
Input domain
System
under test Output
Input test data test data

Fig. 3: Black box

testing

13
Boundary Value Analysis
Consider a program with two input variables x and y. These input
variables have specified boundaries as:
a≤x≤b
c≤y≤d

Input domain

d
y
c

a b
x
Fig.4: Input domain for program having two input
variables 14
Boundary Value Analysis…
The boundary value analysis test cases for our program with two inputs variables (x
and y) that may have any value from 100 to 300 are: (200,100), (200,101), (200,200),
(200,299), (200,300), (100,200), (101,200), (299,200) and (300,200).

This input domain is shown in Fig. 5. Each dot represent a test case and inner rectangle is the
domain of legitimate inputs. Thus, for a program of n variables, boundary value analysis yield 4n + 1
test cases.
Input domain
400
300

y
100
200

0 100 200 400

300
x 15

Fig. 5: Input domain of two variables x and y with boundaries [100,300] each
Robustness testing
It is nothing but the extension of boundary value analysis. Here, we would
like to see, what happens when the extreme values are exceeded with a
value slightly greater than the maximum, and a value slightly less
than minimum. It means, we want to go outside the legitimate boundary of
input domain. This extended form of boundary value analysis is called
robustness testing and shown in Fig. 6

There are four additional test cases which are outside the legitimate input
domain. Hence total test cases in robustness testing are 6n+1, where n is
the number of input variables. So, 13 test cases are:

(200,99), (200,100), (200,101), (200,200), (200,299), (200,300), (200,301),

(99,200), (100,200), (101,200), (299,200), (300,200), (301,200)
24
Robustness testing …

400
300
y 200
100

0 100 200 300 400

x

Fig. 6: Robustness test cases for two variables x and y with range [100,300] each

25
 Worst-case testing
• If we reject “single fault” assumption theory of reliability and may like to
see what happens when more than one variable has an extreme value.
• In electronic circuit analysis, this is called “worst case analysis”.
• It is more thorough in the sense that boundary value test cases are a
proper subset of worst case test cases.
• It requires more effort. Worst case testing for a function of n variables
generate 5n test cases as opposed to 4n+1 test cases for boundary
value analysis.
• Two variables example will have 52=25 test cases. 26
 Equivalence Class Testing
In this method, input domain of a program is partitioned into a finite number of
equivalence classes such that one can reasonably assume, that the test of a
representative value of each class is equivalent to a test of any other value.
Two steps are required for implementing this method:
1. The equivalence classes are identified by taking each input condition and
partitioning it into valid and invalid classes. For example, if an input condition
specifies a range of values from 1 to 999, we identify one valid equivalence class
[1<item<999]; and two invalid equivalence classes [item<1] and [item>999].

2. Generate the test cases using the equivalence classes identified in the previous
step. This is performed by writing test cases covering all the valid equivalence
classes. Then a test case is written for each invalid equivalence class so that no
test contains more than one invalid class. This is to ensure that no two invalid
classes mask each other.
61
Equivalence Partitioning

Invalid inputs

Valid System Outputs

inputs under test

Input domain
Output domain

Fig. 7: Equivalence partitioning

Most of the time, equivalence class testing defines classes of the input
domain. However, equivalence classes should also be defined for
output domain. Hence, we should design equivalence classes
based on input and output domain. 62
Cause Effect Graphing Technique
▪ Consider single input conditions
▪ do not explore combinations of input circumstances
Steps
1. Causes & effects in the specifications are identified.
A cause is a distinct input condition or an equivalence class of input conditions.

An effect is an output condition or a system transformation.

2. The semantic content of the specification is analyzed and transformed into a Boolean
graph linking the causes & effects.

3. Constraints are imposed

4. graph – limited entry decision table
Each column in the table represent a test case.
5. The columns in the decision table are converted into test cases.
Cause Effect Graphing Technique
The basic notation for the graph is shown in fig. 8

93
Fig.8: Basic cause effect graph symbols
 Cause Effect Graphing Technique
Myers explained this effectively with following example. “The characters in column 1 must
be an A or B. The character in column 2 must be a digit. In this situation, the file update is
made. If the character in column 1 is incorrect, message x is issued. If character in column
2 is not a digit, message y is issued”.
The causes are
c1: character in column 1 is A

c2: character in column 1 is 3

c3: character in column 2 is not a digit

and the effects are

e1: update made
e2: message x is issued
e3: message y is issued 94
Sample cause effect graph

95
Fig. 9: Sample cause effect graph
 Structural Testing
A complementary approach to functional testing is called structural / white box
testing. It permits us to examine the internal structure of the program.

Path Testing
Path testing is the name given to a group of test techniques based on
judiciously selecting a set of test paths through the program. If the set of paths is
properly chosen, then it means that we have achieved some measure of test
thoroughness.

This type of testing involves:

1. generating a set of paths that will cover every branch in the program.

2. finding a set of test cases that will execute every path in the set of program
paths.
Flow Graph
The control flow of a program can be analysed using a graphical
representation known as flow graph. The flow graph is a directed graph
in which nodes are either entire statements or fragments of a statement,
and edges represents flow of control.

Fig. 14: The basic construct of the flow graph

Fig. 16: Flow graph example
Mutation Testing
Mutation testing is a fault based technique that is similar to fault seeding, except that
mutations to program statements are made in order to determine properties about
test cases. It is basically a fault simulation technique.

Multiple copies of a program are made, and each copy is altered; this altered copy is
called a mutant. Mutants are executed with test data to determine whether the test
data are capable of detecting the change between the original program and
the mutated program.

A mutant that is detected by a test case is termed “killed” and the goal of mutation
procedure is to find a set of test cases that are able to kill groups of mutant
programs.

167
 Levels of Testing
There are 3 levels of testing:
i. Unit Testing
ii. Integration Testing
iii. System Testing

170
 Unit Testing
There are number of reasons in support of unit testing than testing the entire
product.

1. The size of a single module is small enough that we can

locate an error fairly easily.

2. The module is small enough that we can attempt to test

it in some demonstrably exhaustive fashion.

3. Confusing interactions of multiple errors in widely different

parts of the software are eliminated.
171
 Unit Testing…
There are problems associated with testing a module in isolation. How do
we run a module without anything to call it, to be called by it or,
possibly, to output intermediate values obtained during execution? One
approach is to construct an appropriate driver routine to call if and, simple
stubs to be called by it, and to insert output statements in it.

Stubs serve to replace modules that are subordinate to (called by) the
module to be tested. A stub or dummy subprogram uses the subordinate
module’s interface, may do minimal data manipulation, prints verification of
entry, and returns.

This overhead code, called scaffolding represents effort that is imported

for testing, but does not appear in the delivered product as shown in Fig. 29.
Scaffolding required for testing a program unit

173
Fig. 29 : Scaffolding required for testing a program unit (module)
 Integration Testing
• The purpose of unit testing is to determine that each
independent module is correctly implemented.

• This gives little chance to determine that the interface

between modules is also correct, and for this reason
integration testing must be performed.

• One specific target of integration testing is the interface:

whether parameters match on both sides as to type,
permissible ranges, meaning and utilization.
Integration approaches

Fig. 30 : Three different integration

approaches
 System Testing
• Of the three levels of testing, the system level is closest to everyday
experiences. We test many things; a used car before we buy it, an
on-line cable network service before we subscribe, and so on.
• A common pattern in these familiar forms is that we evaluate a product in
terms of our expectations; not with respect to a specification or a standard.
• Consequently, goal is not to find faults, but to demonstrate performance.
Because of this we tend to approach system testing from a functional
standpoint rather than from a structural one.
• Since it is so intuitively familiar, system testing in practice tends to be less
formal than it might be, and is compounded by the reduced testing interval
that usually remains before a delivery deadline.

Petschenik gives guidelines for choosing test cases during system testing.
 System Testing…
• During system testing, we should evaluate a number of attributes of the software that
are vital to the user as given below.
• These represent the operational correctness of the product and may be part of the
software specifications.

Usable Is the product convenient, clear, and predictable?

Secure Is access to sensitive data restricted to those with authorization?

Will the product work correctly in conjunction with existing data,
Compatible
software, and procedures?
Do adequate safeguards against failure and methods for recovery exist
Dependable
in the product?

Documented Are manuals complete, correct, and understandable?

Attributes of software to be tested during system testing

 Validation Testing
• It refers to test the software as a complete product.

• This should be done after unit & integration testing.

• Alpha, beta & acceptance testing are nothing but the various ways of
involving customer during testing.

• IEEE has developed a standard (IEEE standard 1059-1993) entitled “IEEE

guide for software verification and validation“ to provide specific
guidance about planning and documenting the tasks required by the
standard so that the customer may write an effective plan.

• Validation testing improves the quality of software product in terms of

functional capabilities and quality attributes.
 The Art of Debugging
The goal of testing is to identify errors (bugs) in the program.
The process of testing generates symptoms, and a program’s
failure is a clear symptom of the presence of an error.
After getting a symptom, we begin to investigate the cause and
place of that error.
After identification of place, we examine that portion to
identify the cause of the problem. This process is called
debugging.
180
 Debugging Techniques Characteristics of Bugs
1. “The symptom and the cause may be geographically remote. That is, the symptom
may appear in one part of a program, while the cause may actually be located in other
part. Highly coupled program structures may complicate this situation.

2. The symptom may disappear (temporarily) when another error is corrected.

3. The symptom may actually be caused by non errors (e.g. round off inaccuracies).
4. The symptom may be caused by a human error that is not easily traced.
5. The symptom may be a result of timing problems rather than processing problems.
6. It may be difficult to accurately reproduce input conditions (e.g. a real time application in
which input ordering is indeterminate).
7. The symptom may be intermittent. This is particularly common in embedded system that
couple hardware with software inextricably.
8. The symptom may be due to causes that are distributed across a number of tasks running on
different processors”. 181
 Software Debugging
Induction approach

➢ Locate the pertinent data

➢ Organize the data

➢ Devise a hypothesis

➢ Prove the hypothesis

182
 Software Debugging

Fig. 32 : The inductive debugging

process
 Software Debugging
Deduction approach

➢ Enumerate the possible causes or hypotheses

➢ Use the data to eliminate possible causes

➢ Refine the remaining hypothesis

➢ Prove the remaining hypothesis

184
 Software Debugging

Fig. 33 : The deductive debugging process

 Testing Tools
One way to improve the quality & quantity of testing is
to make the process as pleasant as possible for the
tester. This means that tools should be as concise,
powerful & natural as possible.

The two broad categories of software testing tools are :

➢ Static

➢ Dynamic
 Software Testing Tools
1. Static analyzers, which examine programs systematically and automatically.
2. Code inspectors, who inspect programs automatically to make sure they
adhere to minimum quality standards.
3. Standards enforcers, which impose simple rules on the developer.
4. Coverage analysers, which measure the extent of coverage.
5. Output comparators, used to determine whether the output in a program is
appropriate or not.
6. Test file/ data generators, used to set up test inputs.
7. Test harnesses, used to simplify test operations.
8. Test archiving systems, used to provide documentation about programs.