Software Testing

Dr. Sasmita Padhy

Software Testing
 Testing programs to establish the presence of
system defects.
 “Program testing can be used to show the
presence of bugs, but never to show their
absence!”
Edsger Wybe Dijkstra
 Software Testing is evaluation of the software
against requirements gathered from users and
system specifications.
 Testing is conducted at the phase level in software
development life cycle or at module level in
program code.
 Software testing comprises of Validation and
Software Verification

 Verification is the process of confirming if the software is

meeting the business requirements, and is developed
adhering to the proper specifications and methodologies.
 Verification ensures the product being developed is
according to design specifications.
 Verification answers the question– "Are we developing
this product by firmly following all design
specifications ?"
 Verifications concentrates on the design and system
specifications.
Software Validation
 Validation is process of examining whether or not
the software satisfies the user requirements. It is
carried out at the end of the SDLC. If the software
matches requirements for which it was made, it is
validated.
 Validation ensures the product under development
is as per the user requirements.
 Validation answers the question – "Are we
developing the product which attempts all that user
needs from this software ?".
 Validation emphasizes on user requirements.
Evolution phases of software testing
Debugging-oriented Phase
(Before 1957)
 This phase is the early period of testing.
 Programs were written and then tested by the
programmers until they were sure that all the bugs
were removed.
 The term used for testing was checkout, focused on
getting the system to run.
 Till 1956, there was no clear distinction between
software development, testing, and debugging.
Demonstration-oriented Phase
(1957–78)
 The term ‘debugging’ continued in this phase.
 However, in 1957, the scope of checkout of a
program increased from program runs to program
correctness[Charles Baker ].
 Moreover, the purpose of checkout was to show the
absence of errors.
 There was no stress on the test case design.
 In this phase, there was a misconception that the
software could be tested exhaustively.
Destruction-oriented Phase
(1979–82)
 Myers changed the view of testing from ‘testing is
to show the absence of errors’ to ‘testing is to find
more and more errors.’
 He separated debugging from testing and stressed
on the valuable test cases.
 This phase has given importance to effective testing
in comparison to exhaustive testing.
 The importance of early testing was also realized in
this phase.
Evaluation-oriented Phase
(1983–87)
 In 1983, guidelines by the National Bureau of
Standards were released and the concept of early
testing was realized in the form of verification and
validation activities
 This phase stresses on the quality of software
products such that it can be evaluated at every
stage of development.
Process-oriented Phase
(1996 onwards)
 Testing was established as a complete process rather
than a single phase SDLC.
 The testing process starts as soon as the
requirements for a project are specified and it runs
parallel to SDLC.
 Testing Maturity Model (TMM) for measuring the
performance of a testing process has also been
developed.
 Quantification of various parameters which decide the
performance of a testing process.
 Evolution of software testing
[Hung Q. Nguyen and Rob Pirozzi]
 The evolution of software testing was also discussed
by Hung Q. Nguyen and Rob Pirozzi in three phases,
namely Software Testing 1.0, Software Testing 2.0,
and Software Testing 3.0.
 According to this classification, the current state-of-
practice is Software Testing 3.0.
Software Testing 1.0

 In this phase, software testing was just considered a

single phase to be performed after coding of the
software in SDLC.
 No test organization was there.
 A few testing tools were present but their use was
limited due to high cost.
 Management was not concerned with testing, as
there was no quality goal.
Software Testing 2.0

 In this phase, software testing gained importance in

SDLC and the concept of early testing also started.
 Testing was evolving in the direction of planning the
test resources.
 Many testing tools were also available in this phase.
Software Testing 3.0

 In this phase, software testing is being evolved in

the form of a process which is based on strategic
effort.
 Moreover, it should be driven by quality goals so
that all controlling and monitoring activities can be
performed by the managers.
 Thus, the management is actively involved in this
phase
 Software Testing— Myths And
Facts

 Myth
 Testing is a single phase in SDLC .
 Truth
 It is a myth, at least in the academia, that software testing
is just a phase in SDLC and we perform testing only when
the running code of the module is ready.
 But in reality, testing starts as soon as we get the
requirement specifications for the software
 Myth
 Testing is easy.
 Truth
 This myth is more in the minds of students who have
started a career in testing.
 So the general perception is that, software testing is an
easy job, wherein test cases are executed with testing tools
only.
 But in reality, tools are there to automate the tasks and not
to carry out all testing activities.
 Myth
 Software development is worth more than testing.
 Truth
 This myth prevails in the minds of every team member and
even in fresher's who are seeking jobs. As a fresher, we
dream of a job as a developer.
 But testing has now become an established path for job-
seekers.
 Testing is a complete process like development.
 Myth
 Complete testing is possible.
 Truth
 Almost every person who has not experienced the process
of designing and executing the test cases manually feels
that complete testing is possible.
 But in reality, it is not possible to provide all the possible
inputs to test the software.
 Complete testing has been replaced with effective testing
(to select and run some select test cases such that severe
bugs are uncovered first
 Myth
 Testing starts after program development.
 Truth
 Who are not aware of testing as a process, feels
that testing cannot commence before coding.
 But, the tester performs testing at the end of every
phase of SDLC in the form of verification and plans
for the validation testing.
 Testing after coding is just a part of all the testing
activities.
 Myth
 The purpose of testing is to check the functionality of the
software.
 Truth
 Today, all the testing activities are driven by quality goals.
 But quality does not imply checking only the functionalities
of all the modules.
 There are various things related to quality of the software,
for which test cases must be executed.
Software Testing Definitions

 Myers
 Testing is the process of executing a program with the
intent of finding errors.
 Myers
 A successful test is one that uncovers an as-yet-
undiscovered error.
 W. Dijkstra
 Testing can show the presence of bugs but never their
absence.
 Software Testing Definitions
 James Bach
 Testing is a support function that helps developers look
good by finding their mistakes before anyone else does.
 E. Miller
 Program testing is a rapidly maturing area within software
engineering that is receiving increasing notice both by
computer science theoreticians and practitioners. Its
general aim is to affirm the quality of software systems by
systematically exercising the software in carefully
controlled circumstances.
Software Testing Definitions
 Cem Kaner
 Software testing is an empirical investigation conducted
to provide stakeholders with information about the
quality of the product or service under test, with respect
to the context in which it is intended to operate.
 Craig
 Testing is a concurrent lifecycle process of engineering,
using and maintaining testware (i.e. testing artifacts) in
order to measure and improve the quality of the
software being tested.
Software Testing Terminology

 Failure
 When the software is tested, failure is the first term being
used.
 It means the inability of a system or component to perform
a required function according to its specification.
 In other words, when results or behavior of the system
under test are different as compared to specified
expectations, then failure exists.
 Fault is a condition that in actual causes a system
to produce failure.
 A bug is the result of a coding error.
 A defect is a deviation from the requirements.
 A defect does not necessarily mean there is a bug
in the code, it could be a function that was not
implemented but defined in the requirements of
the software.
 Error Whenever a development team member
makes a mistake in any phase of SDLC, errors are
produced. It might be a typographical error, a
misleading of a specification.
 An error causes a bug and the bug in turn causes
failures.
 Test case Test case is a well-documented
procedure designed to test the functionality of a
feature in the system.
 A test case has an identity and is associated with
a program behavior.
 Test case ID is the identification number given to
each test case.
 Purpose defines why the case is being designed.
 Preconditions for running the inputs in a system
can be defined, if required, in a test case.
 Inputs should not be hypothetical. Actual inputs
must be provided, instead of general inputs.
 For example, if two integer numbers have to be
provided as input, then specifically mention them
as 23 and 56.
 Expected outputs are the outputs which should
be produced when there is no failure.
 Testware The documents created during testing
activities are known as testware. Testware are the
documents that a test engineer produces.
 An Incident is the symptom(s) associated with a
failure that alerts the user about the occurrence of a
failure.
 Test oracle is the means to judge the success or
failure of a test, i.e. to judge the correctness of the
system for some test.
 The simplest oracle is comparing actual results with
expected results by hand.
Manual Vs Automated Testing

 Testing can either be done manually or using an

automated testing tool:
 Manual - This testing is performed without taking
help of automated testing tools. The software tester
prepares test cases for different sections and levels
of the code, executes the tests and reports the result
to the manager. Manual testing is time and resource
consuming. The tester needs to confirm whether or
not right test cases are used. Major portion of testing
involves manual testing.
 Automated This testing is a testing procedure done
with aid of automated testing tools. The limitations
with manual testing can be overcome using
Software Testing Techniques
 Thetechnique which will meet both the
objectives of effective test case design, i.e.
 coverage of testing domain and
 detection of maximum number of bugs.
 Thetechnique used to design effective test
case is called Software Testing Technique.
 Thesetechniques can be categorized into
two parts:
 (a) static testing and
 (b) dynamic testing.
Static Testing

 It is a technique for assessing the structural

characteristics of source code, design specifications
or any notational representation that conforms to
well defined syntactic rules.
 It is called as static because we never execute the
code in this technique.
 For example, the structure of code is examined by
the teams but the code is not executed
Dynamic Testing
 All the methods that execute the code to test a
software are known as dynamic testing
techniques.
 In this technique, the code is run on a number of
inputs provided by the user and the
corresponding results are checked.
 This type of testing is further divided into two
parts:
 (a) black-box testing and
 (b) white-box testing.
Black-box testing
 This technique takes care of the inputs given to a
system and the output is received after processing
in the system.
 Black-box testing is not concerned with
 What is being processed in the system?
 How does the system perform these operations?
 It checks the functionality of the system only.
 It is also known as functional testing. It is used for
system testing under validation.
 Black-box testing attempts to find errors in
the following categories:
 To test the modules independently
 Totest the functional validity of the software so
that incorrect or missing functions can be
recognized
 To look for interface errors
 To
test the system behavior and check its
performance
 To test the maximum load or stress on the system
 Totest the software such that the user/customer
accepts the system within defined acceptable
limits
BOUNDARY VALUE ANALYSIS (BVA)
 BVA is considered a technique
that uncovers the bugs at the
boundary of input values.
 boundary means the maximum
or minimum value taken by the
input domain.
 For example,
 if A is an integer between 10 and
255,
 then boundary checking can be on
10(9,10,11) and on
255(256,255,254).
 Similarly, B is another integer
variable between 10 and 100,
 then boundary checking can be on
BOUNDARY VALUE CHECKING (BVC)
 Inthis method, the test cases are designed
by holding one variable at its extreme
value and other variables at their nominal
values in the input domain.
 The variable at its extreme value can be
selected at:
 (a) Minimum value (Min)
 (b) Value just above the minimum value (Min+ )
 (c) Maximum value (Max)
 (d) Value just below the maximum value (Max−)
Let us take the example of two variables, A and B.

It can be generalized that for n variables in a module,

4n + 1 test cases can be designed with boundary
value checking method.
ROBUSTNESS TESTING METHOD
 The idea of BVC can be extended such that
boundary values are exceeded as:
 A value just greater than the Maximum value
(Max+)
 „A value just less than Minimum value (Min−)
 When test cases are designed considering the
above points in addition to BVC, it is called
robustness testing.
 Add the following test cases to the list of 9 test
cases designed in BVC:

 It can be generalized that for n input variables in a

WORST-CASE TESTING METHOD
 This can again extend the concept of BVC by assuming more
than one variable on the boundary.
 It is called worst-case testing method.
 We can add the following test cases to the list of 9 test cases
designed in BVC as:

n
It can be generalized that for n input variables in a module, 5
test cases can be designed with worst-case testing.
Example
 A program reads an integer number within the range [1,100]
and determines whether it is a prime number or not.
 Design test cases for this program using BVC, robust testing,
and worst-case testing methods.
 Solution
 (a) Test cases using BVC Since there is one variable, the total
number of test cases will be 4n + 1 = 5.
 (b) Test cases using robust testing Since there is one variable, the total
number of test cases will be 6n + 1 = 7. The set of boundary
values is
shown below:

(c) Test cases using worst-case testing Since there is one variable, the
total number of test cases will be 5n = 5. Therefore, the number of test
cases will be same as BVC.
EQUIVALENCE CLASS TESTING
 Equivalence partitioning is a method for deriving test
cases wherein classes of input conditions called
equivalence classes are identified such that each
member of the class causes the same kind of
processing and output to occur.
 Equivalence partitioning method for designing test
cases has the following goals:
 Completeness Without executing all the test cases,
we strive to touch the completeness of testing
domain.
 Non-redundancy Thus, the goal of equivalence
partitioning method is to reduce these redundant test
cases.
 To use equivalence partitioning, one needs to perform two
steps:
 1. Identify equivalence classes
 2. Design test cases
 IDENTIFICATION OF EQUIVALENT CLASSES
 Two types of classes can always be identified as discussed
below:
 Valid equivalence classes: These classes consider valid inputs
to the program.
 Invalid equivalence classes: One must not be restricted to
valid inputs only. We should also consider invalid inputs that
will generate error conditions or unexpected behavior of the
program
Example
 A program reads three numbers, A, B, and C, with a
range [1, 50] and prints the largest number. Design
test cases for this program using equivalence class
testing technique.
 Solution 1. First we partition the domain of input as
valid input values and invalid values, getting the
following classes:
 We can derive another set of equivalence classes based on
some possibilities for three integers, A, B, and C. These are
given below:
White-Box Testing Techniques
 It is also known as glass-box testing, as everything
that is required to implement the software is visible.
 Developers use white-box testing techniques to test
their own design and code. This testing is also
known as structural or development testing.

 LOGIC COVERAGE CRITERIA

 Statement Coverage: Every statement must
execute at least once
 Decision or Branch Coverage: Branch coverage
states that each decision takes on all possible
outcomes (True or False) at least once.
Example

Statement Coverage:
Test case 1: x = y = n, where n is any
number
Test case 2: x = n, y = n1, where n and n1
are different numbers.
Test case 3: x > y
Test case 4: x < y
Decision or Branch Coverage:
Test case 1: x = y
Test case 2: x != y
Test case 3: x < y
Test case 4: x > y
 Condition Coverage: Condition coverage states that each
condition in a decision takes on all possible outcomes at
least once.
 while ((I <= 5) && (J < COUNT))
 Test case 1: I <= 5, J < COUNT
 Test case 2: I > 5, J > COUNT
 Decision/condition Coverage: Each condition in a decision
takes on all possible outcomes at least once, each decision
takes on all possible outcomes at least once, and each point
of entry is invoked at least once.
 if (A && B)
 Test case 1: A = True, B = True
 Test case 2: A = True, B = False
 Test case 3: A = False, B = True
 Test case 4: A = False, B = False
BASIS PATH TESTING
 The guidelines for effectiveness of path testing are
discussed below:
 Path testing is based on control structure of the
program for which flow graph is prepared.
 Choose enough paths in a program such that maximum
logic coverage is achieved.

 CONTROL FLOW GRAPH

 The control flow graph is a graphical representation of
control structure of a program.
 Flow graphs can be prepared as a directed graph.
A directed graph (V, E) consists of a set of vertices V
and a set of edges E that are ordered pairs of elements
of V.
Notations

 Node: It represents one or more procedural statements.

The nodes are denoted by a circle. These are numbered
or labeled.
 Edges or links: They represent the flow of control in a
program. This is denoted by an arrow on the edge. An
edge must terminate at a node.
 Decision node: A node with more than one arrow leaving
it is called a decision node.
 Junction node: A node with more than one arrow entering
it is called a junction.
 Regions: Areas bounded by edges and nodes are called
regions. When counting the regions, the area outside the
graph is also considered a region.
FLOW GRAPH NOTATIONS FOR DIFFERENT
PROGRAMMING CONSTRUCTS

 Using the above notations, a fl ow graph can be constructed. The flow

graph is also known as decision-to-decision-graph or DD graph.
 PATH TESTING TERMINOLOGY
 Path: A path through a program is a sequence of
instructions or statements that starts at an entry, junction,
or decision and ends at another, or possibly the same,
junction, decision, or exit.
 Segment: Paths consist of segments. The smallest
segment is a link, that is, a single process that lies
between two nodes
 Path segment: A path segment is a succession of
consecutive links that belongs to some path.
 Length of a path: The length of a path is measured by the
number of links in it and not by the number of instructions
or statements executed along the path.
 Independent path: An independent path is any path
through the graph that introduces at least one new set of
 CYCLOMATIC COMPLEXITY

 measure the complexity by considering

the number of paths in the control
graph of the program.
 there are six possible paths:
 acei, acgh, acfj, bdei, bdgh, bdfj.
 the number of independent paths is 4.
 For strongly connected graph, the
number of independent paths is given
by
 V(G) = e – n + 1
 V(G) = e – n + 2, the graph which is
not strongly connected
 Cyclomatic complexity number can be derived
through any of the following three formulae

 1. V(G) = e – n + 2p
 where e is number of edges, n is the number of nodes
in the graph, and p is number of components in the
whole graph.

 2. V(G) = d + p
 where d is the number of decision nodes in the graph.

 3. V(G) = number of regions in the graph

Example 1
Solution
(b) Cyclomatic complexity

(i) V(G) = e – n + 2 * p = 10 – 8 + 2

=4

(ii)V(G) = Number of predicate nodes

+1 = 3 (Nodes

B, C, and F ) + 1 = 4

(iii)V (G) = Number of regions = 4

Example 2
Example
Example
 Unit Testing

 Unit testing is undertaken after a module has been coded

and successfully reviewed. Unit testing (or module testing)
is the testing of different units (or modules) of a system in
isolation.
 In order to test a single module, a complete environment is
needed to provide all that is necessary for execution of the
module.
 That is, besides the module under test itself, the following
steps are needed in order to be able to test the module:
 The procedures belonging to other modules that the module under
test calls.
 Nonlocal data structures that the module accesses.
 A procedure to call the functions of the module under test with
appropriate parameters.
Unit Validation Testing
 Since unit is the smallest building block of the
software system, it is the first piece of system to be
validated.
 Unit tests ensure that the software meets at least a
baseline level of functionality prior to integration
and system testing.
 Though software is divided into modules but a
module is not an isolated entity.
 The module under consideration might be getting
some inputs from another module or the module is
calling some other module.
 It means that a module is not independent and
cannot be tested in isolation.
Driver Module
 Module where the required inputs for the
module under test are simulated for the
purpose of unit testing is known as a
driver module.
 The driver module may print or interpret
the results produced by the module under
testing.
Driver Module : Example
 Stubs Module
 The module under testing may also call
some other module which is not ready at the
time of testing.
 Therefore, these modules need to be
simulated for testing.
 Inmost cases, dummy modules instead of
actual modules, which are not ready, are
prepared for these subordinate modules.
 These dummy modules are called stubs.
Stubs Module: Example
 Module B under test needs to call module D and
module E. But they are not ready. So there must
be some skeletal structure in their place so that
they act as dummy modules in place of the actual
modules.
 Therefore, stubs are designed for module D and
module E, as shown in Fig.
Benefits of Designing Stubs and
Drivers
 Stubs allow the programmer to call a method in the
code being developed, even if the method does not
have the desired behavior yet.
 By using stubs and drivers effectively, we can cut
down our total debugging and testing time
Exercise
 (a) Suppose main() module is
not ready for the testing of
calsum() module. Design a
driver module for main().

 (b) Modules calavg() and

calper() are not ready when
called in main(). Design
stubs for these two modules.
Integration testing
 During integration testing, different modules of a system are
integrated in a planned manner using an integration plan. The
integration plan specifies the steps and the order in which
modules are combined to realize the full system. After each
integration step, the partially integrated system is tested.
 Integration test approaches
 There are four types of integration testing approaches. Any
one (or a mixture) of the following approaches can be used to
develop the integration test plan. Those approaches are the
following:
 • Big bang approach
 • Top-down approach
 • Bottom-up approach
 • Mixed-approach
Big-Bang Integration Testing
 It is the simplest integration testing approach,
where all the modules making up a system are
integrated in a single step.
 However, this technique is practicable only for very
small systems.
 The main problem with this approach is that once
an error is found during the integration testing, it is
very difficult to localize the error as the error may
potentially belong to any of the modules being
integrated.
 Therefore, debugging errors reported during big
bang integration testing are very expensive to fix.
Bottom-Up Integration Testing
 The primary purpose of testing each subsystem is to
test the interfaces among various modules making
up the subsystem. Both control and data interfaces
are tested. The test cases must be carefully chosen
to exercise the interfaces in all possible manners.
 A principal advantage of bottom-up integration
testing is that several disjoint subsystems can be
tested simultaneously. In a pure bottom-up testing
no stubs are required, only test-drivers are required.
 A disadvantage of bottom-up testing is the
complexity that occurs when the system is made up
of a large number of small subsystems.
Top-Down Integration Testing
 Top-down integration testing starts with the main routine
and one or two subordinate routines in the system. After
the top-level ‘skeleton’ has been tested, the immediately
subroutines of the ‘skeleton’ are combined with it and
tested.
 Top-down integration testing approach requires the use of
program stubs to simulate the effect of lower-level
routines that are called by the routines under test.
 A pure top-down integration does not require any driver
routines.
 A disadvantage of the top-down integration testing
approach is that in the absence of lower-level routines,
many times it may become difficult to exercise the top
level routines in the desired manner since the lower-level
Mixed Integration Testing

 A mixed (also called sandwiched) integration testing

follows a combination of top down and bottom-up
testing approaches.
 In top-down approach, testing can start only after
the top-level modules have been coded and unit
tested. Similarly, bottom-up testing can start only
after the bottom level modules are ready.
 The mixed approach overcomes this shortcoming of
the top-down and bottom-up approaches. In the
mixed testing approaches, testing can start as and
when modules become available.
 Therefore, this is one of the most commonly used
integration testing approaches.
System testing
 System tests are designed to validate a fully
developed system to assure that it meets its
requirements.
 There are essentially three main kinds of system
testing:
 Alpha Testing. Alpha testing refers to the system
testing carried out by the test team within the
developing organization.
 Beta testing. Beta testing is the system testing
performed by a select group of friendly customers.
 Acceptance Testing. Acceptance testing is the
system testing performed by the customer to
System Test: The Different Types

• Functional testing
• Performance
testing
• Stress testing
• Configuration
testing
 Functional Testing

 Functional tests at the system level are used to ensure

that the behavior of the system adheres to the
requirements specification.
 All types or classes of legal inputs must be accepted by
the software.
 All possible classes of system output must exercised and
examined.
 Configuration Testing
 Configuration testing allows developers/testers to
evaluate system performance and availability when
hardware exchanges and reconfigurations occur.
 Configuration testing also requires many resources
including the multiple hardware devices used for the
tests.
Performance Testing

 The goal of system performance tests is to see if the

software meets the performance requirements.
 Performance testing allows testers to tune the
system; that is, to optimize the allocation of system
resources.

 Stress Testing
 When a system is tested with a load that causes it
to allocate its resources in maximum amounts, this
is called stress testing.
 For example, if an operating system is required to
handle 10 interrupts/second and the load causes 20
interrupts/second, the system is being stressed.
 Security Testing
 Security testing evaluates system characteristics that
relate to the availability, integrity, and confidentially
of system data and services.
 Users/clients should be encouraged to make sure
their security needs are clearly known at
requirements time, so that security issues can be
addressed by designers and testers.
 Recovery Testing
 Recovery testing subjects a system to losses of
resources in order to determine if it can recover
properly from these losses.
 This type of testing is especially important for
transaction systems, for example, on-line banking
software.