Software Testing

1
Validation & Verification (V & V)
Validation Verification

• Validation & verification is the name given to the

checking & analysis processes that ensure that
software conforms to it’s specification and
meets the needs of the customers who are
paying for the software.
• This is a whole lifecycle process. It starts with
requirements reviews and continues through
design reviews and code inspections to product
testing. There should be V & V activities at each
stage of software process.
• Validation
– Are we building the right product?
• Verification
– Are we building the product right?
2
Validation & Verification - further
• Verification involves checking that the software
conforms to it’s specification. It is to be checked
that the system meets it’s specified functional
and non-functional requirements.
• Validation is a more general process. It is to be
ensured that the software meets the
expectations of the customer. It goes beyond
checking conformance of the system to it’s
specification to showing that the software does
what the customer expects as distinct from what
has been specified. 3
Validation & Verification - Illustration
Software
specification

Verification

Actual
Customer
Performance of
expectations
software

Validation

4
 Validation & Verification
• Early validation of the system requirements is
very important. It is easy to make errors &
omissions in the system’s requirements and in
such cases the final software will not meet it’s
customers expectations.
• But, requirements validation is unlikely to
discover all requirements problems. Some faults
in requirements can sometimes only be
discovered when the system implementation is
complete.
5
 Static and dynamic V & V
• Within the V & V process, two techniques of system
checking & analysis may be used.
• Software inspection
– Analyze & check, system representations such as the
requirements document, design diagrams and the program
source code.
– This is to be applied at all stages of the process. Inspections
may be supplemented by associated documents.
– Software inspection is a static V & V technique as it does not
require the system to be executed.
• Software testing
– Involves executing an implementation of the software with test
data and examining the outputs of the software and it’s
operational behavior to check that it is performing as required.
– Testing is a dynamic V & V technique , because it works with
an executable representation of the system.
6
 Software inspection & testing

• Next slide shows the software inspection

and testing in the software process. Arrows
are pointed to the stages in the process
where inspection & testing may be used.
• Inspections can be used at all stages of
the software process.
• But testing can only be used when a
prototype or an executable program is
available.
7
 Software inspection & testing
Software
Software
inspections
inspections

Requirements
Requirements High-level
High-level Formal
Formal Detailed
Detailed Program
Program
specification
specification design
design specification
specification design
design

Program
Program
Prototype testing
testing

8
 Software inspection
• Software inspections do not require the program to be
executed, so may be used as a verification technique
before programs are implemented.
• During an inspection, a representation of the system is
examined. Eg: System model, language source code, a
requirement document etc. Errors can be considered in
isolation, without being concerned about how it will affect
the behavior of the system.
• Inspections have proved to be an effective technique of
error detection. Errors can be found more cheaply
through inspection than by program testing.
• In other words, reviews & inspections are usually more
effective and less expensive than dynamic testing for
discovering defects.
9
 Why inspection is more effective
than dynamic testing?
1. Many different defects may be detected in a
single inspection. In testing, it can often detect
only one error per test, because defects can
cause the program to crash or interfere with
other program defects.
2. Inspections use domain & programming
knowledge. The reviewers are likely to have
seen the types of errors that commonly occur
in particular type of applications. They can
therefore focus on these error types.
10
 Inspection & testing
• But inspections cannot replace software testing.
Inspections of the software can check conformance
(match up) with a specification but they cannot validate
dynamic behavior.
• It is often impractical to inspect a complete system that is
integrated from a number of different sub-systems.
Testing is the only possible V & V technique at the
system level. Testing is also necessary for reliability
assessment, performance analysis, user interface
validation and to check that the software requirements
are what the user really wants.
• Inspection & testing are not competing (rival) V & V
techniques. Inspection plays one part and testing plays
another part in V & V. They have their own pluses &
minuses.
11
 Program inspections
• Software inspections may be applied to
requirements specification, high-level design,
detailed design definitions, programs etc.
• When the program source code is inspected for
defects it is called as program inspections.
• Two main techniques in program inspections.
– Reviews
• The author distributes the code lists to team members. At a
review meeting improvements are suggested.
– Walkthroughs
• The author explains the code to other team members. They
examine the code and suggest improvements.
12
 Program inspection
• Software inspection includes program
inspections, but static techniques can only check
the correspondence between a program & it’s
specification (a static verification); they cannot
demonstrate that the software is operationally
useful.
• Nor can they check non-functional
characteristics of the software such as
performance. (to check performance, it is
needed to run the program; which indicates it is
not static).
• Therefore some testing, is always required to
validate a software system. 13
 Software testing

• Large systems are built out of sub-systems,

which are built out of modules, which are
composed of procedures & functions. The
testing process should therefore proceed in
stages where testing is carried out incrementally
in conjunction with system implementation.
• Next slide shows a five stage testing process
where system components are tested, the
integrated system is tested and finally the
system is tested with the customer’s data.
14
 Software testing process (what
parts are tested) - Illustration
Unit
testing

Module
testing

Component testing
Sub-system
testing

System
testing

Integration testing Acceptance

testing 15
User testing
Software testing process -Overview
• Ideally, component defects are discovered early
in the process.
• Interface problems are discovered when the
system is integrated.
• But as defects are discovered, the program must
be debugged and this may require other stages
in the testing process to be repeated. Eg: errors
in program components may come to light
during integration testing. The process is
therefore an iterative one with information being
fed back from later stages to earlier parts of the
process.
16
 Software testing procedure- Overview
• Testing procedures should be established at the start of
any software project. All testing carried out should be
based on a test plan, which should detail which tests are
to be carried out. For each test, the following information
should be included in the test plan.
– The pre-requisites for the tests
– The steps required to carryout the tests
– The expected results of the test
• The outcome of any tests should be recorded in a test
results document that include whether the test
succeeded or failed and a description of the failure.
• Test results for all passes through the test plan must be
recorded to allow accurate records to be kept of where
problems occur and when they were identified and
corrected.
17
Software testing process- Overview
1. Run the tests as defined by the test plan.
2. Testing should not stop when the first problem is
encountered, unless it is so severe that the rest of
tests would be meaningless. Rather all testing in the
test plan should be carried out and then the errors
addressed.
3. Record the outcome of each test in the test report,
both success & failure should be reported. For failed
tests, the nature of the problem should be described in
sufficient detail to allow it to be corrected and to allow
analysis of the types of errors being found.
4. Correct the errors that were documented from the test
run.
5. Repeat the process until no errors are identified or
error rate is sufficiently low. If the error rate is low then
it may be sufficient to simply re-test the failed errors. If
the error rate is high then all tests should be re-run. 18
Software testing process- by stages
• Unit testing
– Individual components are developed & tested to
ensure that they operate correctly. Each component is
tested independently, without other system
components. This phase is based on the design of the
software for a piece of code. Unit testing should prove
the following about the code.
– Robustness : code should not fail under any
circumstance
– Functionally correct :code should carry out the task
defined by the code design
– Correct interface : inputs & outputs from the code
should be as defined in the design
• Unit test plan must be based on the design of
the code and not the code itself. Therefore the
test plan will be written after the completion of
the design but before the start of the coding . 19
Software testing process- by stages
• Module testing
– A module is a collection of dependent components
such as object classes or some collection of
procedures & functions. Modules can be tested
separately because they encapsulate related
components.
• Sub-system testing
– This phase involves testing collections of modules
which have been integrated into sub-systems. The
most common problems which are in large software
systems are mismatch of interfaces. The sub-system
test process should therefore concentrate on the
detection of module interface errors.
More details on slide 45: Integration testing 20
Software testing process- by stages
• System testing
– The sub-systems are integrated to make up the system. This
process is concerned with finding errors that result from
unexpected interactions between sub-systems and sub-system
interface problems. The purpose of this testing is to prove that
the software meets the agreed user requirements (both
functional & non-functional) and works in the target environment.
– System test plan covers functional & non-functional
requirements such as performance.
– System test plan will also include tests to cover
• Recovery
– Force the system to crash & then try to recover to a sensible state.
• Security
– Attempt to access the system without correct authority.
• Stress
– Attempts to break the system by overloading it.
• Performance
– Ensure the system meets the performance requirement.
21
 Stress testing
• Once a system has been completely integrated it is
possible to test the system for properties such as
performance & reliability.
• Performance tests have to be designed to ensure that
the system can process it’s intended load. This usually
involves planning a series of tests where the load is
steadily increased until the system performance
becomes unacceptable.
• Some systems are designed to handle a specified load.
Eg: a transaction processing system may be designed to
process up to 100 transactions per second; an operating
system may be designed to handle up to 200 separate
terminals.
• Stress testing continues these tests beyond the
maximum design load of the system until the system
fails.
22
Software testing process- by stages
• Acceptance testing
– This is the final stage in the testing process before the
system is accepted for operational use. The system is
tested with data supplied by the customer rather
than simulated test data. Testing is done at the
customer’s site with the customer in attendance.
– Acceptance testing may reveal errors & omissions in
the system requirements definition, because the real
data exercise the system in different ways from the
test data.
– Acceptance testing may also reveal requirements
problems where the system’s facilities do not really
meet the user’s needs or the system performance is
unacceptable.
23
Variants of acceptance testing - Alpha
& Beta
• When a software is built for one customer, a
series of acceptance tests could be conducted
by end-users to enable the customer to validate
all requirements.
• But if software is developed as a product to be
used by many customers, it is impractical to
perform formal acceptance tests with each one.
• Software product builders use Alpha & Beta
testing to uncover errors that only the end-users
are able to find.
24
 Alpha & Beta tests
• Alpha test
– Conducted at the developer’s site by a customer. The
software is used in a natural setting with the
developer “looking over the shoulder” of the user &
record errors and usage problems. Alpha tests are
conducted in a controlled environment.
• Beta test
– Conducted at one or more customer sites by the end-
user of the software. The developer is not present.
The beta test is a live application of the software
in an environment that cannot be controlled by
the developer. The customer records all problems
that are encountered and reports to the developer at
regular intervals. After this feedback, the system is
modified & released either for further beta testing or
for general sale. (Have you heard about beta
versions of software?) 25
 Who will test?
• Unit & module testing are usually the
responsibility of the programmers who develop
the component. Programmers make up their
own test data and incrementally test the code as
it is developed. The programmer knows the
component best and is therefore the best person
to generate test data.
• Later stages of testing involve integrating work
from a number of programmers & must be
planned in advance. An independent team of
testers should work from pre-formulated test
plans which are developed from the system
specification and design.
26
Test plans and their links between
testing & development activities
Requirements
Requirements System
System System
System Detailed
Detailed
specification
specification specification
specification design
design design
design

Module
Module&&unit
unit
System
System Sub-system
Sub-system Code
Acceptance
Acceptance Code
Integration
Integration Integration
Integration &&
Test
Testplan
plan Test
Testplan
plan Test
Testplan
plan test
test

Acceptance
Acceptance System
System Sub-system
Sub-system
Service test Integration
Service test Integrationtest
test Integration
Integrationtest
test

27
 Software testing - Details
• Software testing process started with the testing
of individual program units such as functions or
objects.
• These were then integrated into sub-systems &
systems and the interactions of these units were
tested.
• Finally after completion of the system, the
customer may carry out a series of acceptance
tests to check that the system performs as
specified.
Component Integration
testing testing
Software developer Independent testing team 28
Component & integration testing
• The component testing phase is concerned with
testing the functioning of clearly identifiable
components. These may be functions or groups
of methods collected together into a module or
object.
• During integration testing, these components are
integrated to form sub-systems or the complete
system. At this phase, testing should focus on
interactions between the components and on the
functionality & performance of the system as a
whole.
• Defects in components, that have been missed
during earlier testing are discovered during
integration testing.
29
Component & integration testing
• Integration testing must be based on a
written system specification. This can be a
detailed system requirements specification
or a user-oriented specification of the
features that should be implemented in the
system.
• A separate team, who use the user &
system requirements documents to
develop detailed integration test plans is
responsible for integration testing.
30
 Defect testing
• The objective of defect testing is to expose
hidden defects in a software system
before the system is delivered.
• A successful defect test is a test, which
causes a system to perform incorrectly
and hence exposes a defect.
• Defect testing demonstrates the
presence, not the absence of program
faults.
31
 Defect testing process

Test
Test Test
Test Test Test
cases data Test Test
cases data results
results report
report

Design
Designtest
test Prepare
Preparetest
test Run
Runprogram
program Compare
Compareresults
results
cases
cases data
data with
with testdata
test data To test cases
To test cases

• Test cases
Specification of the inputs to the test & the expected output from the system
and a statement of what is being tested.
• Test data
Inputs which have been devised to test the system.
32
 Defect testing
• Full testing, where every possible program
execution sequence is tested is impractical.
Therefore it is to be based on a subset of
possible test cases.
• That subset should be selected based on the
organizational policies. Alternatively, policies
may be based on features of the operational
system. Eg:
– All system functions that are accessed through menus
should be tested.
– Where user input is provided, all functions must be
tested with both correct & incorrect input.
33
 Black-box testing
• This is an approach to testing where the test
cases are derived from system specifications.
• The system is a ‘Black-box’, whose behavior can
only be determined by studying it’s inputs and
related outputs.
• This test concerns only about the functionality
and not the implementation of the software.
• The key problem is to select inputs, which are
likely to reveal defects.( In the subset I e of the
diagram) Test engineers use their previous
experience & domain knowledge to identify such
test cases.
34
Black-box testing
Inputs causing
Input
Inputtest
testdata
data Ie abnormal
behavior

System is a black
box
System
System

Outputs which
reveal the
presence of
Output test
results Oe defects

35
 Equivalence Partitioning
• The input data to a program usually fall into a
number of different groups. Eg: positive
numbers, negative numbers.
• Programs normally behave in a comparable way
for all members within a group. Because of this
equivalent behavior, these groups are
sometimes called equivalence partitions.
• One systematic approach to defect testing is
based on identifying all equivalence partitions
which must be handled by a program. Test
cases are designed so that the inputs or outputs
lie within these partitions. 36
 Equivalence Partitioning
• Eg: If input is a 5 digit integer between 10,000 and 99,999
equivalence partitions are
<10000, 10000-99999 , >99999
• Once the set of partitions are identified, test cases should be
chosen from each of these partitions. A good guideline is,

– Select test cases on the boundaries of the partitions

+
– Test cases close to the mid-point of the partition

• The rationale for this rule is

– programmers tend to consider usual values of inputs when developing a
system. (close to mid-points are selected because of this).
– Boundary values are often uncommon, so are overlooked by
developers. Program errors often occur when processing these
uncommon values.

• The equivalence partitions may be identified, by using the program

specifications or based on the experience of tester.
37
 Equivalence Partitioning
Possible test inputs

3 4 7 10 11

Less
Lessthan
than44 Between
Between44and
and10
10 More
Morethan
than10
10

Input values Possible test inputs

9999 10000 50000 99999 100000

Less
Lessthan
than10,000
10,000 Between
Between10,000
10,000and
and99,999
99,999 More
Morethan
than99,999
99,999

Input values
38
 Glass-box testing
• This is sometimes called as ‘structural testing’ or
‘white-box’ testing or ‘clear-box’ testing.
• This is an approach to testing, where the tests
are derived from the knowledge of the software’s
structure and implementation.
• This is usually applied to relatively small
program units or the operations associated with
an object.
• The tester can analyze the code and use
knowledge about the structure of a component
to derive test data. The analysis of the code can
be used to find out how many test cases are
needed to guarantee that all of the statements
in the program are executed at least once
during the testing process. 39
 Glass-box testing

Test
Testdata
data

Tests
Derives

Component
code Test
Testoutputs
outputs

40
 Path testing
• Path testing is a glass-box testing strategy/
technique whose objective is to exercise every
independent execution path through a program.
If every independent path is executed then all
statements must have been executed at least
once.
• All conditional statements are tested for both
true & false cases.
• In an object oriented development process, path
testing may be used when testing methods
associated with objects.
41
 Path testing
• The number of paths through a program is
proportional to it’s size. As modules are
integrated into systems, it becomes difficult to
use path testing techniques.
• Path testing techniques are mostly used at the
unit testing and module testing phases of the
testing process.
• The starting point for path testing is a program
flow graph. This is a skeletal model of all paths
through the program. A flow graph consists of
nodes representing decisions and edges
showing flow of control.
• The flow graph is constructed by replacing
program control statements by equivalent
diagrams.
42
 Path testing
Flow chart Flow graph

1 1

2
2

3 4
3 4

5
6

43
 Path testing
• Sequential statements can be ignored in the flow-graph
construction. (assignments, I/O statements).
• Each branch in a conditional statement (if-then-else OR
case) is shown as a separate path and loops are
indicated by an arrow looping back to the loop condition
node.
• The objective of structural testing is to exercise all
program statements by executing each independent
program path at least once. Therefore all path
combinations are not needed to consider.
• An independent program path is one which traverses at
least one new edge in the flow graph. This means
exercising one or more new conditions.
• Both the true & false branches of all conditions must be
executed.
44
 Path testing
Sequence If=then-else While loop Until loop Case selection

1 1 1 1 1

R1 R2 R1 R1 R1 R2 R3
2 R1 3 2 3 4 5
2 2
2 R2
4 R2 R4
6

R – Region bounded by edges & nodes

45
 Path testing
• If all of these paths are executed,
– Every statement has been executed at least once
– Every branch has been exercised for true & false conditions

• The number of independent paths in a program can be

discovered by computing the ‘Cyclomatic complexity’ (CC) of the
program flow graph. It can be computed by using following
alternative formulas.
1. CC=Number(edges) -Number(nodes)+2
Does not 2. CC= Number of regions of the flow graph bounded by edges & nodes;
work for when counting regions, area outside of the graph is also considered as a
case region
selection 3. CC=Number of predicate nodes +1

• CC indicates the number of independent paths of the program.

• The next step is to design test cases to execute each of these
paths. The minimum number of test cases required to test all
program paths is equal to the CC.
• There are testing tools, which reveal which parts of the program
have and have not been tested using test cases.
46
 Integration testing
• Integration testing is carried out after the
separate software modules have been
unit. Integration testing is based on the
functional specifications of the software.
Integration testing should prove the
following.
– Integration
• The modules of the system should interact as
designed.
– Functionally correct
• System should behave as defined in the functional
specification
47
 Incremental integration testing
• Initially integrate a minimal system configuration
and test the system. Then add components to
this minimal configuration and test after each
added increment.
• T1, T2 & T3 tests are carried out after integration
of components A & B. If the tests are successful,
add C & carry out tests T1, T2, T3 & new test
T4. If new errors are introduced, that is due to
the integration of component C. Before adding
another component, these errors are to be
removed.
• After removing these errors, component D
should be added and all existing tests T1, T2,
T3, T4 & new test T5 should be performed in the
same way. 48
 Incremental integration testing -
Illustration
T! T!
T! A A
A

T2 T2 T2

B B
B T3
T3 T3

C C
T4
T4

D
T5

49
 Regression testing
• Carried out after any changes are made to
a software application.
• The purpose of regression test is to prove
that the change has been made correctly
and that the change has not introduced
any new errors.
• Regression test plans are usually a sub-
set of the integration or system test plans.
It is designed to cover enough functionality
to give confidence that the change has not
affected any other part of the system. 50
 Top-down integration testing
• In top-down integration, the high level components of a
system are integrated and tested, before design &
implementation of some of the low level components.

• Top-down testing is an integral part of a top-down

development process where the development process
starts with high-level components & works down the
component hierarchy. The program is represented as a
single abstract component with sub-components.

• After the top-level component has been programmed &

tested, it’s sub-components are implemented and tested
in the same way. This process continues until the
bottom-level components are implemented. The whole
system has then been completely tested. 51
 Bottom-up integration testing
• Low level components are integrated &
tested before the higher level components
have been developed.
• Bottom-up testing involves integrating &
testing the modules at the lower levels in
the hierarchy and then working up the
hierarchy of modules until the final module
is tested.
• This approach does not require the
architectural design of the system to be
complete.
52
 Test phase Test plan Author Technique Run by

Unit test Code design Designer Black box Programmer

Glass box
Integration test Functional Author of Black box Programming
specification specification Glass box team
Top-down
Bottom-up

System test Requirements Analyst Black box System test

Stress testing team
Performance
testing
Acceptance test Requirements Analyst Black box Analyst
Customer Customer
Alpha test No test plan Black box Potential users

Beta test No test plan Black box Any user

Regression test Functional Analyst Black box Development

specification team
Requirements System test
53
team
54