Software Testing Strategies

1
 Software Testing Strategies
- A strategic approach to testing
- Test strategies for conventional software
- Test strategies for object-oriented software
- Validation testing
- White box testing , basic path testing
- Black box testing
- System testing

2
 A Strategic Approach to Testing
• To perform effective testing, a software team should
conduct effective formal technical reviews
• Testing begins at the component level and work outward
toward the integration of the entire computer based system
• Different testing techniques are appropriate at different
points in time
• Testing is conducted by the developer of the software and
(for large projects) by an independent test group
• Testing and debugging are different activities, but
debugging must be accommodated in any testing strategy

3
 Verification and Validation
• Software testing is part of a broader group of activities
called verification and validation that are involved in
software quality assurance
• Verification (Are the algorithms coded correctly?)
– The set of activities that ensure that software correctly
implements a specific function or algorithm
• Validation (Does it meet user requirements?)
– The set of activities that ensure that the software that has
been built is traceable to customer requirements.
• Verification: “Are we building the product right?”
• Validation: “Are we building the right product?”
4
 Organizing for Software Testing
• Testing should aim at "breaking" the software
• Common misconceptions
– The developer of software should do no testing at all
• that the software should be “tossed over the wall” to
strangers who will test it mercilessly,
• that testers get involved with the project only when the
testing steps are about to begin.
• software architecture is complete does an independent test
group become involved.
• The role of an independent test group (ITG) is to remove
the inherent problems associated with letting the builder test
the thing that has been built. Independent testing removes
the conflict of interest that may otherwise be present. 5
 Software testing strategy
• Unit testing
– Concentrates on each component/function of the
software as implemented in the source code
• Integration testing
– Focuses on the design and construction of the software
architecture
• Validation testing
– Requirements are validated against the constructed
software
• System testing
– The software and other system elements are tested as a
whole 6
Testing strategy

7
 Criteria for Completion of testing
when is testing completed ??
• A classic question arises every time software testing is
discussed: “When are we done testing—how do we know
that we’ve tested enough?” Sadly, there is no definitive,
• answer to this question, but there are a few pragmatic
responses and early attempts at empirical guidance.
• . By collecting metrics during software testing and making
use of existing software reliability models, it is possible to
develop meaningful guidelines for answering the question:
“When are we done testing?”

8
 STRATEGIC ISSUES
• Specify product requirements in a quantifiable
manner long before testing commences.
• State testing objectives explicitly.
• Understand the users of the software and
develop a profile for each user category.
• Develop a testing plan that emphasizes “rapid
cycle testing.”

9
 Test Strategies for
Conventional Software
Unit testing :
• Focuses testing on the function or software module
• Concentrates on the internal processing logic and data
structures
• Is simplified when a module is designed with high cohesion
– Reduces the number of test cases
– Allows errors to be more easily predicted and uncovered
• Concentrates on critical modules and those with high
cyclomatic complexity when testing resources are limited

10
Unit testing

11
 Unit testing considerations
• Module interface
– Ensure that information flows properly into and out of
the module
• Local data structures
– Ensure that data stored temporarily maintains its
integrity during all steps in an algorithm execution
• Boundary conditions
– Ensure that the module operates properly at boundary
values established to limit or restrict processing
• Independent paths (basis paths)
– Paths are exercised to ensure that all statements in a
module have been executed at least once
• Error handling paths
– Ensure that the algorithms respond correctly to specific
error conditions 12
 Unit test procedures
• Driver
– A simple main program that accepts test case data,
passes such data to the component being tested, and
prints the returned results
• Stubs
– Serve to replace modules that are subordinate to (called
by) the component to be tested
– It uses the module’s exact interface, may do minimal
data manipulation, provides verification of entry, and
returns control to the module undergoing testing
• Drivers and stubs both represent testing overhead.
– Both must be written but don’t constitute part of the
installed software product
13
Unit-test environment

14
 Integration testing

• Defined as a systematic technique for constructing the

software architecture
– At the same time integration is occurring, conduct tests
to uncover errors associated with interfaces
• Objective is to take unit tested modules and build a program
structure based on the prescribed design
• Two Approaches
– Non-incremental Integration Testing
– Incremental Integration Testing

15
 Non-incremental
Integration Testing
• Uses “Big Bang” approach
• All components are combined in advance
• The entire program is tested as a whole Chaos results
• Many seemingly-unrelated errors are encountered
• Correction is difficult because isolation of causes is
complicated
• Once a set of errors are corrected, more errors occur, and
testing appears to enter an endless loop

16
 Incremental Integration Testing
• The program is constructed and tested in small increments
• Errors are easier to isolate and correct
• Interfaces are more likely to be tested completely
• A systematic test approach is applied
• Different incremental integration strategies
– Top-down integration
– Bottom-up integration
– Regression testing
– Smoke testing

17
 Top-down Integration
• Modules are integrated by moving downward through the
control hierarchy, beginning with the main module
• Subordinate modules are incorporated in two ways :
– depth-first : All modules on a major control path are
integrated
– breadth-first : All modules directly subordinate at each
level are integrated
• Advantages
– This approach verifies major control or decision points
early in the test process
• Disadvantages
– Stubs need to be created to substitute for modules that
have not been built or tested yet; this code is later
discarded
– Because stubs are used to replace lower level modules,
no significant data flow can occur until much later in the
integration/testing process 18
 Forexample,selecting
the left-hand path, components M1, M2 , M5 would be
integrated first. Next,M8 or (if
necessary for proper functioning of M2) M6 would be
integrated. Then, the central
and right-hand control paths are built.
19
 Bottom-up Integration
• Integration and testing starts with the most atomic modules in
the control hierarchy
• Advantages
– This approach verifies low-level data processing early in
the testing process
– Need for stubs is eliminated
• Disadvantages
– Driver modules need to be built to test the lower-level
modules; this code is later discarded or expanded into a
full-featured version
– Drivers inherently do not contain the complete algorithms
that will eventually use the services of the lower-level
modules; consequently, testing may be incomplete or
more testing may be needed later when the upper level
modules are available
20
 Integration follows the pattern illustrated in Figure
Components are combined to form clusters 1, 2, and 3. Each
of the clusters is tested using a driver (shown
as a dashed block). Components in clusters 1 and 2 are
subordinate to Ma. Drivers D1and D2 are removed and the
clusters are interfaced directly to Ma. Similarly, driver D3
for cluster 3 is removed prior to integration with module Mb.
Both Ma and Mb will ultimately be integrated with
21
component Mc, and so forth.
 Regression Testing
• Each new addition or modification of data may cause problems
with functions that previously worked flawlessly
• Regression testing re-executes a small subset of tests that have
already been conducted
– Ensures that changes have not propagated unintended side
effects
– Helps to ensure that changes do not introduce unintended
behavior or additional errors
– May be done manually or through the use of automated
capture/playback tools
• Regression test suite contains three different classes of test
cases
– A representative sample of tests that will exercise all
software functions
– Additional tests that focus on software functions that are
likely to be affected by the change
– Tests that focus on the actual software components that
have been changed
22
 Smoke testing
• Designed as a pacing mechanism for time-critical projects
– Allows the software team to assess its project on a
frequent basis
• Includes the following activities
– The software components that have been translated into
code and linked into a build
– A series of breadth tests is designed to expose errors that
will keep the build from properly performing its function
• The goal is to uncover “show stopper” errors that have
the highest likelihood of throwing the software project
behind schedule
– The build is integrated with other builds and the entire
product is smoke tested daily

23
 Benefits of Smoke Testing
• Integration risk is minimized
– Daily testing uncovers incompatibilities and show-
stoppers early in the testing process, thereby reducing
schedule impact
• The quality of the end-product is improved
– Smoke testing is likely to uncover both functional
errors and architectural and component-level design
errors
• Error diagnosis and correction are simplified
– Smoke testing will probably uncover errors in the
newest components that were integrated
• Progress is easier to assess
– As integration testing progresses, more software has
been integrated and more has been demonstrated to
work
– Managers get a good indication that progress is being
made
24
 Validation Testing
• Validation testing follows integration testing
• The distinction between conventional and object-oriented
software disappears and Focuses on user-visible actions and
user-recognizable output from the system
Validation test criteria :
• Demonstrates conformity with requirements
• Designed to ensure that All functional requirements are
satisfied,All behavioral characteristics are achieved,All
performance requirements are attained
• Documentation is correct
• Usability and other requirements are met (e.g.,
transportability, compatibility, error recovery, maintainability)
• After each validation test
– The function or performance characteristic conforms to
specification and is accepted
– A deviation from specification is uncovered and a
deficiency list is created
27
Configuration review:
• The intent of this review is to ensure that all elements of the
software configuration have been properly developed, are
cataloged, and have the necessary detail to bolster the
support activities
Alpha and beta testing :
• Alpha testing conducted at the developer’s site by end users
– Software is used in a natural setting with developers
watching intently
– Testing is conducted in a controlled environment
• Beta testing conducted at end-user sites
– Developer is generally not present
– It serves as a live application of the software in an
environment that cannot be controlled by the developer
– The end-user records all problems that are encountered
and reports these to the developers at regular intervals
• After beta testing is complete, software engineers make
software modifications and prepare for release of the
28
software product to the entire customer base
 System testing
System testing is a series of different tests whose purpose is to fully
exercise the computer based system
• Recovery testing
– Tests for recovery from system faults
– Forces the software to fail in a variety of ways and verifies
that recovery is properly performed
– Tests reinitialization, check pointing mechanisms, data
recovery, and restart for correctness
• Security testing
– Verifies that protection mechanisms built into a system will, in
fact, protect it from improper access
• Stress testing
– Executes a system in a manner that demands resources in
abnormal quantity, frequency, or volume

27
• Performance testing
– Tests the run-time performance of software within the
context of an integrated system
– Often coupled with stress testing and usually requires both
hardware and software instrumentation
– Can uncover situations that lead to degradation and
possible system failure
• Deployment testing
– Also known as configuration testing
– It examines all installations procedures that will be used by
customers

28
 White box testing
• White box testing is also called as glass-box testing
• Using white-box testing methods can derive test cases that
– guarantee that all independent paths within a module have been
exercised at least once
– exercise all logical decisions on their true and false sides
– execute all loops at their boundaries and within their operational
bounds
– exercise internal data structures to ensure their validity

29
 Basis path testing
• Basis path testing is a white-box testing technique
• The basis path method enables the test-case designer to
derive a logical complexity measure of a procedural design
and use this measure as a guide for defining a basis set of
execution paths
• Flow Graph Notation:
– a simple notation for the representation of control flow, called a flow
graph
– It also know as program graph

30
• Arrows called edges or links represent flow of control
• Circles called flow graph nodes represent one or more actions
• Areas bounded by edges and nodes called regions
• A predicate node is a node containing a condition

31
• Independent program paths:
– An independent path is any path through the program that
introduces at least one new set of processing statements
or a new condition
– independent path must move along at least one edge that
has not been traversed before the path is defined

32
• Independent program paths:
– An independent path is any path through the program that
introduces at least one new set of processing statements
or a new condition
– independent path must move along at least one edge that
has not been traversed before the path is defined
– Example:

33
Three methods to compute Cyclomatic complexity
number
1.V(G)=E-N+2(E is number of edges, N is number of
nodes
2.V(G)=Number of regions
3.V(G)= Number of predicates +1

The flow graph has four regions V(G)=4

V(G)=11 edges-9 nodes+2=4
V(G)=3 predicate nodes
• Deriving test cases
– Using the design or code as a foundation, draw a
corresponding flow graph.
– Determine the cyclomatic complexity of the resultant flow
graph.
– Determine a basis set of linearly independent paths.
– Prepare test cases that will force execution of each path in
the basis set
• Graph matrices
– A data structure, called a graph matrix, can be quite useful
for developing a software tool that assists in basis path
testing
– A graph matrix is a square matrix whose size (i.e., number
of rows and columns) is equal to the number of nodes on
the flow graph
35
 Control Structure Testing

Basis path testing is simple and effective .It is not sufficient in

itself

Control structure broadens the basic test coverage and improves

the quality of white box testing
 Condition Testing
 Data flow Testing
 Loop Testing
Condition Testing

Exercise the logical conditions contained in a

program module
--Focuses on testing each condition in the
program to ensure that it does contain errors

Data flow Testing

 Selects test paths according to the locations of definitions and use of

variables in a program
 Aims to ensure that the definitions of variables and subsequent use is
tested
 First construct a definition-use graph from the control flow of a program
 Def(definition):definition of a variable on the left-hand side of an
assignment statement
 C- use: Computational use of a variable like read, write or variable on
the right hand of assignment statement
 P- use: Predicate use in the condition
 Every DU chain be tested at least once.
Loop Testing

Focuses on the validity of loop constructs

Four categories can be defined
[Link] loops
[Link] loops
[Link] loops
[Link] loops
 Black box testing
• Black-box testing, also called behavioral testing

• Black-box testing attempts to find errors in the following

categories
– incorrect or missing functions
– interface errors
– errors in data structures or external database access
– behavior or performance errors
– initialization and termination errors.

40
 Graph-based testing methods
Draw a graph of objects and relations
Devise test cases t uncover the graph such that each object and its relationship exercised.

Directed
link
Object# Object#
1ob 2

Node
Undirected weight
link
Parallel Links
object#
3

41
 Equivalence Partitioning
• Equivalence partitioning is a black-box testing method that
divides the input domain of a program into classes of data
from which test cases can be derived
• equivalence classes for an input condition. Using concepts
introduced in the preceding section, if a set of objects can be
linked by relationships that are symmetric, transitive, and
reflexive, an equivalence class is present

42
• Equivalence classes may be defined according to the following
guidelines:
1. If an input condition specifies a range, one valid and two
invalid equivalence classes are defined.
2. If an input condition requires a specific value, one valid and
two invalid equivalence classes are defined.
3. If an input condition specifies a member of a set, one valid
and one invalid equivalence class are defined.
4. If an input condition is Boolean, one valid and one invalid
class are defined.

43
 Boundary Value Analysis
• A greater number of errors occurs at the boundaries of the
input domain rather than in the “center.” It is for this reason
that boundary value analysis (BVA) has been developed as a
testing technique
• Boundary value analysis leads to a selection of test cases that
exercise bounding values
• Guidelines for BVA are similar in many respects to those
provided for equivalence partitioning:
– If an input condition specifies a range bounded by values a and b, test
cases should be designed with values a and b and just above and just
below a and b.
– If an input condition specifies a number of values, test cases should
be developed that exercise the minimum and maximum numbers.
Values just above and below minimum and maximum are also tested.
– Apply guidelines 1 and 2 to output conditions
– If internal program data structures have prescribed boundaries (e.g., a
table has a defined limit of 100 entries), be certain to design a test
case to exercise the data structure at its boundary. 39
 Orthogonal Array Testing
• Orthogonal array testing can be applied to problems in which
the input domain is relatively small but too large to
accommodate exhaustive testing
• The orthogonal array testing method is particularly useful in
finding region faults-an error category associated with faulty
logic within a software component.
• a single parameter value makes the software malfunction.
These faults are called single mode faults
• If there exists a consistent problem when specific levels of
two parameters occur together, it is called a double mode
fault

47
Example 1
A Web page has three distinct sections (Top, Middle, Bottom) that can be
individually shown or hidden from a user
•No of Factors = 3 (Top, Middle, Bottom)
•No of Levels (Visibility) = 2 (Hidden or Shown)
•Array Type = L4
(4 is the number of runs arrived after creating the OAT array)

If we go for Conventional testing technique, we need test cases like 2 X 3 = 6 Test

Cases
Test Cases Scenarios Values to be tested
Test #1 HIDDEN Top
Test #2 SHOWN Top
Test #3 HIDDEN Bottom
Test #4 SHOWN Bottom
Test #5 HIDDEN Middle
Test #6 SHOWN Middle
If we go for OAT Testing we need 4 Test cases as shown below:

Test Cases TOP Middle Bottom

Test #1 Hidden Hidden Hidden

Test #2 Hidden Visible Visible

Test #3 Visible Hidden Visible

Test #4 Visible Visible Hidden

Differences Between Black Box Testing & White Box Testing
 Debugging

Debugging occurs as a consequence of

successful testing.
That is, when a test case uncovers an error,
debugging is the process that results in the
removal of the error.
Debugging has the objective: to find and
correct the cause of a software error.
 The Art of Debugging
Debugging occurs as a consequences of
successful testing.
Debugging Strategies
1) Brute Force Method.
2) Back Tracking
3) Cause Elimination
4) Automated debugging

55
Brute force
-- Most common and least efficient
-- Applied when all else fails
-- Memory dumps are taken
-- Tries to find the cause from the load of information
Back tracking
-- Common debugging approach
-- Useful for small programs
-- Beginning at the system where the symptom has been
uncovered, the source code traced backward until the site of the
cause is found.
Cause Elimination
-- Based on the concept of Binary partitioning
-- A list of all possible causes is developed and tests are
conducted to eliminate each
 The Debugging process

Execution of test
cases
Results

Test
cases Additional Suspected
tests causes

Regression tests Debugging

Identified
Corrections causes