A strategic approach to software testing
Testing is a set of activities that can be planned in advance and conducted
systematically.
A number of software testing strategies have been proposed in the literature. All provide
you with a template for testing and all have the following generic characteristics:
To perform effective testing, you should conduct effective technical reviews. By doing
this, many errors will be eliminated before testing commences.
Testing begins at the component level and works “outward” toward the integration of
the entire computer-based system.
Different testing techniques are appropriate for different software engineering
approaches and at different points in time.
Testing is conducted by the developer of the software and (for large projects) an
independent test group.
Testing and debugging are different activities, but debugging must be accommodated
in any testing strategy.
A strategic approach to software testing
Software testing is one element of a broader topic that is often referred to as verification and
validation (V&V). Verification refers to the set of tasks that ensure that software correctly
implements a specific function. Validation refers to a different set of tasks that ensure that the
software that has been built is traceable to customer requirements.
Boehm states this another way:
Verification: “Are we building the product right?”
Validation: “Are we building the right product?”
The definition of V&V encompasses many software quality assurance activities.
Verification and validation includes a wide array of SQA activities: technical reviews, quality
and configuration audits, performance monitoring, simulation, feasibility study,
documentation review, database review, algorithm analysis, development testing, usability
testing, qualification testing, acceptance testing, and installation testing.
Although testing plays an extremely important role in V&V, many other activities are also
necessary.
Organizing for Software Testing
• Testing should aim at "breaking" the software
• Common misconceptions
• The developer of software should do no testing at all
• The software should be given to a secret team of testers who will test it
unmercifully
• The testers get involved with the project only when the testing steps are about to
begin
• Reality: Independent test group
• Removes the inherent problems associated with letting the builder test the software
that has been built
• Removes the conflict of interest that may otherwise be present
• Works closely with the software developer during analysis and design to ensure
that thorough testing occurs
A software testing strategic for conventional software approach
to software
The software process may be viewed as the spiral illustrated in Figure above.
• Initially, system engineering defines the role of
software and leads to software requirements
analysis, where the information domain, function,
behavior, performance, constraints, and
validation criteria for software are
established.
• Moving inward (meaning proceeding towards
inside) along the spiral, you come to design and
finally to coding. To develop computer software,
you spiral inward (counterclockwise) along
streamlines that decrease the level of abstraction
on each turn.
A strategy for software testing may also be viewed in the context of the spiral.
• Unit testing begins at the vortex of the spiral and concentrates on each unit (e.g.,
component, class, or WebApp content object) of the software as implemented in
source code.
• Testing progresses by moving outward along the spiral to integration testing, where
the focus is on design and the construction of the software architecture.
• Taking another turn outward on the spiral, you encounter validation testing, where
requirements established as part of requirements modeling are validated against the
software that has been constructed
Finally, you arrive at system testing, where the software and other system elements are
tested as a whole. To test computer software, you spiral out in a clockwise direction
along streamlines that broaden the scope of testing with each turn.
Testing Strategy applied to Conventional Software
• Unit testing
• Exercises specific paths in a component's control structure to ensure complete
coverage and maximum error detection
• Components are then assembled and integrated
• Integration testing
• Focuses on inputs and outputs, and how well the components fit together and
work together
• Validation testing
• Provides final assurance that the software meets all functional, behavioral, and
performance requirements
• System testing
• Verifies that all system elements (software, hardware, people, databases) mesh
properly and that overall system function and performance is achieved
7
Testing Strategy applied to Object-Oriented Software
• Must broaden testing to include detections of errors in analysis and design
models
• Unit testing loses some of its meaning and integration testing changes
significantly
• Use the same philosophy but different approach as in conventional software
testing
• Test "in the small" and then work out to testing "in the large"
• Testing in the small involves class attributes and operations; the main
focus is on communication and collaboration within the class
• Testing in the large involves a series of regression tests to uncover errors
due to communication and collaboration among classes
• Finally, the system as a whole is tested to detect errors in fulfilling
requirements
8
When is Testing Complete?
• There is no definitive answer to this question
• Every time a user executes the software, the program is being tested
• Sadly, testing usually stops when a project is running out of time, money, or both
• One approach is to divide the test results into various severity levels
• Then consider testing to be complete when certain levels of errors no longer occur or
have been repaired or eliminated
9
Ensuring a Successful Software Test Strategy
• Specify product requirements in a quantifiable manner long before testing
commences
• State testing objectives explicitly in measurable terms
• Understand the user of the software (through use cases) and develop a profile for
each user category
• Develop a testing plan that emphasizes rapid cycle testing to get quick feedback to
control quality levels and adjust the test strategy
• Build robust software that is designed to test itself and can diagnose certain kinds
of errors
• Use effective formal technical reviews as a filter prior to testing to reduce the
amount of testing required
• Conduct formal technical reviews to assess the test strategy and test cases
themselves
• Develop a continuous improvement approach for the testing process through the
gathering of metrics 10
Test Strategies for
Conventional Software
Unit Testing
• Focuses testing on the function or software
module
• Concentrates on the internal processing
logic and data structures
• Is simplified when a module is designed
with high cohesion
• Reduces the number of test cases
• Allows errors to be more easily
predicted and uncovered
• Concentrates on critical modules and those
with high cyclomatic complexity when
testing resources are limited
12
Targets for Unit Test Cases
• Module interface
• Ensure that information flows properly into and out of the module
• Local data structures
• Ensure that data stored temporarily maintains its integrity during all steps in an
algorithm execution
• Boundary conditions
• Ensure that the module operates properly at boundary values established to
limit or restrict processing
• Independent paths (basis paths)
• Paths are exercised to ensure that all statements in a module have been
executed at least once
• Error handling paths
• Ensure that the algorithms respond correctly to specific error conditions
13
Common Computational Errors in Execution Paths
• Misunderstood or incorrect arithmetic precedence
• Mixed mode operations (e.g., int, float, char)
• Incorrect initialization of values
• Precision inaccuracy and round-off errors
• Incorrect symbolic representation of an expression (int vs.
float)
14
Other Errors to Uncover
• Comparison of different data types
• Incorrect logical operators or precedence
• Expectation of equality when precision error makes equality unlikely (using ==
with float types)
• Incorrect comparison of variables
• Improper or nonexistent loop termination
• Failure to exit when divergent iteration is encountered
• Improperly modified loop variables
• Boundary value violations
15
Problems to uncover in Error Handling
• Error description is unintelligible or ambiguous
• Error noted does not correspond to error encountered
• Error condition causes operating system intervention prior to error
handling
• Exception condition processing is incorrect
• Error description does not provide enough information to assist in the
location of the cause of the error
16
Drivers and Stubs for Unit Testing
• Driver
• A simple main program that accepts test case data, passes such data to the
component being tested, and prints the returned results
• Stubs
• Serve to replace modules that are subordinate to (called by) the component to
be tested
• It uses the module’s exact interface, may do minimal data manipulation,
provides verification of entry, and returns control to the module undergoing
testing
• Drivers and stubs both represent overhead
• Both must be written but don’t constitute part of the installed software product
17
Integration Testing
• Defined as a systematic technique for constructing the software architecture
• At the same time integration is occurring, conduct tests to uncover errors
associated with interfaces
• Objective is to take unit tested modules and build a program structure based on the
prescribed design
• Two Approaches
• Non-incremental Integration Testing
• Incremental Integration Testing
18
Non-incremental Integration Testing
• Commonly called the “Big Bang” approach
• All components are combined in advance
• The entire program is tested as a whole
• Chaos results
• Many seemingly-unrelated errors are encountered
• Correction is difficult because isolation of causes is
complicated
• Once a set of errors are corrected, more errors occur, and
testing appears to enter an endless loop
19
Incremental Integration Testing
• Three kinds
• Top-down integration
• Bottom-up integration
• Sandwich integration
• The program is constructed and tested in small increments
• Errors are easier to isolate and correct
• Interfaces are more likely to be tested completely
• A systematic test approach is applied
20
Top-down Integration
• Modules are integrated by moving downward through the control hierarchy,
beginning with the main module
• Subordinate modules are incorporated in either a depth-first or breadth-first
fashion
• DF: All modules on a major control path are integrated
• BF: All modules directly subordinate at each level are integrated
• Advantages
• This approach verifies major control or decision points early in the test process
• Disadvantages
• Stubs need to be created to substitute for modules that have not been built or
tested yet; this code is later discarded
• Because stubs are used to replace lower level modules, no significant data
flow can occur until much later in the integration/testing process
21
Bottom-up Integration
• Integration and testing starts with the most atomic modules in the control
hierarchy
• Advantages
• This approach verifies low-level data processing early in the testing process
• Need for stubs is eliminated
• Disadvantages
• Driver modules need to be built to test the lower-level modules; this code is
later discarded or expanded into a full-featured version
• Drivers inherently do not contain the complete algorithms that will eventually
use the services of the lower-level modules; consequently, testing may be
incomplete or more testing may be needed later when the upper level modules
are available
22
Sandwich Integration
• Consists of a combination of both top-down and bottom-up integration
• Occurs both at the highest level modules and also at the lowest level modules
• Proceeds using functional groups of modules, with each group completed before the next
• High and low-level modules are grouped based on the control and data
processing they provide for a specific program feature
• Integration within the group progresses in alternating steps between the high
and low level modules of the group
• When integration for a certain functional group is complete, integration and
testing moves onto the next group
• Reaps the advantages of both types of integration while minimizing the need for drivers and stubs
• Requires a disciplined approach so that integration doesn’t tend towards the “big bang” scenario
23
Regression Testing
• Each new addition or change to baselined software may cause problems with functions that
previously worked flawlessly
• Regression testing re-executes a small subset of tests that have already been conducted
• Ensures that changes have not propagated unintended side effects
• Helps to ensure that changes do not introduce unintended behavior or
additional errors
• May be done manually or through the use of automated capture/playback
tools
• Regression test suite contains three different classes of test cases
• A representative sample of tests that will exercise all software functions
• Additional tests that focus on software functions that are likely to be affected
by the change
• Tests that focus on the actual software components that have been changed
24
Smoke Testing
• Taken from the world of hardware
• Power is applied and a technician checks for sparks, smoke, or
other dramatic signs of fundamental failure
• Designed as a pacing mechanism for time-critical projects
• Allows the software team to assess its project on a frequent basis
• Includes the following activities
• The software is compiled and linked into a build
• A series of breadth tests is designed to expose errors that will keep
the build from properly performing its function
• The goal is to uncover “show stopper” errors that have the highest likelihood of throwing
the software project behind schedule
• The build is integrated with other builds and the entire product is
smoke tested daily
• Daily testing gives managers and practitioners a realistic assessment of the progress of
the integration testing
• After a smoke test is completed, detailed test scripts are executed
25
Benefits of Smoke Testing
• Integration risk is minimized
• Daily testing uncovers incompatibilities and show-stoppers
early in the testing process, thereby reducing schedule
impact
• The quality of the end-product is improved
• Smoke testing is likely to uncover both functional errors and
architectural and component-level design errors
• Error diagnosis and correction are simplified
• Smoke testing will probably uncover errors in the newest
components that were integrated
• Progress is easier to assess
• As integration testing progresses, more software has been
integrated and more has been demonstrated to work
• Managers get a good indication that progress is being made
26