Software Testing
Mateen Ahmed Abbasi
Department of Computer Science
Khwaja Fareed University Of Engineering and Information Technology
Observations about Testing
• “Testing is the process of executing a program
with the intention of finding errors.” – Myers
• “Testing can show the presence of bugs but
never their absence.” - Dijkstra
Good Testing Practices
• A good test case is one that has a high probability
of detecting an undiscovered defect, not one that
shows that the program works correctly
• It is impossible to test your own program
• A necessary part of every test case is a
description of the expected result
Good Testing Practices (cont’d)
• Avoid non-reproducible or on-the-fly testing
• Write test cases for valid as well as invalid input
conditions.
• Thoroughly inspect the results of each test
• As the number of detected defects in a piece of
software increases, the probability of the
existence of more undetected defects also
increases
Good Testing Practices (cont’d)
• Assign your best people to testing
• Ensure that testability is a key objective in
your software design
• Never alter the program to make testing
easier
• Testing, like almost every other activity, must
start with objectives
Program testing
• Testing shows that a program does what it is intended to
do and to discover program defects before it is put into
use. It is a dynamic validation and verification (V&V)
technique.
• To test, you execute a program using artificial data
• Check the results of the test run for errors, anomalies or
information about the program’s non-functional attributes
• Can reveal the presence of errors NOT their absence
• Testing is part of a more general verification and
validation process, which also includes static V&V
techniques
Program testing goals
• To demonstrate that software meets its
requirements
– For custom software, at least one test for every
requirement in the requirements document
– For generic software products, have tests for all of the
system features AND feature combinations
• To discover situations in which the behavior of the
software is incorrect or undesirable
– Defect testing is concerned with rooting out
undesirable system behavior such as system
crashes, unwanted interactions with other systems,
incorrect computations and data corruption
Validation and defect testing
• The first goal leads to validation testing
– You expect the system to perform correctly
using a given set of test cases that reflect the
system’s expected use
• The second goal leads to defect testing
– The test cases are designed to expose
defects. Tests can be obscure and address
unusual use of software.
Testing process goals
• Validation testing
– To demonstrate to the developer and the system
customer that the software meets its
requirements
– A successful test shows that the system operates
as intended
• Defect testing
– To discover faults or defects in the software
where its behavior is incorrect or doesn’t follow its
specification
– A successful test is a test that makes the system
perform incorrectly
An input-output model of program
testing
Verification vs validation
• Verification:
"Are we building the product right?”
– The software should conform to its
specification
• Validation:
"Are we building the right product?”
– The software should do what the user really
requires
V & V confidence
• Aim of V & V is to establish confidence that
the system is ‘fit for purpose’
• Depends on:
– Software purpose
• The level of confidence depends on how critical the
software is to an organization
– User expectations
• Users may have low expectations of certain kinds of
software
– Marketing environment
• Getting a product to market early may be more
important than finding defects in the program
Inspections and testing
• Software inspections: Analyze the static
system representation to discover
problems (static verification)
– May be supplemented by tool-based document
and code analysis.
• Software testing: Exercise and observe
product behaviour (dynamic verification)
– The system is executed with test data and its
operational behaviour is observed.
Inspections and testing
Software inspections
• People examine the source code to
discover anomalies and defects
• Inspections do not require execution of a
system.
• They may be applied to any representation
of the system (requirements, design,
configuration data, test data, etc.)
• They have been shown to be an effective
technique for discovering program errors
Software Inspections
[Link]
Advantages of inspections
• During testing, errors can mask (hide) other errors.
Because inspection is a static process, you don’t have
to be concerned with interactions between errors.
• Incomplete versions of a system can be inspected
without additional costs. If a program is incomplete,
then you need to develop specialized test harnesses
to test the parts that are available.
• As well as searching for program defects, an
inspection can also consider broader quality attributes
of a program, such as compliance with standards,
portability and maintainability.
Inspections and testing
• Inspections and testing are complementary and not
opposing verification techniques
• Both should be used during the V & V process
• Inspections can check conformance with a
specification but not conformance with the
customer’s real requirements
• Inspections cannot check non-functional
characteristics such as performance, usability, etc.
Stages of testing
• Development testing - the system is tested
during development
• Release testing - a separate testing team
tests a complete version of the system
before it is released
• User testing - users or potential users of a
system test the system in their own
environment
Development testing
• Development testing includes all testing
activities that are carried out by the
developers
– Unit testing: focuses on testing the functionality of
objects or methods
– Component testing: create components from
object combinations. Focuses on testing
component interfaces.
– System testing: the components in a system are
integrated and the system is tested as a whole.
Focuses on testing component interactions.
Unit testing
• Unit testing is the process of testing
individual components in isolation
• The purpose is to discover defects
• Units may be:
– Individual functions or methods within a class
– Classes with several attributes and methods
– Composite components with defined
interfaces used to access their functionality
Object class testing
• Complete test coverage of a class involves
– Testing all operations associated with an
object
– Setting and interrogating all object attributes
– Exercising the object in all possible states
• Inheritance makes it more difficult to
design object class tests
Automated testing
• Whenever possible, unit testing should be
automated so that tests are run and checked
without manual intervention
• In automated unit testing, the testers make
use of a test automation framework (such as
JUnit) to write and run your program tests
• Unit testing frameworks can run all the
implemented tests and report on the success
of the tests
Unit test effectiveness
• The test cases should show that component does what it
is supposed to do
• Should reveal defects in the component, if there are any
• This leads to 2 types of unit test cases:
– Tests that reflect normal operation of a program to
show that the meets expectations.
– Tests based on testing experience of where common
problems arise. Should use abnormal inputs, check
that these are properly processed, and not crash the
component.