Software Testing and
Quality Engineering (CMP 454)
Introduction to
Software and Protocol Testing
Khaled El-Fakih
1
First: The Context
• Software Testing in the context of SOFTWARE
ENGINEERING
• Software Engineering – a disciplined approach for
the production of quality software systems and
applications within time and budget in the context of
constant change
2
Project
Initiation
Process
The Waterfall Model of
Concept
Exploration
the Software
Process
System
Allocation
Development Life Cycle
Process
Requirements
Process
Design
Process
Implementation
Process
Verification
& Validation
Process
Installation
Process
Operation &
Support Process
3
Project
Initiation
Process
The Waterfall Model of
Concept
Exploration
the Software
Process
System
Allocation
Development Life Cycle
Process
Requirements
Process
Design
Process
Life-cycle activities: Implementation
Process
Verification
& Validation
Project Management Process
Testing Installation
Process
Quality Assurance
Operation &
Support Process
4
Software Testing
• Testing includes activities performed after completion
of each phase of the software development life cycle
• Testing aims at detecting and correcting defects found
during the development of software
• Testing cannot prove the absence of errors in a given
software
• Correctness is only with respect to the test data
Software is correct if it behaves properly for the given
test data – it may fail for some other test data
5
Software Quality Assurance
(SQA)
• Ongoing software project activities whose aim is
to ensure the delivered end product satisfy some
preset quality standard
Internal company standards
Local, regional or international standards for
software quality (like ISO and IEEE standards)
• SQA group is normally an independent group
from the development team
To ensure unbiased judgment and evaluation
6
Software Quality Issues
Quality of the Product
Quality of the Process
Quality of the Business
Software Quality Engineering is Important
7
Qualities of the Product
• Hidden qualities
Modularity, simplicity, efficiency, fault-
tolerance, machine-independence, self-
documentation, …
• Observable qualities
User-friendliness, Usability, Consistency,
Correctness, Completeness, …
8
Quality of the Process
• Quality of the Process
Capability Maturity Level (CMM Level) of the
process
Process definition, monitoring, and management
Monitoring quality of work products
Training, Certification . .. .
9
Why do we test ?
• Increase confidence in the delivered software
• Catch as many errors as possible
• Company reputation
• Consumer (software user) safety
Safety critical software in airplanes, air traffic controllers,
medical instruments, …
10
Coding errors
• NASA Fortran error – loss of a satellite in orbit
DO 2 I=1.5 instead of: Do 2 I=1,5
2 continue 2 continue
• AT&T 1990 error – loss of millions of $ and reputation
while () {
switch () {
case … : if …. break;
…
case … :
….
}
11 }
Specification Error
• Input specification: n1, n2, n3, ….. , nI
• Output specification: sorted numbers
For j = 1 to I do read n[j];
For j = 1 to I do { n[j] = 1; print n[j]; }
The output specification should have said:
Output specification: the same input but sorted in
ascending order
12
Training on Software Testing
• Techniques and Methods
• Tools
• Standards
Current State in Industry
• Beginning to appreciate the importance of testing
• Complex systems require more (intelligent testing)
• Lack of tools (getting better)
• Often a management decision
13
Stakeholders in Software
Development
Users Developers
Software
Product
Client
14
Distribution of Efforts
Requirements and
Specification (8%)
Design (8%)
Coding (7%)
Maintenance
Testing (10%)
15
Basic Definitions
Error : mistake
Fault : result of an error
Failure: occurs when a fault executes
Test Case: pre-amble, input(s), expected output(s), post-amble
16
Testing Activities
• Testing of the Requirements
• Testing of the Specification
• Testing of the Design
• Testing of the Implementation (or code)
17
Testing Requirements and
Specification
• Check that the client’s real needs are satisfied
Can be done by prototyping
• Check that specifications are correct
Can be done by walkthrough or inspection
Walkthrough is going through the specification
document page by page and trying to find errors
Inspection is checking the specification document
against an inspection CHECKLIST
• Specification review meetings should include:
SQA, client and developers
18
Testing the Design
• Verify that specifications have been completely and
correctly (faithfully) incorporated into the different
design documents
• Ensure correctness and consistency of design
Use of automated design verification tools
• Perform a design walkthrough and inspection
19
Design Walkthrough
• Client is not involved
It is an internal developer process
• Mapping specification to design - cross list
• Check effect of design choices on performance
Choice of algorithms and data structures
20
Design Inspection
• Overview of design documents by designers,
programmers and may be testers
• Independent reviewing is desirable
• Checklist of logic and interface errors
• May want to check error handling, exceptions, …
• Resolve all design errors and schedule follow up
reviews as needed
• Inspection team: designers, programmers, testers,
SQA
21
Testing the Implementation
• Referred to as Software Testing
• Two ways of testing:
Dynamic testing
Static testing
22
Static Testing
• Manual inspection of code
• Code walkthrough
• Use of automated tools to find common errors
Can be done by a ‘smart’ compiler or static
checker programs
Uninitialized variables, unused variables, dead-
code, dangling pointers, … Test
• No execution of the code is needed
------
Report
Static
Source ------
Checker
Software ------
Tool
23 Under Test ------
Dynamic Testing
• Execute the program under test using a set of selected
test data controlled by a tester or a testing tool
• Evaluate the output results to determine conformity to
the expected output
• Better confidence in the software – see the software
running
Program
------ output
------ Executable ------
------ Under Test ------
------
24 Test input data
Issues in Testing
• Very important for safety-critical systems
• Affects system’s reliability and usability
• When to stop testing ?
Often a management decision – do we have time left ?
No idea on how error-free is the software under test is
Cannot prove absence of errors
Exhaustive and time consuming activity
Probability of finding more errors decreases when
more testing is done
Finding errors VS Time elapsed is not linear
‘hard’ errors will require more time to catch
No quantification of test coverage
25
Errors Found VS Time Elapsed
No. of
Errors
time
26
The Elements of a Test Plan
• Better be well-documented and structured
• Basic element is a test case
• Test cases form a test suite
• A test case consists of:
27
Types of Testing
• Acceptance Testing • Regression Testing
• Unit Testing • Robustness Testing
• Integration Testing • Usability Testing
• Installation Testing • User Interface
• Performance Testing Testing
• Load / Stress Testing • Reliability Testing
• Interoperability
28
• Conformance testing
System-Level Black-box-Based
Test Plans and Activities
• Acceptance Test Plan (ATP)
The reference point for the client’s acceptance
Could be developed once the functional
specifications are accepted by the client
ATP is executed before delivering the product
ATP is also executed by the client before formal
acceptance
Normally executed by SQA
29
System-Level Black-box-Based
Test Plans and Activities
• Unit Test Plan (UTP)
To test each individual module or function of the
software
Developed by the module designer (preferably) or
by the programmer
Executed by the programmer
30
System-Level Black-box-Based
Test Plans and Activities
• Integration Test Plan (ITP)
To test the overall functionality of the product –
modules work together smoothly
The strategy for building up the software from
components
Executed by SQA and developers
31
System-Level Black-box-Based
Test Plans and Activities
• Load Test Plan (LTP)
Also referred to as Stress Testing
Test against requirements specifications
Examining the behavior of the system under test
when exposed to a heavy load (large number of
simultaneous users)
Very important for real-time and multi-user
online systems, example, web-based applications
32
System-Level Black-box-Based
Test Plans and Activities
• Performance Test Plan
Testing against performance requirements
normally specified in the requirements and
specifications documents
Test the response time behavior of the system
under low, medium and heavy load / usage
Can be done together with load testing
Very important for real-time systems
33
System-Level Black-box-Based
Test Plans and Activities
• Robustness Test Plan
Testing against illegal input and abnormal
situations and operating conditions
Behavior of system when a software or hardware
failure
Indication of correct behavior should be specified
in the specification document
34
System-Level Black-box-Based
Test Plans and Activities
• Usability Testing
How easy is to use the product ?
How user-friendly is the product ?
• Reliability Testing
Measure of frequency and criticality of product
failure
How often does it fail ?
How long in average does it take to repair an
error ?
35
System-Level Black-box-Based
Test Plans and Activities
• Regression Testing Plan
What to test if parts of the software modules
change ?
Plan for partial testing – important for large
systems
• Installation Testing Plan
How to test that software installation and
configuration will work at the client site ?
Important for distributed systems and multi-user
systems
36
Sources of Generating Tests
37
Classification of Faults by Severity
38
Testing Tools and Languages
• Please check the course folder for a comprehensive
list of industrial strength tools
• TTCN Language , Java JUnit,
• Other Languages ?
39
Areas of Research on Testing
• Test generation/design strategies
How to automatically generate test cases from
code or from specifications
• Test selection strategies
How to obtain the minimal and most significant
and error-revealing (effective) test cases from the
automatically generated test cases
• Test execution and architectures
How to automate the testing processes in different
setups (local, remote, distributed, …)
• Software Design for Testability
Self-testable software, observability and
controllability
40
Areas of Research on Testing
• Test specification languages
Formal description of test cases to facilitate test
execution and analysis
• Test result analysis
How to make sense and verdict out of the output of
the test process
Automate the diagnosis and error localization
• Test coverage and metrics
Quantify the effectiveness of the test cases
• Test correctness
How to ensure that the test cases themselves are
41
correct !