Software Testing & Quality

Dr. Shaukat Hayat

Department of Software Engineering
IIU-Islamabad
 QUALITY REVOLUTION
People seek quality in every man-made artifact.
Certainly, the concept of quality did not originate with software systems.
 Rather, the quality concept is likely to be as old as human endeavor

• Historical Background: Quality concepts have roots in mass production; the quality
revolution accelerated with the Internet.

• Global Competition: Increased competition, outsourcing, off-shoring and customer

expectations emphasize the need for high-quality products.

• Process Integration: Quality must be integrated throughout the entire product

development cycle, not just at the end.

Dr. Hayat 2
QUALITY REVOLUTION
• Key Quality Focus Areas
• Understanding customer requirements
• Continuous improvement and waste elimination
• Measurement integration with design
• Organizational commitment to quality at all levels
• System-level perspective on methodology and process

Dr. Hayat 3
QUALITY REVOLUTION
• Pioneers of Quality: Deming, Juran, and Ishikawa shaped quality
management practices, promoting statistical methods and total
quality control (TQC). 1940s and the 1950s,Japan

• 1950, Deming seminar based on the Shewhart methods of statistical

quality control and introduced the plan – do – check – act (PDCA)
cycle

Dr. Hayat 4
 QUALITY REVOLUTION
• The Shewhart cycle
• Setting goals,
• Assigning them to measurable milestones,
• Assessing the progress against those milestones.

• W. A. Shewhart. Economic Control of Quality of Manufactured Product . Van Nostrand, New York, 1931.
• W. A. Shewhart. The Application of Statistics as an Aid in Maintaining Quality of a Manufactured Product. Journal of American Statistical Association , December
1925, pp. 546 – 548.
Dr. Hayat 5
QUALITY REVOLUTION
• Lean Principles: Originated in Japan (Toyota) to minimize waste and
optimize resource use, now applied across various industries.

• Quality Circles: Encouraged participatory management and problem-

solving among workers to enhance quality. [in Japan in 1962 by Kaoru Ishikawa]

Dr. Hayat 6
SOFTWARE QUALITY
• Quality is a complex concept — it means different things to different
people, and it is highly context dependent.

• Garvin’s Analysis: how software quality is perceived in different ways

in different domains, such as philosophy, economics, marketing, and
management. [D. A. Garvin. What Does “Product Quality” Really Mean? Sloan Management Review , Fall
1984, pp. 25 – 43.]

Dr. Hayat 7
SOFTWARE QUALITY
• Five Views of Quality by Kitchenham and Pfleeger’s B. Kitchenham and S. L. Pfleeger. Software Quality:
The Elusive Target. IEEE Software, January 1996, pp. 12 – 21.

• Transcendental View: Quality is recognizable but hard to define.

• User View: Quality is fitness for purpose; does it meet user needs?
• Manufacturing View: Quality is conformance to specifications.
• Product View: Quality ties to inherent characteristics of the product.
• Value-Based View: Quality is determined by customer willingness to
pay.

Dr. Hayat 8
SOFTWARE QUALITY
• Historical Context: Efforts to measure software quality began in the
mid-1970s.
• Quality Factors: Behavioral characteristics such as correctness ,
reliability , efficiency , testability , maintainability ,and reusability

• Quality Criteria: Attributes related to development, e.g., modularity

improves maintainability.

Reference: [McCall, Richards, and Walters, J. A. McCall, P. K. Richards, and G. F. Walters. Factors in
Software Quality, Technical Report RADC-TR-77-369. U.S. Department of Commerce, Washington, DC,
1977.]

Dr. Hayat 9
SOFTWARE QUALITY
• Key Models: Various software quality models have been proposed to define quality and its related attributes.

• ISO 9126: Defines six quality characteristics: functionality, reliability, usability, efficiency, maintainability,
and portability.
• CMM: Assesses development process maturity on a scale of 1-5.
 S. Land. Jumpstart CMM/CMMI Software Process Improvement . Wiley, Hoboken, NJ, 2005.
 International Organization for Standardization (ISO). Quality Management Systems — Fundamentals and Vocabulary, ISO 9000:2000. ISO,
Geneva, December 2000.
• Testing Models:

• TPI Model [test process improvement]: Assesses and improves software testing processes.
• TMM [test maturity Model]: Guides organizations in test process improvement.
 T. Koomen and M. Pol. Test Process Improvement . Addison-Wesley, Reading, MA, 1999.
 I. Burnstein. Practical Software Testing . Springer, New York, 2003.

Dr. Hayat 10
ROLE OF TESTING
• Importance of Testing:

• Essential for achieving and assessing software quality.

• Involves a cycle of testing, defect identification, and fixes.

• Purpose of Software Testing:

• Verification process for software quality assessment.

• Supports product quality improvement.

Dr. Hayat 11
ROLE OF TESTING
• Categories of Software Quality Assessment
• Static Analysis:
• Examines documents without executing the code.
• Involves:
• Requirements documents
• Software models
• Design documents
• Source code
• Techniques include:
• Code review
• Inspection
• Walk-through
• Algorithm analysis
• Proof of correctness
• Examples:
• Compiler optimizations.

Dr. Hayat 12
ROLE OF TESTING
• Dynamic Analysis:
• Involves executing the program to identify failures.
• Observes:
• Behavioral properties
• Performance properties
• Uses both typical and specially chosen inputs.
• Key considerations:
• Practical selection of a finite test set.
• Represents behaviors to assess system quality.

Dr. Hayat 13
ROLE OF TESTING
• Static analysis and dynamic analysis are complementary in nature,
and for better effectiveness, both must be performed repeatedly and
alternated.
• Practitioners and researchers need to remove the boundaries
between static and dynamic analysis and create a hybrid analysis that
combines the strengths of both approaches

Dr. Hayat 14
VERIFICATION AND VALIDATION
• Both are essential concepts in software testing.

• They involve different activities and focus on different aspects of the

software development process.

Dr. Hayat 15
VERIFICATION AND VALIDATION
• Verification:
• Evaluates whether the product of a development phase meets
specified requirements.
• Focus: Checks correctness of interim products (e.g., requirements,
design, code).
• Activities:
• Inspections Quality Attributes: Consistency, completeness, and correctness.
Goal: Confirms that the product is being built correctly.
• Walkthroughs When: Conducted throughout the development process.
• Reviews
Dr. Hayat 16
VERIFICATION AND VALIDATION
• Validation:
• Confirms that the final product meets its intended use and customer
expectations.
• Focus: Assesses the overall product from the user's perspective.
• Activities:
• Acceptance testing
• System testing • Goal: Ensures the product is the right one for users.
• When: Typically performed towards the end of the development cycle but can be
done early (e.g., in extreme Programming).

Dr. Hayat 17
VERIFICATION AND VALIDATION
• Verification: Primarily static analysis with possible dynamic elements.
• Validation: Dynamic testing in real environments.

• Both verification and validation are critical for ensuring software

quality.
• Effective testing practices should integrate both to minimize risks and
meet customer expectations.

Dr. Hayat 18
Failure, Error, Fault, and Defect
• Key Terms in Software Testing:

• Failure: A failure occurs when the system's behavior does not match
its specifications.

Example: If a weather app is supposed to display the current

temperature but shows an error message instead, this is a failure
because it does not conform to what users expect from the app.

Dr. Hayat 19
Failure, Error, Fault, and Defect
• Error: An error is a state within the system that could lead to a failure
if not corrected.

• Example: If a web application retrieves data from a database but

receives incorrect data, it enters an error state. If no action is taken to
correct the data retrieval process, this error could lead to a failure
when the app tries to display the wrong information.

Dr. Hayat 20
Failure, Error, Fault, and Defect
• Fault: A fault is the underlying cause of an error.

• Example: If the code responsible for retrieving data from the

database has a bug (like a typo in the query), that bug is a fault. It is
the reason for the error state that could ultimately lead to a failure in
the application.

Dr. Hayat 21
Failure, Error, Fault, and Defect
• Behavior Chain:
• The process can be summarized as:
• Fault → Error → Failure
• failure of one component can lead to a failure of another interacting
component.

Dr. Hayat 22
Failure, Error, Fault, and Defect
• Customer Perspective:
• A failure can occur even with a fault-free implementation if the specifications don’t meet
customer expectations.

• Broader Context:
• Software failure means failing to meet customer expectations or hindering useful work.
• Defect vs. Fault:

• Defects are often linked to design issues.

• Faults relate to incorrect decisions or signals in the system.

• In industry, "defect" is commonly used, while researchers prefer "fault."

Dr. Hayat 23
Week-02
Dr. Hayat 24
NOTION OF SOFTWARE RELIABILITY
 Software reliability is defined as the probability of failure-free
operation of a software system for a specified time in a specified
environment
 The level of reliability of a system depends on those inputs that
cause failures to be observed by the end users.
 Software reliability can be estimated via random testing, as
suggested by Hamlet [R. Hamlet. Random Testing. In Encyclopedia of Software Engineering , J.
Marciniak, Ed. Wiley, New York, 1994, pp. 970 – 978]

Dr. Hayat 25
NOTION OF SOFTWARE RELIABILITY
• Capturing the future usage pattern of a system in a general sense is
described in a form called the operational profile.

Dr. Hayat 26
OBJECTIVES OF TESTING
Stakeholders in the Test Process
• Programmers
• Test Engineers
• Project Managers
• Customers
Key Testing Objectives
1. It Does Work
I. Test to ensure the unit or system works under normal circumstances.
II. Focus on showing that the system performs its basic functions.
2. It Does Not Work
I. Test to find faults and make the unit or system fail.
II. Identifying failures leads to discovering weaknesses.
Dr. Hayat 27
OBJECTIVES OF TESTING
Additional Testing Objectives
1. Reduce the Risk of Failure
I. More tests reduce system failure rates.
II. Objective: Lower risk of failure to an acceptable level.
2. Reduce the Cost of Testing
I. Minimize costs related to designing, maintaining, and executing test cases.
II. the cost of analyzing the result of executing each test case,
III. the cost of documenting the test cases, and
IV. the cost of actually executing the system and documenting it.
V. Focus on producing fewer, effective test cases for cost-efficient testing.

Dr. Hayat 28
OBJECTIVES OF TESTING
1. Test Case Effectiveness
I. The goal is to deliver low-risk software with carefully selected, effective
test cases.

Dr. Hayat 29
Test Case
• A test case is a pair of <input, expected outcome>.
• Example: A program calculating square roots of non-negative
numbers.

Stateless Systems
• Outcomes depend solely on current input.
• Example: A compiler or a square root program.

Dr. Hayat 30
Test Case
 State-Oriented Systems
• The outcome depends on both the current state and the current
input.

Key Considerations for State-Oriented Systems

• Test cases may involve sequences, decisions, and timing.
• Examples: Telephone systems, ATM machines.

Dr. Hayat 31
Expected Outcome in Program Execution
• An outcome of program execution includes:
- Values produced by the program
- Outputs for local observation (e.g., integer, text, audio,
image)
- Outputs (messages) for remote storage, manipulation, or
observation

Dr. Hayat 32
Expected Outcome in Program Execution
• State changes (program, database)
Examples of State Changes:
• Initialization: a program starts, it moves from an idle state to an active
state as it begins executing.
• Variable Modification: inputs data  program processes internal
variables (or states) change to reflect the new information
• Completion of Tasks: task (processed user request) is completed  the
state of the program might change from "processing" to "idle" or
"completed.“

Dr. Hayat 33
Expected Outcome in Program Execution
• State changes (program, database)
• Add Operation
• Delete Operation
• Update Operation
•
• Sequence of values interpreted together for outcome validity
(sometimes the outputs or results from a program or database operation are not just standalone
values)

Dr. Hayat 34
Oracles in Test Design
• Role of an Oracle:

- Any entity (program, process, human, or data) providing the expected

outcome.
- Test cases are only meaningful if we can judge the acceptability of
results.
- Ensures that the result produced by the program is correct.

Dr. Hayat 35
Precomputing Expected Outcomes
• The test outcome is computed before the program is executed with
the selected test input.

• The idea here is that one should be able to compute the expected
outcome from an understanding of the program’s requirements.

• Pre-computation of the expected outcome will eliminate any

implementation bias in case the test case is designed by the
developer.

Dr. Hayat 36
 Exceptional cases
• 1. Execute program with selected input.
• 2. Observe the actual outcome.
• 3. Verify that the actual outcome matches the expected.
• 4. Use the verified outcome for subsequent tests.

Dr. Hayat 37
CONCEPT OF COMPLETE TESTING
Complete or exhaustive testing means testing a program so thoroughly that
no bugs or problems are left undiscovered by the end of the testing phase.
However, for most systems, achieving complete testing is nearly impossible
for several reasons:

1. Too Many Inputs:

• Programs can accept a huge variety of inputs, both correct (valid) and
incorrect (invalid).
• Some inputs can only be correct at certain times. For example, an input
that works in the morning might not work in the evening.
• Because there are so many different inputs to test, it’s almost impossible to
check them all.

Dr. Hayat 38
CONCEPT OF COMPLETE TESTING
2. Complex Design Issues:
• The design may include implicit decisions and assumptions.
• Programmers might use global or static variables to control program
execution, making testing more complex.
3. Environmental Constraints
• Varied Execution Environments:
• It might not be possible to recreate all execution environments for
testing.
• This becomes significant when software behavior depends on
external factors like weather, temperature, altitude, or pressure.

Dr. Hayat 39
Central Issue in Testing
• Complete testing (finding all faults) is highly desirable.
• However, it is almost impossible to achieve full coverage of all faults.
• A more practical approach is selecting a subset of inputs to test.

Dr. Hayat 40
Central Issue in Testing
• Input domain D contains all possible inputs for a program P.
• Select a subset of the input domain D₁ to test the program.
• D₁ may only cover a part of the program's behavior P₁, leaving other
parts P₂ untested.

Dr. Hayat 41
Systematic Subset Selection
• The goal is to deduce properties of the entire program P by testing a
subset P₁.
• Careful selection of D₁ ensures that the test coverage is as complete
and accurate as possible.
• Use coverage criteria to systematically select test cases to ensure
broader behavior coverage.

Dr. Hayat 42
TEST ACTIVITIES
• Testing involves multiple key
activities to assess the
program.
• Main steps:
• Identify an objective to be
tested.
• Select inputs based on the
objective.
• Compute expected outcomes
based on the inputs.
Dr. Hayat 43
TEST ACTIVITIES
• Execution Process
• Set up execution environment:
• Ensure proper system setup (e.g., network, database).
• Prepare external systems if needed (e.g., servers in client-server
apps).
• Execute the program:
• Run the program with selected inputs.
• Observe the actual output.

Dr. Hayat 44
TEST ACTIVITIES
• Analyze Results and Verdict
• Analyze the test results:
• Compare actual outcome with expected outcome.
• Assign a test verdict:
• Pass: Outcome matches expectations.
• Fail: Outcome differs.
• Inconclusive: Further testing needed.
• Write a test report:
• Document the failure, expected outcome, and how to reproduce the
issue.
Dr. Hayat 45
TEST LEVELS

Dr. Hayat 46
Week-03

Dr. Hayat 47
SOURCES OF INFORMATION FOR TEST CASE SELECTION

• Test Case Design: Focus of both researchers and practitioners in

software development.

• Information Sources:
• Requirements and Functional Specifications
Describes what the system should do.[intended behavior ]
Can be informal (plaintext, diagrams) or formal (use cases, ER
diagrams, Z notation). Formal specification language: mathematical symbols and logic to
precisely specify the behavior, data, and constraints of a system.
Example: Bluetooth specification uses both plaintext and equations.

Dr. Hayat 48
SOURCES OF INFORMATION FOR TEST CASE SELECTION

• Source Code
Describes the system's actual behavior.
Converts high-level requirements into concrete implementations.
Test cases designed from code must consider how it's implemented
(e.g., sorting methods).

• For example, a step in the detailed design can be “sort array A.” To sort an array,
there are many sorting algorithms with different characteristics, such as iteration,
recursion, and temporarily using another array. Therefore, test cases must be
designed based on the program
Dr. Hayat 49
SOURCES OF INFORMATION FOR TEST CASE SELECTION

• Input and Output Domains in Software Testing

• Some values in the input domain of a program have special meanings, and hence
must be treated separately
• Example: Factorial Function
• Correct: factorial(0) = 1
• Incorrect Implementation: May ignore n = 0.

Dr. Hayat 50
SOURCES OF INFORMATION FOR TEST CASE SELECTION

• Special Output Values:

• Some outputs also have special meanings.
• a program must be tested to ensure that it produces the special
values for all possible causes.
• Example: output value 1 has special significance:
• (i) it is the minimum value computed by the factorial function and
• (ii) it is the only value produced for two different inputs.

Dr. Hayat 51
SOURCES OF INFORMATION FOR TEST CASE SELECTION

• Operational Profile
• It is created to guide test engineers in selecting test cases (inputs) using samples
of system usage
• A quantitative representation of how a system is actually used.
• Created to improve software reliability by considering real-world operation
during testing.
• Developed by:
• Mills (IBM, Cleanroom Software Engineering).
• Musa (AT&T Bell Labs).

Dr. Hayat 52
Fault Model in Software Testing
• Previously encountered faults serve as a great source for designing
new test cases.
• Faults are categorized into types like:
 Initialization faults
 Logic faults
 Interface faults

Dr. Hayat 53
Fault Model in Software Testing
• Error Guessing:
• Test engineers use their experience to guess where faults might be and
create tests to expose those faults.
• Fault Seeding:
• Inject known faults into a program to check how well a test suite can find
them.
• If a test suite finds seeded faults, it's likely to find other faults.
• Mutation Analysis:
• Make small mutations to program statements to assess if test cases can
find these faults.
• If not, add more test cases to detect them.

Dr. Hayat 54
WHITE-BOX AND BLACK-BOX TESTING
White-Box Testing (Structural Testing)
• Focuses on examining the internal structure of the program.

• Control Flow: The order in which individual statements, instructions, or

function calls are executed.
• Data Flow: The movement of data between variables and constants.

• Scope: units of the program

Dr. Hayat 55
WHITE-BOX AND BLACK-BOX TESTING
• Black-Box Testing (Functional Testing)
• Tests the software without access to internal code; treated as a black
box
• Inputs: Based on program requirements
• Outputs: Focus on expected outcomes
• Scope: entire system or individual program units

Dr. Hayat 56
TEST PLANNING AND DESIGN
• Test Planning
• Purpose: Prepare and organize for test execution.
• Components of a Test Plan:
• Framework: Ideas and facts within which tests will be conducted.
• Scope: Outlines the extent of test activities
• Resource Details: Identifies required resources
• Effort Required: Estimation of the workload
• Schedule: Timeline for testing activities
• Budget: Financial plan for testing.

Dr. Hayat 57
TEST PLANNING AND DESIGN
• Test Design
• A critical phase where system requirements are analyzed to identify:

• Features to Test: Determining what aspects of the system need testing.

• Test Objectives: Defined from requirement and functional specifications.
• Test Cases:
• Designed for each test objective.
• Composed of Test Steps: Modular components that can be combined for
complex tests.
• Reusability: Each test case is clearly specified for easy understanding and
reuse.

Dr. Hayat 58
TEST PLANNING AND DESIGN
• Test-Driven Development (TDD)
• test-centric approach emerging in software development.
• Key Characteristics:
• Incremental Development: Building software in small increments.
• Coding Tests: Unit and acceptance tests created by programmers alongside
customers.
• Frequent Regression Testing: Regular testing to ensure existing
functionalities remain intact.
• Test First Approach: Writing test code before the actual production code.

Dr. Hayat 59
Task: Theory of Program Testing
• THEORY OF GOODENOUGH AND GERHART
• THEORY OF WEYUKER AND OSTRAND
• THEORY OF GOURLAY

A detailed concepts of the above mentioned theories about the

program testing, make a an abstract report and present in a class.

Dr. Hayat 60
Dr. Hayat 61