UNIT-IV

Software Quality Assurance

 Quality
• Quality refers to “how good something is”
compared to other similar things.
• Quality is an essential and distinguishing
attribute or set of attributes of something or
someone.
• Quality is a degree or grade of excellence or
worth of something or someone.
• Quality is a perceptual, conditional, and
somewhat subjective attribute and may be
understood differently by different people.
 Quality Concept
• Variation control is the heart of quality control.
• Software engineers strive to control the
– process applied.
– resources expended.
– end product quality attributes.
• Quality of Design
– refers to characteristics, designers specify for
the end product to be constructed.
• Quality of Conformance
– degree to which design specifications are
followed in manufacturing the product.
 Quality Concept
• Quality Control
– Series of inspections, reviews, and tests used
to ensure conformance of a work product to
its specifications.
• Quality Assurance
– Auditing and reporting procedures used to
provide management with data needed to
make proactive decisions.
– Reducing the possibility of creeping in
faults/errors in the development phases.
Software Quality
 Software Quality
• Two definitions of the quality, which are consistent and
have been adopted & used .
– Quality is conformance to requirements.
– Quality is fitness to use.
• Because of the two perspectives of quality the de
facto definition of quality consists of two levels.
• The first is the intrinsic product quality, often
operationally limited to product defect rate and
reliability. This narrow definition is referred to as the
“small q”.
• The broader level of the definition of quality includes
product quality, process quality and customer
satisfaction. It is referred to as “big Q.
 Software Quality
• Big Q
- Conformance to explicitly stated functional and
performance requirements (Build the software as
described in the system Requirements and
Specifications).
- Conformance to explicitly documented
development standards (Build the software the
right way)
- Conformance to implicit characteristics of
professionally developed software i.e. Build
software that meets the expectations of a
reasonable person.
 Software Quality
 The narrowest sense of product quality is
commonly recognized as lack of “bugs” in the
product.
 This definition is usually expressed in two ways:
 Defect Rate/Density (Number of defects per
millions of lines of source codes, per function
point, or other units).
 Reliability (Probability of failure-free
operation in a specified time and
environment).
Software Quality Attributes
Software Quality Attributes
Attribute Description
Attribute Description
Attribute Description
Software Quality Factors
McCall’s Quality Criteria
ISO 9126 Quality Model
 Software Quality Attributes
 In addition to overall customer satisfaction with
software product, satisfaction towards specific
attributes is also measured.
 IBM monitors the CUPRIMDSO (capability,
usability, performance, reliability, installability,
maintainability, documentation/information,
service, and overall)
 HP focuses on FURPS (Functionality, usability,
reliability, performance, and serviceability).
 High Quality Software?
• It must be useful (to the original customer).
• It must be portable (work at all the customer’s sites).
• It must be maintainable.
• It must be reliable.
• It must have integrity (produces correct results, with a high
degree of accuracy).
• It must be efficient.
• It must have consistency of function (it does what the user
would, reasonably expect it to do).
• It must be accessible (to the user).
• It must have good human engineering (easy to learn and
easy to use).
 Software Quality Metrics
• There have been some efforts to develop numerical metrics
for software quality.
• Tabulate some easily measurable attribute of the software.
– Source size or compiled size.
– Number of files or modules or classes.
– Number of each type of statement.
– Complexity of the control flow graph.
– Amount of coupling, degree of cohesion
• Relate the value of these attributes to software quality.
Usually larger attribute value means lower quality.
• There is no underlying theoretical basis that
establishes any easily countable metric as a valid
measure of software quality.
 Software Quality Metrics
 Software Quality Factors requiring
Measures:
 Correctness (defects per KLOC)
 Maintainability
 mean time to change (MTTC)
 spoilage = (cost of change / total cost of system)
 Integrity
threat = probability of attack (that causes failure)
security = probability that the attack is repelled
Integrity =  [1 - threat * (1 - security)]
 Software Quality Metrics
• Usability
easy to learn and use (time)
productivity increase (quantity)
user attitude (questionnaire score)
• Defect Removal Efficiency
DRE = E / (E + D).
E = # of errors found before delivery
D = # of errors found after delivery
 Quality Costs
• Prevention Costs: quality planning, formal
technical reviews, test equipment, training.
• Appraisal Costs: in-process and inter-process
inspection, equipment calibration and
maintenance, testing.
• Failure Costs: rework, repair, failure mode
analysis.
• External Failure Costs: complaint resolution,
product return and replacement, help line
support, warranty.
 Quality Assessment
• Assessing software quality is often based
on subjective criteria.
• As yet there is no generally accepted
computable metric for software quality.
• Identifying quality attributes, quality
carrying properties, quality metrics, and
defining mapping that connects these
aspects by relating them together.
Identify Select Select
Quality Quality Quality
Attributes Properties Metrics

Total Quality Index

 Quality Factors
• Lot of work has been done on identifying subjective factors that
contribute to software quality and these are hard to precisely
define.
• McCall developed one of the more widely used taxonomy of
software quality attributes
– Quality Factors - high level quality attributes
– Quality Criteria - lower level attributes used to help
measure quality factors.
• The Quality Factors/Criteria can only be evaluated subjectively
(e.g. on an arbitrary 1 .. 10 scale. The combining of factors into
a single measure is also a subjective process.
• There is difficulty in mapping what we can measure easily (or
at all) into what we want to know about software quality.
 The SEI’s Capability Maturity
Model (SEI-CMM)
The CMM is a five level model laying out a generic path to
process improvement for a software organization.
1. Initial: ad hoc approach of development.
2. Repeatable: basic management processes.
3. Defined: management and engineering
processes are documented, standardized,
integrated, and actually used.
4. Managed: measured, monitored and controlled
using measurements.
5. Optimizing: Continuous process improvement is
enabled by quantitative feedback from the
process and from piloting innovative ideas and
technologies.
CMM Levels: Key Process Areas
1. Initial level
– No formalized procedures, project plans, cost estimates.
– Tools not adequately integrated.
– Many problems overlooked/ignored.
– Maintenance very difficult.
– Generally ad-hoc processes.
2. Repeatable level
– Requirements management.
– Software Project planning.
– Software project tracking and oversight.
– Software subcontract management.
– Software quality assurance.
– Software configuration management.
CMM Levels: Key Process Areas
3. Defined level
– Organization process focus
– Organization process definition
– Training Program
– Integration software management
– Software product engineering
– Inter group coordination
– Peer reviews
4. Managed level
– Quantitative process management
– Software Quality management
5. Optimizing level
– Defect prevention
– Technology change management
– Process change management
 ISO 9000
• A set of quality standards developed so that purchasers
of goods can have confidence that suppliers have
produced something of acceptable quality.
• ISO 9000 certification has become a widely required
international standard.
• Any supplier who is not ISO 9000 certified will find it
difficult to sell their goods.
• The ISO 9000-3 standard describes how to apply the
general ISO 9000 standard to the software industry.
• The ISO standard addresses design, development,
production, installation and maintenance issues.
• The emphasis in the ISO standard is on documentation of
the process and the managing of the process.
 ISO 9001 Components
1. Management responsibility
2. Quality system
3. Contract review
4. Design control
5. Document and data control
6. Purchasing
7. Control of customer-supplied
8. Product identification
9. Process control
10. Inspection and testing
 ISO 9001 Components
11. Control of inspection, measuring, test equipment
12. Inspection and test status
13. Control of non-conforming product
14. Corrective and preventive action
15. Handling, storage, packaging, preservation, delivery
16. Control of Quality records
17. Internal Quality Audits product
18. Training and traceability
19. Servicing
20. Statistical techniques
 Clean Room Process
• The cleanroom software engineering process is
a software development process intended to
produce software with a certifiable level of
reliability.
• The focus of the cleanroom process is on
defect prevention, rather than defect removal.
• Basic principle behind the cleanroom process is
that “Prevention is always better than cure”.
Cleanroom Process
Cleanroom Process
 Software Quality Assurance
(SQA)
• SQA is a collection of activities during software
development that focus on increasing the
quality of the software being produced.
• SQA is often conducted by an independent
group in the organization.
• Often this group has the final veto over the
release of a software product.
 SQA includes
• Analysis, design, coding & testing methods and
tools.
• Formal Technical reviews during software
development.
• A multi-tiered testing strategy.
• Control of software documentation and the
changes made to it.
• Procedures to ensure compliance with software
development standards.
• Software measurement and reporting
mechanisms.
Software Quality Assurance
 SQA Activities
1. Application of Technical Methods
– Tools to aid in the production of a high quality specification
i.e. specification checkers and verifiers.
– Tools to aid in the production of high-quality designs i.e. design
browsers, checkers, cross-references, verifiers.
– Tools to analyze source code for quality.
2. Formal Technical Reviews
– Group analysis of a specification or design to discover errors.
3. Software Testing
4. Enforcement of standards
– Specification and design standards.
– Implementation standards, e.g portability.
– Documentation standards.
– Testing standards.
 SQA Activities
5. Control of Change
– Formal management of changes to the software
and documentation.
– Changes require formal request to approving
authority.
– Approving authority makes decision on which
changes get implemented and when.
– Opportunity to evaluate the impact and cost of
changes before committing resources.
– Evaluate effect of proposed changes on
software quality.
 SQA Activities
6. Measurement
– Ongoing assessment of software quality.
– Track quality changes as system evolves.
– Warn management if software quality appears to be
degrading.
7. Record Keeping and Reporting
– Collect output and reports of SQA activities.
– Disseminate reports to software managers.
– Maintain archive of SQA reports.
– Maintain log of software development activity. (especially
testing) to satisfy legal requirements.
– Maintain institutional memory of the software development
effort.
Software Quality Assurance
 Statistical Quality Assurance
• Information about software defects is collected
and categorized.
• Each defect is traced back to its cause.
• Using the Pareto principle (80% of the defects can
be traced to 20% of the causes).
• Isolate the "vital few” causes (only 20%) of defect.
• Move to correct the problems that caused the
defects.
 Phase wise Statistics of Errors
Error Total Serious Moderate Minor
Causes # % # % # % # %

IES n1 p1 n11 p11 n12 p12 n13 p13

MCC n2 p2 n21 p21 n22 p22 n23 p23

MIS n12 p12 n121 p121 n122 p122 n123 p123

Ei Si Mi Ti

Ei: Total number of errors Si:

Number of serious errors
Mi: Number of moderate errors Ti: Number of
 Quality Indicator
• S: Size of the product
• Weighing factors Ws=10, Wm=3, Wt=1 are
assigned to serious, moderate and minor errors
respectively.
• Phase Index for the ith phase (PIi) is
PIi = Ws(Si/Ei)+Wm(Mi/Ei)+Wt(Ti/Ei)
• Error Index (EI) is calculated by assigning
weights to different phases (Wi)
EI = Sum(Wi*PIi)/S
• Error index acts as an indicator for the quality
of the software.
Defect Amplification Model
 Defect Prevention/ Removal
• S/w contains 200K lines
• Inspection time = 7053 Hrs.
• Defects prevented = 3112
• Programmer cost = 40.00 per hr.
• Total cost of defect prevention = 7053 x
40.00 =
282120.00
• Cost per defect prevention = 282120/3112

= 91.00
 Defect Removal
• Defect escaped into product = 1 per 1K
• Total defects escaped = 200K/1000
= 200
• Cost of removal of single defect = 25000
• Total defect removal cost = 25000 x 200
= 5000000
• Ratio of defect removal to prevention = 18
Relative Cost of Defect
Removal
Relative Cost of Defect
Removal
Scope Creep Effect
Scope Creep
 Scope Creep
• After the implementation/programming has started,
users and stakeholders make changes.
• Each change is easy to describe, so it sounds
“small” and the programmers agree to it.
• But, later it is found to be very difficult to incorporate
the change.
• Eventually, the project slows to a crawl.
– It’s 90% done – with 90% left to go.
• Programmers know that if they had been told from
the beginning what to build, they could have built it
quickly from the start.
 Formal Analysis
• Refers to tool-based methods used to explore,
debug, and verify formal specifications.
• Methods
– Theorem proving
– Proof checking
– Model checking
– Animation and simulation
 Formal Proof
Deductive Reasoning: Proofs are based on a
formal system that includes
– Set of Primitives
• finite strings from a fixed alphabet.
– Set of Axioms
• specifying the rules of behavior for the
primitives.
– Set of Inference Rules
• allow deduction of additional true
statements (known a theorems) within the
system.
 Formal Proof
• Deductive system
– axioms and inference rules for a formal system.
• Theory
– axioms and derived theorems in a formal
system.
• Proof of Theorem
– sequence of statement transformations that
adheres to the system’s inference rules.
• s1, s2, s3, … , sn |- T
– theorem T is provable following the sequence s i
 Formal System Properties
• Consistent
– not possible to derive a statement and it’s
contradiction from the same set of initial
statements.
• Complete
– every true statement is provable.
• Decidable
– there is an algorithm for determining whether
any legal statement is true.
• Note:- consistency must be present (Essential),
completeness and decidability would be nice
(desirable).
 Proof Construction
• Forward Argument (deductive calculus)
– starting with axioms and proven results the
inference rules are used to prove the desired
consequent.
• Backward Argument (test calculus)
– starting with the desired result and applying the
inference rules to derive a known result, axiom,
or theorem.
 Validation and Program
Verification
• Validation refers to the process to ensure that customer
requirements have been correctly transformed to the
specifications.
• Verification refers to the process to ensure that
specifications have been correctly implemented or
transformed to the code.
• Similar to writing a mathematical proof.
• Must present a valid argument that is believable to the
reader.
• The argument must demonstrate using evidence that the
algorithm is correct.
• Algorithm is correct if code correctly transforms initial
state to final state.
 State of Computation
• Most programming algorithms are based on the
notion of transforming the algorithm inputs to the
corresponding outputs.
• The state of computation may be defined by
examining the contents of the key variables
before and after the execution of each statement.
 Assertions
• Assertions are the facts about the state of the
program variables.
• It is wasteful to spend time looking at variables
that are not affected by a particular statement.
• Default Assertion
– any variables not mentioned in the assertion
for a statement do not affect the state of
computation.
 Use of Assertions
• Pre-condition
– assertion describing the state of computation
before a statement is executed.
• Post condition
– assertion describing the state of computation
after a statement is executed.
 Simple Algorithm
• Model {P} A {Q}
– P = pre-condition
– A = Algorithm
– Q = post-condition

• Sum Algorithm
{pre: x = x0 and y = y0}
z=x+y
{post: z = x0 + y0}
 Sequence Algorithm
• Model
if {P} A1 {Q1} and {Q1} A2 {Q}
then {P} A1 ; A2 {Q} is true

• Swap Algorithm
{pre: x = x0 and y = y0}
temp = x
x=y
y = temp
{post: temp = x0 and x = y0 and y = x0}
 Intermediate Assertions
• Swap Algorithm
{pre: x = x0 and y = y0}
temp = x
{Int.: temp = x0 and x = x0 and y = y0}
x=y
{Int.: temp = x0 and x = y0 and y = y0}
y = temp
{post: temp = x0 and x = y0 and y = x0}
 Conditional Statements
• Absolute Value
{pre: x = x0}
if x < 0 then
y = - x0
else
y = x0
{post: y = | x0 |}
 Intermediate Assertions
{x = x0 and x0< 0}
if x < 0 then
y = - x0
{y = | x0 |}
else
{x = x0 and x0>= 0}
y = x0
{y = | x0 |}
 Algorithm Correctness (Example)
• Suppose you have a software component that
accepts as input an array T of size N.
• The component produces an array T` which
contains the elements of T arranged in ascending
order as output.
• How would we convert the code to its logical
counterpart and prove its correctness?
 Bubble Sort Algorithm
T` = T
more = true
lab1: i=0
if (more ~= true) then go to end
not(more) = true
//** assertion needed
lab2: i=i+1
if i >= N then go to lab1
if T`(i) < T`(i+1) then go to lab2
//* assertion needed
exchange T`(i) with T`(i+1)
more = true
go to lab2
end:
 Step 1
Write assertions to describe the components
input and output conditions.
•Input assertion
A1: (T is an array) & (T is of size N)
•Output assertion
Aend: (T` is an array) &
( i if i < N then (T`(i) <= T`(i+1)) &
( i if i < N then j(T`(i) = T(j)) &
(T` is of size N)
 Step 2 (Process)
• Draw a flow diagram to represent the logical
flow through the component.
• Indicate points where data transformations
will occur and write the corresponding
assertions.
• Ex. Assuming a bubble sort is used two
assertions might be
– [(not(more) = true)) & (i < N) & T`(i) > T`(i+1))]
 [T`(i) is exchanged with T`(i+1)] //*
– [(not(more) = true)) & (i >= N)]  [T`(i) sorted]
 Step 3
• From the assertions we generate a series of
theorems to be proved.
• If your first assertion is A1 and the first
transformation point has assertion A2 associated
with it, the theorem to be proved is
A1  A2
• Once the list of theorems is established each
must be proved individually (order does not
matter)
 Steps 4 and 5
• We need to locate each loop in the flow
diagram and write an if-then assertion for
each loop condition.
• To prove correctness, identify each logic path
beginning with A1 and ending with Aend.
• Following each of these paths allows us to
demonstrate that the code shows that the truth of
the input condition will lead to the truth of the
output condition.
 Steps 6 & 7
• After identifying each logic paths the truth of
each path is proved rigorously (showing that
the input assertion implies the output assertion
according to the logic transformations found on
that path)
• Finally you need to prove that the program
terminates (which may mean an induction
argument if loops are involved)