Unit - VI Software Quality Assurance

6.0 INTRODUCTION

Software Quality Assurance (SQA) consists of a means of monitoring the software

engineering processes and methods used to ensure quality.

Quality assurance (QA) is a way of preventing mistakes and defects in manufactured

products and avoiding problems when delivering solutions or services to customers.

ISO 9000 defines part of quality management focused on providing confidence that quality
requirements will be fulfilled.

6.1 SOFTWARE QUALITY MANAGEMENT VS SOFTWARE QUALITY

ASSURANCE

Software Quality Management (SQM) is a management process that aims to develop and
manage the quality of software in such a way so as the best ensure the product meets the
quality standards expected by the customer.

SQM is a process that ensures the required level of software quality is achieved when it
reaches the users, so that they are satisfied by its performance.

Quality software refers to a software which is reasonably bug or defect free, is delivered in
time and within the specified budget, meets the requirements and/or expectations, and is
maintainable.

Software quality management activities are generally split up into three core components:
quality assurance, quality planning, and quality control.

Quality refers to the features and characteristics of product (service) which define its ability
to satisfy user requirements.

• Various quality concepts are listed below:

1. Ease of use and operation.

2. Conformity with established standards and end user requirements.

3. Well documented.

4. Performance as expected by the user.

5. Ease of modification.

6. Well tested for functionality, user interface performance etc.

Quality also can be looked at in terms of user satisfaction as given below:

User Satisfaction = Compliant Product + Good quality + Delivery within budget and schedule
SQA (Software Quality Assurance) define as, "conformance to explicitly stated functional
and performance requirements, explicitly documented development standards, and implicit
characteristics that are expected of all professionally developed software".

This definition emphasizes three points:

1. Software requirements are the foundation from which quality is measured; lack of
conformance to requirements is lack of quality.

2. Specified standards define a set of development criteria that guide the manner in which
software is engineered; if the criteria are not followed, lack of quality will almost surely
result.

3. A set of implicit requirements often goes unmentioned; if software fails to meet implicit
requirements, software quality is suspect.

6.1.1 Quality

Quality is measurable characteristics of a product. It can be defined as, "a characteristic or

attribute of something".

Quality means consistently meeting customer needs in terms of - requirements, cost, service
and delivery schedule.

Software quality refers to the extent to which the software is able to perform correctly along
with required features and functions.

Quality/software is reasonably bug-free, delivered on time and within budget, meets

requirements and/or expectations and is maintainable.

IEEE defines software quality as "the degree to which a system, component or process meets
specified requirements" OR "the degree to which a system, components or process meets
customer or user needs or expectation".

Software quality is also defined in terms of three factors namely, Failures, Reliability, and
Customer Satisfaction.

A software product is said to have good quality if:

1. It has few failures when used by the customer.

2. It is reliable, meaning that it hardly ever crashes or demonstrates unexpected behaviour

when used in the customer environment.

3. It satisfies a majority of customers.

There are two kinds of quality:

1. Quality of Design: Refers to characteristics and designs specified for the end product to be
constructed. It includes requirements, specifications and the design of the system.
2. Quality of Conformance: Degree to which design specifications are followed in
manufacturing the product. It primarily focuses on implementation.

6.1.2 Quality Control (QC)

Quality control focuses on operational techniques and the activities used to fulfill and verify
requirements of quality.

It involves the series of inspections, reviews, and tests used throughout the software process
to ensure that each work product meets its specifications/requirements.

Key concept of quality control is that - "all work products have defined and measurable

specifications", which are compared with the output of each process.

Quality control is the process of testing software intensive systems to uncover defects and
hence measuring actual quality.

Quality control activities include:

1. Defining and classifying the severity of defects,

2. Inspecting documents,

3. Inspecting code (either manually or via automatic static code analysis),

4. Testing executable software. For example: module, unit, integration, system and
acceptance testing,

5. Recording of defects,

6. Setting quality control limits outside of which corrective action must be taken,

7. Tracking corrective action on defects, and

8. Defect data analysis.

Software Quality Control (SQA) is the set of procedures used by organizations to ensure that
a software product will meet its quality goals at the best value to the customer, and to
continually improve the organization's ability to produce software products in the future.

Software quality control refers to specified functional requirements as well as non-functional

requirements such as supportability, performance and usability.

6.1.3 Quality Assurance (QA)

QA consists of auditing and reporting procedures, which are used to provide necessary data
to management in order to make proactive decisions.

The goal of QA is to provide adequate confidence that a product or service is of the type and
quality expected by the customer.
 If the data provided through quality assurance identify problems, it is management's
responsibility to address the problems and apply the necessary resources to resolve quality
issues.

Functions of Quality Assurance:

1. Consists of a set of auditing and reporting functions that assess the effectiveness and
completeness of quality control activities.

2. Provides management personnel with data that provides insight into the quality of the
products.

3. Alerts management personnel to quality problems so that they can apply the necessary
resources to resolve quality issues.

Elements of QA:

1. Standards: Ensure that standards are adopted and followed.

2. Reviews and Audits: Audits are reviews performed by SQA personnel to ensure hat quality
guidelines are followed for all software engineering work.

3. Testing: Ensure that testing id properly planned and conducted.

4. Error/Defect Collection and Analysis: Collects and analyses error and defect data to better
understand how errors are introduced and can be eliminated.

5. Changes Management: Ensures that adequate change management practices have been
instituted.

6. Education: Takes lead in software process improvement and educational program.

7. Vendor Management: Suggests specific quality practices vendor should follow and
incorporates quality mandates in vendor contracts.

8. Security Management: Ensures use of appropriate process and technology to achieve

desired

security level.

9. Safety: Responsible for assessing impact of software failure and initiating steps to reduce
risk.

10. Risk Management: Ensures risk management activities are properly conducted and that
contingency plans have been established.
Cost of Quality:

Cost incurred or occurred in performing quality related activities are referred to as quality
costs. The cost of quality is the total amount the company spends to achieve and scope with
the quality of its product.

Quality costs are divided into costs associated with prevention, appraisal, and failure.

1. Prevention Costs: These costs are incurred while examining the activities that affect
software quality during the software development.

Includes: Quality planning, formal technical reviews, test equipment, staff training and clear
specification. i.e. Cost of preventing customer dissatisfaction, including errors or weaknesses
in software, design, documentation and support.

2. Appraisal Costs: These costs are incurred in verifying the product before the software is
delivered to the user or client or customer.

Includes: Equipment maintenance, testing, in-process and inter-process inspection, i.e. cost of
inspection and testing.

3. Failure Costs: These costs are incurred when the product does not meet user requirements.

Failure costs would disappear if no defects appeared before shipping a product to customers.
Failure costs are subdivided into Internal failure costs and External failure costs as shown in
Fig. 6.1.

Internal Failure Costs: Include rework, repair and failure mode analysis.

External Failure Costs: Include complaint resolution, product return and replacement, help
line support and warranty work. External failure costs are associated with the defects found
after the product has been shipped to the customer.

6.1.4 Software Quality Assurance (SQA)

Software Quality Assurance (SQA) is a planned and systematic way to evaluate quality of
software product standards, processes and procedures.

Objectives of SQA:

1. To determine the objectives of software to be accomplished.

2. To establish or develop software plans, when the objectives are determined.

3. To monitor and adjust software plans to satisfy user requirements or specifications.

SQA includes the process of assuring that standards and procedures are established and are
followed throughout the software acquisition life cycle.

1. Standards:

Standards are the criteria to which the software products are compared. Specified standards
are used to define the development criteria that are used to guide the manner in which
software is engineered.

Types of standards includes:

Documentation Standards: Specify form and content for planning, analysis and product
documentation and provide consistency throughout a project.

Design Standards: Specify the form and content of the design product. They provide rules
and methods for translating the software requirements into the software design and for
representing it in the design documentation.

Code Standards: Specify the language in which the code is to be written and define any
restrictions on use of language features. They define legal language structures, style
conventions, rules for data structures and interfaces and internal code documentation.

2. Procedures:

Procedures are the established criteria to which the development and control processes are
compared.

Procedures are explicit steps to be followed in carrying out a process. All processes should
have documented procedures.

Examples of processes for which procedures are needed are configuration management, non-
conformance reporting and corrective action, testing, and formal inspections.

6.2 PHASES OF SOFTWARE QUALITY ASSURANCE

Software Quality Management is a process that ensures the required level of software quality
is achieved when it reaches the users, so that they are satisfied by its performance.

In this section we will study phases of SQA.

6.2.1 Planning

Planning the quality management approach for the project is accomplished in parallel with
developing the scope and requirements for the product and the project.
The project assumptions, dependencies, and risks are inputs to the test strategy. Other inputs
include application and architecture models as well as integration requirements.

Development activities during the planning phase, such as conducting a proof of concept or
demonstrating a prototype, provide opportunities to assess whether the project requires
changes and additions to existing quality policies, test environments, test tools, and test
methodologies.

The fundamental quality outputs from the planning phase are the anal quality policy (quality
standards) for the project and the test strategy derived from the functional and nonfunctional
requirements.

Planning for quality assurance and quality control is incomplete until resources are assigned
to the quality tasks in the work breakdown structure (project schedule).

Planning the QA tests and user acceptance tests should be coordinated to avoid unnecessary
duplication of effort.

6.2.2 SQA Activities

The SEI (Software Engineering Institute) recommends a set of SQA activities that address
quality assurance planning, record keeping, analysis, and reporting.

Various SQA activities, which are performed by independent SQA group are as mentioned
below:

1. Prepare SQA plan for the project: The plan identifies:

(i) Evaluations to be performed.

(ii) Audits and reviews to be performed.

(iii) Standards that are applicable to the project.

(iv) Procedures for error reporting and tracking.

(v) Documents to be produced by the SQA group.

2. Participate in the development of the project's software process description: The software
team selects a work process to be performed. SQA group reviews the process description for
compliance with organizational policy, internal and external software standards.

3. Review software engineering activities, (Analysis, Design, Construction, Verification and

Management) to verify compliance with the defined software process.

4. Audit designated software work products to verify compliance with those defined as part
of the software process: SQA group also verifies that corrections have been made; and
periodically reports the results of its work to the project manager.
5. Ensure that any deviations in software or work products are documented and handled
according to a documented procedure: Deviations may be encountered in the project plan,
process description, applicable standards or technical work products.

6. Record any evidence of noncompliance and reports them to management: Noncompliance

items are tracked until they are resolved.

7. Configuration management monitoring: Configuration management is also called as

'change control management'. It is the systematic way of controlling the changes to the
software items. It also helps to store and retrieve the configurable items, (i.e. Software code,
Defect log, Test reports etc.). SQA assures that software configuration management activities
are performed in accordance with the configuration management plans, standards and
procedures. SQA reviews the configuration management plans for compliance with software
configuration management policies and requirements and provides follow-up for non-
conformances.

The configuration management activities monitored and audited by SQA include:

1. Baseline control: Base lining is done when the product is ready to go to next phase.
Current state of product is frozen and further changes to the product are done along with
change in version of the product.

2. Configuration identification: Software configuration identification is consistent and

accurate with respect to the numbering or naming of programs, software modules, software
units, and associated software documents. Example tools for configuration management are
Win CVS (Concurrent Version System), VSS (Visual Source Safe). SQA assures Verification
and Validation (V&V) activities by monitoring technical reviews, inspections and
walkthroughs.

6.2.3 Audit

Audit is an independent examination of a work product or set of work products, to verify

whether or not the product is in compliance with policies, specifications, standards,
contractual agreements, or other criteria.

Audits should be planned on a regular basis. High-risk areas should be audited more often to
ensure conformance. Audits are also used to check any previously identified non-
conformances.

It is always best to choose a team of auditors from within the organization; your audit team
should comprise people from all areas and levels of the company. Audits need to be
documented.

During an audit, evidences are used to verify that the processes are being done in accordance
to the procedures and policies. Evidence should be recorded against each section being
audited.
Recording of evidence needs to have a description of the documentation sighted, number,
date and any other information that will assist in identifying that document.

Non-conformances found should be reported for further action. A date should be established
for the correction, a follow-up audit should be carried out to ensure that the non-conformance
has been fixed.

Quality Audit Documentation:

Audit documentation does not need to be complicated, normally there are three documents
namely, the audit plan, audit notes and audit report.

1. Audit Plan: The audit plan is sent to the department being audited a few days prior, it
should include the date of the audit, the planned time, duration, auditor's names, location and
the policies and procedures that will be used during the audit. It should also mention any non-
conformances that were found during last audit.

2. Audit Notes: The notes are the Auditor's questions that will be asked during the audit. It
should include references to particular policies and procedures and what will be asked during
the audit. The same document should be used to record the findings and any comments
during the audit.

3. Audit Report: The audit report should include details of the audit, date, Auditor's name,
policies and procedures and findings against them.

6.2.4 Reviews

Software reviews are a filter for the software engineering process. Reviews are applied at
various points to find the errors before they are passed on to another software engineering
activity or released to the customer.

Reviews provide a powerful way to improve quality by providing a means by which defects
can be detected (and corrected) early in development.

A review is a way to:

1. Point out needed improvements in the product.

2. Confirm those parts of a product in which improvement is either not desired or not needed.

Objectives of Review:

1. To uncover errors in function, logic, or implementation for any representation of the

software.

2. To verify that the software under review meets its requirements.

3. To ensure that the software has been represented according to predefined standards.

4. To achieve software that is developed in a uniform manner.

5. To make projects more manageable.

Classes of Reviews:

• Reviews are divided into following classes:

1. Informal Reviews:

An informal meeting is a form of review if technical problems are discussed.

It is also called as peer-review i.e. simply giving a document to a colleague and asking them
to look at it closely which will identify defects, we might never find on our own.

It is generally one-on-one meeting between author of a work product and a peer. No agenda
is required and results are also not formally reported.

2. Formal Reviews:

A formal review can range from a simple meeting between two programmers to a detailed
inspection of the code.

6.3 QUALITY EVALUATION STANDARDS

In order to achieve the desired software quality, it is necessary to have information about
standards.

6.3.1 Six Sigma Principles

Now-a-days, six sigma is regarded as the most widely used strategy or technique for
evaluating software quality.

It prefers to six standard derivations, and these derivations describes the distribution of the
process used in software development. Six sigma is a statistical strategy to ensure quality
assurance of processes in the organization.

Six sigma is used to eliminate error or faults or bugs in the software by evaluating the
performance of the software process.

The objective of six sigma is to reduce variance and improve the processes in an
organization. In addition six sigma, minimizes the cost of poor quality.

It monitors day-to-day activities of an organization, in order to make optimal use of

resources. Below six standards of six sigma are described:

1. Good business strategy that balances cost, quality, features and constraints as well as
matches business priorities.

2. Decisions made are tied to bottom line of the organization.

3. Exercise care to use correct measurements in each situation.

4. Consider measuring output for a longer period.

5. Limited scope with adequate importance and reasonable time.

6. Clear quantitative measures to define success.

Six Sigma uses data and statistical analysis to measure and improve a company's operational
performance.

Three core steps of six sigma are:

1. Define customer requirements, deliverables, and project goals via well-defined methods of
customer communication.

2. Measure the existing process and its output to determine current quality performance
(collect defect metrics).

3. Analyze defect metrics and determine the vital few causes (the 20%).

Two additional steps are added for existing processes (and can be done in parallel):

1. Improve the process by eliminating the root causes of defects, and

2. Control the process to ensure that future work does not reintroduce the causes of defects.

All these steps mentioned in Fig. 6.2, need to be performed so that we can manage the
process to accomplish something.

Concept of DMAIC and DMADV:

Six sigma uses following approaches for processes in an organization:

1. DMAIC (Define, Measure, Analyze, Improve, Control) – used when existing processes
have fallen below acceptable levels and incremental improvement is desired; and

2. DMADV (Define, Measure, Analyze, Design, Verify) – used when new processes or
dramatic
1. DMAIC Method:

The letters in the abbreviation DMAIC stand for *Define, Measure, Analyze, Improve,
Control*, the steps in ordered process.

DMAIC checks whether a process is performing correctly. DMAIC improves the system by
improving the process that is not able to meet user requirements.

DMAIC used when a product or process is in existence and is not meeting customer
specification or is not performing adequately.

Many techniques in the Six sigma toolkit are directly applicable to software and are already
in use in the software industry. For instance, “voice of the client” and “quality function
deployment”are useful for developing customer requirements.

There are numerous charting/calculation techniques that can be used to scrutinize cost,
schedule, and quality (project-level and personal-level) data as a project proceeds. And for
technical development, there are quantitative methods for risk analysis and concept/design
selection.

2. DMADV:

The strength of “six sigma” comes from consciously and methodically deploying these tools
in a way that achieves customer satisfaction.

The letters in the abbreviation DMADV stand for *“Define, Measure, Analyze, Design,
Verify”*, the steps in the ordered process.
The DMADV methodology should be used when a product or process is in existence at an
organization but is not meeting customer specification or requirements.

DMADV is a process defined by Motorola as part of their Six Sigma management

philosophy.

DMADV is applied to new processes to make sure that they achieve Six Sigma quality. Six
Sigma sets extremely ambitious goals to minimize the occurrence of flaws in products and
services.

When to use DMADV?

1. A product or process is not in existence at an organization and one needs to be developed.

2. The existing product or process exists and has been optimized (using either DMAIC or not)
and still does not meet the level of customer specification or Six Sigma level.

Steps of DMADV are given below:

D: Define – State the problem, specify the customer set, identify the goals, and outline the
target process.

M: Measure – Decide what parameters need to be quantified, work out the best way to
measure them, collect the necessary data, and carry out the measurements by experiment.

A: Analyze – Identify performance goals and determine how process inputs are likely to
affect process outputs.

D: Design – Work out details, optimize the methods, run simulations if necessary, and plan
for design verification.

V: Verify – Check the design to be sure it was set up according to plan, conduct trials of the
processes to make sure that they work, and begin production or sales.

ISO 9000 Quality Standards:

The International Organization for Standardization established the term ISO 9000, which
refers to a set of quality management standards.

The ISO 9000 standards are maintained by ISO and administered by accreditation and
certification bodies. ISO’s purpose is to facilitate international trade by providing a single set
of standards that is recognized everywhere.

ISO first published its quality standards in 1987, revised them in 1994 and then republished
an updated version in 2000. These new standards are referred to as the “ISO 9000: 2000
Standards”.

ISO 9000:2000 Standards apply to all kinds of organizations in all kinds of areas. Some of
these areas include manufacturing, processing, printing, forestry, electronics, banking,
telecommunications, agriculture, government, education, software development,
biotechnology and so on.

6.3.2 CMMI

CMMI stands for Capability of Maturity Model Integration. CMMI is a process model that
provides a clear definition of what an organization should do to promote behaviors that lead
to improved performance.

The CMMI project is a collaborative effort to provide models for achieving product and
process improvement.

The primary focus of the project is to build tools to support improvement of processes used to
develop and sustain systems and products.

The output of the CMMI project is a suite of products, which provides an integrated approach
across the enterprise for improving processes, while reducing the redundancy, complexity
and cost resulting from the use of separate and multiple Capability Maturity Models (CMMs).

Software engineering and System engineering (CMMs) that integrates other engineering
models and it has two methods:

1. Staged CMMI, and

2. Managed CMMI.

The CMMI model is a framework for process improvement that has broad applicability
across a range of companies.

1. Process areas 2. Goals 3. Practices

Two methods of software engineering and system engineering which integrates other
engineering models are given below.

Staged CMMI Model:

The staged CMMI model is comparable with software. CMM in that it provides a means to
assess an organization’s process capability at one of five levels.

It prescribes the goals that should be achieved at each of these levels. Process improvement is
achieved by implementing practices at each level, moving from the lower to the higher levels
in the model.

Each maturity level has an associated set of process areas and generic goals. The five levels
in the staged CMMI model are shown in Fig. 6.3.
Advantages:

(i) It defines a clear improvement pathway for organizations.

(ii) The move from the second to third level and so on.

Disadvantages:

(i) It may be more appropriate to introduce goals and practices at higher levels before lower-
level practices.

(ii) When an organization does this, a maturity assessment will give a misleading picture of
its capability.

2. Continuous CMMI Model:

Continuous maturity models do not classify an organization according to discrete levels.

Rather they are finer-grained models that consider individual or groups of practices and
assess the use of each practice as shown in Fig. 6.4.

The maturity assessment is not, therefore a single value but a set of values showing the
organization’s maturity for each process or process group.

[Link] CMMI Levels

The six-point scale assigns a level to a process as follows:

1. Level 0: Not performed – One or more specific goals associated with the process area is
not satisfied.

2. Level 1: Performed – The goals associated with process area are satisfied and for all
processes the scope of the work to be performed is explicitly set out and communicated to the
team members.

3. Level 2: Managed – At this level, the goals associated with the process areas are met and
organizational policies are in place that define when each process should be used. There must
be documented plans and resource management and process monitoring products must be in
place across the institution.

4. Level 3: Defined – This level focuses on organizational standardization and deployment of

processes. Each project in an organization has a managed process that is tailored from a
defined set of organizational processes. Processes assets and process measurements must be
collected and used for future process improvements.

5. Level 4: Quantitatively Managed – At this level, there is an organizational responsibility to

use statistical and other qualitative methods to control sub-processes. That is collected
process and product measurements must be used in process management.

6. Level 5: Optimizing– At this highest level, the organization must use the process and
product measurements to drive process improvement. Trends must be analyzed and the
processes adapted to changing business needs.

[Link] Process Areas

CMMI defines each process area and these areas are defined in terms of specific goals and
specific practice required to achieve these goals.

Specific goal establishes the characteristics that must exist if the activities implied by process
are to be effective while specific practices refine a goal into a set of process related activities.

Fig. 6.5 shows Key Process Area (KPA) that exists in CMMI maturity level.

KPA determines a group of related activates, which are performed collectively to

accomplish/achieve goals for establishing the process at that maturity level.