2.

Testing throughout the software life cycle (K2)

2.1 Software development models (K2)
2.1.1 V-model (sequential development model) (K2)
2.1.2 Iterative-incremental development models (K2)
2.1.3 Testing within a life cycle model (K2)
2.2 Test levels (K2)
2.2.1 Component testing (K2)
2.2.2 Integration testing (K2)
2.2.3 System testing (K2)
2.2.4 Acceptance testing (K2)
2.3 Test types (K2)
2.3.1 Testing of function (functional testing) (K2)
2.3.2 Testing of non-functional software characteristics
(non-functional testing) (K2)
2.3.3 Testing of software structure/architecture (structural
testing) (K2)
2.3.4 Testing related to changes (confirmation testing
(retesting) and regression testing) (K2)
2.4 Maintenance testing (K2)
[Link] throughout the software life cycle
(K2)

We view testing as an activity we normally do after writing the software.

We base this assumption on the view that we cannot test anything
intangible. In reality, testing activities include more than just running
tests. We need to design and write the tests before we can execute them.

2.1Software development & Testing models (K2)

Testing does not exist in isolation; test activities are related to software
development activities.
Different development life cycle models need different approaches to
testing.

Waterfall Model
A software process model is an abstract representation of software. The
basic concept of the waterfall model considers software development as a
sequence of stages, and that software developers must clearly delineate
the stages: First we design and after we finish designing, we begin
coding. After we finish coding, we begin the testing and after we finish
the testing, we deploy.

Using the Waterfall Model has some advantages. The testing is inherent
to every phase of the model. We use an enforced discipline approach. We
produce documentation at every stage.

Requirements in software change more often and more rapidly than in

other engineering disciplines. In the Waterfall Model, the biggest
challenge is that we must have complete requirements from the
customers right from the start. Thus it is hard to modify and implement
changes.

2.1.1V-Model (Sequential Development Model)(K2)

SDLC
VS
TLC

Stages
Fig. 3: V-Model

The above figure shows V-Model testing. Every phase of the Testing Life
Cycle (TLC) in this model corresponds to some activity in the Software
Development Life Cycle (SDLC). The V proceeds from left to right,
depicting the basic sequence of development and testing activities. This
model highlights the existence of several levels of testing and depicts the
way each relates to a different development phase.

We use the output of requirement analysis, the requirement specification

document, as the input for writing system level test cases. We use the
output of the design phase of SDLC, the design specification document,
as the input for writing integration level test cases. We use the Program
specifications created, as an input for writing unit level test cases.

Inspite of these activities, the testing activity formally starts at the end of
the Coding phase or the Unit Testing activity by the developers. Note
that V-model does not say that testing activity must formally start after
coding. It is just a mapping of development phase with respect to various
testing levels.

As a key point, we must conduct testing in conjunction with V diagram

phases to minimize the testing effort in later stages.

When we do the designing later, fixings the defects found in requirement

specification becomes more expensive. The program requires a lot of
rework in all the subsequent stages in SDLC.

Early Run
Tests Tests
Design
Fig. 4: Early Test Design

Ideally, the tests must be designed as early as possible. The earlier a

defect is found, the cheaper the cost to fix it. Early design of tests leads
to early finding of defects and hence makes it cheaper to fix defects. It
also prevents defect multiplication because it prevents defects from
leaking into the next stage. E.g. we can prevent a defect in the coding by
finding it in the design stage. This ensures better quality and less time in
running tests because fewer defects are found. This results in an overall
cost and effort reduction.

In practice, a V-model may have more, fewer or different levels of

development and testing, depending on the project and the software
product. For example, there may be component integration testing after
component testing, and system integration testing after system testing.
Software work products (such as business scenarios or use cases,
requirement specifications, design documents and code) produced during
development are often the basis of testing in one or more test levels.
References for generic work products include Capability Maturity Model
Integration (CMMI) or ‘Software life cycle processes’ (IEEE/IEC 12207).
Verification and validation (and early test design) can be carried out
during the development of the software work products.

Verification, Validation and Testing

Verification and Validation, denoted as V and V, assures that a
software system meets the needs of the users.

Verification is defined as a ‘process of evaluating a system or its

components, to determine whether the products of the given
development phase, satisfy the conditions imposed at the start of that
phase’. In other words, the software must conform to its specification.
Verification ensures that each function works correctly. Verification
checks the ‘quality of the built product’. In other words, we check that
we built the product right.

Validation is defined as ‘the determination of the correctness of the

products of software development with respect to the user needs and
requirements’. Validation is the process of evaluating a system or a
component, during or at the end of the development process, to
determine whether it satisfies the specified requirements. Validation
ensures that the system has implemented each of the requirements and
that we can trace back each of the system functions to a particular
requirement. In other words, we check that we have built the right
product.

The following diagram indicates the V&V activities that can be conducted
during a typical SDLC.. Please note although the following diagram is
similar to the v-model, it is just a representation of the major
development and testing phases and there relevant V & V activities

Fig. 5: Validation and Verification

The Impact of Early Test Design on SDLC

By designing test cases early, we can specify the order in which to
develop the software before we build it. This helps in executing the
testing and development process in a parallel fashion. This enables us to
have total test execution schedules of a shorter duration and gives a
more even distribution of test effort across the SDLC.

2.1.2 Iterative-Incremental development models (K2)

Iterative-Incremental development is the process of establishing
requirements, designing, building and testing a system, done as a series
of shorter development cycles. Examples are: prototyping, rapid
application development (RAD) and agile development models. The
resulting system produced by an iteration may be tested at several levels
as part of its development. An increment, added to others developed
previously, forms a growing partial system, which should also be tested.
Regression testing is increasingly important on all iterations after the
first one. Verification and validation can be carried out on each
increment.

Prototyping Model is a systems development method (SDM) in which a

prototype (an early approximation of a final system or product) is built,
tested, and then reworked as necessary until an acceptable prototype is
finally achieved from which the complete system or product can now be
developed. This model works best in scenarios where not all of the
project requirements are known in detail ahead of time. It is an iterative,
trial-and-error process that takes place between the developers and the
users.

There are several steps in the Prototyping Model:

1. The new system requirements are defined in as much detail as

possible. This usually involves interviewing a number of users
representing all the departments or aspects of the existing system.
2. A preliminary design is created for the new system.
3. A first prototype of the new system is constructed from the
preliminary design. This is usually a scaled-down system, and
represents an approximation of the characteristics of the final
product.
4. The users thoroughly evaluate the first prototype, noting its strengths
and weaknesses, what needs to be added, and what should to be
removed. The developer collects and analyzes the remarks from the
users.
5. The first prototype is modified, based on the comments supplied by
the users, and a second prototype of the new system is constructed.
6. The second prototype is evaluated in the same manner as was the first
prototype.
7. The preceding steps are iterated as many times as necessary, until the
users are satisfied that the prototype represents the final product
desired.
8. The final system is constructed, based on the final prototype.
9. The final system is thoroughly evaluated and tested. Routine
maintenance is carried out on a continuing basis to prevent large-
scale failures and to minimize downtime.

RAD
RAD is an acronym for Rapid Application Development. As the name
suggests this methodology deals with developing applications within a
very short time frame. This is achieved by a number of ways like :
 collecting all the requirements right at the beginning so that the
delivered application is complete
 Making use of programming systems which have rich features to
reduce the development time
 Making use of reusable components instead of coding the entire
application
 Making use of COTS ( explained below)

COTS
COTS is an acronym for Commercial off-the-shelf software. COTS based
systems are systems wherein multiple off-the shelf parts from multiple
suppliers are integrated to achieve new or expanded system
functionality.

The main advantage of COTS based systems is that the delivery time and
cost is reduced because of the reduced total development time.

The challenges of COTS based systems are as follows :

 Limited visibility into the COTS code and behaviour
 Limited control of frequency or content of COTS releases
 ‘New Business’ issues like licensing, data rights, warranties
 Frequent, continual change of COTS products and market place
 COTS products are driven by the marketplace and not by the systems
whose development is based on them
 Dependencies between components to be moved to the proper
location.

Agile methodology is a development methodology which attempts to

minimize risk by developing software in short time boxes, called
iterations. Each iteration is like a miniature software project of its own. It
includes all the tasks necessary to release the mini-increment of new
functionality: planning, requirements analysis, design, coding, testing,
and documentation. While an iteration may not add enough functionality
to warrant releasing the product, an agile software project intends to be
capable of releasing new software at the end of every iteration. At the
end of each iteration, the team reevaluates project priorities. Agile
methods emphasize real-time communication, preferably face-to-face,
over written documents

Most agile teams are located in a bullpen and include all the people
necessary to finish software. At a minimum, this includes programmers
and their "customers." (Customers are the people who define the
product. They may be product managers, business analysts, or actual
customers.) The bullpen may also include testers, interaction designers,
technical writers, and management.

Agile methods also emphasize working software as the primary measure

of progress. Combined with the preference for face-to-face
communication, agile methods produce very little written documentation
relative to other methods. This has resulted in criticism of agile methods
as being undisciplined hacking.

RUP
RUP is an acronym for Rational Unified Process. It is a Software
Engineering Process. Its goal is to ensure the production of high-quality
software that meets the needs of its end-users, within a predictable
schedule and budget. It is observed that most of projects take a longer
development time because of improper communication amongst
development teams and also between development teams and customers.
RUP aims at improving this by the following means :
 providing a disciplined approach to assigning tasks and
responsibilities within a development organization
 providing every team member with easy access to a knowledge base
with guidelines, templates and tool mentors for all critical
development activities
 ensuring that all team members share a common language, process
and view of how to develop software
 focusing on creating and maintaining models (semantically rich
representations of the software system under development) rather
than focusing on production of large amount of paper documents
 using UML (Unified Modeling language) which is a industry-standard
language that allows to clearly communicate requirements,
architectures and designs.

2.1.3Testing within a life cycle model (K2)

In any life cycle model, there are several characteristics of good testing:
 For every development activity there is a corresponding testing
activity.
Eg in V-model we use the output of requirement analysis, the
requirement specification document, as the input for writing system
level test cases.

 Each test level has test objectives specific to that level.

Eg in iterative development model, the objective of testing at the
proto-type level is to get the application flow and GUI reviewed by the
user.

 The analysis and design of tests for a given test level should begin
during the corresponding development activity.
Eg. The analysis and design of system test should begin after the
requirement specifications are ready and high-level design is in
progress. This will help to find out gaps, if any , in requirements,
early enough.

 Testers should be involved in reviewing documents as soon as drafts

are available in the development life cycle.
Eg If the testers are involved in reviewing the draft program
specifications, they can find out topics missed out and also ensure that
the document is written in an unambiguous way. If this is not done
then there is a possibility that the missed out requirements are not
discovered until the User acceptance testing phase and / or the
developers misinterpret the specifications while coding and the coding
defects may get detected during component/unit
testing.

Test levels can be combined or reorganized depending on the nature of

the project or the system
architecture. For example, for the integration of a commercial-off-the-shelf (COTS) software
product into a system, the purchaser may perform integration testing at the system level (e.g.
integration to the infrastructure and other systems, or system deployment) and acceptance
testing (functional and/or non-functional, and user and/or operational testing).

2.2Test levels (K2)

For each of the test levels, the following can be identified: their generic
objectives, the work product(s) being referenced for deriving test cases
(i.e. the test basis), the test object (i.e. what is being tested), typical
defects and failures to be found, test harness requirements and tool
support, and specific approaches and responsibilities.

2.2.1Component Testing (K2)

Component/Unit Testing is testing at the lowest level. It is at the bottom on the V-Model
software development life cycle. Component testing searches for defects in, and verifies the
functioning of, software (e.g. modules, programs, objects, classes, etc.) that are separately
testable. It may be done in isolation from the rest of the system, depending on the context of
the development life cycle and the system. Stubs, drivers and simulators may be used. In this
type of testing, generally, programmers test their own code (in case of White Box Testing).
We can use both black box and white box testing techniques. It is also known as Unit /
Module / Program testing.

Component testing may include testing of functionality and specific non-

functional characteristics, such as resource-behavior (e.g. memory leaks)
or robustness testing, as well as structural testing (e.g. branch
coverage). Test cases are derived from work products such as a
specification of the component, the software design or the data model.

Typically, component testing occurs with access to the code being tested
and with the support of
the development environment, such as a unit test framework or
debugging tool, and, in practice,
usually involves the programmer who wrote the code. Defects are
typically fixed as soon as they
are found, without formally recording incidents.

The programmer who wrote the code generally performs component

testing making it a sensible and the most economic approach. A
programmer, who executes test cases on his/her own code, can usually
track down and fix any defects revealed by the tests relatively quickly. If
a tester executes the test cases, he/she must document each failure.
Eventually the programmer investigates each of the defect reports and
perhaps reproduces them in order to determine their causes. Once fixed,
the tester re-tests the software to confirm that each defect had indeed
been fixed. This amounts to more effort and yet has the same outcome.

But, an organization must bring some independence into the test

specification activity. Someone other than the programmer must also
specify test cases. Both functional and structural test case design
techniques are appropriate, though we must define the extent of their
use during the test planning activity. This depends on the risks involved.

One approach in component testing is to prepare and automate test

cases before coding. This is
called a test-first approach or test-driven development. This approach is
highly iterative and is based on cycles of developing test cases, then
building and integrating small pieces of code, and executing the
component tests until they pass.

2.2.2Integration testing (K2)

Integration testing tests interfaces between components, interactions to

different parts of a system, such as the operating system, file system,
hardware or interfaces between systems.
It is performed to expose defects in the interfaces and in the interaction
between integrated components. Testers should understand the
architecture to plan the integration testing. It is highly recommended
that They be included in the integration testing planning and design.

Integration is concerned with the process of combining components into

an overall system. In software, we normally have integration at two
levels. First the integration of components at the module level into a
system, sometimes known as component integration testing or
integration in the small. Second the integration of systems into a larger
system, sometimes known as system integration testing or integration
testing in the large.

Integration testing is all encompassing. It not only includes the correct

implementation of the interface between components, but also checks
whether the integrated components, now a system, behave as specified.
This behavior covers both functional and non-functional aspects of the
integrated system. Fig. 7 shows two components interacting to form an
integrated system.

Fig. 7: Integration

E.g.: We consider two components (actually one or more) as integrated

when:
1 They have been compiled, linked, and loaded together.
2 They have successfully passed the integration tests at the interface
between them.
Thus, we create a new, larger component (A, B) by integrating
components A and B. Note that this does not conflict with the idea of
incremental integration. It just means that we added a big component ‘A’
and a small component ‘B’. This level of testing is also called Link
Testing or Sub-System Testing. A better name for Link Testing as
mentioned in the Practitioner Syllabus: Component Integration
Testing.

The greater the scope of integration, the more difficult it becomes to

isolate failures to a specific
component or system, which may lead to increased risk.

As we combine more and more components together, we form a

subsystem. This has a more system-like functionality that we can test.
Testing non-functional aspects, such as performance, at this stage may
prove useful. For integration testing in the small, we have two choices to
make:
1 How many components to combine in one go?
2 In what order to combine the components?

There are three main reasons for Integration Testing:

1 To find defects not found in component testing because they become
apparent only after integrating the components.
2 To generate credible information about the software under test, so
that technical and business decisions can be made.
3 Confirm that even as the system grows, there is a diminished risk of
failure.

The decision over which choices are made is called the ‘integration
strategy’. The two main integration strategies are Big Bang and
Incremental.

Big Bang Integration

In “Big Bang” Integration, we put together all the components in one go.
We assume that since all components have already undergone testing at
the unit level and have no defects, we can now put them together.

The Big-Bang approach has a very simple philosophy where we basically

construct and test all the modules or builds independent of each other
and on finishing, we put them all together at the same time. The main
advantage of this approach is that it is very quick. No drivers or stubs are
needed and this cuts down development time. However it increases the
difficulty of locating and fixing a defect in case a problem arises,
ultimately making it much more time consuming.

Incremental Integration
We define Incremental Integration as combining a small number of
components all at once. The minimum number of components added at
each increment is one.

The main advantage of integration testing is that it makes the location as

well as fixing of defects and recovery by reverting to the last known good
baseline very easy. Three main incremental integration strategies
determine the order of combination of the components:
1 Top Down.
2 Bottom-Up.
3 Functional Incrementation.

Top Down Integration and Stubs

In Top down integration we develop modules and tested them starting at
the top level of the programming hierarchy, continuing on to the lower
levels. In this type of integration, we combine components starting with
the highest level in the hierarchy. We integrate all components at a
certain level and tested them before moving to the next level. We use
stubs, temporary programs for testing purpose in Top Down Integration,
in the place of unavailable components. Stubs are similar to programs.
Drivers are also known as test harness or scaffolding.

Driver Works like a program

Module to be tested Results

Stub
Stub

Fig. 8 : Stubs and Drivers

Bottom Up Integration and Drivers

Bottom-up strategy, as the name suggests, is the opposite of the Top-
down method. This process starts with the building and testing of the low
level modules, working its way up the hierarchy. As we start with the
lowest level components, all calling components may not be available.
Hence we make use of drivers. We can also use stubs in bottom-up
integration, though not frequently. We use a Driver, a temporary
program, for testing purpose. It replaces a calling program, in its
absence.

Top Down Bottom Up

Program A Driver Program A is

not ready
Fig. 9:Top down and Bottom Up Strategy

Functional Incrementation
The third integration strategy is Functional Incrementation. Use this to
achieve a basic functionality with a minimum number of components
integrated. Functional Incrementation is defined as “a strategy for
combining components in integration testing in the small where they are
combined to achieve some minimum capability or to follow a thread of
execution of transactions”.

Consider the following points when performing integration testing:

 Adding small increments to the baselines.
 Considering use of larger increments if using stubs and drivers.
 Keeping stubs and drivers as simple as possible.
 Planning the Integration Testing in the small as early as possible in
SDLC. This saves a lot of time and effort.

Integration Testing in rhe large , also called as “System Integration

Testing”, where integration testing of two or more system components
takes place. Specifically, system integration testing is the testing of
software components distributed across multiple platforms (e.g., client,
web server, application server, and database server) to produce failures
caused by system integration defects (i.e., defects involving distribution
and back-office integration).

The typical objectives of system integration testing are to:

 Cause failures involving the distribution of integrated system
components.
 Cause failures involving integration of the system with other
applications (e.g., legacy back-office systems).
 Report these failures to the development team so that they can fix
them.
 Help the development team to ensure that the software has been
successfully distributed to improve the effectiveness of system and
launch testing.
 Minimize the number of low-level defects that prevent effective
system and launch testing.

System integration testing can typically begin when the following

preconditions hold:
 The integration team is adequately staffed and trained in system
integration testing.
 The integration environment is ready.
 At least two software components have:
 Passed the relevant software integration testing.
 Been ported to the integration environment.
 Been integrated.

System integration testing is typically complete when the following post

conditions hold:
 All system components have been integrated (e.g., software
components have been distributed to their hardware components).
 A system integration test suite of test cases exists for each interface
between software components distributed to different hardware
components.
 All system integration test suites execute successfully (i.e., the tests
completely execute and the actual test results match the expected test
results).

The system passes the subset of the black box functional system tests
provided by the independent test team, whereby these tests exercise
major system interfaces, especially between distributed software
components and with external server systems. Now after identification of
all of the package's components, we can finalize the package through the
inclusion of scripts and other external components, and then automate
the package for deployment. This refined package serves to support the
next battery of tests.

It ensures that:
 Our system can interact with different networks, operating systems
and communication middleware.
 Our system can work with other systems like inventory, payroll etc
 Our system works with 3rd party packages.

Planning for system integration testing , includes:

 Testing of connections/interfaces one at a time.
 Identifying risks and testing most critical areas first.
 Identifying the various resources required.
 Considering different operating systems.
 Considering different machine configurations.
 Considering different network configurations.
 Considering a tie-up with some hardware manufacturers to use their
sites for testing.
 2.2.3System testing (K2)
In System Testing, we test the entire product or a major portion of it at
one time.

System testing is concerned with the behaviour of a whole

system/product as defined by the scope of a development project or
programme.

In system testing, the test environment should correspond to the final

target or production
environment as much as possible in order to minimize the risk of
environment-specific failures not being found in testing.

System testing may include tests based on risks and/or on requirements

specifications, business
processes, use cases, or other high level descriptions of system behavior,
interactions with the
operating system, and system resources.

It is divided into two categories: Functional System Testing and Non-

Functional System Testing. Functional System Testing is further divided
into two more categories: Requirements-based testing and Business
Process-based testing.

Functional System Testing

In Functional System Testing, we test the completed application as a
whole, to determine that it provides all of the behaviors required of it. It
also covers testing of completed increments that provide some degree of
end-user functionality.

In this type of Testing, we search for defects or variances between the

actual operation of the system and the requirements for the system. We
treat the system as a black box and analyze the expected behavior of the
system, according to its functional specification and generate a test
procedure for each of the possible usage scenarios.

It also corresponds to use case scenarios. Here we analyze how a change

in one part of the system affects other parts of the test cases. The result
of one test case produces the data that serves as the input for the next
test case.

Requirements-Based Testing
Requirements-based testing uses a specification of the functional
requirements for the system, as the basis for designing tests. Use the
table of contents of the requirement specification, as an initial test
inventory or list of items to test (or not to test). We must also prioritize
the requirements, based on risk criteria, and use this to prioritize the
tests. This ensures that the most important and most critical tests are
included in the system testing effort. Requirement based testing mainly:
1 Is based on functional requirements.
2 Uses specification of functional requirements.

Business Process-Based Testing

Business process-based testing is defined as “testing based on expected
user profiles such as scenarios or use cases used in system testing and
acceptance testing.” This type of testing is based on business process.
Here we use knowledge of business profiles (situations in day to day
business use of system). We also create test cases from a business
perspective.

Non-Functional System Testing

Non-functional system testing is defined as “testing of system
requirements that do not relate to functionality, i.e. performance,
usability, etc. Also known as quality attributes.” The various types of non-
functional system testing include:
 Load, performance & stress testing.
 Usability testing.
 Configuration & Installation testing.
 Reliability testing.
 Back-up & Recovery testing.
 Documentation testing.

We have mentioned about non-functional system testing in detail later in

section 2.3.2

2.2.4Acceptance testing (K2)

Acceptance testing is often the responsibility of the customers or users of
a system; other
stakeholders may be involved as well.

The goal in acceptance testing is to establish confidence in the system,

parts of the system or specific non-functional characteristics of the
system. Finding defects is not the main focus in acceptance testing.
Acceptance testing may assess the system’s readiness for deployment
and use, although it is not necessarily the final level of testing. For
example, a large-scale system integration test may come after the
acceptance test for a system.

Acceptance testing may occur as more than just a single test level, for
example:
 A COTS software product may be acceptance tested when it is
installed or integrated.
 Acceptance testing of the usability of a component may be done
during component testing.
 Acceptance testing of a new functional enhancement may come before
system testing.

Typical forms of acceptance testing include the following:

User Acceptance Testing

In User Acceptance Testing (UAT), we test the software, against certain
pre-determined criteria for acceptance. The UAT is the final stage of
validation. We must involve the users in the test specification of UAT. In
this type, we conduct testing in the replica of a real environment. UAT is
also known as Beta Testing (usually in the PC world) QA Testing,
Application Testing or End User Testing. This phase focuses on
functionality testing to check whether the system meets user acceptance
criteria or not.

By users we mean the real business users, who operate the system like
the staff of an organization, the suppliers or customers. They understand
the business exactly, and know how it operates and therefore, they are
the only people qualified to check a system to see if it delivers any benefit
to the business or organization.

It is unlikely that system developers know much about the realities of

running the organization, other than what they acquire from
requirements specifications, and similar documents. In addition they
have close involvement in the design compromises that always take place
during system development, and so have a commitment to the system as
it is. In UAT, business users try and make a system fail, taking into
account the real organization it must work in. UAT checks the system in
the context of the business environment it must operate in.

In short we must protect the organization from harm. Any changes in a

business, especially installing new computer systems, expose it to many
risks like:
 Reputation Risk: When external people such as customers, suppliers,
or legal authorities perceive a problem with the organization, and
decide not to use it, or in the case of legal authorities, give it more
scrutiny then they have before.
 Legal Risk: The possibility that the system could break laws, leaving
the organization open to legal proceedings.
 Time Risk: The system may not meet key business deadlines. We must
discover this in testing and not when a system goes live.
 Resource Risk: If a system does not properly integrate with our
organization, we may have to expend a lot of resources for working
around the system, adding to the cost.
Therefore the main reason for UAT is to find out what a system does to
our organization before we implement it. The final decision is then made
based on the evidence presented by the testing.

Operational (acceptance) testing

The acceptance of the system by the system administrators, including:
 testing of backup/restore;
 disaster recovery;
 user management;
 maintenance tasks;
 periodic checks of security vulnerabilities.

Contract and regulation Acceptance Testing

If a system is the subject of a legally binding contract, there may be
certain areas/aspects of the system that we have to test. In this type, we
conduct testing as per the contract. When the contract is negotiated,
both customer and supplier focus on agreeing about the specifications
and the timetable for delivery. Often the parties are content to deal with
the important area of acceptance on an “agreement to agree” basis, but
this is legally unsatisfactory, as an “agreement to agree” is not
enforceable. If the parties insist on taking this approach, the lawyer (for
the customer) must try to use contractual wording to at least describe
the parameters and timing of the testing process.
Regulation acceptance testing is performed against any regulations which must be adhered to,
such as governmental, legal or safety regulations

Alpha and Beta (Field) Testing

We perform Alpha and Beta testing when the software seems stable. The
people who represent our market use the product as if they have bought
the finished version and give us their comments. This helps the
developers of market, or COTS, software get feedback from potential or
existing customers in their market before the software product is put up
for sale commercially Alpha tests are performed at the developer’s site,
while beta tests are performed at the testers’ sites. The customer, in the
presence of the developer, conducts Alpha Testing at the developer’s
location. We conduct testing in a controlled environment. Alpha testing is
the software prototype stage when the software is first able to run. It
does not have all the intended functionality, but has core functions and is
able to accept inputs and generate outputs.

Beta Testing is conducted at customer’s location by the end user. The

developer is not present during testing. The end user, in live condition,
conducts pre-release test. It is useful in making the final release
acceptable to the end user.

Both Alpha and Beta testing are performed by potential customers, not
the developers of the [Link] may use other terms as well,
such as factory acceptance testing and site acceptance testing for
systems that are tested before and after being moved to a customer’s
site.

2.3Test types (K2)

A group of test activities can be aimed at verifying the software system
(or a part of a system) based on a specific reason or target for testing.

A test type is focused on a particular test objective, which could be the

testing of a function to be performed by the software; a non-functional
quality characteristic, such as reliability or usability, the structure or
architecture of the software or system; or related to changes, i.e.
confirming that defects have been fixed (confirmation testing) and
looking for unintended changes (regression testing).

A model of the software may be developed and/or used in structural and

functional testing. For
example, in functional testing a process flow model, a state transition
model or a plain language
specification; and for structural testing a control flow model or menu
structure model.

2.3.1Testing of function (functional testing) (K2)

The functions that a system, subsystem or component are to perform may
be described in work
products such as a requirements specification, use cases, or a functional
specification, or they may be undocumented. The functions are “what”
the system does.

Functional tests are based on functions and features (described in

documents or understood by the testers), and their interoperability with
specific systems, and may be performed at all test levels (e.g. tests for
components may be based on a component specification).

Specification-based techniques may be used to derive test conditions and

test cases from the
functionality of the software or system. (See Chapter 4.) Functional
testing considers the external
behaviour of the software (black-box testing). A type of functional testing,
security testing, investigates the functions (e.g. a firewall) relating to
detection of threats, such as viruses, from malicious outsiders. It
attempts to verify that protection mechanisms built into a system do, in
fact, protect it from improper penetration.
Another type of functional testing , interoperability testing, evaluates the
capability of the software product to interact with one or more specified
components or systems..
 2.3.2Testing of non-functional software characteristics (non-
functional testing) (K2)
Non-functional testing includes, but is not limited to, performance
testing, load testing, stress
testing, usability testing, maintainability testing, reliability testing and
portability testing. It is the testing of “how” the system works .

Non-functional testing may be performed at all test levels. The term non-
functional testing describes the tests required to measure characteristics
of systems and software that can be quantified on a varying scale, such
as response times for performance testing. These tests can be referenced
to a quality model such as the one defined in ‘Software Engineering –
Software Product Quality’ (ISO 9126).

Non-Functional System Testing

Non-functional system testing is defined as “testing of system
requirements that do not relate to functionality, i.e. performance,
usability, etc. Also known as quality attributes.” The various types of non-
functional system testing include:
 Load, performance & stress testing.
 Usability testing.
 Configuration & Installation testing.
 Reliability testing.
 Back-up & Recovery testing.
 Documentation testing.

Load, Performance & Stress testing

Performance testing is defined as the “testing conducted to evaluate
the compliance of a system or component with specified performance
requirements.”

In performance testing we measure and evaluate response times,

transaction rates, and other time-sensitive requirements. Performance
testing of a software system usually includes:
 Resource usage.
 Throughput.
 Stimulus-response time.
 Queue lengths detailing the average or maximum number of tasks
waiting to be serviced by selected resources.

The goals of performance testing include:

 Performance.
 Bottleneck identification.
 Performance comparison and evaluation.

Typically, we conduct performance testing by using a benchmark, a

program or workload designed to be representative of the typical system
usage.
In Load testing we subject the target of the test, to varying workloads,
to measure and evaluate the performance behaviors and ability of the
target and of the test to continue to function properly under these
different workloads.

Stress Testing is defined as the “testing conducted to evaluate a system

or component at or beyond the limits of its specified requirements.” In
stress testing we implement and execute tests to find errors due to low
resources or competition for resources. Low memory or disk space may
reveal defects in the target-of-test. Stress Testing determines the
breaking point or unacceptable performance point of a system to reveal
the maximum service level it can achieve.

Usability Testing
Usability testing refers to evaluating the ease with which users can learn
and use a product. The scope of this phrase, however, can mean different
things to different people. Some people refer to define usability testing as
the process that employs participants who are representative of the
target population, to evaluate the degree to which a product meets
specific usability criteria (Barnum, 2002). Other practitioners use the
term more globally to refer to any technique for evaluating the ease of
use of a product or system. The authors tend to concur with the latter
definition because our research shows participant involvement as a
desirable, but not essential, part of evaluating all aspects of system
usability.

Usability testing helps to check:

 How people interact with the system we are testing?
 What do people find easy or difficult?
 What makes sense about the system?
 What is exciting about the system?
 What changes do the users want to see?
 What do they really dislike about it?

The way subjects actually use our system may reveal bugs invisible to us.
It may also suggest enhancements that seemed un-necessary when we
tested the system. To perform usability testing, we require inputs from
real users.

Configuration and Installation Testing

We define configuration testing as the system testing of different
variations of an integrated, black box application against its
configurability requirements.

The typical objectives of configuration testing are to:

 Validate the application partially (i.e., to determine if it fulfills its
configurability requirements).
 Cause failures concerning the configurability requirements that help
identify defects not efficiently found during unit and integration
testing like:
 Functional Variants.
 Internationalization (e.g., multiple languages, currencies, taxes
and tariffs, time zones, etc.).
 Personalization.
 Report these failures to the development teams so that they can fix
the associated defects.
 Determine the effect of adding or modifying hardware resources such
as:
 Memory.
 Disk and tape resources.
 Processors.
 Load balancers.
 Determine an optimal system configuration.

We can test with different hardware and software configurations,

different upgrade paths, distribution media (floppy, CD, FTP site), and
perform installation/un-installation using the installation guide.

We perform Installation Testing to ensure successful ship-out of all

components of software and installation of software with help of
installation note provided. Objects under test include:
 Installation / Release notes / Release procedure.
 Documentation: User / System / Training.
 Feature / Function / Requirements list.
 Source / Object / Executable.
 Interfaced objects / Utilities.
 Media provided for Installation (Floppy/CD).
 Hardware Keys.

In Installation Testing we test:

 Installation on Various Operating Systems.
 Database Creation.
 Detection of Hardware key at the time of installation.
 Creation of System Data.

Reliability Testing
We define Reliability Testing as testing that the software does not cause
failure of the system for a specified time under specified conditions.
Other quality attributes are:
 Maintainability.
 Portability.
 Availability.

Back-Up and Recovery Testing

Backup Testing is a test of the back up procedures. In this we check if
the backups taken manually or automatically and test that the backups
are restored and verified.

Recovery Testing is a system testing that forces the software to fail in a

variety of ways and verifies that recovery is properly performed. This
ensures a defect tolerant system. We find the recovery time after failure
and check if recovery is automatic or manual.

Documentation Testing
With documentation testing we thoroughly review all written material in
the final solution like user manual or application maintenance manual.
People unfamiliar with the application can perform documentation
testing by following documented procedures. We must write the
documented procedures in a way that provides step-by-step guidance on
how to accomplish the given task. We need to improve the procedures if
the testers cannot successfully complete them.

We must try all the alternative ways documented to accomplish a task. In

many cases, the.
primary way works, but the alternatives do not. We must review the
material for language,
punctuation, and grammar, and for professionalism and readability.
Include a separate section
for errors in documentation in the defect report under the heading
Documentation Slippage or
Documentation Defect

2.3.3Testing of software structure/architecture (structural

testing) (K2)
Structural (white-box) testing may be performed at all test levels.
Structural techniques are best used after specification-based techniques,
in order to help measure the thoroughness of testing through assessment
of coverage of a type of structure.

Coverage is the extent that a structure has been exercised by a test suite,
expressed as a percentage of the items being covered. If coverage is not
100%, then more tests may be designed to test those items that were
missed and, therefore, increase coverage. Coverage techniques are
covered in Chapter 4.
At all test levels, but especially in component testing and component
integration testing, tools can be used to measure the code coverage of
elements, such as statements or decisions. Structural testing may be
based on the architecture of the system, such as a calling hierarchy.
Structural testing approaches can also be applied at system, system
integration or acceptance testing levels (e.g. to business models or menu
structures).

2.3.4Testing related to changes (confirmation

testing(retesting) and regression testing) (K2)

After a defect is detected and fixed the software should be retested to

confirm that the original defect has been successfully removed. This is
called confirmation testing. Debugging (defect fixing) is a development
activity, not a testing activity. Regression testing is the repeated testing
of an already tested program, after modification, to discover any defects
introduced or uncovered as a result of the change(s). These defects may
be either in the software being tested, or in another related or unrelated
software component. It is performed when the software, or its
environment, is changed. The extent of regression testing is based on the
risk of not finding defects in software that was working previously.

Tests should be repeatable if they are to be used for confirmation testing

and to assist regression
testing. Regression testing may be performed at all test levels, and
applies to functional, non- functional and structural testing. Regression
test suites are run many times and generally evolve slowly, so regression
testing is a strong candidate for automation.

2.4Maintenance Testing (K2)

A systems requirements may change when after the system is developed
because of changes in the environment. Therefore, a deployed system
does not meet its the changed requirements or environment. Systems
are tightly coupled with their environment. Such software systems must
be maintained if we want them to remain useful in an environment

The purpose of the Maintenance Testing is to sustain the capability of the

system to provide a service. Software Maintenance consists of activities
required to keep a software system operational and responsive, after the
customer accepts it and it is in production. Maintenance Testing is
preserving the quality we have already achieved. Implementing and
managing changes to legacy systems, presents as a key problem for
organizations.
Fig. 10: Software Maintenance
Maintenance testing is done on an existing operational system, and is triggered by
modifications, migration, or retirement of the software or system.

We perform Maintenance Testing to:

 Repair software defects (Corrective): Changing a system to correct
deficiencies in the way it meets requirements.
 Adapt software to a different operating environment (Adaptive):
Changing a system so that it operates in a different environment
(computer, OS, etc.) from its initial implementation. This is called
Migration of systems.
 Add to or modify the system’s functionality (Perfective): Modifying the
system to satisfy new requirements.
 Prevent a future problem or to make the system maintainable
(Preventive): Y2K
 Check whether or not the new system meets the requirements which
the retired system met.

Maintenance Testing includes a lot of re-testing and regression testing. Determining how the
existing system may be affected by changes is called impact analysis, and is used to help
decide how much regression testing to do.

At the time of development, testing starts with the partial system and
progresses to complete system. For maintenance testing, we have the
entire system available. We do the testing using live data.

We can adopt two methodologies here:

 Breadth test: Broad test over entire system, not in detail though.
 Depth test: Testing only specific areas in detail.

We use Depth tests to analyze the impact of change, identify affected

areas and carry out depth testing in higher risk areas.

To conduct Breadth tests, we can use regression tests.

Fig. 11: For Continuous Development

Poor or Missing Specifications

The lack of specifications in old systems along with out-of-date
documentation makes maintenance testing difficult. Generally, we do not
have regression scripts and have limited knowledge of the system. So
talking to users and understanding the behavior of the system presents
as the best possible solution to the above problems. We must document
everything, including our views, and explore the current system to
identify expected behavior. We must also refer to any available
documentation, including user guides and manuals and explore the
possibility of asking the experts.