Designing Test Suites

ntroduction

Development of a new software, like any other product, remains

incomplete until it subjected to exhaustive tests. The primary objective
of testing is not to verify that all desired features have been
implemented correctly. However, it also includes verification of the
software behavior in case of "bad inputs".

In this experiment we discuss in brief about different types of testing,

and provide mechanisms to have hands-on experience on unit testing.

Objectives

After completing this experiment you will be able to:

 Learn about different techniques of testing a software

 Design unit test cases to verify the functionality and locate bugs, if
any

Time Required

Around 3.00 hours

Software Testing

Testing software is an important part of the development life cycle of a

software. It is an expensive activity. Hence, appropriate testing
methods are necessary for ensuring the reliability of a program.
According to the ANSI/IEEE 1059 standard, the definition of testing is
the process of analyzing a software item, to detect the differences
between existing and required conditions i.e. defects/errors/bugs and
to evaluate the features of the software item.

The purpose of testing is to verify and validate a software and to find

the defects present in a software. The purpose of finding those
problems is to get them fixed.

 Verification is the checking or we can say the testing of software

for consistency and conformance by evaluating the results against
pre-specified requirements.

 Validation looks at the systems correctness, i.e. the process of

checking that what has been specified is what the user actually
wanted.

 Defect is a variance between the expected and actual result. The

defect’s ultimate source may be traced to a fault introduced in
the specification, design, or development (coding) phases.

Standards for Software Test Documentation

IEEE 829-1998 is known as the 829 Standard for Software Test

Documentation. It is an IEEE standard that specifies the form of a set of
documents for use in software testing [i]. There are other different
standards discussed below.

 IEEE 1008, a standard for unit testing

 IEEE 1012, a standard for Software Verification and Validation

 IEEE 1028, a standard for software inspections

 IEEE 1044, a standard for the classification of software anomalies

 IEEE 1044-1, a guide to the classification of software anomalies

  IEEE 830, a guide for developing system requirements
specifications

 IEEE 730, a standard for software quality assurance plans

 IEEE 1061, a standard for software quality metrics and

methodology

 IEEE 12207, a standard for software life cycle processes and life
cycle data

 BS 7925-1, a vocabulary of terms used in software testing

 BS 7925-2, a standard for software component testing

Testing Frameworks

Following are the different testing frameworks:

 jUnit - for Java unit test [ii]

 Selenium - is a suite of tools for automating web applications for

software testing purposes, plugin for Firefox [iii]

 HP QC - is the HP Web-based test management tool. It familiarizes

with the process of defining releases, specifying requirements,
planning tests, executing tests, tracking defects, alerting on
changes, and analyzing results. It also shows how to customize
project [iv]

 IBM Rational - Rational software has a solution to support

business sector for designing, implementing and testing
software [v]

Need for Software Testing

There are many reasons for why we should test software, such as:

 Software testing identifies the software faults. The removal of

faults helps reduce the number of system failures. Reducing
failures improves the reliability and the quality of the systems.

 Software testing can also improves the other system qualities

such as maintainability, usability, and testability.

 In order to meet the condition that the last few years of the 20th
century systems had to be shown to be free from the ‘millennium
bug’.

 In order to meet the different legal requirements.

 In order to meet industry specific standards such as the

Aerospace, Missile and Railway Signaling standards.

Test Cases and Test Suite

A test case describes an input descriptions and an expected output

descriptions. Input are of two types: preconditions (circumstances that
hold prior to test case execution) and the actual inputs that are
identified by some testing methods. The set of test cases is called a test
suite. We may have a test suite of all possible test cases.

Types of Software Testing

Testing is done in every stage of software development life cycle, but

the testing done at each level of software development is different in
nature and has different objectives. There are different types of testing,
such as stress testing, volume testing, configuration testing,
compatibility testing, recovery testing, maintenance testing,
documentation testing, and usability testing. Software testing are
mainlyof following types [1]

1. Unit Testing

2. Integration Testing

3. System Testing

Unit Testing

Unit testing is done at the lowest level. It tests the basic unit of
software, that is the smallest testable piece of software. The individual
component or unit of a program are tested in unit testing. Unit testing
are of two types.

 Black box testing: This is also known as functional testing , where

the test cases are designed based on input output values only.
There are many types of Black Box Testing but following are the
prominent ones.

- Equivalence class partitioning: In this approach, the domain of input

values to a program is divided into a set of equivalence classes. e.g.
Consider a software program that computes whether an integer
number is even or not that is in the range of 0 to 10. Determine the
equivalence class test suite. There are three equivalence classes for this
program. - The set of negative integer - The integers in the range 0 to
10 - The integer larger than 10

- Boundary value analysis : In this approach, while designing the test

cases, the values at boundaries of different equivalence classes are
taken into consideration. e.g. In the above given example as in
equivalence class partitioning, a boundary values based test suite is { 0,
-1, 10, 11 }

 White box testing: It is also known as structural testing. In this

testing, test cases are designed on the basis of examination of the
[Link] testing is performed based on the knowledge of how
the system is implemented. It includes analyzing data flow,
control flow, information flow, coding practices, exception and
error handling within the system, to test the intended and
unintended software behavior. White box testing can be
performed to validate whether code implementation follows
intended design, to validate implemented security functionality,
and to uncover exploitable [Link] testing requires
access to the source code. Though white box testing can be
performed any time in the life cycle after the code is developed,
but it is a good practice to perform white box testing during the
unit testing phase.

Integration Testing

Integration testing is performed when two or more tested units are

combined into a larger structure. The main objective of this testing is to
check whether the different modules of a program interface with each
other properly or not. This testing is mainly of two types:

 Top-down approach

 Bottom-up approach

In bottom-up approach, each subsystem is tested separately and then

the full system is tested. But the top-down integration testing starts
with the main routine and one or two subordinate routines in the
system. After the top-level ‘skeleton’ has been tested, the immediately
subroutines of the ‘skeleton’ are combined with it and tested.

System Testing

System testing tends to affirm the end-to-end quality of the entire

system. System testing is often based on the functional / requirement
specification of the system. Non-functional quality attributes, such as
reliability, security, and maintainability are also checked. There are
three types of system testing

 Alpha testing is done by the developers who develop the

software. This testing is also done by the client or an outsider with
the presence of developer or we can say tester.

 Beta testing is done by very few number of end users before the
delivery, where the change requests are fixed, if the user gives
any feedback or reports any type of defect.

 User Acceptance testing is also another level of the system

testing process where the system is tested for acceptability. This
test evaluates the system's compliance with the client
requirements and assess whether it is acceptable for software
delivery

An error correction may introduce new errors. Therefore, after every

round of error-fixing, another testing is carried out, i.e. called
regression testing. Regression testing does not belong to either unit
testing, integration testing, or system testing, instead, it is a separate
dimension to these three forms of testing.

Regression Testing
The purpose of regression testing is to ensure that bug fixes and new
functionality introduced in a software do not adversely affect the
unmodified parts of the program [2]. Regression testing is an important
activity at both testing and maintenance phases. When a piece of
software is modified, it is necessary to ensure that the quality of the
software is preserved. To this end, regression testing is to retest the
software using the test cases selected from the original test suite.

Example

Write a program to calculate the square of a number in the range 1-100

#include <stdio.h>

int

main()

int n, res;

printf("Enter a number: ");

scanf("%d", &n);

if (n >= 1 && n <= 100)

res = n * n;

printf("\n Square of %d is %d\n", n, res);

}
 else if (n<= 0 || n > 100)

printf("Beyond the range");

return 0;

Output

Inputs Outputs

I1 : -2 O1 : Beyond the range

I2 : 0 O2 : Beyond the range

I3 : 1 O3 : Square of 1 is 1

I4 : 100 O4 : Square of 100 is 10000

I5 : 101 O5 : Beyond the range

I6 : 4 O6 : Square of 4 is 16

I7 : 62 O7 : Square of 62 is 3844

Test Cases

T1 : {I1 ,O1}

T2 : {I2 ,O2}

T3 : {I3, O3}

T4 : {I4, O4}

T5 : {I5, O5}
T6 : {I6, O6}

T7 : {I7, O7}

Some Remarks

A prevalent misconception among the beginners is that one should be

concerned with testing only after coding ends. Testing is, in fact, not a
phase towards the end. It is rather a continuous process. The efforts for
testing should begin in the form of preparation of test cases after the
requirements have been finalized. The Software Requirements
Specification (SRS) document captures all features to be expected from
the system. The requirements so identified here should serve as a basis
towards preparation of the test cases. Test cases should be designed in
such a way that all target features could be verified. However, testing a
software is not only about proving that it works correctly. Successful
testing should also point out the bugs present in the system, if any.

SIMULATION

Exhaustive testing of a software is required to determine it is working

as per expectations and requirements. Developers often do not have
enough time (or at times interest) to test their codes thoroughly. To
handle such scenarios, most projects usually have a dedicated testing
team. However, unit testing, at least, is done by the developers.

Irrespective of who performs testing, or what is being tested, testing

usually involve some generic steps. In this simulation we provide a
broad overview of the testing process.
As already discussed under the theory section, test case preparation could begin
right after requirements identification stage. It is desirable (and advisable) to
create a Requirements Traceability Matrix (RTM) showing a mapping from
 individual requirement to test case(s). A simplified form of the RTM is shown in
table 1 (the numbers shown in this table are arbitrary; not specific to LIS).
Table 1: A simplified mapping from requirements to test
cases
Requirement # Test Case #
R1 TC1

R2 TC2, TC3, TC4

R3 TC5

R4 TC6
Table 1 states which test case should help us to verify that a specified feature
has been implemented and working correctly. For instance, if test case # TC6
fails, that would indicate requirement # R4 has not fully realized yet. Note that it
is possible that a particular requirement might need multiple test cases to verify
whether it has been implemented correctly.
To be specific to our problem, let us see how we can design test cases to verify
the "User Login" feature. The simplest scenario is when both user name and
password have been typed in correctly. The outcome will be that the user could
then avail all features of LIS. However, there could be multiple unsuccessful
conditions:
 User ID is wrong
 Password is wrong
 User ID & password are wrong
 Wrong password given twice consecutively
 Wrong password given thrice consecutively
 Wrong password given thrice consecutively, and security question answered
correctly
 Wrong password given thrice consecutively, and security question answered
incorrectly
We would create test case for the above stated login scenarios. These test cases
together would constitute a test suite to verify the concerned requirement. Table
2 shows the details of this test suite.
Table 2: A test suite to verify the "User Login" feature
# TS1
Title Verify "User Login" functionality
Description To test the different scenarios that might arise while an user is trying to login
Post-
Summary Dependency Pre-condition Execution Steps Expected Outp
condition
Verify that user Employee User is 1. Type in employee ID "Home" page for the u
already ID 149405 is a logged in as 149405 is displayed
 Table 2: A test suite to verify the "User Login" feature
# TS1
Title Verify "User Login" functionality
Description To test the different scenarios that might arise while an user is trying to login
Post-
Summary Dependency Pre-condition Execution Steps Expected Outp
condition
registered with 2. Type in
the LIS is able to registered user of LIS; password this_is_passw
login with correct user's password ord
user ID and is this_is_password 3. Click on the 'Login'
password button
1. Type in employee ID
Verify that an as 149405xx The "Login" dialog is s
Employee
unregistered user User is not 2. Type in with a "Login failed! C
ID 149405xx is not a
of LIS is unable logged in password whatever your user ID and
registered user of LIS
to login 3. Click on the 'Login' password" message
button
Verify that user 1. Type in employee ID
already Employee as 149405 The "Login" dialog is
registered with ID 149405 is a
User is not 2. Type in shown with a "Login f
the LIS is unable registered user of LIS;
to login with user's password
logged in password whatever Check your user ID a
3. Click on the 'Login' password" message
incorrect is this_is_password
password button
Verify that user
already 1. Type in employee ID
registered with This test case is as 149405 The "Login" dialog is s
the LIS is unable executed after
User is not 2. Type in with a "Login failed! C
to login with TC3 execution of TC3
logged in password whatever2 your user ID and
incorrect before executing any
3. Click on the 'Login' password" message
password given other test case
twice button
consecutively
Verify that user
already 1. Type in employee ID The "Login" dialog is
registered with This test case is as 149405 shown with a "Login f
the LIS is unable executed after Check your user ID a
User is not 2. Type in
to login with TC4 execution of TC4 password" message;
incorrect before executing any
logged in password whatever3 security question and
password given other test case 3. Click on the 'Login' box for the answer ar
thrice button displayed
consecutively
Verify that a TC5 This test case is Email sent 1. Type in the answer Login dialog is display
registered user executed after containing as my_answer an email containing th
can login after execution of TC6 new 2. Click on the 'Email password is received
 Table 2: A test suite to verify the "User Login" feature
# TS1
Title Verify "User Login" functionality
Description To test the different scenarios that might arise while an user is trying to login
Post-
Summary Dependency Pre-condition Execution Steps Expected Outp
condition
password.
three consecutive
before executing any The email is
failures by
other test case. Answer expected to
correctly
to the security question be received
Password' button
answering the
is my_answer. within 2
security question
minute.
Verify that a
registered user's
account is
Execute the test cases 1. Type in the answer
blocked after User The message "Your
TC3, TC4, and TC5
three account has as not_my_answer account has been blo
once again (in order)
consecutive been 2. Click on the 'Email Please contact the
before executing this
failures and blocked Password' button administrator." appea
test case
answering the
security question
incorrectly
In a similar way, test suites corresponding to other user requirements could be
created as well. A good test plan can reduce the burden of testing team by
specifying what exactly they should focus on.