SOFTWARE TESTING -

Module 2
 Module – 2 : SYLLABUS

➢ Concept of Unit testing

➢ Static Unit testing
➢ Dynamic Unit testing - Control Flow testing, Data Flow testing,
Domain testing, Functional Program testing
➢ Mutation testing - Mutation and Mutants, Mutation operators,
Mutation score
➢ Junit - Framework for Unit testing
➢ Case Study - Mutation testing using Junit and Muclipse
 Unit testing -
Basics
► Developer focuses on individual units or components of
a software system
► To validate the smallest possible unit of code, such as a
function or a method
► To test a Module/unit in isolation from the rest of the
system
► Quickly identify and fix any issues, early in the
development process
► To validate each unit of software
➢ works as intended
➢ meets the requirements
► performed early in the development process
➢ before the code is integrated and tested as a whole
system
 Unit testing -
Concept
➢ Unit testing has a limited scope

➢ Programmer will need to verify if the written code works correctly

➢ Why Unit testing is done ?

✓ Execute every line of code.

✓ Execute every predicate(a function that returns a boolean) in the unit to

evaluate them to true and false separately.

✓ Unit performs its intended function and ensure that it contains no known
errors
 Unit testing - Objective

► To isolate a section of code

► To verify the correctness of the code

► To test every function and procedure

► To fix bugs early in the development cycle

➢ which save’s time and money (Ex : Intel Pentium Bug)

► Developers to understand the code base

► To help developers enable code to make changes quickly

► To help with code reuse

 UNIT TESTING - Types

Unit testing is conducted in two phases:

1. Static unit testing
2. Dynamic unit testing
1. Static unit testing:
➢ NOT executed
➢ The code is examined over all possible behaviors that might arise during run time
2. Dynamic unit testing:
✓ Its Execution based
 STATIC UNIT TESTING

➢ A programmer does NOT execute the unit(Hardware / complete system)

➢ The code is examined over all possible behaviors that might arise during run time
➢ Its also known as non-execution-based unit testing
➢ Code review is done, even before unit(s) are individually tested by actually
executing them
✓ The code of each unit is validated against requirements of the unit by reviewing
the code
✓ Why code Review (The objective of code review is to review the code, and not to
evaluate the Developer)
❑potential issues are identified and resolved

✓ 2 Techniques for code is review:

1. Inspection:
Using predetermined criteria

2. Walkthrough:
Uses predefined scenarios
 Stakeholder/Role’s in Review Meeting
• Moderator:
A review meeting is chaired by the moderator Guides the pace of the review
process
Selects the reviewers and schedules the review meetings

• Author:
Person who has written the code to be reviewed

• Presenter:
Someone other than the author of the code Presents the author’s code in the review
meeting
 Stakeholder/Role’s in Review Meeting
• Recordkeeper:
Documents the problems found during the review process and the follow-up
actions suggested should be different than author and moderator

• Reviewers:
Experts in the subject area of code under review

• Observers:
People who want to learn about the code in review Do not participate in review
process but are observers
 CR and Code Review Metrics

Change request (CR)

➢ To track all stakeholder requests (new features, enhancement requests,
defects, changed requirements…)
➢ To track related status information throughout the project lifecycle

Code Review Metrics

• Number of lines of code (LOC) reviewed per hour
• Number of CRs generated per thousand lines of code (KLOC)
• Number of CRs generated per hour
• Total number of CRs generated per project
• Total number of hours spent on code review per project
STATIC UNIT TESTING – Code
Review Process
STATIC UNIT TESTING – Code
Review Process
Readiness:
To ensure the unit under test is ready for review A unit is ready if it satisfies
the following criteria: Completeness, Minimal Functionality, Readability,
Complexity, Requirements and Design Documents

Preparation
Before the meeting, each reviewer carefully reviews the work package
It is expected that the reviewers read the code and understand its organization
and operation before the review meeting
STATIC UNIT TESTING – Code
Review Process
Examination
Examination process consists of the following activities:
• Author presents logic used in the code
• Presenter reads the code line by line
• Reviewers raise questions if code has defects
• Recordkeeper documents CR and suggestions for fixing the problems (if there are
any)
Rework
At the end of the meeting, the recordkeeper produces a summary of the meeting
with following information:
• A list of all the CRs, the dates by which those will be fixed, and the names of
the persons responsible for validating the CRs
• A list of improvement opportunities
• The minutes of the meeting (optional)
STATIC UNIT TESTING – Code
Review Process
Validation
• The CRs are independently validated by the moderator or another person
designated for this purpose
• To ensure the suggested improvements have been implemented correctly
• The revised and final version of the outcome of the review meeting is
distributed to all the group members

Exit
• Summarizing the review process
• Take updates on code review metrics
 Code Review – Checklist
1. Does the code do what has been specified in the design specification?

2. Does the procedure used in the module solve the problem correctly?

3. Does a software module duplicate another existing module which could be reused?

4. If library modules are being used, are the right libraries and the right versions
of the libraries being used?

5. Does each module have a single entry point and a single exit point? Multiple exit
and entry point programs are harder to test.
 Code Review – Checklist
[Link] the cyclomatic complexity of the module more than 10? If yes, then it is
extremely difficult to adequately test the module.

[Link] each atomic function be reviewed and understood in 10–15 minutes? If

not, it is considered to be too complex.

[Link] naming conventions been followed for all identifiers, such as pointers,
indices, variables, arrays, and constants?
It is important to adhere to coding standards

9. Has the code been adequately commented upon?

[Link] all the variables and constants been correctly initialized? Have
correct types and scopes been checked?
 Code Review – Checklist
[Link] the global or shared variables, if there are any, carefully controlled?

[Link] there data values hard coded in the program? Rather, these should be
declared as variables.

13. Are the pointers being used correctly?

[Link] the dynamically acquired memory blocks deallocated after use?

[Link] the module terminate abnormally? Will the module eventually terminate?

[Link] there a possibility of an infinite loop, a loop that never executes, or a loop with
a premature exit?
 Code Review – Checklist

[Link] all the files been opened for use and closed at termination?

[Link] there computations using variables with inconsistent data types? Is overflow or
underflow a possibility?

[Link] error codes and condition messages produced by accessing a common table
of messages?

20. Is the code portable?

Example : same code on different operating

21. Is the code efficient?

clarity, readability, or correctness
 DYNAMIC UNIT TESTING

✓ Its referred as Execution-based unit testing

✓ Static testing is performed before compilation, whereas Dynamic testing is
performed after compilation
✓ In Static testing, code is verified BUT during Dynamic testing we validate
the code

Dynamic unit is actually executed in isolation

This execution differs from ordinary execution, due to below reasons:
➢ A unit under test is taken out of its actual execution environment.
➢ The actual execution environment is emulated by writing more code
➢ The above compiled aggregate is executed with selected inputs
How Environment is emulated ?

Why ?
• UNIT has to be tested in isolation.
• Any fault found during running
has to be associated only with
the UNIT under test (UUT)

The Context of a UNIT test consists of two parts:

1. A caller of the unit --> Test Driver
2. All the units called by the unit --> Stubs

Note :
Test driver and Stubs are together called scaffolding
Each unit should have one dedicated test driver and several stubs
 Functions of Test Driver and Stubs

Test Driver:
• Its a program that invokes the UUT
• Executes with input values it got from Driver and returns
a value to Driver
• Driver compares actual outcome with expected, and
reports test result
• Functions as MAIN unit in the execution process
• Facilitates compilation
• Driver provides input data to UUT(in expected format)
 Functions of Test Driver and Stubs

Stubs:
❑Its a “dummy subprogram” that replaces a unit that is called by UUT
❑Stubs replace the units called by the UUT
❑A stub performs two tasks:
1. It shows an evidence(prints a message) that the stub was actually
called
2. It returns a precomputed value to caller so that UUT can continue
its execution
 Test Driver & Stubs - Features
❖Test driver and Stubs are together called as scaffolding
❖NOT discarded after the unit test is completed
❖Its Tightly coupled with UUT
❖It accompanies unit throughout its life cycle (released with actual software)
❖It should be reusable and maintainable
❖If a unit is modified, we should check IF we need to modify the
corresponding driver and stubs
❖If there is a new fault in the unit:
✓ Corresponding test driver should be updated with new set of input data
➢ Detect that similar faults in the future
❖If Unit is expected to run on different platforms, the test driver and stubs
should also be built to test the unit on new platforms
❖Test driver and stubs are checked in to the version control system as a product
along with the unit
 Test data – Dynamic unit Testing
❖Design document provides guidance for the selection of input test data
❖The selection of input test data (might) find bugs
❖Techniques used for selecting Input data:
1) Control Flow Testing
2) Data Flow Testing
3) Domain Testing
4) Functional Program Testing
Test either Input OR output domains of a program
 DYNAMIC UNIT TESTING-CONTROL FLOW TESTING

✓ Control flow testing is a type of software testing that uses a program’s

control flow as a model.
✓ Control flow testing is a structural testing strategy. This testing technique
comes under white box testing.
✓ For the type of control flow testing, all the structure, design, code and
implementation of the software should be known to the testing team.
✓ This type of testing method is often used by developers to test their own code
and own implementation as the design, code and the implementation is better
known to the developers.
 DYNAMIC UNIT TESTING-CONTROL FLOW TESTING

Control Flow Testing Process

• Control Flow Graph Creation: From the given source code a control flow
graph is created either manually or by using the software.
• Coverage Target: A coverage target is defined over the control flow graph
that includes nodes, edges, paths, branches etc.
• Test Case Creation: Test cases are created using control flow graphs to cover
the defined coverage target.
• Test Case Execution: After the creation of test cases over coverage target,
further test cases are executed.
• Analysis: Analyze the result and find out whether the program is error free or
has some defects.
 DYNAMIC UNIT TESTING-DATA FLOW TESTING

• Is a type of structural testing. It is a method that is used to find the test paths
of a program according to the locations of definitions and uses of variables in
the program.
• Data Flow Testing uses the control flow graph to find the situations that can
interrupt the flow of the program.
• Reference or define anomalies in the flow of the data are detected at the
time of associations between values and variables.
• These anomalies are:
❖ A variable is defined but not used or referenced,
❖ A variable is used but never defined,
❖ A variable is defined twice before it is used
 DYNAMIC UNIT TESTING-DATA FLOW TESTING
Types of Data Flow Testing:
1. Testing for All-Du-Paths: It Focuses on “All Definition-Use Paths. All-Du-Paths
is an acronym for “All Definition-Use Paths.” Using this technique, every
possible path from a variable’s definition to every usage point is tested.
2. All-Du-Path Predicate Node Testing: This technique focuses on predicate
nodes, or decision points, in the control flow graph.
3. All-Uses Testing: This type of testing checks every place a variable is used in
the application.
4. All-Defs Testing: This type of testing examines every place a variable is
specified within the application’s code.
5. Testing for All-P-Uses: All-P-Uses stands for “All Possible Uses.” Using this
method, every potential use of a variable is tested.
 DYNAMIC UNIT TESTING-DATA FLOW TESTING
6. All-C-Uses Test: It stands for “All Computation Uses.” Testing every possible
path where a variable is used in calculations or computations is the main goal of
this technique.
7. Testing for All-I-Uses: All-I-Uses stands for “All Input Uses.” With this method,
every path that uses a variable obtained from outside inputs is tested.
8. Testing for All-O-Uses: It stands for “All Output Uses.” Using this method,
every path where a variable has been used to produce output must be tested.
9. Testing of Definition-Use Pairs: It concentrates on particular pairs of
definitions and uses for variables.
10. Testing of Use-Definition Paths: This type of testing examines the routes
that lead from a variable’s point of use to its definition.
 DYNAMIC UNIT TESTING-DOMAIN TESTING
• It is a software testing technique where minimum numbers of inputs are used
to access appropriate output of a system, to ensure the system does not
accept invalid input values.
• The system is expected to give required outputs blocking the invalid inputs.

Importance:
• Protection of Input Space: A software application’s complete input area
should be sufficiently covered, and domain testing helps to verify this.
• Error detection: It works well for identifying mistakes or irregularities that
could happen in particular input domains.
 DYNAMIC UNIT TESTING-DOMAIN TESTING
• Preventing Bugs: Software can be designed with fewer vulnerabilities and
critical situations if developers have a better understanding of the properties
and limits of input domains.
• Enhanced Efficiency of Tests: Domain testing aids in organizing testing
efforts according to the software’s most important and pertinent sections.
 MUTATION TESTING

• Mutation testing is a white box method in software testing where we insert errors
purposely into a program (under test) to verify whether the existing test case can
detect the error or not.
• In this testing, the mutant of the program is created by making some modifications
to the original program.
• The primary objective of mutation testing is to check whether each mutant created
an output, which means that it is different from the output of the original program.
• We will make slight modifications in the mutant program because if we change it on
a massive scale than it will affect the overall plan.
 MUTATION TESTING

• When we detected the number of errors, it implies that either the program is
correct or the test case is inefficient to identify the fault.
• Mutation testing purposes is to evaluate the quality of the case that should be able
to fail the mutant code hence this method is also known as Fault-based testing as it
used to produce an error in the program and that why we can say that the mutation
testing is performed to check the efficiency of the test cases.
 MUTATION TESTING

Objective of Mutation Testing:

To identify pieces of code that are not tested properly.
• To identify hidden defects that can’t be detected using other testing methods.
• To discover new kinds of errors or bugs.
• To calculate the mutation score.
• To study error propagation and state infection in the program.
• To assess the quality of the test cases.
 MUTATION TESTING

Types of Mutation Testing:

Mutation testing is basically of 3 types:
1. Value Mutations:
• In this type of testing the values are changed to detect errors in the program.
• Basically a small value is changed to a larger value or a larger value is changed to a
smaller value. In this testing basically constants are changed.
 MUTATION TESTING

Types of Mutation Testing:

Mutation testing is basically of 3 types:
2. Decision Mutations:
• In decisions mutations are logical or arithmetic operators are changed to detect
errors in the program.
 MUTATION TESTING

Types of Mutation Testing:

Mutation testing is basically of 3 types:
3. Statement Mutations:
• In statement mutations a statement is deleted or it is replaces by some other
statement.
 MUTATION TESTING

Advantages of Mutation Testing:

• It brings a good level of error detection in the program.
• It discovers ambiguities in the source code.
• It finds and solves the issues of loopholes in the program.
• It helps the testers to write or automate the better test cases.
• It provides more efficient programming source code.
 MUTATION TESTING

Disadvantages of Mutation Testing:

• It is highly costly and time-consuming.
• It is not able for Black Box Testing.
• Some, mutations are complex and hence it is difficult to implement or run against
various test cases.
• Here, the team members who are performing the tests should have good
programming knowledge.
• Selection of correct automation tool is important to test the programs.
 MUTATANT

❖ Mutant :
➢ Modification of code/unit/program, is called as Mutant
❖ Mutants can be equivalent to original code
❖ Mutants can also be faulty or modified version of original program
❖ A mutant is said to be killed when the execution of a test case causes it to fail and
the mutant is considered to be dead
MUTATANT Score
• If the tests with the mutants detect the same number of issues as the test with the
original program, then either the code has failed to execute, or the software
testing suite being used has failed to detect the mutations.
• If this happens, the software test is worked on to become more effective.
• A successful mutation test will have different test results from the mutant code.
After this, the mutants are discarded.
• The software test tool can then be scored using the mutation score. The mutation
score is the number of killed mutants divided by the total number of mutants,
multiplied by 100.
• Mutation score = (number of killed mutants/total number of mutants killed or
surviving) x 100
• A mutation score of 100% means the test was adequate.
MUTATANT Operators
❖Mutant Operators :
✓ Operator insertion (Like – and ++)
✓ Adding or replacement of operators
✓ Inside if-else statement, remove else part
✓ GOTO replacement
✓ Return statement replacement / deletion
✓ Comparable array name replacement
✓ Statement replacement by changing the data
✓ Data Modification for the variables
✓ Modification of data types in the program
JUnit
• JUnit is a Java testing framework that makes writing reliable and efficient tests
easy.
• It can be used for applications made in most languages but is particularly suited
for testing Java applications.
• It is an open-source testing framework for java programmers. The java
programmer can create test cases and test his/her own code.
• It is one of the unit testing framework. Current version is JUnit 4.
• To perform unit testing, we need to create test cases. The unit test case is a code
which ensures that the program logic works as expected.
JUnit
Types of unit testing
There are two ways to perform unit testing: 1) manual testing 2) automated testing.
1) Manual Testing
If you execute the test cases manually without any tool support, it is known as
manual testing. It is time consuming and less reliable.
2) Automated Testing
If you execute the test cases by tool support, it is known as automated testing. It is
fast and more reliable.
JUnit

JUnit provides several features that make it easy to create and run tests, including:
Assertions: Assertions are used to verify the expected behavior of a system. JUnit
provides a set of assertion methods that can be used to check the results of a test.
Test runners: Test runners are used for executing the tests and reporting the results.
JUnit provides a graphical test runner that can run tests and view the results.
Test suites: Test suites are used to group related tests. JUnit provides a way to
create test suites that can be run together
Reporting: When you run your tests, JUnit can help you analyze the results. It
provides a built-in reporter that prints out information about the executed tests.
JUnit

Benefits of using JUnit

There are many benefits to using JUnit, the most notable of which is its ability to
help you develop reliable and testable code. Here are some other reasons why you
will definitely want to start using JUnit:
• JUnit can help you keep your code organized and easy to read.
• JUnit can help you detect and fix errors in your code.
• JUnit can help you improve the quality of your software.
• JUnit can help you work more efficiently and improve your testing process.
JUnit
Annotations for Junit testing
The Junit 4.x framework is annotation based, so let's see the annotations that can be
used while writing the test cases.
@Test annotation specifies that method is the test method.
@Test(timeout=1000) annotation specifies that method will be failed if it takes
longer than 1000 milliseconds (1 second).
@BeforeClass annotation specifies that method will be invoked only once, before
starting all the tests.
@Before annotation specifies that method will be invoked before each test.
@After annotation specifies that method will be invoked after each test.
@AfterClass annotation specifies that method will be invoked only once, after
finishing all the tests.
Unit Test Frameworks
1. Junit
Junit is an open-source unit testing framework that supports Java (5 and
above) and is known for its efficacy in facilitating test-driven development (TDD). It
belongs to a family of unit testing frameworks popularly known as xUnit.
Advantages
• Can be used for both integration and unit tests.
• Exceptionally effective in enabling TDD or Test Driven Development.
• Allows development of anomaly-free and easily readable code.
• Supported by most major IDEs – Eclipse, NetBeans, Maven, Ant, and IntelliJIDEA.
• Compatible with Java 8, but also allows execution of test cases written in earlier
JUnit versions.
Limitations
• You cannot use JUnit to run dependency tests.
Unit Test Frameworks
2. JBehave
JBehave is a fairly well-known unit testing framework, known for its ability to
support Behavior Driven Development (BDD). This test approach intends to design
software from the primary POV of behavior rather than tests, which makes it a software
development AND design perspective.
Advantages
• Since it supports BDD, JBehave tests prioritize product intuitiveness and reasoning.
• Enables integration between multiple dev teams working with similar requirements in
different projects.
• You can run JBehave in the same format as your product specs – created by
stakeholders and project managers.
Limitations
Any BDD testing framework requires seamless and frequent communication between team
members using the tool. Getting all members onboard poses a management challenge,
especially when a team is entirely new to the BDD approach.
Unit Test Frameworks
3. Nunit
Like JUnit, NUnit also belongs to the xUnit family. However, this framework is
written in C#. It supports all .Net languages and is similar to JUnit in functionality.
Advantages
• NUnit works flawlessly with multiple build platforms and custom test runners.
• Is known for speedy test running and sophisticated annotations to specify multiple
inputs for a test easily.
Limitations
There are reports of NUnit integrating inadequately with Visual Studio. Users have
had to put in extra effort and install a few tools to get it to work.
Unit Test Frameworks
4. Xunit
xUnit is an umbrella name, referring to multiple unit testing frameworks that
draw their hierarchy and ability from SUnit, a framework created back in 1998. It works
best with .Net aligned languages like C#, [Link], and F#.
Advantages
• It is more flexible and extensible than the other .Net Unit test frameworks. It allows
you to create new attributes to control tests.
• Unlike other .Net Unit testing frameworks, xUnit test methods are isolated.
• XUnit requires a reduced number of custom attributes.
• Lightweight, easy to use, and supported by an active developer community.
Limitations
Some developers and industry leaders have claimed that xUnit test reports provide
insufficient information, such as what kind of test failed and at what juncture in the
execution
Unit Test Frameworks
5. Robot
This open-source unit testing framework is developed and operates using Python. This
multi-platform unit testing framework can run tests at different levels in real or
simulated user environments. Designed to facilitate test-driven development, Robot is
known for offering tabular data syntax, keyword-driven testing, rich libraries & toolsets,
and parallel testing.
Advantages
• Quite easy to install and use, even for individuals with no technical expertise.
• It supports many test libraries and is easily extensible with custom libraries. Easy to
integrate with other tools and systems.
• Integrates flawlessly with essential development and testing systems – Selenium,
Appium, Jenkins, etc.
• Provides detailed test reports that make it easy to identify the location and cause of
failure.
Unit Test Frameworks
• Supports data-driven and parallel testing.
• Is supported by a large community of devs and testers.
Limitations
• If you want to extend the Robot framework with custom libraries or test libraries,
you’ll have to go through a steep learning curve.
• You get limited support for database testing.
• Robot’s keyword-driven testing approach may not work well for tests requiring a high
level of customization. Debugging for such tests is especially complex.
• You get limited community support for certain Robot libraries.
Unit Test Frameworks
6. PyUnit
PyUnit/ Unittest is essentially Python’s unit testing framework. It is a Python
language version of JUnit, also part of the Python Standard Library as of Python version
2.1. It is the default testing framework that you get out of the box with the Python
package, so most Python devs start running tests with it before using other frameworks.
Advantages
• Comes out of the box with the Python package.
• No additional modules are required to run PyUnit, since it is part of the standard
Python library.
• Provides flexible and uncomplicated test execution.
• Generates quick but detailed reports in XML as well as unittest-sml-reporting.
Limitations
• It requires a rather intimidating amount of boilerplate code to work.
Unit Test Frameworks
7. CyPress
Cypress web testing framework allows the tester to carry out two essential
testing flows: End-to-End testing and Component testing. A complete test result is usually
comprised of the E2E and Component tests.
Advantages
• Quick, easy setup and execution of tests.
• Writing time for tests is lower compared to other frameworks.
• It captures screenshots of test execution automatically in case of test failure for
effective debugging.
• Automatically reloads Test Runner once the changes have been made to the test.
Limitations
• Parallel Testing setup is complicated in Cypress and not straightforward.
• It doesn’t support multiple tabs and windows.