UNIT IV

TEST AUTOMATION
1. INTRODUCTION TO AUTOMATION
a. AUTOMATION TESTING TOOLS
b. TEST SCRIPT DEVELOPMENT AND MAINTENANCE

II. DEFECT MANAGEMENT

a. DEFECT LIFE CYCLE
b. DEFECT REPORTING AND TRACKING
c. ROOT CAUSE ANALYSIS
d. DEFECT METRIC AND ANALYSIS

III. TEST METRICS AND MEASUREMENT

a. METRICS FOR TEST PLANNING AND PROGRESS TRACKING
b. TEST COVERAGE METRICS

I. INTRODUCTION TO AUTOMATION
Automation Testing plays a crucial role in modern software development,
especially in Agile and DevOps environments, by improving reliability.

Benefits of Automation Testing

 Cost Savings: Reduces long-term costs by minimizing manual effort and

speeding up development cycles.

 Early bug detection: Identifies defects early, reducing the cost and effort of
fixing issues later.

 Faster Feedback: Provides quick feedback, enabling faster decision-making

and reduced development time.
  Better Resource Utilization: Automates repetitive tasks, freeing up
resources for more complex work.

 Improved Accuracy: Eliminates human errors, ensuring reliable and

consistent test results.

 Increased Coverage: Tests more scenarios, improving the quality and

reliability of the software.

 Speed: Automated tests run faster, ideal for time-consuming tests like
regression or load testing.

Types of Automation Testing

Automated testing covers various testing methods to ensure software quality,

functionality, and security. Some key types of Automation testing are:

 Regression Testing
 Smoke testing
 Data-Driven Testing
 Performance Testing
 Functional Testing
 Security Testing
 Acceptance Testing
 PROCESS OF AUTOMATION TESTING

A step-by-step method of the process involved in automation testing:

1. Define Goals: Before starting automation, be clear about your long-term and
short-term goals of test automation by discussing the things clear with
higher management and stakeholders .

2. Test Approach Planning: There are different testing levels; automation is

often confused with end-to-end testing, but there are many levels. The best
approach is to create test pyramids starting from Unit testing to Acceptance
testing.

3. Choose a Framework: Once you know the business value, the most critical
part is the type of framework you choose. There are different types
of automation frameworks and should be discussed with the team and
choose the right one.

4. Test Automation Environment: You should also know which

environment the application under test will be used for validation

5. Risk Analysis: Analyzing and communicating the risk to management is

essential. Automation testing may have some risks based on your tools and
strategy.

6. Create, Execute, and Maintain Tests: Start creating the automation tests
for the critical features first. Once you create test cases, execute them often.

7. DevOps Integration: A DevOps integration also reduces the dependency of

the testing team and anybody can execute and get the report.
 8. Test Reporting: The test report must have numbers of total test cases,
passed, failed, etc. The report must be understandable for any non-technical
person.

9. Review : Continuously review your test scripts, and understand what is

causing false failures and what is impacting test case creation

(a) AUTOMATION TESTING TOOLS

NEED FOR AUTOMATION TESTING TOOLS

 Simplifies Test Case Execution: Automation testing allows monitoring of

the results at the end of the process and simplifying the overall test
execution and increasing the efficiency of the application.

 Improves Reliability of Tests: Automation testing ensures that there is

equal focus on all the areas of the testing, thus ensuring the best quality end
product.

 Increases amount of test coverage: Using automation testing, more test

cases can be created and executed for the application under test. Thus,
resulting in higher test coverage and the detection of more bugs.

 Minimizing Human Interaction: In automation testing, everything is

automated from test case creation to execution thus there are no changes
for human error due to neglect.

 Saves Time and Money: The initial investment for automation testing will
be high but it is cost-efficient and time-efficient in the long run. This is due
to the reduction in the amount of time required for test case creation and
execution which contributes to the high quality of work.

 Earlier detection of defects: Automation testing documents the defects,

thus making it easier for the development team to fix the defect and give a
faster output. The earlier the defect is identified, the more easier and cost-
efficient it is to fix the defects.
 TOP TOOLS FOR AUTOMATION TESTING

Automation testing relies on various tools to streamline the testing process,

improve accuracy, and reduce manual effort.

Popular Automation Testing Tools

 Selenium: A widely used open-source framework for web application

automation, compatible with multiple browsers and programming languages.

 BrowserStack Automate: Enables automated testing on 3,500+ real devices

and browsers. It works with frameworks like Selenium, Cypress, Playwright,
and Puppeteer for reliable web automation.

 BrowserStack App Automate: Designed for mobile app testing on real

Android and iOS devices. It integrates with Appium, Espresso, for efficient
mobile automation.
 JUnit and TestNG are popular testing frameworks used in Java for writing
and running unit tests.

JUnit and TestNG are popular test automation frameworks in the Java
community and used widely by people. TestNG only provides a few
additional features in comparison to Junit. Hence, the choice of the “right”
testing framework for test automation is purely based on the project
requirements and flexibility.
JUnit

It is an open-source unit testing framework for Java. As the name suggests, it is

used to test small code units. This framework is used by default by Java developers
to write as well as execute test cases.

JUnit follows the approach of “testing first then coding”. This emphasizes setting
up test data and testing the small piece of code first and then implementing the
same. This increases productivity and helps in improving code stability.

Features of JUnit

Here are the features of JUnit:

 Open-Source: JUnit is an open-source framework that helps developers

write and test code quickly, enhancing code quality.

 Annotations: It offers a variety of test annotations (e.g., @Test,

@BeforeEach) to mark and organize test methods.

 Assertions: JUnit provides built-in assertions to verify expected outcomes in

tests (e.g., assertEquals, assertTrue).

 Test Runners: It includes test runners that execute test cases and report results

automatically.

 Automation & Feedback: Tests run automatically and provide instant

feedback to developers, helping catch issues early.
  Debugging Support: JUnit supports debugging, allowing developers to
quickly identify and fix issues during test development.

 Latest Version: The latest version, JUnit 5, is designed to leverage features

of Java 8 and above, offering more modularity and flexibility.

Test Setup of JUnit

1. Installing and Setting Up

 JUnit requires Java Version 8 and above.

 Download Java

 Download JUnit 5 jars (for Java project)

 Open your project and add these external jars through the build path
configuration.

 Set up environment variables.

 Add the below dependency in [Link] for running JUnit tests.

2. Test Suites

 The test suite consists of a collection of test cases that lets you run the test
cases simultaneously. It’s like a logical grouping of test cases.

 JUnit 5 uses @RunWith and @Suite Classes for creating Test Suites
 3. Test Annotations

 JUnit 5 is an annotation-based framework. Annotations are tags that provide

additional information about the test methods or the class. They are usually
represented by ‘@’.

Some popular annotations of JUnit 5 include:

 @Test – The MAIN business logic of a program resides here.

 @BeforeAll – Executes before the first test method of the class runs

 @AfterAll – Executes after all the test methods of the current class runs

 @BeforeEach – Executes before each test method

 @AfterEach – Executes after each test method

4. Exceptions Handling

Exception handling is done via [Link]() API.

assertThrows() method asserts that the code throws an exception of the given
type when executed. This method fails if there is no exception or exception
of some other type.

5. Ignoring Tests

Ignoring tests is important in an automation framework since some test cases

can become redundant due to changed requirements or may have to be
avoided for some reason. This can be done using @Ignore in JUnit 5.
Grouping Tests

Grouping tests help in identifying the tests and executing them quickly. JUnit 5
uses @Tag to group the tests.

Parameterizing Tests

Parameterizing Tests helps test the same test scenario with different sets of inputs.
In JUnit 5, the following dependency needs to be added to parameterize the tests
along with @ParameterizedTest.
TestNG

TestNG is a Java-based test automation framework that was inspired by JUnit. It

overcomes all the limitations of JUnit along with additional functionalities. This
makes it more powerful and easier to use. NG here stands for “Next Generation”.

It is designed to cover a range of test categories such as Unit testing, Functional

testing, Integration testing, etc. It is extremely popular and helps testers organize
the test cases in a structured way. This helps in maintaining the readability of the
scripts. The current version of TestNG is 7.6.0.
Features of TestNG

1. Open-Source: TestNG is an open-source testing framework, freely available

for use and customization.

2. Java Integration: It makes extensive use of advanced Java features,

enhancing flexibility and performance.

3. Annotations: TestNG supports multiple @Before and @After annotations

for detailed test configuration (e.g., @BeforeSuite, @AfterClass).

4. XML Configuration: Tests can be configured and organized through XML

files, making suite management easier.

5. Parameterization: TestNG allows parameterized testing using XML or

@DataProvider, enabling dynamic test input.

6. Multithreading: It supports multi-threaded test execution for faster test

runs.

7. Data-Driven Testing: Allows testing with multiple sets of data through

features like @DataProvider.

8. Report Generation: Automatically generates detailed HTML and XML

reports for test execution results.

9. Open APIs: Provides publicly available APIs, enabling integration with

other tools and custom extensions.

[Link] Execution: Supports parallel test execution, improving testing

efficiency in large projects.
Test Setup of TestNG

1. Installing and setting up

 Download Java from

 Download TestNG jar (for Java project)

 Open your project and add these external jars through Build path
configuration.

 Add the below dependency in [Link] for running TestNG tests.

2. Test Suites

In TestNG, Test Suites are defined in an XML file.

3. Test Annotations

Some popular annotations of TestNG include:

 @Test – MAIN program logic place

 @BeforeClass – Executes before the first test method of the class

 @AfterClass – Executes after all the test methods of the current class

 @BeforeMethod – Executes before each test method

 @AfterMethod – Executes after each test method

4. Exceptions Handling

Exception handling is done via the expectedExceptions parameter along with

@Test annotation.
5. Ignoring Tests

Ignoring tests can be done by passing a parameter in the @Test method.

[Link] Tests

Grouping tests can be done by passing parameters in the @Test method.

7. Parameterizing Tests

Parameterizing tests are done by @Parameters. The parameter value is passed in

the TestNG XML file.

Parameterizing tests can also be done by @DataProviders. It enables multiple input

values to test.
Reason to choose BrowserStack for Automation Testing

BrowserStack stands out as a comprehensive testing solution due to:

 Real Device Testing: Run tests on actual devices instead of emulators or

simulators.

 Seamless Integrations: Works with Selenium, Appium, Cypress, and other

leading frameworks.

 Scalability: Supports parallel testing to speed up test execution and reduce

build times.

 Reliable and Secure: Enterprise-grade security with zero maintenance

hassles.
With BrowserStack, teams can execute automated tests efficiently, ensuring high-
quality software delivery at scale.

CI/CD PIPELINE

Components of a CI/CD Pipeline

1. Commit Change
Developers make code changes and commit these changes to a version
control system (e.g., Git). This step initiates the CI/CD pipeline.
Committing code changes ensures that they are tracked and versioned
properly.

2. Trigger Build
The version control system detects the new commit and triggers the build
process automatically. Automated triggering of the build process ensures
that new changes are continuously integrated and tested.
3. Build
Common tools include Maven, Gradle, Ant for Java projects, and Docker
for containerized applications. The build step verifies that the code
compiles correctly and that all dependencies are resolved.

4. Notify of Build Outcome

The CI/CD system notifies the team of the build results, whether it passed
or failed. Immediate feedback on the build status helps developers quickly
identify and resolve any build issues.

5. Run Tests
Automated tests are executed on the build. These can include unit tests,
integration tests, end-to-end tests, and more. Testing frameworks like
JUnit, Selenium, TestNG, pytest, etc. Running tests ensures that the new
code does not introduce any bugs.

6. Notify of Test Outcome

The results of the test suite are reported back to the development team.
Quick feedback on test outcomes allows developers to address any failures
promptly.

7. Deliver Build to Staging

If tests pass, the build artifact is deployed to a staging environment. The
staging environment simulates the production environment, allowing for
final validation before production deployment. Deployment tools like
Ansible, Chef, Puppet, Kubernetes, or cloud-specific services like AWS
CodeDeploy.

8. Deploy to Production
After successful validation in staging, the build is automatically or
manually promoted to the production environment. This step makes the
new features and fixes available to end-users.
 EXTRA MATERIAL
SETTING UP A CI/CD PIPELINE
Setting up a CI/CD pipeline involves several stages, from configuring your
version control system to deploying your application. Below are detailed steps to
set up a CI/CD pipeline using a common tool like Jenkins.

Step 1. Install and Configure Jenkins

 Download Jenkins: Go to the Jenkins website and download the appropriate
installer for your operating system.
 Install Jenkins: Follow the installation instructions for your operating
system.
 Start Jenkins: After installation, start Jenkins and access it via
[Link]

Step 2. Set Up Version Control System (VCS)

 Choose a VCS: Use Git, GitHub, GitLab, Bitbucket, etc.
 Create a Repository: Create a new repository or use an existing one.
 Commit Code: Ensure your code is committed to the repository.

Step 3. Install Required Plugins in Jenkins

 Git Plugin: For connecting Jenkins with your Git repository.
 Pipeline Plugin: For defining Jenkins pipelines.
 Other Plugins: Depending on your project requirements (e.g., Maven,
Docker, NodeJS).

Step 4. Create a New Jenkins Pipeline Job

 New Item: From the Jenkins dashboard, click on “New Item”.
 Pipeline: Select “Pipeline” and give your job a name.
 OK: Click “OK” to create the job.

Step 5. Configure the Pipeline Job

 Pipeline Script from SCM: Under the “Pipeline” section, select “Pipeline
script from SCM”.
 SCM: Select “Git” and provide the repository URL.
 Branch Specifier: Specify which branch to build (e.g., */main).

Step 6. Define the Pipeline Script

 Jenkinsfile: Create a Jenkinsfile in your repository root. This file will define
the stages of your CI/CD pipeline.
 EXTRA MATERIAL

Top Automation Testing Tools

1. BrowserStack Automate

It is a cloud-based service that allows users to run automated Selenium tests on

browsers and mobile devices. It eliminates the need for maintaining an in-house
infrastructure, providing a seamless way to test web applications across different
platforms and devices.

Advantages:

1. Real Device Testing: BrowserStack Automate allows testing on a wide

range of real devices, including the latest smartphones and tablets ensuring
that tests reflect real user conditions.

2. Live Debugging: With BrowserStack Automate, users can debug tests in

real time using rich artifacts like Text, Visual & Video Logs. It also offers
Selenium Logs, Console Logs, Telemetry & Network Logs that help in
better Root Cause Analysis and fixing of issues.

3. Comprehensive Browser Coverage: It supports a vast array of browser

versions and operating systems, enabling cross-browser testing to ensure
consistent performance across different environments.
Limitations:

1. Concurrency Restrictions: Depending on the plan, the number of parallel

sessions (concurrent tests) is limited. For large-scale test execution you need
to update number of parallel sessions in the subscription.

2. Subscription-Based Model: Continuous usage requires a subscription,

which might not be ideal for intermittent users.

Use Cases:

1. Cross-Browser Compatibility: It ensures that the web applications are

consistent and work seamlessly across different Browser-Device-OS
combinations. BrowserStack Automate helps check cross browser
compatibility on 3500+ combinations.

2. Internationalization Testing: Ensures web applications work correctly in

different languages and locales.

3. Accessibility Testing: Verifies that web applications are accessible to users

with disabilities.

4. Testing in different Network Conditions: Assesses how web applications

perform under various Network conditions and slow internet using Network
Throttling feature of BrowserStack Automate.
 5. Local Testing: You can test your website hosted locally in Dev environment
with BrowserStack Automate’s Local Testing features.

2. Selenium

Selenium is a widely-used automation testing framework for web applications. It

provides a suite of tools and libraries for automating web browsers, enabling
testers to simulate user interactions and validate web application behavior
automatically.

Advantages:

1. Open Source: Selenium is free to use and has a large community for support
and continuous improvement.

2. Language Support: Selenium supports multiple programming languages

such as Java, Python, C#, and JavaScript, providing flexibility for
developers to write tests in their preferred language.

3. Vast Community and Resources: Selenium has a large and active

community, offering extensive documentation, tutorials, and support forums,
making it easier for users to get help and resources.
 4. Integration with Testing Frameworks: Selenium integrates well with
popular testing frameworks like TestNG and JUnit, allowing for seamless
integration into existing testing environments and CI/CD pipelines.
Limitations:

1. Flakiness: Selenium tests can sometimes be flaky due to factors like

browser inconsistencies, timing issues, and dynamic web elements, requiring
careful test design and maintenance.

2. Complexity in Setup: Setting up Selenium WebDriver and configuring

browser drivers can be complex, especially for beginners or for testing
across multiple browsers and environments.

3. Handling Dynamic Content: Selenium may face challenges in handling

dynamic content or single-page applications (SPAs) that heavily rely on
JavaScript, requiring additional effort to ensure test reliability.

4. Limited Support for Mobile Testing: While Selenium WebDriver supports

mobile testing through third-party tools like Appium, native support for
mobile testing is limited compared to web testing.
Use Cases:

1. Testing Browser Extensions: Selenium can be used to test browser

extensions or add-ons, ensuring their compatibility, functionality, and
performance across different browsers.

2. Automating Online Tasks: Selenium can automate repetitive online tasks

such as form filling, data entry, or interacting with web-based applications,
saving time and effort for users.
 3. Interactive Demos and Tutorials: Selenium can create interactive demos
or tutorials by automating user interactions on a website, guiding users
through specific features or workflows.

4. Web Scraping and Data Extraction: Selenium can be used to automate

web scraping tasks, extracting data from websites for analysis, research, or
data mining purposes.

3. BugBug

It is a tool for testing web applications, focusing on ease of use and effectiveness
in test automation. It is tailored for technical and non-technical users, featuring a
user-friendly interface for creating, editing, and running automated tests without
requiring extensive coding skills. One of its most notable features is the test
recorder, which provides a simple method for users to automate test scenarios for
web applications.

Advantages:

1. Intuitive Record and Playback: record user interactions as test steps,

allowing users to replay them to identify bugs and ensure functionality.
 2. Automatic Selector Validation and Correction: verifies selector accuracy
automatically and allows users to re-record or manually edit invalid
selectors.

3. Waiting Conditions and Active Waiting: uses active waiting conditions to

ensure elements are ready before proceeding, reducing flaky tests caused by
timing issues.

4. Cloud Execution and Local Testing: run tests locally or use their cloud
infrastructure for parallel testing across different environments, speeding up
the process.

5. Collaboration and Integration: allow team collaboration on tests and

integration with CI/CD pipelines for seamless continuous integration and
deployment.
Limitations:

1. No mobile app testing: BugBug only supports web applications and

website testing.

2. No cross-browser testing: BugBug only supports Chrome browser.

3. No visual regression testing: some changes in the application’s UI may go

unnoticed.
Use Cases:

1. Creating and Maintaining Tests: simplifies test creation and maintenance

with a user-friendly Chrome plugin, supporting cost-effective team
collaboration.
 2. Automating Test Scenarios: The test recorder feature allows users to
automate test scenarios for web apps easily.

3. Ensuring Cross-Environment Functionality: BugBug’s cloud execution

enables parallel testing across different environments..

4. Collaborative Testing: Teams can collaborate on tests within BugBug and

integrate them into CI/CD pipelines for continuous integration and
deployment.

4. Cypress

Cypress is a modern JavaScript-based end-to-end testing framework for web

applications. It offers fast, reliable testing with built-in features for simulating user
interactions and real-time test execution. Its user-friendly interface makes it ideal
for developers and QA engineers to write and maintain tests effectively.

Advantages:

1. Fast and Efficient: Cypress offers quick test execution directly in the
browser, improving feedback speed.

2. Easy Setup and Use: Its intuitive API and debugging tools make test
writing and maintenance simple.
 3. Automatic Waiting: Cypress waits for elements to load, reducing flakiness
and enhancing test stability.

4. Real-time Feedback: Developers get instant feedback during test execution,

speeding up the development process.
Limitations:

1. Browser Limitations: Cypress only supports Chrome-based browsers and

Electron, which may limit testing scenarios for applications that need to be
tested across multiple browsers like Firefox, Safari, or Edge.

2. Cross-Domain Testing Challenges: Cypress does not support testing across

different domains in a single test run due to security restrictions.

3. Limited Support for File Uploads: Handling file uploads in Cypress can
be challenging as it doesn’t have built-in support for file upload testing,
requiring workarounds to manage this functionality.

4. Slower Execution with Large Data Sets: While Cypress is known for fast
execution, it may experience performance issues when dealing with large
data sets or complex test scenarios.
Use Cases:

1. Real-Time Collaboration Testing: Cypress allows multiple testers to

collaborate in real time on the same test suite which is particularly useful for
distributed teams working on complex web applications.

2. Network Traffic Control: With Cypress, testers can control and manipulate
network traffic during tests. This is beneficial for testing scenarios like
network failures, slow connections,etc.
 3. Debugging Capabilities: Cypress provides advanced debugging capabilities
with features like time-traveling, which allows testers to inspect every step
of the test execution in detail, making it easier to identify and debug issues.

4. Accessibility Testing: Cypress can be extended with plugins like cypress-

axe to perform accessibility testing.

5. Playwright

Playwright is a modern automation testing framework developed by Microsoft,

designed to streamline the testing process for web, mobile, and desktop
applications. It offers a comprehensive set of features and cross-browser support,
making it a versatile choice for testing across different platforms.

Advantages:

1. Multiple Browser Support: Playwright supports multiple browsers,

including Chromium, Firefox, and WebKit, allowing for extensive cross-
browser testing.

2. Automation of Web, Mobile, and Desktop Apps: Playwright can automate

testing for web applications as well as mobile and desktop applications,
offering versatility in testing.
 3. Rich API and Easy Setup: Playwright offers a rich API that is easy to use,
with straightforward setup and configuration, making it accessible for both
beginners and experienced testers.

4. Rich Set of Features: Playwright offers a rich set of features for testing,
including native support for handling file uploads, downloads, and browser
context management.
Limitations:

1. Limited Community and Resources: As a relatively newer framework,

Playwright may have fewer resources and community support compared to
more established frameworks like Selenium.

2. Complex Setup for Mobile and Desktop Testing: Setting up mobile and
desktop testing environments with Playwright can be more complex
compared to web testing.

3. Compatibility with Older Browsers: Playwright may not fully support

older browser versions, which could be a limitation for projects requiring
compatibility with legacy systems.

4. Tool Maturity: Playwright may still be evolving in terms of stability,

performance optimizations, and feature completeness which could lead to
occasional bugs in certain scenarios.
Use Cases:

1. Testing Progressive Web Apps (PWAs): Playwright is well-suited for

testing PWAs, which combine the best features of web and mobile
applications.
 2. Multi-Platform Testing: Playwright is suitable for testing web, mobile, and
desktop applications within a single framework, streamlining testing
processes.

3. End-to-End Testing: Playwright excels at end-to-end testing by simulating

user interactions across various platforms and scenarios.

4. Browser Automation for Data Scraping: Beyond testing, Playwright can

automate browser-based tasks like data scraping and web crawling.

CHOOSING THE RIGHT AUTOMATION TESTING TOOLS

Involves evaluating various factors to ensure the selected tool meets your
project’s specific needs. Here are key considerations to help you make the right
choice.

1. Project Requirements: Understand the scope of your project. Determine

the type of application (web, mobile, desktop) and the testing needs
(functional, performance, security).

2. Ease of Use: The tool should have an intuitive interface and be easy to learn
and use. This reduces the learning curve for your team.

3. Supported Platforms and Languages: Ensure the tool supports the

platforms (Windows, macOS, Linux) and programming languages (Java,
Python, JavaScript) used in your project.

4. Integration Capabilities: The tool should integrate well with your existing
CI/CD pipeline, version control systems, and other development tools.
 5. Scalability: Choose a tool that can handle your current needs and scale as
your project grows.

6. Cost: Consider the budget for your project. There are both open-source and
commercial tools available, each with its own cost-benefit ratio.

COMPARISON WITH TRADITIONAL APPROACH

1. CODE INTEGRATION
Traditional Approach (Waterfall style)
Developers work separately for weeks/months. Merge all code at the end of
the project.
Result: Huge “merge conflicts,” broken features, lots of debugging.
CI/CD Approach
 Developers integrate small code changes daily or even hourly.
 Automated build + test runs immediately.
 Result: Conflicts caught early, easier fixes.

2. TESTING
Traditional Approach
 Testing happens after the full system is built. Manual QA takes days/weeks.
 Bugs found late = more costly to fix.
CI/CD Approach
 Automated tests run for every single code push (unit tests, integration tests,
UI tests).
 Bugs found within minutes of writing the code.

3. DEPLOYMENT
Traditional Approach
 Releases happen every few months.
 Example: Midnight deployment, 5-hour downtime, angry customers.

CI/CD Approach
 Small, frequent deployments (multiple times a day if needed).
 Example: Deploy wishlist feature at 3 PM, monitor, roll back instantly if
needed.
 (b) TEST SCRIPT DEVELOPMENT AND MAINTENANCE

Test script development and maintenance are crucial components of effective

software quality assurance, particularly within test automation.
Test scripts are a set of instructions or procedures written in a scripting language
that are used to execute a series of test cases on a software application or system.
In other words, test scripts are line-by-line descriptions that include related
information and must be performed to validate the application under test. It lists
the steps that need to be taken to meet the expected results.

Script Development and Maintenance: Refers to the process of creating,

organizing, and updating automated scripts used in software testing. These scripts
help ensure that applications work correctly by running a series of tests
automatically.

IMPORTANCE OF SCRIPT DEVELOPMENT

In the world of software test automation, script development is crucial. It involves

writing code that instructs a computer to perform specific tasks within a software
application. By developing effective scripts, testers can quickly find bugs or issues
in the software. This not only saves time but also improves the quality of the
software being tested.

Key Aspects of Script Development

1. Choosing the Right Tools: Testers often use different automation tools depending
on the software they are testing. It’s important to choose tools that are compatible
and efficient.
2. Testing the Script: Once the script is written, it needs to be tested to ensure it
works correctly. This includes checking if it can handle different scenarios and still
produce the expected results.
 IMPORTANCE OF SCRIPT MAINTENANCE

Script maintenance is just as important as script development. After scripts are

created, they need regular updates to stay relevant and functional. Software
changes over time, which can break older scripts.

Key Aspects of Script Maintenance

1. Updating Scripts: When the software undergoes updates or changes, scripts may
need adjustments. Regularly updating ensures that tests remain accurate.
2. Debugging: Scripts will not work as expected. Debugging involves identifying and
fixing issues within the scripts to maintain their functionality.
3. Refactoring: This means reorganizing the script code to make it more efficient
without changing what the script actually does. Refactoring can help improve
performance and maintainability.

TYPES OF TEST SCRIPTS

Scripts are divided into two types related with manual and automated testing
approaches.
Manual Test Scripts: These scripts are mainly written and executed by
human testers without assistance from automation tools. This approach proves
valuable for exploratory testing, Usability testing, ad-hoc testing.
Automated Test Scripts: These scripts are particularly written in a
programming language and run with the help of automation testing tools. These
scripts use automation testing frameworks to interact with the application and
automatically verify expected outcomes. There are further subtypes on the level of
testing.

Unit Testing Scripts - these concentrate on individual units or components of the

software applications composed by developers.
Integration Scripts - these focus on interaction between multiple components.
Functional Testing - these scripts validate that the software application function
correctly.
Regression Testing - these scripts confirm that existing functionalities remain
unaffected following new changes or updates done in the software.
Performance Testing - these scripts help the application performance under
different conditions.
End-to-End testing scripts- These scripts simulate real world scenarios to
evaluate the entire application flow from starting to completion.
Smoke testing scripts - These scripts are aimed at verifying the basic functionality
of the software application.
Security testing scripts- these assess the software applications functionalities
against security threats.

THE TEST SCRIPT TEMPLATE

It gives the format and specifies the information to be included in each test case.
A test script template has sections as follows-
1. Test case name- It is a unique identifier or name of the test case.
2. Test objective- It highlights the objective of the test cases.
3. Test steps- It gives detailed steps to run during the test.
4. Expected result- It includes information on specific outcomes expected when
test steps are run.
5. Test data- It has inputs needed for test cases. It includes values, parameters or
variables.

Utilizing a standard template within the team helps save time and reduce errors in
software testing. It also provides consistent script creation and effective
communication among team members.
Information or Key Components in Test Script:
 1. Inputs and expected results: The test script contains the raw content, including
inputs and sometimes the expected results.
2. User information: It outlines the user related details, like data to be provided to
the script.
3. Usage Information: Testers may include additional useful information, like
explanations of specific usage scenarios to related tests that might require
modifications.
4. Annotation: Comments included within the script explain the logical steps and
actions performed with the software that is tested.

Test Script Development steps:

 Requirement Analysis:
Understand the application's functionalities, user stories, and acceptance criteria
to define the scope of testing.
 Test Case Design:
Create detailed test cases outlining steps, input data, and expected outcomes for
each scenario.
 Scripting:
Translate test cases into executable test scripts using a chosen automation
framework and programming language. This involves identifying UI elements,
simulating user interactions, and validating results.
 Modularity and Reusability:
Design scripts to be modular and reusable, breaking down complex
functionalities into smaller, manageable functions or components. This improves
maintainability and reduces redundancy.
 Data Management:
Implement strategies for managing test data, separating it from the script logic to
enhance flexibility and reusability.
 Error Handling:
Incorporate robust error handling mechanisms within scripts to manage
unexpected behaviors and provide clear failure messages.
Test Script Maintenance:
 Alignment with Application Changes:
Regularly update test scripts to reflect changes in the application's UI,
functionalities, or underlying architecture. This ensures tests remain relevant and
accurate.
 Bug Fix Verification:
Update scripts to verify fixes for identified defects and prevent regression,
ensuring resolved issues do not re-emerge.
 New Feature Validation:
Expand or create new test scripts to validate the functionality of newly introduced
features and ensure they integrate seamlessly without introducing new defects.
 Optimizing Test Coverage:
Periodically review and optimize test coverage to ensure all critical
functionalities are adequately tested, identifying and addressing any gaps.
 Regular Reviews and Refactoring:
Schedule regular reviews of test scripts to identify outdated or redundant tests,
refactor code for clarity and efficiency, and improve overall script quality.
 Version Control:
Utilize version control systems (e.g., Git) to track changes, enable collaboration,
and facilitate easy rollback to previous versions if needed.
 Integration with CI/CD:
Integrate test maintenance activities into the Continuous Integration/Continuous
Delivery (CI/CD) pipeline to automate script updates and regression testing after
code changes.
 Monitoring and Analysis:
Regularly analyze test results to identify trends, potential areas for improvement,
and address flaky tests that provide inconsistent results.

Example of a Test Script:

Let us take a simple test case of testing the file uploading functionality of a web
application.
Manual Test Script is written as
1. Test Script title: Verify whether the user can upload a PDF file to the
application.
 2. Preconditions: The user should have a valid login username(testUser) and
password(testPassword).
3. Test Steps
a. Load the web application from a web browser. Log into the
application
b. Navigate to the file upload section
c. Browse a PDF file and select
d. Click on the file upload button
e. Verify whether the success message is received.

4. Expected Results: The user should be able to upload a PDF file to the web
application successfully.

II. DEFECT MANAGEMENT

DEFECT MANAGEMENT: It is the structured process of identifying, tracking

and resolving the defects to ensure software quality and reliability.

Importance of Defect Management in Software Testing-

Enhances Software Quality: It makes sure a stable, bug-free application.

Reduces Development Costs: Fixing defects early prevents costly rework.

Improves User Experience: A defect free product leads to better customer

satisfaction.

Streamlines Development Workflow: Organized defect tracking optimizes

debugging and fixes.

Ensures Compliance: Helps meet industry standards and regulations.

The Defect Management Process is a systematic approach to identify, track, and

resolve defects in software development.
 Fig: Defect Management Process
1. Defect identification - Defects are identified through various testing
activities, such as unit testing, integration testing, and user acceptance testing.

2. Defect logging - Defects are logged in a defect tracking system, along with
details such as description, severity, and priority.

3. Defect triage - The triage process involves evaluating the defects to determine
their priority and the resources required to resolve them.

4. Defect assignment - Defects are assigned to developers or testers for

resolution, based on their expertise and availability.

5. Defect resolution - The assigned personnel work on resolving the defects by

fixing the code, updating the documentation, or performing other necessary
actions.

6. Defect verification - Once the defect is resolved, it is verified by the tester to

ensure that it has been fixed correctly and does not introduce any new defects.

7. Defect closure - Once the defect has been verified, it is closed and the status
is updated in the defect tracking system.

8. Defect reporting - Regular reports on the status of defects, including the

number of open defects, the number of defects resolved, and the average time
 to resolve defects, are generated to provide visibility into the defect
management process.

(a) DEFECT LIFE CYCLE

DEFECT/BUG LIFE CYCLE

It is the specific set of states that defect or bug goes through in its entire life. The
purpose of Defect life cycle is to easily coordinate and communicate current status
of defect and make the defect fixing process systematic and efficient.
STAGES OF BUG LIFE CYCLE IN TESTING
The exact number of stages in your bug life cycle will depend on the type of
software project you’re working on.

1. New
This is the status of any bug identified and flagged for the first time, documented
in detail, and brought to the development team’s attention.

2. Assigned
This is the second step in the defect life cycle, where the bug is authorized and
assigned to the development team, either to an individual developer or the lead,
who will then pick a developer and assign him the responsibility to fix the bug..

3. Open
This is when the assigned developer runs tests on the bug to determine where the
mistake is and how to repair it. If the defect is identified as inappropriate, it’s
moved to either the “Deferred” or the “Rejected” state.

4. Fixed
A bug is marked as “fixed” after the developer has made a code change and
verified it.

5. Test
Now, the bug is tested to see if the error has been resolved.

6. Verified
The tester re-tests the bug after it got fixed by the developer. If there is no bug
detected in the software, then the bug is fixed and the status assigned is “verified.”

7. Closed
This is the final stage in the software bug life cycle, where the defect has been
resolved. The goal is to reach this stage as soon as possible. If the bug is no longer
exists then tester assigns the status “Closed.”

8. Reopen
This status is assigned if the bug recurs even after the resolution. If this happens,
all the stages of the bug life cycle need to be repeated until the problem is solved.
If the bug persists even after the developer has fixed the bug, the tester changes the
status to “reopened”. Once again the bug goes through the life cycle.

9. Duplicate
This is when the same bug gets reported twice by mistake. If the defect is repeated
twice or the defect corresponds to the same concept of the bug, the status is
changed to “duplicate.”

10. Deferred
If the bug isn’t of top priority or if there’s a plan to modify it in the next release of
the software, the testing team might “defer” it. However, it’s important to note that
high-priority bugs shouldn’t be deferred.

11. Rejected

A non-reproducible bug may occur due to platform/data/build mismatch . If the

developer feels the defect is not a genuine defect then it changes the defect to
“rejected.”

12. Cannot be fixed

In some cases, this status is assigned if the development team lacks the skills or
resources or if fixing the bug is expensive.

13. Not a defect

If the bug doesn’t impact other app functions, this status is assigned. Ultimately,
it’s “Rejected.” If it does not affect the functionality of the application then the
status assigned to a bug is “Not a bug”.

14. Not reproducible

The development team assigns this state to the bug if it can’t be reproduced due to
a platform/build mismatch, inconsistent defects, or an incomplete guideline defined
by the tester.

(b) DEFECT REPORTING AND TRACKING

Reporting and tracking bugs is crucial for effective software testing, and it largely
depends on the tools you use. A professional bug-tracking tool allows you to
efficiently log, report, and monitor defects throughout the testing process.
A bug-tracking tool ensures that all issues are detected, managed, and resolved.

With BrowserStack, you gain access to a powerful debugging toolkit, making it

2easier to verify, fix, and debug issues directly on real devices.
DEFECT TRACKING

Defect tracking is the process of identifying, documenting, and managing Defects

in software during its development or maintenance.

It involves systematically recording information about a Defect define, prioritize,

analyze, and track its status from discovery through resolution to ensure a more
reliable and high-quality end product.

Different Phases of Defect Tracking

1. New: Defects are initially identified and reported, starting in the “New”
state.

2. Assigned: A team lead or manager assigns the defect to the appropriate

developer or development team.

3. Open: The assigned developer works on fixing the defect, moving it to the
“Open” state.

4. Fixed: Once the developer has implemented a solution, the defect is marked
as “Fixed.”

5. Verified: The testing team verifies the fix; if successful, the defect is
marked as “Verified.” If not, it may be reopened or assigned back to the
developer.

6. Closed: After successful verification, the defect is marked as “Closed,”

indicating that it has been resolved, and the code is ready for release.
 7. Deferred: The decision is made to postpone the resolution of the defect to a
future release or update.

DEFECT TRACKING PARAMETERS

The most commonly used test tracking parameters are:

 ID: Uniquely identified each defect.

 Title: Name of the defect that is also usually descriptive.

 Description: A larger description of the defect, as well as the step required

to reproduce the defect.

 Severity: Marking the intensity of the defect – critical, major, minor.

 Priority: Defining how important it is to fix the defect, depending on how

much it impacts the software system – high, medium, low.

 Status: The current status of the defect – open, closed, deferred.

 Assigned: The developer to whom the defect is assigned for resolution.

 Due Date: The date by which the defect must be resolved.

 Comments: Comments providing further information about the defect.

Defects are generally tracked in the following ways:

1. Logging Defects: make sure that information related to the defect is captured
with enough details and the screenshots if possible.
2. Prioritizing Defects: Assign levels of priority of bugs that ensures higher ones
to be resolved first.

3. Assigning Defects: appropriate team members handle the defects for resolution.

4. Tracking Status: Make a note on the status of the defect, like whether it is a
new, in progress, fixed or closed defect.

5. Reviewing and Closing: Check the fix, retest the defect and then close it after
confirming that defect is resolved.

(c) ROOT CAUSE ANALYSIS

RCA (Root Cause Analysis) helps in improving software quality by improving the
development processes and communication . RCA involves a structured
investigation process that examines various aspects of the software development
and testing lifecycle, including requirements, design, testing and development.

It is a structured and effective process to find the root cause of issues in a Software
Project team. If performed systematically, it can improve the performance and
quality of the deliverables and the processes, not only at the team level but also
across the organization.

RCA (Root Cause Analysis) is a mechanism of analyzing the Defects, to identify

its cause. It digs into the defect to identify whether the defect was due to “testing
miss”, “development miss” or was a “requirement or designs miss”.
TYPES OF ROOT CAUSES

1) Human Cause: Human-made error. Factors attributed to human actions,

decisions, or knowledge. For example, Data entry error in the Database
Examples:
 Under skilled.
 Instructions not duly followed.
 Performed an unnecessary operation.

2) Organizational Cause: A process that people use to make decisions that were
not proper. Factor that arises from problems within organizations. For Example,
poor communication and weak policies
Examples:
 Vague instructions were given from Team Lead to team members.
 Picking the wrong person for a task.
 Monitoring tools not in place to assess the quality.

3) Physical Cause: Any physical item failed in some way. Factor related to
malfunctioning or breakdown of hardware. For Example, hard disk failure.
Examples:
 The computer keeps restarting.
 The server is not booting up.
 Strange or loud noises in the system.
 3RS OF ROOT CAUSE ANALYSIS (RCA)
The 3R’s of Root Cause Analysis (RCA) are “Recognize,” “Rectify,” and
“Replicate“. They are the key steps in Root Cause Analysis used to identify and
address the underlying causes of a problem.

 Recognize: This initial step involves acknowledging a problem or defect in the

software. It requires keen observation and effective communication between
team members to ensure that issues are identified.
 Rectify: After identifying the issue, the next step is to take corrective action to
deal with the underlying causes. This entails implementing fixes that won’t just
address the current problem but also stop it from happening again.
 Replicate: This phase ensures the identified root cause is effectively rectified
and allows thorough software testing to prevent any future recurrence of issues.

PERFORMING ROOT CAUSE ANALYSIS (RCA) IN SOFTWARE TESTING OR

WORKING:
The essential steps to perform the Root Cause Analysis (RCA) in software testing
are:-

1. Define the Problem: The first step is identifying and defining the problem or
defect during software testing.
2. Collect Data: After finding the problem, collect problem evidence, such as
problem test cases, logs, screenshots, and reports.
3. Identify Possible Causes: List all potential causes contributing to the problem
or defects.
4. Identify the Root Cause: Employ various Root Cause Analysis (RCA)
techniques to identify the underlying issue accurately.
5. Implement Root Cause Corrective Action (RCCA): After identifying the root
cause, implement corrective action to address identified root causes and
proactively prevent the recurrence of issues.
6. Implement Root Cause Prevention Action (RCPA): After resolving the root
cause, implement preventive measures such as improved testing procedures or
process enhancements to avoid similar issues in the future.

Root-Cause Analysis Templates

1. Ishikawa diagram

The Ishikawa diagram, commonly referred to as a fish-bone diagram or a

cause-and-effect diagram, was developed by Kaoru Ishikawa .

The six Ms—man, measures, material, milieu, methods, and machines—are

essential considerations in an Ishikawa diagram.

2. Pareto chart

The 80/20 rule, commonly referred to as the Pareto principle, is the foundation of
the Pareto chart. The 80/20 rule states that 20 percent of the issues cause 80
percent of the expenses. The Pareto chart is used to pinpoint the problems where

improvement efforts will have the most significant impact.

 failure costs, or other categories as necessary.

 Issue prioritization is the primary application of this tool.

3. 5 WHYs

The 5 Whys approach is used to explore further until the actual root cause of an
event is found. When an event has more than one reason, the process might be
divided, or the core cause can be sought by asking why repeatedly 5 times.

Here are the considerations to remember while using this root cause analysis tool:
  You can find the real root reason for an occurrence by asking why five
times.
 This technique must be used in conjunction with more accurate,
quantitative approaches.
 One may use it to identify failure as well as incidence.

4. Fault Tree Analysis (FTA)

Fault tree analysis (FTA) investigates the root causes of system failures.

Risks are ranked according to importance in fault tree analysis, allowing the most
significant threats to be fixed first. It takes a top-down strategy to pinpoint the
component level failures (basic event) that lead to the system level failure (top
event), combining them using Boolean logic.

5. Failure mode and effects analysis (FMEA)

FMEA is a proactive root cause analysis method that guards against a

probable system or equipment failure. An FMEA diagram shows:

 Failure scenarios, effects, and causes

 The current safeguards against every kind of failure
 Ratings for severity (S), occurrence (O), and detection (D) enable you
to determine the risk priority number (RPN) and the next course of
action.

Example of RCA: Analyzing the causes behind website downtime

Let us imagine a media company’s website is available 97% of the time

compared to its competitors. Every time the website goes down, a reason is given,
such as a failed change, a person making a mistake, data problems, or service
failures. The organization conducts a gap analysis to identify the underlying
reasons for these failures.

The analysis reveals that the website’s infrastructure, platform, development

procedures, and code have problems resulting in an unstable environment. For
instance, the company hired a company with a high turnover rate to handle its
development and web application security management outsourcing.
 Each developer only spends an average of two weeks working on the code since
the company often moves its personnel around. The platform’s developers’ lack of
familiarity leads to more stinky code and issues. In such scenarios, root cause
analysis helps prioritize which problems to fix first and where to invest resources.

(d) DEFECT METRIC AND ANALYSIS

DEFECT METRICS
Defect metrics are measurements used to track and evaluate defects in products,
especially in software development. They help teams understand how many defects
are present, how severe they are, and how quickly they can be fixed.

Importance of Defect Metrics

Defect metrics are crucial for improving product quality. By keeping track of
defects, teams can find patterns and signs of problems earlier in the development
process. This allows them to fix issues before they become bigger problems.

Key Components of Defect Metrics

1. Defect Density: This measures the number of defects found in a given size of the
product, like per 1,000 lines of code. It helps to understand how healthy the code
is.
Definition of Defect Density: Understanding what defect density means in
the context of software.
Calculating Defect Density: Methods for calculating defect density, often
expressed per 1,000 lines of code.
Importance of Defect Density: The role defect density plays in assessing
code quality.

2. Defect Life Cycle: This tracks a defect from when it is found until it is fixed. By
monitoring this life cycle, teams can see how long it takes to resolve issues.
 Stages of the Defect Life Cycle: Overview of stages such as discovery,
classification, resolution, and closure.
Tracking Defects: Tools and techniques for monitoring defects throughout
their life cycle.
Analyzing Life Cycle Data: How to use life cycle data to improve
development processes.

3. Severity Levels: Not all defects are the same. Some defects are critical and need
immediate attention, while others can wait. Understanding severity levels helps
prioritize fixes.
Types of Severity Levels: Explanation of different severity levels, from
critical to minor defects.
Categorizing Defects: Methods for classifying defects based on their impact
on the system.
Prioritization of Fixes: Techniques for prioritizing defect fixes according to
severity.

4. Defect Discovery Rate: This metric tracks how quickly defects are found. A high
discovery rate can indicate more testing is needed or that the development process
may have flaws.
Definition and Importance: What defect discovery rate means and its
significance in software testing.
Measurement Techniques: Methods for measuring how quickly defects are
discovered.
Implications of High Discovery Rates: Understanding what a high
discovery rate indicates about the testing process.

5. Defect Trends and Analysis

Identifying Patterns: How to identify recurring patterns in defect
occurrences.
 Using Data for Predictions: Techniques for predicting future defects based
on historical data.
Reporting and Visualization: Best practices for reporting defect data
effectively.

Benefits of Measuring Defect Metrics

 Improved Quality: By analyzing defect metrics, teams can improve the overall
quality of the product.
 Better Communication: Defect metrics provide clear data that everyone on the
team can understand. This helps foster better communication among team
members.
 Resource Management: Understanding defect trends can help teams allocate
resources more effectively. They can prioritize their time and efforts toward fixing
more critical issues.

III. TEST METRICS AND MEASUREMENT

Without measurement, it is impossible to determine whether QA efforts are

effective.

TEST METRICS PROVIDES:

 Progress Insights: and what is left to test. Identify what has been tested in the
software
 Better Decision-Making: Make informed decisions about testing strategies based
on data and tool selection
 Identifying bottlenecks: Find out the areas for better efficiency and performance.
Benefits of Measurement in Software Testing

• Identification of testing strengths and weaknesses.

• Providing insights into the current state of the testing process.
• Evaluating testing risks.
• Benchmarking.
• Improving planning.
• Improving testing effectiveness.
• Evaluating and improving product quality.
•Measuring productivity.
• Determining level of customer involvement and satisfaction.
• Supporting controlling and monitoring of the testing process.
• Comparing processes and products with those both inside and outside the
organization.

TYPES OF SOFTWARE TESTING METRICS

1. PROCESS METRICS

Process metrics focus on the efficiency and effectiveness of testing activities.

1. Test Case Effectiveness: Measures how well test cases detect defects.
Formula: (Defects Detected / Test Cases Run) x 100
Use Case: Helps evaluate the quality of test cases and refine them for better
defect detection.
2. Cycle Time: Tracks the time taken to complete the testing process.
Insight: Helps teams understand the efficiency of test runs and identify
delays.
Example: If a testing cycle takes longer than expected, it may indicate
inefficiencies in the test environment or setup.
3. Defect Fixing Time: Measures the time taken to resolve a defect from
detection to closure.
Use Case: Identifies delays in defect resolution, aiding in optimizing
workflows.
2. PRODUCT METRICS

Product metrics evaluate the quality of the software under test.

1. Number of Defects: Indicates the quality and efficiency of the software
product.
Insight: Helps teams identify problem areas and implement necessary
improvements.
2. Defect Severity: Classifies defects based on their impact.
Example Categories: Critical, Major, Minor.
Use Case: Prioritizes fixes based on severity to ensure critical issues are
addressed first.
3. Passed/Failed Test Case Metrics: Provides data on the stability and
functionality of the software.
Formula for Passed Cases: (Passed Test Cases / Total Test Cases) x 100

Use Case: Identifies areas of improvement by analyzing failed test cases.

3. PROJECT METRICS

Project metrics provide insights into the broader scope of the testing process,
focusing on team and resource efficiency.
1. Test Coverage: Measures the percentage of tested functionalities.
Formula: (Tested Functionalities / Total Functionalities) x 100
Use Case: Ensures that all critical functionalities are tested.
2. Cost of Testing: Assesses total testing expenditure, including infrastructure
and resource costs.
Use Case: Helps in budget allocation and identifying cost-saving
opportunities.
3. Budget/Schedule Variance: Tracks deviations from planned costs and
timelines.
Use Case: Aids in maintaining project timelines and avoiding budget
overruns.
Calculating Key Testing Metrics
 SOFTWARE TEST METRICS LIFE CYCLE

Different stages in the test metrics life cycle:

1. Analysis: Identify the most suitable metrics for your process. This step
involves understanding project requirements and determining key performance
indicators.
 The metrics must be recognized.
 Define the QA metrics that have been identified.

2. Communication: Educate the team and stakeholders about the metrics. Ensure
alignment and clarity on the objectives.
 Stakeholders and the testing team should be informed about the
requirement for metrics.
 Educate the testing team on the data points that must be collected in order
to process the metrics.

3. Evaluation: Collect and validate data to assess metric performance. Regularly

review metrics to ensure their relevance and accuracy.

 Data should be captured and verified.

 Using the data collected to calculate the value of the metrics

4. Reporting: as visualizations can help with understanding. with stakeholders

for input. Graphs and charts Document discoveries and share

 Create a strong conclusion for the paper.

 Distribute the report to the appropriate stakeholder and representatives.
 Gather input from stakeholder representatives.
Formula for Test Metrics:
To get the percentage execution status of the test cases, the following formula can
be used:
Percentage test cases executed = (No of test cases executed / Total no of test
cases written) x 100

Similarly, it is possible to calculate for other parameters also such as test cases
that were not executed, test cases that were passed, test cases that were failed, test
cases that were blocked, and so on. Below are some of the formulas:

1. Test Case Effectiveness:

Test Case Effectiveness = (Number of defects detected / Number of test cases
run) x 100

2. Passed Test Cases Percentage: Test Cases that Passed Coverage is a metric
that indicates the percentage of test cases that pass.
Passed Test Cases Percentage = (Total number of tests ran / Total number
of tests executed) x 100

3. Failed Test Cases Percentage: This metric measures the proportion of all
failed test cases.
Failed Test Cases Percentage = (Total number of failed test cases / Total number
of tests executed) x 100

4. Blocked Test Cases Percentage: During the software testing process, this
parameter determines the percentage of test cases that are blocked.
Blocked Test Cases Percentage = (Total number of blocked tests / Total number
of tests executed) x 100

5. Fixed Defects Percentage: Using this measure, the team may determine the
percentage of defects that have been fixed.
Fixed Defects Percentage = (Total number of flaws fixed / Number of defects
reported) x 100

6. Rework Effort Ratio: This measure helps to determine the rework effort ratio.
Rework Effort Ratio = (Actual rework efforts spent in that phase/ Total actual
efforts spent in that phase) x 100

7. Accepted Defects Percentage: This measures the percentage of defects that

are accepted out of the total accepted defects.
Accepted Defects Percentage = (Defects Accepted as Valid by Dev Team / Total
Defects Reported) x 100

8. Defects Deferred Percentage: This measures the percentage of the defects

that are deferred for future release.
Defects Deferred Percentage = (Defects deferred for future releases / Total
Defects Reported) x 100
Example of Software Test Metrics Calculation:
Consider an example to calculate test metrics:
1. Percentage test cases executed = (No of test cases executed / Total no of test
cases written) x 100
= (164 / 200) x 100
= 82

2. Test Case Effectiveness = (Number of defects detected / Number of test cases

run) x 100
= (20 / 164) x 100
= 12.2

3. Failed Test Cases Percentage = (Total number of failed test cases / Total
number of tests executed) x 100
= (60 / 164) * 100
= 36.59

4. Blocked Test Cases Percentage = (Total number of blocked tests / Total

number of tests executed) x 100
= (4 / 164) * 100
= 2.44

5. Fixed Defects Percentage = (Total number of flaws fixed / Number of defects

reported) x 100
 = (12 / 20) * 100
= 60

6. Accepted Defects Percentage = (Defects Accepted as Valid by Dev Team /

Total Defects Reported) x 100
= (15 / 20) * 100
= 75

7. Defects Deferred Percentage = (Defects deferred for future releases / Total

Defects Reported) x 100
= (5 / 20) * 100
= 25

(a) METRICS FOR TEST PLANNING AND PROGRESS TRACKING

A process-oriented approach for software testing ensures that the work products
are tested as they are complete at key process points. Attributes identification of
software test planning process leads to generation of useful metrics which satisfies
the four key aspects of managing a test planning effort.

1. Establishing goals for test planning process,

2. Execution of the test planning process,
3. Measurement of the results of test planning process and
4. Changing the test planning process .
FIG: Managing software test planning process through metrics.

The system test planning process takes input from the development process in the
form of knowledge of where to optimally utilize the testing resources.
The test planning can take advantage of focusing its effort on those error areas and
over complex modules.

The attributes to be measured is dependent on three factors:

 First, the attributes to measure is dependent on the time and phase in the
software development lifecycle.
 Secondly, new attributes to measure emerges from new business needs.
 Thirdly, attributes to be measured are aligned to ultimate goal of the project.

The Attributes that are useful for measurement during Software Test Planning are:
1. Progress
Following attributes are defined under the progress category of metrics:

a. The Suspension Criteria for Testing

The suspension criteria of testing are based on the measurement of certain
attributes. Therefore, metrics that establish conditions for suspending testing are to
be identified at the test planning phase.
b. The Exit Criteria
The exit criteria for testing needs to be established during test planning. The
decision is to be based on metrics, so that an exit criterion is enabled after meeting
a condition.
c. Scope of Testing
The test planning activity also determines the scope of testing. So metrics is how
much of it is to be tested and required .
d. Monitoring of Testing Status
On time and in budget completion of testing tasks is essential to meet schedule and
cost goals. Therefore, the testing status needs to be monitored for getting the
correct status to schedule and cost.

2. COST
Following attributes are defined under the cost category of metrics:

a. Testing Cost Estimation

The test planning phase also has to establish the estimates for budgets. Therefore,
metrics supporting testing budget estimation need to be established early. This
measurement helps answering the question of how much will it cost to test?

b. Duration of Testing
Estimating a testing schedule is one of the core activities of test planning.
Therefore, metrics that assist in the creation of a testing schedule is required. As
part of the testing schedule, the time required to develop a test plan is also to be
estimated.

c. Resource Requirements
The test planning activity has to estimate the number of testers required for
carrying out the system testing activity.

d. Training Needs of Testing Group and Tool Requirement

The test planning must identify the training needs for the testing group and tool
requirement. Metrics indicating the need of training and tool requirement needs to
be established.
3. QUALITY
Following attributes are defined under the quality category of metrics:

a. Test Coverage
During test planning, test coverage decisions are to be made. Test coverage helps
to answer how much of the requirements, design, code and interfaces is tested?
There are different forms of coverage namely, e.g. code coverage, requirements
coverage, design coverage and interface coverage [6]. Therefore, test coverage is
an attribute that a testing group wants to measure to assess the percentage of code,
requirements, design or interface covered by a test set.

b. Effectiveness of Smoke Tests

Smoke tests establish that stability of application for carrying out testing. Metrics
establishing the effectiveness of smoke tests is required to be certain that
application is stable enough for testing.

(b) TEST COVERAGE

Test Coverage refers to the level to which test cases verify the application's
functionality and make sure that tests are developed for every part of the
application.

A test manager assures that all the feature requirements link with their relative test
cases. The goal is to keep the mapping of requirements to test cases to 100%.

Test Coverage Metrics in Software Testing

1. Test execution progress:

It gives an idea about the total number of test cases executed compared to the
number of test cases still pending. This metric is measured during test execution
and determines the coverage of testing.

Formula:

Test Execution Coverage= (Total number of executed test cases or scripts/ Total
number of test cases or scripts planned to be executed)*100
2. Functional Coverage: It defines how much coverage the test plan provides for
the business and functional requirements. Functional coverage is a metric
measuring the functions involved during software testing.

Formula : The number of functions executed by a test suite is divided by the total
number of functions in the software under testing.

Example: Software Application- Online Banking Website

To ensure functional coverage the following features and functionalities
should be tested -

Login and Logout- Users should be able to log in and out of their accounts
using username and password.

Account Management- Users should be able to view their account balances,

transaction history.

Funds transfer- Users should be able to transfer money between their accounts
and to other accounts within the same bank or at other banks.

Bill pay- Users should be able to schedule and pay bills electronically.

Check deposit- Users can deposit checks remotely using their smart phone or
computer.

Customer support- Users should have access to customer support through various
channels such as phone, email or live chat.

3. Product Coverage: It focuses on testing different aspects of software product

which includes dimensions like user interface, hardware configurations, operating
systems, browsers and databases.
Example- Web application

Product coverage dimensions:

 User interfaces- Desktop and mobile versions

 Operating systems- Windows and Linux
 Browsers - Chrome, Safari and Firefox
 Databases - MySQL

4. Requirements Coverage: Thoroughly testing the software in different scenarios

and environments, testers make sure that all the required functionalities and
features work as per the requirement of the users.

Example- Users must be able to transfer money between their accounts is the
requirement.

 Users must be able to view their account balance and transaction history
 Users must be able to pay bills online.
 Users must be able to deposit checks online.
 Users must be able to receive alert for account activity.
 Users must be able to contact customer support for assistance.

All these will ensure sufficient requirement coverage for the online banking
website.

5. Risk Coverage:

By conducting thorough testing, testers can identify potential risks associated with
the software. This includes identifying security loopholes, system failures and
software's performance and user data.

Example- for online banking website, risk coverage covers security vulnerabilities,
Fraudulent transactions .