Software Testing Metrics Life Cycle

 Analysis: Identify the most suitable metrics for your testing

process.

 Communication: Educate the testing team and stakeholders about

the chosen metrics.

 Evaluation: Collect and verify relevant data, then use this data to
assess the value of the metrics.

 Reporting: Create a comprehensive report summarizing the

findings and distribute it to stakeholders for feedback.

How to Calculate Test Metrics?

Metric Description Formula

Measures the number

of defects relative to Defect Density = Total
Defect the size of the Defects / Size of the
Density software module. Module

Evaluate the efficiency Test Case Effectiveness

Test Case of test cases in = (Defects Detected /
Effectiveness detecting defects. Test Cases Run) x 100

Defect Leakage =
Quantifies defects (Defects Found Post-
Defect missed during testing Testing / Defects Found
Leakage but found later. Pre-Testing) x 100

Assesses the Defect Removal

Defect effectiveness of the Efficiency = (Defects
Removal testing process in Removed / Total Defects
Efficiency removing defects. at Start) x 100

Indicates the extent to

which the software Test Coverage = (Tested
Test functionalities are Functionalities / Total
Coverage tested. Functionalities) x 100
 Passed Test Measures the Passed Test Cases =
Cases proportion of test (Passed Tests / Total
Percentage cases that passed. Tests Executed) x 100

Failed Test Indicates the Failed Test Cases =

Cases proportion of failed (Failed Tests / Total Tests
Percentage test cases. Executed) x 100

Test Execution Rate =

Test (Test Cases Executed /
Execution Tracks the speed of Planned Test Cases) x
Rate the testing process. 100

Defect Age = Time of

Measures the time to Defect Fix – Time of
Defect Age fix a defect. Defect Discovery

Assesses the effort Rework Effort Ratio =

Rework Effort spent on reworking or (Rework Effort / Total
Ratio correcting defects. Testing Effort) x 100

Examples of Software Test Metrics Calculation

Example 1:

Consider a scenario where you have the following data from test case
development:

 Total Test Cases Written: 200

 Total Test Cases Executed: 180

 Passed Test Cases: 100

 Failed Test Cases: 80

Using this data, you can calculate various metrics. For example:

 Percentage of Test Cases Executed: (180 / 200) x 100 = 90%

 Test Case Effectiveness: (Number of Defects Detected / Number of

Test Cases Run) x 100 = (10 / 180) x 100 ≈ 5.55%.

Example 2:

Let's take another example for calculating software test metrics:

  Total Number of Defects Identified: 20

 Defects Accepted as Valid by the Development Team: 15

 Defects Deferred for Future Releases: 5

 Defects Fixed: 12

From this data, you can calculate metrics like:

 Fixed Defects Percentage: (12 / 20) x 100 = 60%

 Accepted Defects Percentage: (15 / 20) x 100 = 75%

 Defects Deferred Percentage: (5 / 20) x 100 = 25%

Types of Software Testing Metrics

Since software testing metrics offer quantitative measurement data to

estimate the progress, quality, and health of the software testing process,
they help teams in making better decisions for future testing activities. By
getting their hands on reliable data about the testing process, they help
understand the type and scale of improvements needed and effectively
plan the next steps.

There are different types of metrics that teams can use for ensuring QA.
Let’s look at the top 10 metrics for measuring quality assurance success:

Process Metrics

Process metrics provide important information on the testing process,

indicating common bottlenecks and the overall level of process efficiency.

1. Test case effectiveness is a software testing metric that measures how

effectively test cases detect problems. It helps in answering how good
tests are, assessing if high-value test cases are being run, and
determining the bug-finding ability of the test case.

2. Cycle time measures the amount of time it takes to complete the entire
length of the testing process. It can be in hours, days, or even weeks. It
also delivers insight into passed, failed, blocked, incomplete, and
unexecuted test executions for easy absorption of the test run status.

3. Defect fixing time showcases the amount of time it takes for testing
teams to remedy a problem – from the time of detection to the time of
resolution. It also gives a better idea of how soon they can release the
product to the market – under budget and time.

Product Metrics

Product metrics shed vital light on the quality of the product and lead to
better testing decisions.
1. Number of defects is a software testing metric that helps test engineers
dig deeper into the many aspects of software quality. By having
information on the number of defects, teams can determine the quality
and efficiency of the software product and implement remediation
measures accordingly.

2. Defect severity and cause is another metric that allows testers to see
how severe defects are. It also provides insight into their root cause and
how they will affect the software’s quality.

3. Passed/failed test case metrics inform testing teams of the number of

test cases that have passed. Such insight helps ascertain the functionality,
performance, stability, usability, and compatibility of the software under
test and the efforts needed to reduce the frequency of failed tests.

Project Metrics

Project metrics provide information on the productivity and efficiency of

the team, tools, and the project in general.

1. Test coverage metrics measure the test effort. They provide insight into
how much of the product or application is successfully tested. It is a clear
indicator of the quality of the test plan and estimates the amount of
testing performed, including functional requirements and risk coverage.

2. Rework effort metrics help testing teams calculate the actual rework
efforts spent in resolving a defect after product delivery. It also helps
tweak the product or process to ensure they don’t recur. Using this metric,
teams also get vital insight into the time and cost needed to fix a defect.
As a result, they can arrive at an approximate deadline for deployment.

3. Cost of testing metric specifies the overall expenditure of the testing

process. Since people, infrastructure, and tools contribute toward the total
cost of testing, teams need to know how much they intend to spend and
how much they end up spending.

4. Budget/schedule variance is yet another important software testing

metric that states the difference between actual cost/schedule and
planned cost/schedule. It helps teams improve financial and operational
planning, work within budgets, and pursue opportunities that accelerate
time-to-test.

Manual Test Metrics: What Are They and How Do They Work?

Manual test metrics involve collecting and analyzing data from manual
testing activities. These metrics help in understanding the effectiveness
and efficiency of the manual testing process. They include metrics like the
total number of test cases, the number of test cases executed, and the
number of defects found. These metrics are crucial for assessing the
manual testing process thoroughness and identifying areas for
improvement.

Choose the Best Software Testing Metrics

Software testing metrics play a massive role in QA. It allows testers, agile
teams, and test managers to take their testing to the next level.
Constantly measuring the test process across different metrics enables
teams to establish a baseline and track progress over the next days,
weeks, and months.

While there are hundreds of testing metrics that you can choose from,
focusing on key process, product, and project metrics can help you in
ensuring that your software product meets necessary functionality,
performance, reliability, and security standards and delivers a seamless
user experience.

Looking to score high across all your metrics? Here’s how ACCELQ’s low-
code, AI-powered test automation framework can help!

Test Economy

1. Total Allocated Cost: The amount approved by QA Directors for

testing activities and resources for a certain project or period of
time.

2. Actual Cost: The actual amount used for testing. Calculate this on
the basis of cost per requirement, per test case or per hour of
testing.

3. Budget Variance: The difference between the Allocated Cost and

Actual Cost

4. Time Variance: The difference between the actual time taken to

finish testing and planned time.

5. Cost Per Bug Fix: The amount spent on a defect per developer.

6. Cost of Not Testing: Say, a set of new features that went into prod
need to be reworked, then the cost of the reworking activities is
basically, the cost of not testing.

7. Test Execution = Number of tests already run/Total tests to be run) x

100
8. Requirements Coverage = Number of requirements covered by existing
tests/Total number of requirements) x 100
 9. Number of tests run per (duration): Number of tests executed / Total
duration
10. Test review rate: Number of tests reviewed / Total duration
11. Defect capture rate: Total defects captured / Total test run
duration
12. Average bugs per test: Total number of bugs / Total number of
tests
13. Total cost allocated for testing: The monetary amount that
management has approved for QA activities for a specific duration
(quarter, year, etc.)
14. Actual cost of testing: The actual monetary amount that went into
running necessary tests. This calculation can include the cost of testing
per hour, per test case or per requirement.

For example: if your total allocated cost is 2000 dollars and you have to test
200 requirements,

Cost of testing per requirement: 2000/200 = 10 dollars

Cost per test hour: 2000/number of test hours (let’s say 200) = 100 dollars

Cost per test case: 2000/ number of test cases (let’s say 50) = 40 dollars

Cost of not testing

Calculating the cost of not testing might seem counterintuitive but is a

great way to establish the necessity of QA functions. Monitoring this QA
metric is especially important if you have to justify increased budgetary or
hiring requests to stakeholders.

The cost of not testing refers to the cost of fixing any feature that went
into production without testing, failed and needed fixing.

Not only can you calculate cost based on the number of dev hours spent
to fix the flaw but you can also include subjective costs such as:

 More time devoted to customer calls & support requests

 Product downtime

 Loss of customer trust, loyalty and brand credibility

 15. Defect Age

 Defect age measures the average time developers take to fix a defect,
from when it starts to when the bug is actually resolved.
 Defect Age = Difference between the time of bug creation and time of
bug resolution
 Generally, defect age is measured in days. Let’s say the bug was
identified on 6/4/2022 and was fixed on 23/4/2022. In this case, the
defect age is 17 days.
 Progressively low defect age is a powerful indicator of a QA team’s
maturation. It means bugs take less time to fix with every test cycle.
 16. Test Case Effectiveness
 Derived as a percentage, this number indicates the efficacy of test
cases in detected bugs. In other words, how many test cases executed
by your QA team have successfully identified bugs in a single test
cycle?
 The formula is simple:

Test Case Effectiveness = (Number of bugs found/Number of test

cases executed) x 100
 An important measure of test case quality, the number should
incrementally grow over progressive test cycles. It is one of the most
obvious indicators of a QA team’s performance.
 17. Defect Leakage
 Defect leakage may seem similar to the first metric in this list ( i.e.
escaped bugs). However, in this case, you’re monitoring the number of
bugs that escape to the UAT (User Acceptance Testing) stage.
Therefore, dealing with defect leakage is far less serious than handling
escaped bugs.
 Essentially, this refers to the number of bugs that show up in UAT, after
the app has been through multiple layers of testing. Ideally, your test
cases should filter them out before potential users touch your product.
 Calculate this as:

Defect Leakage = (Total numbers of defects in UAT/ Total number of

defects found before UAT) x 100
 8. Test Case Productivity
 You probably won’t have to report this metric to management, but
measuring it helps with setting realistic expectations for your team.
 Test case productivity evaluates the effort required to build test cases
for a specific sprint/cycle. The formula is:

Test Case Productivity = (Number of Test Cases/Effort required per

Test Case) x 100
 Obviously, “Effort Required per Test Case” won’t be an exact number.
Certain test cases require more design work than others. But you can
ask your testers to provide a fair average. This metric will give you a
sense of what is reasonably possible for your team to accomplish per
cycle.
est Completion Status

Not every test case your team designed will be executed to completion.
Some tests will pass, some will fail, and some will end up not being
executed or blocked—monitoring test completion status is another KPI
indicator of overall team performance.

A few different formulas whose results will combine to provide the big
picture of test completion status:

 % of Test Cases Executed = (Number of test cases

executed/Number of test cases created) x 100

 % of Test Cases Not Executed = (Number of test cases not

executed/Number of test cases created) x 100

 % of Test cases Passed = (Number of test cases passed / Number of

test cases executed) x 100

 % of Test cases Failed = (Number of test cases failed / Number of

test cases executed) x 100

 % of Test cases Blocked = (Number of Test cases blocked / Number

of test cases executed) x 100

With these numbers in hand, you can quickly judge the current state of QA
operations. For example, if the % of test cases passing is lower than % of
test cases blocked, there might be a foundational issue with the test case
design or the test environment. Now you know what problem to zero in on
to improve outcomes for the next sprint.

20. Test Review Efficiency

Even though test cases may have flagged bugs, each such flag requires
some review by a tester, even if it takes just a few minutes – and it usually
takes longer. However, depending on the software and its development
stage, tests may return a large number of bugs. The time to review each
does add up, which is why you need to calculate test review efficiency.

Test review efficiency % = (Number of tests reviewed/ Total number of

tests requiring review) x 100

Of course, the formula for this QA metric must be applied in the context of
a certain duration. Let’s say that in a test sprint of 7 days, 58 bugs were
detected but given the nature of these bugs, your team could only review
and forward 45 of them for resolution; Your test review efficiency then
stands at 77%.
Again, this is a good number to measure your team’s performance, and
what they need to review more defects.

Closing Thoughts

The necessity of measuring a QA team’s performance cannot be

overstated. Like every investment, QA must show adequate returns to
account for its existence in any SDLC. Fortunately, the necessity and
efficacy of a QA function has been proven innumerable times, as long as it
adheres to best, evolving practices.

Measure the QA metrics described above, and you will have profound
clarity on how testing teams are performing and the absolute value they
bring to the table.

SDLC

What is software testing in SDLC?

Software Testing Life Cycle (STLC) is a process used to test software and
ensure that quality standards are met. Tests are carried out systematically
over several phases. During product development, phases of the STLC
may be performed multiple times until a product is deemed suitable for
release.
When discussing your reason for changing jobs as a QA Manager, it’s
important to frame your answer positively and focus on your professional
growth. Here’s a structure you might follow:

1. Focus on Growth: Emphasize your desire for new challenges or

opportunities to advance your skills. You might say, “I’m looking for
an opportunity to expand my expertise in quality assurance and
take on new challenges that align with my career goals.”

2. Alignment with Values: Talk about how the new role aligns with
your values or vision. For example, “I’m excited about the prospect
of working in an organization that prioritizes innovation and quality,
which I believe is crucial in today’s market.”

3. Contribution: Highlight what you can bring to the new role. You
could say, “I believe my experience in leading teams and
implementing effective QA processes can contribute significantly to
your organization.”

4. Positive Transition: Avoid speaking negatively about your current

job. Instead, you can say, “I’ve learned a lot in my current position,
and I’m ready for a new environment where I can leverage that
experience and continue to grow.”

Here’s a sample answer:

“I’m looking for a new opportunity to advance my career in quality

assurance and take on new challenges that align with my professional
goals. I’m particularly drawn to your organization because of its
commitment to innovation and quality. I believe my experience in leading
QA teams and implementing successful processes can make a meaningful
contribution here. I’ve gained valuable insights in my current role, and I’m
excited about the chance to apply that knowledge in a new environment.”

This approach keeps the focus on your aspirations and what you can offer
rather than any negatives about your current situation.

Performance testing metrics are essential for evaluating the speed,

scalability, reliability, and overall performance of a system. Here are some
key metrics along with examples:

1. Response Time

 Definition: The time taken to receive a response after a request is

made.
  Example: If a user clicks a button on a webpage and the server
responds in 1.2 seconds, that’s the response time for that action.

2. Throughput

 Definition: The number of requests processed by the system in a

given time frame, typically measured in requests per second (RPS).

 Example: A web application handles 500 requests in 10 seconds,

resulting in a throughput of 50 RPS.

3. Error Rate

 Definition: The percentage of requests that result in errors, often

expressed as a ratio or percentage.

 Example: If 10 out of 1,000 requests fail, the error rate is 1%.

4. Latency

 Definition: The time delay from when a request is sent until the
first byte of the response is received.

 Example: If a user sends a request and receives the first byte back
in 300 milliseconds, that’s the latency.

5. Concurrent Users

 Definition: The number of users simultaneously interacting with the

system.

 Example: During a load test, if 200 users are logged in and

performing actions at the same time, that’s the level of concurrency
being tested.

6. CPU and Memory Usage

 Definition: The percentage of CPU and memory resources being

utilized by the application during testing.

 Example: A performance test shows CPU usage at 85% and

memory usage at 70% while handling a peak load.

7. Network Bandwidth

 Definition: The amount of data that can be transmitted over the

network in a given time, usually measured in Mbps (megabits per
second).

 Example: If a system can handle 50 Mbps during peak usage, that’s

the network capacity.

8. Scalability
  Definition: The system's ability to handle increased loads by
adding resources, either vertically (upgrading existing resources) or
horizontally (adding more machines).

 Example: If a system can handle 1000 users with 2 servers and

scales to handle 2000 users with 4 servers without degradation, it
demonstrates good scalability.

9. Load Time

 Definition: The total time taken to fully load a web page or

application.

 Example: A webpage takes 3 seconds to fully load all elements,

which is measured during a performance test.

10. Percentiles (e.g., P95, P99)

 Definition: The percentage of requests that are completed within a

certain time. For example, P95 means 95% of requests are
completed within that time frame.

 Example: If the P95 response time is 1.5 seconds, it indicates that

95% of requests are completed in 1.5 seconds or less.

Conclusion

By monitoring these performance testing metrics, teams can ensure that

applications meet their performance goals and provide a seamless user
experience. Regular analysis and reporting of these metrics help identify
bottlenecks and guide optimization efforts.