Chapter 2: Performance

Testing in the Software

Lifecycle
Author: Quoc Bao Nguyen
2.1 Typical Metrics Collected in Performance
Testing
Why do we need Performance Metrics?
 The metrics which are derived from those measurements are essential for defining the goals of
performance testing
 Evaluate the results of performance testing
 Performance testing should not be undertaken without first understanding which measurements
and metrics are needed.
2.1 Typical Metrics Collected in Performance
Testing
Project risks if there is no Performance metrics:
 It is unknown if the levels of performance are acceptable to meet operational objectives
 The performance requirements are not defined in measurable terms
 It may not be possible to identify trends that may predict lower levels of performance
 The actual results of a performance test cannot be evaluated by comparing them to a baseline set
of performance measures that define acceptable and/or unacceptable performance
 Performance test results are evaluated based on the subjective opinion of one or more people
 The results provided by a performance test tool are not understood
2.1 Typical Metrics Collected in Performance
Testing
Collecting Performance Metrics and Measurements
 As with any form of measurement, it is possible to obtain and express metrics in precise ways.
 Therefore, any of the metrics and measurements described in this section can and should be defined
to be meaningful in a particular context.
 This is a matter of performing initial tests and learning which metrics need to be further refined and
which need to be added.
For example, the metric of response time likely will be in any set of performance metrics.
However, to be meaningful and actionable such as:
 The response time metric will need to be further defined in terms of time of day
 Number of concurrent users
 The amount of data being processed and so forth.
2.1 Typical Metrics Collected in Performance
Testing
The metrics collected in a specific performance test will vary based on the
 Business context (business processes, customer and user behavior, and stakeholder expectations)
 Operational context (technology and how it is used)
 Test objectives
For example, the metrics chosen for the performance testing of an international e-commerce website
will differ from those chosen for the performance testing of an embedded system used to control
medical device functionality.
A common way to categorize performance measurements and metrics is to consider the technical
environment, business environment, or operational environment in which the assessment of
performance is needed.
2.1 Typical Metrics Collected in Performance
Testing
Technical Environment • Response time
• e.g., per transaction, per concurrent user, page load
The type of the technical environment, times
as shown in the following list: • Resource utilization
Web-based • e.g., CPU, memory, network bandwidth, network
Mobile latency, available disk space, I/O rate, idle and busy
threads
Internet-of-Things (IoT) Metrics • Throughput rate of key transaction
Desktop client devices • i.e., the number of transactions that can be
Server-side processing processed in a given period of time
Mainframe • Batch processing time
Databases • e.g., wait times, throughput times, data base
Networks response times, completion times
The nature of software running in the • Numbers of errors impacting performance
environment (e.g., embedded) • Completion time
• e.g., for creating, reading, updating, and deleting
data
• Background load on shared resources (especially in
virtualized environments)
• Software metrics
• e.g., code complexity
2.1 Typical Metrics Collected in Performance
Testing
Business Environment
From the business or functional perspective, performance metrics may include the following:
 Business process efficiency
 e.g., the speed of performing an overall business process including normal, alternate and exceptional use case flows
 Throughput of data, transactions, and other units of work performed
 e.g., orders processed per hour, data rows added per minute
 Service Level Agreement (SLA) compliance or violation rates
 e.g., SLA violations per unit of time
 Scope of usage
 e.g., percentage of global or national users conducting tasks at a given time
 Concurrency of usage
 e.g., the number of users concurrently performing a task
 Timing of usage
 e.g., the number of orders processed during peak load times
2.1 Typical Metrics Collected in Performance
Testing
Operational Environment
The operational aspect of performance testing focuses on tasks that are generally not considered to be
user-facing in nature. These include the following:
 Operational processes
e.g., the time required for environment start-up, backups, shutdown and resumption times
 System restoration
e.g., the time required to restore data from a backup
 Alerts and warnings
e.g., the time needed for the system to issue an alert or warning
2.1 Typical Metrics Collected in Performance
Testing
Selecting Performance Metrics
 It should be noted that collecting more metrics than required is not necessarily a good thing.
 Each metric chosen requires a means for consistent collection and reporting.
 It is important to define an obtainable set of metrics that support the performance test objectives.
2.1 Typical Metrics Collected in Performance
Testing
Goal-Question-Metric (GQM) approach
 The idea is to first establish the goals, then ask questions to know when the goals have been achieved
 Metrics are associated with each question to ensure the answer to the question is measurable.
2.1 Typical Metrics Collected in Performance
Testing
Goal-Question-Metric (GQM) approach example
2.1 Typical Metrics Collected in Performance
Testing
Some disadvantages of GQM approach
 It should be noted that the GQM approach doesn’t always fit the performance testing process.
For example, some metrics represent a system’s health and are not directly linked to goals.
Valuable conclusion
It is important to realize that after the definition and capture of initial measurements further
measurements and metrics may be needed to understand true performance levels and to determine where
corrective actions may be needed.
2.2 Aggregating Results from Performance
Testing
The purpose of aggregating performance metrics
 It is to be able to understand and express them in a way that accurately conveys the total picture of
system performance.
 When performance metrics are viewed at only the detailed level, drawing the right conclusion may
be difficult—especially for business stakeholders.
 For many stakeholders, the main concern is that the response time of a system, web site, or other
test object is within acceptable limits.
2.2 Aggregating Results from Performance
Testing
What will we gain after aggregating performance metrics?
Once deeper understanding of the performance metrics has been achieved, the metrics can be
aggregated so that:
 Business and project stakeholders can see the “big picture” status of system performance
 Performance trends can be identified
 Performance metrics can be reported in an understandable way
2.3 Key Sources of Performance Metrics
System performance should be no more than minimally impacted by the metrics collection effort
(known as the “probe effect”)
In addition, the volume, accuracy and speed with which performance metrics must be collected
makes tool usage a requirement.
While the combined use of tools is not uncommon, it can introduce redundancy in the usage of test
tools and other problems
2.3 Key Sources of Performance Metrics
Performance Test Tools
 All performance test tools provide measurements and metrics as the result of a test.
 Tools may vary in the number of metrics shown, the way in which the metrics are shown, and
the ability for the user to customize the metrics to a particular situation
 Some tools collect and display performance metrics in text format, while more robust tools
collect and display performance metrics graphically in a dashboard format. Many tools offer
the ability to export metrics to facilitate test evaluation and reporting.
2.3 Key Sources of Performance Metrics
Performance Monitoring Tools
 Performance monitoring tools are often employed to supplement the reporting capabilities of
performance test tools.
 In addition, monitoring tools may be used to monitor system performance on an ongoing basis
and to alert system administrators to lowered levels of performance and higher levels of
system errors and alerts.
 These tools may also be used to detect and notify in the event of suspicious behavior (such as
denial of service attacks and distributed denial of Service attacks).
2.3 Key Sources of Performance Metrics
Log Analysis Tools
 There are tools that scan server logs and compile metrics from them.
 Some of these tools can create charts to provide a graphical view of the data.
Errors, alerts and warnings are normally recorded in server logs. These include:
 High resource usage, such as high CPU utilization, high levels of disk storage consumed, and
insufficient bandwidth
 Memory errors and warnings, such as memory exhaustion
 Deadlocks and multi-threading problems, especially when performing database operations
 Database errors, such as SQL exceptions and SQL timeouts
2.4 Typical Results of a Performance Test
Differences between Functional Testing and Performance Testing
 In functional testing, particularly when verifying specified functional requirements or functional
elements of user stories  the expected results usually can be defined clearly and the test results
interpreted to determine if the test passed or failed.
 In performance testing often lacks this source of information. Not only are the stakeholders
notoriously bad at articulating performance requirements, many business analysts and product
owners are bad at eliciting such requirements  Testers often receive limited guidance to define
the expected test results.
2.4 Typical Results of a Performance Test
Evaluating performance test results
 It is important to look at the results closely. Initial raw results can be misleading with
performance failures being hidden beneath apparently good overall results.
 For example, resource utilization may be well under 75% for all key potential bottleneck resources,
but the throughput or response time of key transactions or use cases are an order-of-magnitude too
slow.