Chapter 3: Performance Testing

in the Software Lifecycle

Author: Quoc Bao Nguyen
3.1 Principal Performance Testing Activities
Nature of Performance Testing
 Performance testing is iterative in nature. Each test provides valuable insights into application and
system performance
 The information gathered from one test is used to correct or optimize application and system
parameters.
 The next test iteration will then show the results of modifications, and so on until test objectives are
reached

Performance testing activities align with the ISTQB test process

3.1 Principal Performance Testing Activities
Test Planning
Why do we need Test Planning?
 The need for the allocation of test environments, test data, tools and human resources.
 The scope of performance testing is established.
What do we need to do in Test Planning?
 Risk identification and risk analysis activities are completed and relevant information is updated
in any test planning documentation (e.g., test plan, level test plan).
 Test Planning is revisited and modified as needed, so are risks, risk levels and risk status
modified to reflect changes in risk conditions. (aka Risk Assessment)
3.1 Principal Performance Testing Activities
Test Monitoring and Control
Control measures are defined to provide action plans should issues be encountered which might impact
performance efficiency, such as
 Increasing the load generation capacity if the infrastructure does not generate the desired loads as
planned for particular performance tests
 Changed, new or replaced hardware
 Changes to network components
 Changes to software implementation
The performance test objectives are evaluated to check for exit criteria achievement.
3.1 Principal Performance Testing Activities
Test Analysis

Test Basis
SLAs Compliance
IT Architecture Test conditions
Business Process .. - Load levels
- Timing
Types of Performance Test
conditions
- Transactions to
be tested

Modeling and analysis of

system resource
requirements and/or
behavior using
spreadsheets or
capacity planning tools
3.1 Principal Performance Testing Activities
Test Design
 Performance test cases are designed.
 These are generally created in modular form so that they may be used as the building blocks of
larger, more complex performance tests.
3.1 Principal Performance Testing Activities
Test Implementation
What should we do to build a workflow?
 Performance test cases are ordered into performance test procedures
 These performance test procedures should reflect the steps normally taken by the user and other
functional activities that are to be covered during performance testing.
What should we do to prepare test environment and test data?
 A test implementation activity is establishing and/or resetting the test environment before each
test execution.
 Performance testing is typically data-driven, a process is needed to establish test data that is
representative of actual production data in volume and type so that production use can be
simulated.
3.1 Principal Performance Testing Activities
Test Execution
 It occurs when the performance test is conducted, often by using performance test tools.
 Test results are evaluated to determine if the system’s performance meets the requirements
and other stated objectives. Any defects are reported
3.1 Principal Performance Testing Activities
Test Completion
 Performance test results are provided to the stakeholders (e.g., architects, managers, product
owners) in a test summary report.
 The results are expressed through metrics which are often aggregated to simplify the meaning of
the test results.
 Visual means of reporting such as dashboards are often used to express performance test results in
ways that are easier to understand than text-based metrics.
3.1 Principal Performance Testing Activities
Remember
 Performance testing is often considered to be an ongoing activity in that it is performed at
multiple times and at all test levels (component, integration, system, system integration and
acceptance testing).
 At the close of a defined period of performance testing, a point of test closure may be reached
where designed tests, test tool assets (test cases and test procedures), test data and other test
ware are archived or passed on to other testers for later use during system maintenance
activities.
3.2 Categories of Performance Risks for
Different Architectures
As mentioned previously, application or system performance varies considerably based on the
architecture, application and host environment.
While it is not possible to provide a complete list of performance risks for all systems, the list below
includes some typical types of risks associated with particular architectures:
3.2 Categories of Performance Risks for
Different Architectures
Single Computer System Excessive resource consumption:
• Memory leaks
• Background activities such as
• Security software
• Slow storage subsystems
• Operating system mismanagement.
• Inefficient implementation of algorithms
3.2 Categories of Performance Risks for
Different Architectures
Multi-tier System

Performance can degrade due to:

• Poor or non-scalable database design
• Network bottlenecks
• Inadequate bandwidth or capacity on any single
server.
3.2 Categories of Performance Risks for
Different Architectures
Distributed System

Experience performance problems due to

• Critical workflows or data flows to, from
• Unreliable or unpredictable remote servers
• Servers suffer periodic connection
problems or intermittent periods of intense
load.
3.2 Categories of Performance Risks for
Different Architectures
Virtualized System

Performance risks that arise specifically

from virtualization include
• Excessive load on the hardware across
all the virtual machines
• Improper configuration of the host
virtual machine resulting in inadequate
resources.
3.2 Categories of Performance Risks for
Different Architectures
Dynamic/Cloud-based Systems
• These systems are typically distributed and virtualized multi tier systems
• Ability to scale on demand, increasing capacity as the level of load increases
• There are risks associated with failures to properly configure these features during initial
setup or subsequent updates.
3.2 Categories of Performance Risks for
Different Architectures
Client-Server Systems

Performance risks exist due to:

• Connection speed
• Reliability issues
• Network congestion at the client connection
point (e.g., public Wi-Fi)
• Potential problems due to firewalls packet
inspection and server load balancing
3.2 Categories of Performance Risks for
Different Architectures
Mobile Applications
Performance issues can arise due to:
• The limited and variable resources and connectivity available on
the mobile device by:
 Location
 Battery life,
 Charge state
 Available memory on the device
 Temperature
• Slow data flows from:
 Device sensors or radios such as accelerometers or Bluetooth
• Performance efficiency bottleneck due to:
 Mobile applications often have heavy interactions with other
local mobile apps and remote web services
3.2 Categories of Performance Risks for
Different Architectures
Embedded Real-time Systems

These systems often have many of the risks of mobile

devices, including (increasingly) connectivity related
issues since these devices are connected to the
Internet.

However the diminished performance of a mobile

video game is usually not a safety hazard for the user,
while such slowdowns in a vehicle braking system
could prove catastrophic.
3.2 Categories of Performance Risks for
Different Architectures
Mainframe Applications
These are applications supporting often mission-critical business functions in a
data center, sometimes via batch processing. Most are quite predictable and fast
when used as originally designed

Many of these are now accessible via APIs, web services, or through their
database  Unexpected loads
3.2 Categories of Performance Risks for
Different Architectures
Note that:
 Any particular application or system may incorporate two or more of the architectures listed
above  all relevant risks will apply to that application or system
 Two areas where extreme levels of interaction and connection is the rule
 Architecture is clearly an important technical decision with a profound impact on performance
risks, other technical decisions also influence and create risks
 Memory leaks are more common with languages that allow direct heap memory management, such as
C/C++
 Performance issues are different for relational versus non-relational databases
As a tester, the ability to know about or even influence such decisions will vary, depending on the
roles and responsibilities of testers within the organization and software development lifecycle.
3.3 Performance Risks Across the Software
Development Lifecycle
Risk Management
In this section, the focus is on performance-related risks to product quality, including ways that the
process, the participants, and the considerations change. For performance-related risks to the quality of
the product, the process is:
1) Identify risks to product quality, focusing on characteristics such as time behavior, resource utilization,
and capacity. (Risk Identification)
2) Assess the identified risks, ensuring that the relevant architecture categories are addressed. Evaluate
the overall level of risk for each identified risk in terms of likelihood and impact using clearly defined
criteria. (Risk Assessment)
3) Take appropriate risk mitigation actions for each risk item based on the nature of the risk item and the
level of risk. (Risk Mitigation)
4) Manage risks on an ongoing basis to ensure that the risks are adequately mitigated prior to release.
The specific risk analysis process chosen should have the appropriate level of formality and rigor.
3.3 Performance Risks Across the Software
Development Lifecycle
What interests of stakeholders related to the Performance Risks?
Interest Business Technical

Objective Awareness of how performance problems in Deep understanding of the performance

production will actually affect customers, implications of relevant requirements,
users, the business, and other downstream architecture, design, and implementation
stakeholders decisions
Affection of Intended usage, business-, societal-, or safety- Architecture, design, and implementation
Performance criticality, potential financial and/or decisions affect performance risks from a
Risks reputational damage, civil or criminal legal technical perspective, creating risks and
liability and similar factors affect risk from a influencing the likelihood of defects.
business perspective, creating risks and
influencing the impact of failures.
3.3 Performance Risks Across the Software
Development Lifecycle
When should we conduct Risk Identification and Analysis for System Performance?
 Risk identification and analysis process be started early and is repeated regularly
 The tester should avoid relying entirely on performance testing conducted towards the end of
the system test level and system integration test level

Why should we conduct Risk Identification and Analysis for System Performance?

 Many larger and more complex systems of systems projects met with unfortunate surprises
due to the late discovery of performance defects which resulted from requirements, design,
architecture, and implementation decisions made early in the project.

So that …
 An iterative approach to performance risk identification, assessment, mitigation, and
management throughout the software development lifecycle.
3.3 Performance Risks Across the Software
Development Lifecycle
Example of early risk identification and analysis:
If large volumes of data will be handled via a relational database, the slow performance of
many-to-many joins due to poor database design may only reveal itself during dynamic testing
with large-scale test datasets, such as those used during system test. However, a careful
technical review that includes experienced database engineers can predict the problems prior
to database implementation. After such a review, in an iterative approach, risks are identified
and assessed again.
3.3 Performance Risks Across the Software
Development Lifecycle
Some notes for Risk Mitigation and Management
Risk mitigation and management must span and influence the entire software development process,
not just dynamic testing.
For example, when critical performance-related decisions such as the expected number of transactions
or simultaneous users cannot be specified early in the project  Design and architecture decisions
allow for highly variable scalability (e.g., on-demand cloud based computing resources). This enables
early risk mitigation decisions to be made  early risk mitigation decisions to be made
3.3 Performance Risks Across the Software
Development Lifecycle
The cost of Early Performance Risk Identification and Analysis
Good performance engineering can help project teams avoid the late discovery of critical performance
defects during higher test levels, such as system integration testing or user acceptance testing.
Performance defects found at a late stage in the project can be extremely costly and may even lead to
the cancellation of entire projects.
3.3 Performance Risks Across the Software
Development Lifecycle
Remember
As with any type of quality risk, performance-related risks can never be avoided completely, i.e.,
some risk of performance-related production failure will always exist.
Therefore, the risk management process must include providing a realistic and specific evaluation
of the residual level of risk to the business and technical stakeholders involved in the process.
 For example, simply saying, “Yes, it’s still possible for customers to experience long delays during
check out,” is not helpful, as it gives no idea of what amount of risk mitigation has occurred or of
the level of risk that remains. Instead, providing clear insight into the percentage of customers
likely to experience delays equal to or exceeding certain thresholds will help people understand
the status
3.4 Performance Testing Activities
Sequential Development Models
 The ideal practice of performance testing in sequential development models is to include
performance criteria as a part of the acceptance criteria which are defined at the outset of a
project.
 As the project progresses, each successive performance test activity should be based on items
defined in the prior activities as shown below.

Concept Verify that system performance goals are defined as acceptance criteria

Verify that performance requirements are defined and

Requirement
represent stakeholder needs correctly
Verify that the system design reflects
Analysis & Design the performance requirements

Verify that the code is efficient and

Coding
reflects the requirements and design

Testing
3.4 Performance Testing Activities
Testing in Sequential Development Models

Validate that system performance meets the originally

stated user needs and acceptance criteria
Conduct with the entire system which is
representative of the production environment
Conduct with hardware, software, procedures and data
that are representative of Production environment

Conduct

Conduct
3.4 Performance Testing Activities
Iterative and Incremental Development Models

• Performance risks are identified and assessed, and mitigation measures planned
Release Planning • Planning of any final performance testing before the release of the application.

• Performance testing may be performed within the iteration and as

Iteration/Sprint Planning each iteration is completed.
• Performance risks are assessed in more detail for each user story

User Stories Creation • The specific performance criteria described in the associated acceptance criteria

Design Performance Test • Design test based on performance criteria/user stories/requirements

Coding • Performance testing may be performed at a component level

• Performance testing may be performed as a separate activity, depending on the scope and objectives of performance
Testing testing during the iteration
• If the goal of performance testing is to test the performance of the iteration as a completed set of user stories, a wider
scope of performance testing will be needed than that seen in performance testing a single user story  This may be
scheduled in a dedicated iteration for performance testing.

• Performance will need to be monitored to determine if the application achieves the desired levels of performance
Delivery in actual usage.
3.4 Performance Testing Activities
Continuous Integration in Iterative and Incremental Development Models

Why do we need CI?

 Commonly performed in iterative and incremental software development lifecycles, which facilitates a
highly automated execution of tests
 Objective of testing in CI is to perform regression testing and ensure each build is stable
 Performance testing can be part of the automated tests performed in CI
3.4 Performance Testing Activities
Continuous Integration in Iterative and Incremental Development Models

Some concerns related to Performance Tests which performed in CI:

 The setup of the performance test environment:
 This often requires a test environment that is available on demand, such as a cloud-based performance test
environment.
 Determining which performance tests to automate in CI:
 Due to the short timeframe available for CI tests  CI performance tests may be a subset of more extensive
performance tests that are conducted by a specialist team at other times during an iteration.
 Creating the performance tests for CI:
 The main objective of performance tests as part of CI is to ensure a change does not negatively impact
performance.
 Depending on the changes made for any given build, new performance tests may be required.
 Executing performance tests on portions of an application or system:
 This often requires the tools and test environments to be capable of rapid performance testing including the
ability to select subsets of applicable tests.
3.4 Performance Testing Activities
Commercial Off-the-Shelf (COTS) and other Supplier/Acquirer Models
 Context: Many organizations do not develop applications and systems themselves, but instead are
in the position of acquiring software from vendor sources or from open-source projects.
 Problem: In such supplier/acquirer models, performance is an important consideration that
requires testing from both the supplier (vendor/developer) and acquirer (customer) perspectives.
In the case of customized vendor-developed software, performance requirements and associated
acceptance criteria which should be specified as part of the contract between the vendor and
customer.
In the case of COTS applications, the customer has sole responsibility to test the performance of
the product in a realistic test environment prior to deployment.