Chapter 4: Performance Testing

Tasks
Author: Quoc Bao Nguyen
4.1 Planning
Deriving Performance Test Objectives
Stakeholders may include users and people with a business or technical background  They may
have different objectives relating to performance testing.
Stakeholders set the objectives, the terminology to be used and the criteria for determining
whether the objective has been achieved
4.1 Planning
Deriving Performance Test Objectives

Business Technical

Objective • Focus primarily on end-user satisfaction and • Focus on operational aspects and providing
business goals answers to questions regarding a system’s
• Less concerned about feature types or how a ability to scale
product gets delivered • What conditions degraded performance may
become apparent
Key Objective • Identify potential risks
• Find opportunities for improvement
• Identify necessary changes
4.1 Planning
Deriving Performance Test Objectives
When gathering information from the various stakeholders, the following questions should be answered:
 What transactions will be executed in the performance test and what average response time is
expected?
 What system metrics are to be captured (e.g., memory usage, network throughput) and what values
are expected?
 What performance improvements are expected from these tests compared to previous test cycles?
4.1 Planning
The Performance Test Plan
 The Performance Test Plan (PTP) is a document created prior to any performance testing
occurring.
 The PTP should be referred to by the Test which also includes relevant scheduling information.
It continues to be updated once performance testing begins.
4.1 Planning
The Performance Test Plan
Objective:
Describes the goals, strategies and methods for the performance test  enables a quantifiable answer
to the central question of the adequacy and the readiness of the system to perform under load.
4.1 Planning
The Performance Test Plan
Test Objectives
Overall test objectives for performance efficiency to be achieved by the System Under Test (SUT) are
listed for each type of stakeholder
4.1 Planning
The Performance Test Plan
System Overview
A brief description of the SUT will provide the context for the measurement of the performance test
parameters.
The overview should include a high-level description of the functionality being tested under load.
4.1 Planning
The Performance Test Plan
Types of Performance Testing to be Conducted
 See in Chapter 1.2
4.1 Planning
The Performance Test Plan
Acceptance Criteria
Determine the responsiveness, throughput, reliability and/or scalability of the system under a given
workload
Concerns:
 Response time is a user concern
 Throughput is a business concern
 Resource utilization is a system concern
Acceptance criteria should be set for all relevant measures and related back to the following as
applicable:
 Overall performance test objectives
 Service Level Agreements (SLAs)
 Baseline values
 A baseline is a set of metrics used to compare current and previously achieved performance measurements
4.1 Planning
The Performance Test Plan
Test Data
This data can include the following:
 User account data (e.g., user accounts available for simultaneous log in)
 User input data (e.g., the data a user would enter into the application in order to perform a business
process)
 Database (e.g., the pre-populated database that is populated with data for use in testing)
The test data creation process should address the following aspects:
 Data extraction from production data
 Importing data into the SUT
 Creation of new data
 Creation of backups that can be used to restore the data when new cycles of testing are performed
 Data masking or anonymizing
4.1 Planning
The Performance Test Plan
System Configuration
The system configuration section of the PTP includes the following technical information:
 A description of the specific system architecture, including servers
 e.g., web, database, load balancer
 Definition of multiple tiers
 Specific details of computing hardware including versions
 e.g., CPU cores, RAM, Solid State Disks (SSD), Hard Drive Disks (HDD)
 Specific details of software including version
 e.g., applications, operating systems, databases, services used to support the enterprise
 External systems that operates with the SUT and their configuration and version
 e.g., Ecommerce system with integration to NetSuite
 SUT build / version identifier
4.1 Planning
The Performance Test Plan
Test Environment
Context: The test environment is often a separate environment that mimics production, but at a
smaller scale.
With some systems, the production environment becomes the only viable option for testing, but in this
case the specific risks of this type of testing must be discussed.
 Testing tools sometimes reside outside the test environment  require special access rights in
order to interact with the system components.
 This is a consideration for the test environment and configuration.
 Performance tests may also be conducted with a component part of the system that is capable of operating
without other components  This is often cheaper than testing with the whole system and can be
conducted as soon as the component is developed.
4.1 Planning
The Performance Test Plan
Test Tools
This section includes a description of which test tools (and versions) will be used in scripting, executing
and monitoring the performance tests.
This list normally includes:
 Tool(s) used to simulate user transactions
 Tools to provide load from multiple points within the system architecture (points of presence)
 Tools to monitor system performance, including those described above under system configuration
4.1 Planning
The Performance Test Plan
Profiles
Operational profiles provide a repeatable step-by-step flow through the application for a particular
usage of the system.
Aggregating these operational profiles results in a load profile (commonly referred to as a scenario).
4.1 Planning
The Performance Test Plan
Relevant Metrics
4.1 Planning
The Performance Test Plan
Risks
 Risks can include areas not measured as part of the performance testing as well as limitations to
the performance testing
e.g., external interfaces that cannot be simulated, insufficient load, inability to monitor servers.
 Limitations of the test environment may also produce risks
e.g., insufficient data, scaled down environment
4.1 Planning
Communicating about Performance Testing
Stakeholders with a Business Focus
 Less interested in the distinctions between functional and non-functional quality characteristics.
 Concerning tooling, scripting, and load generation are secondary interests.
 The connection between product risks and performance test objectives must be clearly stated
 Awareness of the balance between the cost of planned performance tests and how representative
the performance testing results will be, compared to production conditions.
 The repeatability of planned performance tests must be communicated.
 Project risks must be communicated.
 Include constraints and dependencies concerning the setup of the tests, infrastructure requirements (e.g.,
hardware, tools, data, bandwidth, test environment, resources), and dependencies on key staff.
 The high-level activities must be communicated together with a broad plan containing costs, time
schedule and milestones.
4.1 Planning
Communicating about Performance Testing
Stakeholders with a Technology Focus
 The planned approach to generating required load profiles must be explained & the expected
involvement of technical stakeholders made clear
 Detailed steps in the setup and execution of the performance tests must be explained to show
the relation of the testing to the architectural risks
 Steps required to make performance tests repeatable must be communicated
 Where test environments are to be shared, the scheduling of performance tests must be
communicated to ensure the test results will not be adversely impacted.
 Mitigations of the potential impact on actual users if performance testing needs to be executed
in the production environment must be communicated and accepted
 Technical stakeholders must be clear about their tasks and when they are scheduled
4.2 Analysis, Design and Implementation
Typical Communication Protocols
Communication protocols define
 A set of communications rules between computers and systems.
 Designing tests properly to target specific parts of the system requires understanding protocols.
4.2 Analysis, Design and Implementation
Typical Communication Protocols
Based on Open Systems Interconnection (OSI) model layers (see ISO/IEC 7498-1).
For performance testing, protocols from Layer 5 (Session Layer) to Layer 7 (Application Layer) are
most commonly used for performance testing. Common protocols include:
 Database - ODBC, JDBC, other vendor-specific protocols
 Web - HTTP, HTTPS, HTML
 Web Service - SOAP, REST
4.2 Analysis, Design and Implementation
Typical Communication Protocols
Additional protocols used in performance testing include:
 Network - DNS, FTP, IMAP, LDAP, POP3, SMTP, Windows Sockets, CORBA
 Mobile - TruClient, SMP, MMS
 Remote Access - Citrix ICA, RTE
 SOA - MQSeries, JSON, WSCL
4.2 Analysis, Design and Implementation
Typical Communication Protocols
It is important to understand the overall system architecture because performance tests can be executed
on an individual system component (e.g., web server, database server) or on a whole system via end-to-
end testing.
Example:
Context: Traditional 2-tier applications built with a client-server model specify the “client” as the GUI
and primary user interface, and the “server” as the backend database.
These applications require the use of a protocol such as ODBC to access the database.

With the evolution of web-based applications and multi-tiered architectures, many servers
are involved in processing information that is ultimately rendered to the user’s browser.
4.2 Analysis, Design and Implementation
Typical Communication Protocols
Depending on the part of the system, an understanding is required of the appropriate protocol to be used
Example:
if the need is to perform end-to-end testing emulating user activity from the browser, a web protocol
such as HTTP/HTTPS will be employed.
In this way, interaction with the GUI can be bypassed and the tests can focus on the communication and
activities of the backend servers.
4.2 Analysis, Design and Implementation
Transactions
Definition
 Transactions describe the set of activities performed by a system from the point of initiation to when
one or more processes (requests, operations, or operational processes) have been completed.
Why do we need it?
 The response time of transactions can be measured for the purpose of evaluating system performance.
 During a performance test these measurements are used to identify any components that require
correction or optimization.
4.2 Analysis, Design and Implementation
Transactions
How can I determine Time of Transaction?
 Simulated transactions can include think time to better reflect the timing of a real user taking
an action (e.g., pressing the “SEND” button).

The elapsed time for transaction = Think time + Transaction Response time
4.2 Analysis, Design and Implementation
Transactions
What is the meaning of transaction response time?
 The transaction response times collected during the performance test show how this
measurement changes under different loads imposed on the system.
 Analysis may show no degradation under load while other measurements may show severe
degradation. By ramping up load and measuring the underlying transaction times, it is possible
to correlate the cause of degradation with the response times of one or more transactions.
4.2 Analysis, Design and Implementation
Transactions
Nesting Transactions
 Transactions can also be nested so that individual and aggregate activities can be measured.
 This can be helpful when understanding the performance efficiency of an system.
For example:
The tester may want to measure the discrete steps in the order process of an online ordering system
(e.g., search for item, add item to cart, pay for item, confirm order) as well as the order process as
a whole. By nesting transactions, both sets of information can be gathered in one test.
4.2 Analysis, Design and Implementation
Identifying Operational Profiles
Introduction
 Operational profiles specify distinct patterns of interaction with an application such as from users
or other system components.
 Multiple operational profiles may be specified for a given application  They may be combined to
create a desired load profile for achieving particular performance test objectives
The following principal steps for identifying operational profiles are:
1. Identify the data to be gathered
2. Gather the data using one or more sources
3. Evaluate the data to construct the operational profiles
4.2 Analysis, Design and Implementation
Identifying Operational Profiles
Identify Data
Identify how they interact with the system
 Different types of user personas and their roles
 Different generic tasks performed by those users/roles. Note that these tasks are generally best
modeled at a high level of abstraction
 Estimated numbers of users for each role/task per unit of time over a given time period. This
information will also be useful for subsequently building load profiles
4.2 Analysis, Design and Implementation
Identifying Operational Profiles
Gather Data
There are some different sources to gather data
 Conducting interviews or workshops with stakeholders
 PO, Sale Manager, Potential end-user will join the interviews
 Reveal the principal operational profiles of users and provide answers to the fundamental question “Who is this
application intended for”.
 Functional specifications and requirements
 Help identify user types and their operational profiles.
 The standard format directly enables types of users to be identified (i.e., As a , I want so that ).
 UML Use Case diagrams and descriptions identify the “actor” for the use case
 Evaluating usage data and metrics gained from similar applications
 Support identification of user types & provide some initial indications of the expected numbers of users.
 Access to automatically monitored data is recommended.
 Monitoring the behavior of users when performing predefined tasks with the application
 Give insights into the types of operational profiles to be modeled for performance testing
 Coordinating this task with any planned usability tests is recommended
4.2 Analysis, Design and Implementation
Identifying Operational Profiles
Construct Operational Profiles
 A top-down approach is taken
 Simple broad operational profiles are initially created and only broken down further if this is needed to
achieve performance test objectives
 Particular user profiles may be singled out as relevant for performance testing if they involve tasks
which are executed frequently
 Critical (high risk) or frequent transactions between different system components, or potentially demand large
volumes of data to be transferred
 Operational profiles are reviewed and refined
 With the principal stakeholders before being used for the creation of load profiles
Example of Constructing Operational Profiles
Context: We have one e-commerce website. We have to make sure that the users will have better
experience with the application. We need to have performance test to make sure that the frequent
behaviors of the users should be covered.
Example of Constructing Operational Profiles
Broad Operational Profile
 Browsing Products:
 Users viewing different clothing items, categories, and promotions
 Searching for Specific Items
 Users enter search terms to find specific products
 Adding to Cart
 Users select items and add them to their online shopping cart
 Checkout
 User finalizing their purchases, entering payment and shipping details
 Account Management
 Users login, update their personal information, check order status, and view order history
Example of Constructing Operational Profiles
Breakdown Broad Operational Profile Through account (username, password)

User login Through Gmail, Facebook

Account Management Update personal information Change username, password, email

Tracking orders
View order history
Displaying previous orders
Searching for Specific Items
Gift Code

Credit Card
Browsing Products
Selecting Payment Method PayPal, ShopeePay, ZaloPay

Adding to Cart Choosing shipping method

Entering Shipping Details Applying promo codes

Checkout Input address

Receiving confirmation receipt
Finalizing Purchases
Confirming Payment
Example of Constructing Operational Profiles
Particular user profiles may be singled out as relevant for performance testing

Through account (username, password)

Account Management User login

Through Gmail, Facebook

Gift Code
Searching for Specific Items
Credit Card
Selecting Payment Method PayPal, ShopeePay, ZaloPay

Choosing shipping method

Checkout Entering Shipping Details
Applying promo codes
Input address

Finalizing Purchases Receiving confirmation receipt

Confirming Payment
Example of Constructing Operational Profiles
Particular user profiles may be singled out as relevant for performance testing
Why do we choose the following operational profiles?
Operational Profile Reason
Account Management – User Login This is a frequent feature user interacts.
In the Black Friday, A lot of users will login to the application concurrently 
This is the critical feature/functionality
Searching for Specific Items At the high traffic, The needs of searching items is critical, the user would like
to find the item quickly as much as possible  This will be a frequent
transaction
Checkout This could be high impact because of payment information, data transfer  It is
a very high risk transaction
4.2 Analysis, Design and Implementation
Off-line Batch Processing Systems
The focus here lies principally on the throughput of the batch processing system and its ability to
complete within a given time period.
Operational profiles focus on the types of processing which are demanded of the batch processes.

For example: the operational profiles for a stock trading system (which typically includes online and
batch-based transaction processing)
It may include those relating to payment transactions, verifying credentials, and checking compliance of
legal conditions for particular types of stock transactions.
 Each of these operational profiles would result in different paths being taken through the overall
batch process for a stock.
 The steps outlined above for identifying the operational profiles of online user-based systems can also
be applied in the batch processing context.
4.2 Analysis, Design and Implementation
Systems of Systems
Components within a multi-system (which may also be embedded) environment respond to different
types of input from other systems or components.
Depending on the nature of the system under test, this may require modeling of several different
operational profiles to effectively represent the types of input provided by those supplier systems.
This may involve detailed analysis (e.g., of buffers and queues) together with the system architects and
based on system and interface specifications.
4.2 Analysis, Design and Implementation
Creating Load Profiles
Definition
 A load profile specifies the activity which a component or system being tested may experience in
production.
Properties of the structure
 The Operational Profiles
 A designated number of instances that will perform the actions of predefined operational profiles over a
specified time period
The principal information to create a realistic and repeatable load profile
 The performance testing objective
 Operational profiles which accurately represent individual usage patterns
 Known throughput and concurrency issues
 The quantity and time distribution with which the operational profiles are to be executed such that the SUT
experiences the desired load. Typical examples are:
o Ramp-ups: Steadily increasing load (e.g., add one virtual user per minute)
o Ramp-downs: Steadily decreasing load
o Steps: Instantaneous changes in load (e.g., add 100 virtual users every five minutes)
o Predefined distributions (e.g., volume mimics daily or seasonal business cycles)
4.2 Analysis, Design and Implementation
Creating Load Profiles
The following example shows the construction of a load profile with the objective of generating stress
conditions (at or above the expected maximum for a system to handle) for the system under test.
4.2 Analysis, Design and Implementation
Creating Load Profiles
At the top of the diagram a load profile is shown which consists of a step input of 100 virtual users.
These users perform the activities defined by Operation Profile 1 over the entire duration of the test.
This is typical of many performance load profiles that represent a background load.
4.2 Analysis, Design and Implementation
Creating Load Profiles
The middle diagram shows a load profile that consists of a ramp-up to 220 virtual users that is
maintained for two hours before ramping down.
Each virtual user performs activities defined in Operational Profile 2.
4.2 Analysis, Design and Implementation
Creating Load Profiles
The lower diagram shows the load profile that results from the combination of the two described above.
The system under test is subjected to a three-hour period of stress.
4.2 Analysis, Design and Implementation
Analyzing Throughput and Concurrency
System Throughput
Definition:
 System throughput is a measure of the number of transactions of a given type that the system
processes in a unit of time
 System throughput should be distinguished from network throughput, which is the amount of data
moved over the network
For Example:
 The number of orders per hour or the number of HTTP requests per second.
4.2 Analysis, Design and Implementation
Analyzing Throughput and Concurrency
System Throughput
Meaning:
 System throughput defines load on the system
 There is a misunderstanding that it is the number of concurrent users is used to define the load for
interactive systems instead of throughput  This is partially true because that number is often easier to
find, and partially because it is the way load testing tools define load
 Unless defining operational profiles, the number of users is not a good measure of load
For Example:
 If there are 500 users running short queries each minute, we have a throughput of 30,000 queries per
hour but the same 500 users, they are run only one query. If the same 500 users are running the same
queries, but one per hour, the throughput is 500 queries per hour.
4.2 Analysis, Design and Implementation
Analyzing Throughput and Concurrency
System Throughput
How to determine
 Workload modeling is usually done by considering the number of virtual users (execution threads) and
the think time (delays between user actions).
 However, system throughput is also defined by processing time, and that time may increase as load
increases.
System throughput = [number of virtual users] / ([processing time] + [think time])
So when the processing time increases, throughput may significantly decrease even if everything else stays
the same
In conclusion
System throughput is an important aspect when testing batch processing systems.
In this case, the throughput is typically measured according to the number of transactions that can be
accomplished within a given time frame (e.g., a nightly batch processing window).
4.2 Analysis, Design and Implementation
Analyzing Throughput and Concurrency
Concurrency
Definition
 Concurrency is an important measure. It represents the number of parallel sessions, each of which may
use its own resources.
How to Measure
 Concurrency is a measure of the number of simultaneous / parallel threads of execution.
 For interactive systems, it may be a number of simultaneous / parallel users.
 Concurrency is usually modeled in load testing tools by setting the number of virtual users
4.2 Analysis, Design and Implementation
Analyzing Throughput and Concurrency
Concurrency
Meaning
 In case the throughput is the same, the amount of resources used may differ depending on concurrency.
For example:
 In case, the closed system, which the number of users in the system is set.
If all users are waiting for the system’s response in a closed system, no new users can arrive.
 In case, the open system, which the number of users in the system is more flexible
if all the current users are waiting for the system’s response, new users are arriving all the time
4.2 Analysis, Design and Implementation
Basic Structure of a Performance Test Script
The structure of test script
 A performance test script should simulate a user or component activity that contributes to the load on
the system under test (which may be the whole system or one of its components).
 It initiates requests to the server in a proper order and at a given pace
There are 3 ways to create performance test script:
 Record & Playback: recording communication between the client and the system or component on the
protocol level and then play it back after the script has been parameterized and documented
 Recording at GUI level: capturing GUI actions of a single client with a test execution tool and running that
script with the load generation tool to represent multiple clients
 Programming: Using protocol requests (e.g., HTTP requests), GUI actions, or API calls. In the case of
programming scripts, the exact sequence of requests sent to and received from the real system must be
determined
Usually a script is one or several sections of code (written in a generic programming language with some
extensions or in a specialized language) or an object, which may be presented to a user by the tool in a GUI.
In both cases the script will include server requests creating load (e.g., HTTP requests) and some
programming logic around them specifying how exactly these requests would be invoked (e.g., in what
order, at what moment, with what parameters, what should be checked)
 The more sophisticated the logic, the more need for using powerful programming languages.
4.2 Analysis, Design and Implementation
Basic Structure of a Performance Test Script
Overall Structure
Usually a script is one or several sections of code (written in a generic programming language with some
extensions or in a specialized language) or an object, which may be presented to a user by the tool in a
GUI.
In both cases the script will include server requests creating load (e.g., HTTP requests) and some
programming logic around them specifying how exactly these requests would be invoked (e.g., in what
order, at what moment, with what parameters, what should be checked)
 The more sophisticated the logic, the more need for using powerful programming languages.
So that the script should have the following structure to make sure it is maintainable
 Initialization section: where everything gets prepared for the main part
 Main sections: It may be executed multiple times, where contains all logic of programming
 Clean-up section: where necessary steps are taken to finish the test properly
4.2 Analysis, Design and Implementation
Basic Structure of a Performance Test Script
Data Collection
To collect response times:
 Timers should be added to the script to measure how long a request or a combination of requests
takes.
 The timed requests should match a meaningful unit of logical work
 For example, a business transaction for adding an item to an order or submitting an order.

It is important to understand what exactly is measured:

 In the case of protocol-level scripts it is server and network response time only
 GUI scripts measure end-to end time (although what exactly is measured depends on the technology
used).
4.2 Analysis, Design and Implementation
Basic Structure of a Performance Test Script
Result Verification and Error Handling
 An important part of the script is result verification and error handling.
 Even in the best load testing tools, default error handling tends to be minimal, it is not enough to see
deep dive the system performance, how the system works
 So that we have to handle to increase error handling, result verification
There are 3 things, we have to do:
 Error handling: any cleanup is required in case of an error. (clearing cache and data if any error)
 Result verification: check to verify what the requests actually return
 Indirect verification: check the database to verify that the proper information was added
Additionally, we have another ways:
 Add other logic specifying rules concerning when and how server requests will be made into the script
 Example: Setting synchronization points, which is done by specifying that the script should wait for an event
at that point before proceeding  The synchronization points may be used to ensure that a specific action is
invoked concurrently or to coordinate work between several scripts.
4.2 Analysis, Design and Implementation
Implementing Performance Test Scripts
What should we do first?
 Based on the PTP and the load profiles
 Using IDE
 To simulate a user or component behavior. Usually the script is created to simulate a specific
operational profile
Approaches to implement performance test scripts
There are 2 ways:
 Recording: ensures that it exactly simulates the real system
 Programming: relies on knowledge of the proper request sequence
4.2 Analysis, Design and Implementation
Implementing Performance Test Scripts
Correlation
 Problem: When recording on the protocol level is used, an essential step after recording in most cases
is replacing all recorded internal identifiers that define context
 Solution: These identifiers must be made into variables that can be changed between runs with
appropriate values that are extracted from the request responses
For example: After we create a new user then the user will have user id, so that we will use user id to
pass it as a variable to the script for update/rename/delete the user
 This is a part of script parameterization, sometimes referred to as ‘correlation’.

Advanced load testing tools may do some correlation automatically, so it may be transparent in some
cases—but in more complex cases, manual correlation or adding new correlation rules may be required.
 But sometimes, adding new correlation rules or adding correlation manually will lead to
incorrect correlation  recorded scripts fail to playback
4.2 Analysis, Design and Implementation
Implementing Performance Test Scripts
Incorrect Correlation/Lack of Correlation
Context: Running multiple virtual users with the same user name and accessing the same set of data
 It will be misleading results
The reason of misleading results:
 The data could be completely cached  results would be much better than in production
 Using the same users and/or data can also cause concurrency issues  results would be much worse
than in production as the software would wait for the lock to free before the next user could lock the
data for update
 To reduce the issue and the impact of caching, we have to use totally different data  So that we need
to parameterize the script
4.2 Analysis, Design and Implementation
Implementing Performance Test Scripts
Parameterization
To avoid/reduce concurrency issues or data caching while executing performance testing
 So scripts and test harnesses should be parameterized
What kind of data we should use to parameterize?
Particularly, each virtual user uses a proper set of data
 The term “proper” here means different enough to avoid problems with caching and concurrency,
which is specific for the system, data, and test requirements
This further parameterization depends on:
 the data in the system
 the way the system works with this data
 So it usually is done manually, although many tools provide assistance here.
There are cases where some data must be parameterized for the test to work more than once
 For example: when an order is created and the order name must be unique. Unless the order’s name is
parameterized, the test will fail as soon as it tries to create an order with an existing (recorded)
name.
4.2 Analysis, Design and Implementation
Implementing Performance Test Scripts
Adding think time
To match operational profiles, think times should be inserted and/or adjusted (if recorded) to generate
a proper number of requests / throughput
In conclusion
 When scripts for separate operational profiles are created, they are combined into a scenario
implementing the whole load profile.
 The load profile controls how many virtual users are started using each script, when, and with what
parameters.
 The exact implementation details depend on the specific load testing tool or harness.
4.2 Analysis, Design and Implementation
Preparing for Performance Test Execution
Activities
The main activities for preparing to execute the performance tests include:
 Setting up the system under test
 Deploying the environment
 Setting up the load generation and monitoring tools and making sure that all the necessary
information will be collected
The importance of test environment
It is important to ensure the test environment is as close to the production environment as possible.
If not the following project risk will occur:
 Lack of projection with test results on the production environment
 The more difficult it becomes to make accurate projections for production performance
4.2 Analysis, Design and Implementation
Preparing for Performance Test Execution
How to prepare good enough environment for performance test?
Depending on the specific goals and the systems to test, one test environment may be preferred over
another. For example,
 To test the effect of a performance improvement (performance optimization), using an isolated lab
environment may be a better option to see even small variations introduced by the change.
 To load test the whole production environment end-to-end to make sure the system will handle the
load without any major issues, testing from the cloud or a service may be more appropriate. (Note
that this only works for SUTs that can be reached from a cloud).
 To minimize costs when performance testing is limited in time, creating a test environment in the
cloud may be a more economical solution.
4.2 Analysis, Design and Implementation
Preparing for Performance Test Execution
Properties of test environment
The most important parts of the test environment are:
 Data
 Hardware and software configuration
 Network configuration.
4.2 Analysis, Design and Implementation
Preparing for Performance Test Execution
Properties of test environment – Data
The size and structure of the data
 Impact: The size and structure of the data could affect load test results dramatically.
 Issue: In case the production system will use a large set of data, using a small sample set of data or a
sample set with a different data complexity for performance tests  It leads to misleading results.
 It is difficult to predict how much the data size affects performance before real testing is performed.
 Suggestion: The closer the test data is to the production data in size and structure, the more reliable
the test results will be.
Restoring data
 Impact: After running performance tests, the data is generated or altered during the test
 Issue: The data is changed, it causes impact to another feature testing
 Suggestion: It may be necessary to restore the original data before the next test cycle to ensure that
the system is in the proper state.
4.2 Analysis, Design and Implementation
Preparing for Performance Test Execution
Properties of test environment – Hardware and software configuration
 Whatever approach to deployment is used, both hardware and software should be configured to meet
the test objective and plan.
 If the environment matches production, it should be configured in the same way.
 However, if there are differences, the configuration may have to be adjusted to accommodate these
differences.
 For example: If test machines have less physical memory than the production machines, software memory
parameters (such as Java heap size) may need to be adjusted to avoid memory paging
 If some parts of the system or some of the data is unavailable for performance tests for whatever
reason, a workaround should be implemented. The process is often referred to as “service
virtualization” and there are special tools available to assist with that process. The use of such tools
are highly recommended to isolate the system under test.
 For example: A stub may be implemented to replace and emulate a third party component responsible for
credit card processing.
4.2 Analysis, Design and Implementation
Preparing for Performance Test Execution
Properties of test environment – Network configuration
Proper configuration / emulation of the network is important for global and mobile systems.
 For global systems (i.e., one which has users or processing distributed world wide) one of
approaches may be to deploy load generators in places where users are located.
 For mobile systems network emulation remains the most viable option due to the variances in
the network types that can be used.
Some load testing tools have built in network emulation tools and there are standalone tools for
network emulation.
4.2 Analysis, Design and Implementation
Preparing for Performance Test Execution
Tool Configuration
The load generation tools should be properly deployed and the monitoring tools should be configured to
collect all necessary metrics for the test.
The list of metrics depends on the test objectives, but it is recommended to collect at least basic metrics
for all tests
Selecting tool
 Depending on the load, specific tool / load generation approach, and machine configuration, more
than one load generation machine may be needed.
 To verify the setup, machines involved in load generation should be monitored too.
 This will help avoid a situation where the load is not maintained properly because one of the load generators
is running slowly.
 Depending on the setup and tools used, load testing tools need to be configured to create the
appropriate load.
For example: specific browser emulation parameters may be set or IP spoofing (simulating that each virtual user
has a different IP address) may be used.
4.2 Analysis, Design and Implementation
Preparing for Performance Test Execution
Checking the tool before executing performance test
Before tests are executed, the environment and setup must be validated.
 This is usually done by conducting a controlled set of tests and verifying the outcome of the tests
as well as checking that the monitoring tools are tracking the important information.
 To verify that the test works as designed, a variety of techniques may be used, including log
analysis and verifying database content.
 Preparing for the test includes checking that required information gets logged, the system is in
the proper state, etc.
For example: if the test changes the state of the system significantly (add / change information in
database), it may be necessary to return the system to the original state before repeating the test.
4.3 Execution
What activities will we do in test execution?
Performance test execution involves generation of a load against the SUT according to a load profile:
 Monitoring all parts of the environment
 Collecting and keeping all results and information related to the test.
Tools can support those activities?
Usually advanced load testing tools / harnesses perform these tasks automatically:
 Some tools provide a console to enable performance data to be monitored during the test and
permit necessary adjustments to be made
 However, depending on the tool used, the SUT, and the specific tests being executed  some
manual steps may be needed.
4.3 Execution
How do we check the load generation tool before execution?
Performance tests are usually focused on a steady state of the system, i.e., when the system’s behavior is
stable
When the load is changing (for example, when new users are added)  the system’s behavior is changing
 It becomes more difficult to monitor and analyze test results.
So that when the system’s behavior is changing  we need to test transient states
For example:
This may apply to the concurrent logging of a large number of users or spike tests. When testing transient
states it is important to understand the need for careful monitoring and analysis of the results, as some
standard approaches—such as monitoring averages—may be very misleading.
4.3 Execution
How does the load generation tool testing work?
There are 2 stages for checking:
 The stage of getting to the steady state is often referred to as the ramp-up
 The stage of finishing the test is often referred to as the ramp-down

During the ramp-up it is advisable to implement incremental load states to monitor the impact of the
steadily increasing load on the system’s response  ensure that sufficient time is allocated for the
ramp-up and that the system is able to handle the load
Once the steady state has been reached, it is a good practice to monitor that both:
 The load and the system’s responses are stable
 That random variations (which always exist) are not substantial
4.3 Execution
How do the monitoring tool & log analysis tool testing work?
Monitoring tool
 If monitoring is built into the load testing tool and it is properly configured  it usually starts at the
same time as the test execution.
 If stand-alone monitoring tools are used  monitoring should be started separately and the necessary
information collected such that subsequent analysis can be carried out together with the test results.
Log analysis tool
The same is true for log analysis. It is essential to time-synchronize all tools used, so that all information
related to a specific test execution cycle can be located.
4.3 Execution
During execution test what should I do?
Test execution:
+ Is monitored using the performance test tool’s console
+ Real time log analysis to check for issues and errors in both the test and the SUT

Why do we need to do that?

Because this tests are expensive  the above actions helps to avoid needlessly continuing with running
large-scale tests, which might even impact other systems if things go wrong
e.g., if failure occur, components fail or the generated loads are too low or high.

While monitoring and check log we need to give one of the following actions:
 stop the test
 make some on-the-fly adjustments to the performance test or the system configuration if the test
deviates from the expected behavior.
4.3 Execution
The way we execute the tests
There are 2 ways:
GUI-level:
 How it works: verifying load tests which are communicating directly on the protocol level is to run
several GUI-level (functional) scripts or even to execute similar operational profiles manually in parallel
to the running load test.
 What we need to check: check that response times reported during the test only differ from the
response times measured manually at the GUI level by the time spent on the client side.

Continuous Integration:
 How it works: The test set up should be able to recognize any deviations or problems and issue an alert
 What we need to check:
 checks must be done automatically, since manual monitoring and intervention may not be possible
 easier to implement for regression performance tests when the system’s behavior is generally known, but may be
more difficult with exploratory performance tests or large-scale expensive performance tests that may need
adjustments to be made dynamically during the test.
4.4 Analyzing Results and Reporting
Overview
 When analyzing the data it is first compared to the performance test objective. Once the behavior is
understood, conclusions can be drawn which provide a meaningful summary report that includes
recommended actions.
 These actions may include:
 Changing physical components (e.g., hardware, routers)
 Changing software (e.g., optimizing applications and database calls),
 Altering the network (e.g., load balancing, routing).
4.4 Analyzing Results and Reporting
Metrics
Type Metrics Definition Meaning
Virtual users status # Passed It is normally expected that all Any interruption to this activity would mimic what an
# Failed simulated users have been able to actual user may experience  This makes it very
accomplish the tasks specified in the important to first see that all user activity is
operational profile completed since any errors encountered may influence
the other performance data.
Transaction  Minimum • Minimum, Maximum: readings • P(90): is used as a goal since it represents the
Response Time  Maximum show the extremes of the system majority of users attaining a specific performance
 Average 90% performance. threshold.
 Percentile • Average: is not necessarily • It is not recommended to require 100% compliance
indicative of anything other than with the performance objectives as the resources
the mathematical average and can required may be too large and the net effect to the
often be skewed by outliers. users will often be minor.

Transactions per # Passed / second System Throughput See in System Throughput

second # Failed / second
# Total / second
Transaction # Passed • This data is used when analyzing • Any failures encountered are a cause for concern
failures # Failed transactions per second. and the root cause must be investigated.
• Failures indicate the expected • Failed transactions may also result in invalid
event or process did not complete, transactions per second data since a failed
or did not execute. transaction will take far less time than a completed
one.
4.4 Analyzing Results and Reporting
Metrics
Type Metrics Definition Meaning

Hits (or requests) Hits / second This provides a sense of the number
per second  Minimum of hits to a server by the simulated
 Maximum users during each second of the test.
 Average
 Total

Network Bits / second This is usually measured in bits by This represents the amount of data the simulated users
throughput  Minimum time interval, as in bits per second. receive from the server every second.
 Maximum
 Average
 Total
HTTP responses. • Measured per second
• Response code such as 200, 302,
304, 404
4.4 Analyzing Results and Reporting
Metrics
Techniques used in analyzing data can include:
 Comparing results to stated requirements
 Observing trends in results
 Statistical quality control techniques
 Identifying errors
 Comparing expected and actual results
 Comparing the results to prior test results
 Verifying proper functioning of components (e.g., servers, networks)
Identifying correlation between metrics can help us understand at what point system performance begins to
degrade.
For example:
 what number of transactions per second were processed when the CPU reached 90% capacity and the system
slowed?
Analysis  identify the root cause of the performance degradation or failure
Confirmation testing will help determine if the corrective action addressed the root cause.
4.4 Analyzing Results and Reporting
Reporting
 Analysis results are consolidated and compared against the objectives stated in the performance
test plan.
 These may be reported in the overall test status report together with other test results, or
included in a dedicated report for performance testing.
 The level of detail reported should match the needs of the stakeholders
It should be include the following parts:
 Executive Summary
 Test Results
 Test Logs/Information Recorded
 Recommendations
4.4 Analyzing Results and Reporting
Reporting
Executive Summary
 This section is completed once all performance testing has been done and all results have been
analyzed and understood.
 The goal is to present concise and understandable conclusions, findings, and recommendations for
management with the goal of an actionable outcome.
4.4 Analyzing Results and Reporting
Reporting
Test Results
 Summary: Providing an explanation and elaboration of the results.
 Baseline:
 Test that serves as “snapshot” of system performance at a given time and forms the basis of
comparison with subsequent tests.
 The results should include:
 the date/time the test started
 the concurrent user goal
 the throughput measured
 key findings: overall error rate measured, response time and average throughput
 A high-level diagram: Showing any architectural components that could (or did) impact test objectives
 A detailed analysis (tables and charts) of the test results:
 Showing response times, transaction rates, error rates and performance analysis.
 The analysis also includes a description of what was observed, such as at what point a stable
application became unstable and the source of failures (e.g., web server, database server).
4.4 Analyzing Results and Reporting
Reporting
Test Logs/Information Recorded
A log of each test run should be recorded. The log typically includes the following:
Basic:
 Date/time of test start
Test Data & Scripts:
 Scripts used for test (including script mix if multiple scripts are used) and relevant script configuration
data
 Test data file(s) used by the test
 Test duration
Configuration:
 HW/SW configuration tested (especially any changes between runs)
Logs:
 Name and location of data/log files created during test
 Average and peak CPU and RAM utilization on web and database servers
 Notes on achieved performance
 Defects identified
4.4 Analyzing Results and Reporting
Reporting
Recommendations
Recommendations resulting from the tests may include the following:
 Technical changes recommended: reconfiguring hardware or software or network infrastructure
 Areas identified for further analysis: analysis of web server logs to help identify root causes of issues
and/or errors
 Required additional monitoring: gateways, servers, and networks so that more detailed data can be
obtained for measuring performance characteristics and trends (e.g., degradation)