The Association of System

Performance Professionals

The Computer Measurement Group, commonly called CMG, is a not for profit, worldwide organization of data processing professionals committed to the
measurement and management of computer systems. CMG members are primarily concerned with performance evaluation of existing systems to maximize
performance (eg. response time, throughput, etc.) and with capacity management where planned enhancements to existing systems or the design of new
systems are evaluated to find the necessary resources required to provide adequate performance at a reasonable cost.

This paper was originally published in the Proceedings of the Computer Measurement Group’s 1996 International Conference.

For more information on CMG please visit [Link]

Copyright Notice and License

Copyright 1996 by The Computer Measurement Group, Inc. All Rights Reserved. Published by The Computer Measurement Group, Inc. (CMG), a non-profit
Illinois membership corporation. Permission to reprint in whole or in any part may be granted for educational and scientific purposes upon written application to
the Editor, CMG Headquarters, 151 Fries Mill Road, Suite 104, Turnersville , NJ 08012.

BY DOWNLOADING THIS PUBLICATION, YOU ACKNOWLEDGE THAT YOU HAVE READ, UNDERSTOOD AND AGREE TO BE BOUND BY THE
FOLLOWING TERMS AND CONDITIONS:

License: CMG hereby grants you a nonexclusive, nontransferable right to download this publication from the CMG Web site for personal use on a single
computer owned, leased or otherwise controlled by you. In the event that the computer becomes dysfunctional, such that you are unable to access the
publication, you may transfer the publication to another single computer, provided that it is removed from the computer from which it is transferred and its use
on the replacement computer otherwise complies with the terms of this Copyright Notice and License.

Concurrent use on two or more computers or on a network is not allowed.

Copyright: No part of this publication or electronic file may be reproduced or transmitted in any form to anyone else, including transmittal by e-mail, by file
transfer protocol (FTP), or by being made part of a network-accessible system, without the prior written permission of CMG. You may not merge, adapt,
translate, modify, rent, lease, sell, sublicense, assign or otherwise transfer the publication, or remove any proprietary notice or label appearing on the
publication.

Disclaimer; Limitation of Liability: The ideas and concepts set forth in this publication are solely those of the respective authors, and not of CMG, and CMG
does not endorse, approve, guarantee or otherwise certify any such ideas or concepts in any application or usage. CMG assumes no responsibility or liability
in connection with the use or misuse of the publication or electronic file. CMG makes no warranty or representation that the electronic file will be free from
errors, viruses, worms or other elements or codes that manifest contaminating or destructive properties, and it expressly disclaims liability arising from such
errors, elements or codes.

General: CMG reserves the right to terminate this Agreement immediately upon discovery of violation of any of its terms.
 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - [Link]/conference

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at [Link]
BENCHMARKING:
Why, What, How, When and Where

Jerry L. Rosenberg
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at [Link]/subscribe

SRM Associates, Ltd.

The availability of tools to predict the performance of computing systems has relegated the art of benchmarking to
a position of an obscure footnote in the lexicon of capacity management skills. Many analysts feel that it is a
function that need only be undertaken by hardware vendors and developers of software products. The author
believes that as long as there are changes occurring in our environments, we will need to perform benchmarks,
analyze their results and utilize this information in a proactive manner. There is certainly enough happening in
the world of available computing solutions to require that this time proven approach be considered in a total
resource planning process.

This paper is an expanded outline of the presentation. More detail will be supplied in the presentation itself. The
presentation will include checklists and “recipes” to facilitate a successful benchmark exercise. Recent
experiences in developing, executing and analyzing benchmarks of Real-time systems in a CICSplex, Sysplex, DB2
Data Sharing environment on CMOS based complexes will be utilized as examples.

Benchmarking is one of several

Background methods/techniques available to perform
In the caveman days of data processing, performance measurement and prediction.
benchmarking was a commonly accepted The spectrum of standard approaches is
form of predictive performance assessment given in Figure 1. You will note that
(remember, this was before we had defined traditional benchmarks fall at the far right
the term - capacity planning). It was so with high accuracy but also high cost. It has
prevalent that vendors had programs (like been felt that simulation and modeling
IBM’s OSTA - On-site/off-site test offered reasonably accurate predictions of
allowance) which afforded the customer performance without the accompanying time,
specified amounts of non-billable usage on manpower and complexity. The problem in
equipment to benchmark its characteristics. this paradigm is that the model is only a valid
With the advent of more powerful predictive predictor if the data within truly reflects the
tools (i.e. analytic models and simulation hardware and software and anticipates how
packages) and more robust measurement they will behave under changing conditions.
data, benchmarking began to fall into disuse It has been the author’s experience that this
as an excessively time and manpower key data is often not available for new
consumptive task. There are, however, still technologies and must be specifically
many scenarios in which this process is the obtained in order to make the model
most effective, if not the only, method for meaningful. So, typically, a benchmark will
acquiring information on the performance of be developed and exercised in order to
a given hardware, software architecture. acquire information for a model that is not

Find a CMG regional meeting near you at [Link]/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - [Link]/conference

available via other means. A recent example

of this situation occurred in the development 1. Rules of Thumb

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at [Link]
of predictive models for CMOS technology
Requirements for ballpark figures will
utilizing Parallel Sysplex, CICSplex and DB2 probably utilize Rule-Of-thumb type
data sharing. While there were some rules of estimation principles. Rules-of-Thumb can
thumb and estimates available from the be applied by comparing against a"similar"
vendor as to the probable overhead of each workload. The similar workload might be a
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at [Link]/subscribe

of these features, the possible errors in projection based on past experience with
estimation from inappropriate combinations sizing a similar system, or comparison
against an existing DB2 application, or using
of overhead values were too great. The only another site executing a vendor package.
way to proceed with confidence was to Often the rules will come from vendors in the
develop a limited benchmark to actually form of expected proportional changes to
measure these conditions with a workload current workloads. Be very careful. There
that resembled production expectations. are usually caveats and conditions that
govern the estimates and these must be
assessed against your workload. How well
Clearly, there are times when the only way to does the test bed really map to your expected
get the data that you need is to go out and production work ?
create an environment in which that data can
be captured. This occurs with new
technologies, unique situations, and custom
applications. The key to modern 2. The Prototyping Method
benchmarks is to reduce the scope so that
just enough benchmarking is done to capture This method involves loading tables from
"production-like" data, taking measurements
data, while not creating a monster that is for a single user per DB2 plan and projecting
bigger than the production system in effort. to the full workload. This technique is
reasonable but will fall short in terms of
To recap, the reasonable approaches are: impact of other work causing contention to
the transaction. For example, multiple
1. Rule of Thumb Principles transactions executing with updates will
2. Prototyping based on single case cause some locking resulting in more CPU
measurements time and longer response times than
3. Benchmarking measured in the simple case.
4. Modeling
5. Combination of the above
3. Modeling
The technique employed is highly dependent This usually involves capturing data from an
on the level of data available. Questions existing system and using a vendor supplied
such as, "Is there sufficient data to create a package to produce a parameter file for the
valid representation of the application to be analytic queuing or simulation model.
modeled?" must be answered. In many Execution of the model allows for
cases, the answer is “no, but I can get the predictions of performance under varying
data with a controlled benchmark”. This workload volumes and hardware/software
leads to option 5 - the combination approach configurations. This is a very effective
which fits very well into a new technology methodology for scenarios which have been
situation defined in the modeling construct. A
problem arises when we are investigating
very new technologies, unique

Find a CMG regional meeting near you at [Link]/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - [Link]/conference

configurations or application specific detailed table specifications and access

conditions. Often, these scenarios require patterns have been defined.. The developers

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at [Link]
that the model be augmented with external will need to determine the representativeness
data to yield truly valid results. Obtaining
this data usually necessitates a benchmark. of the prototypes benchmarked and the
percentage of the application flow that they
represent.
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at [Link]/subscribe

4. Benchmarking
Benchmark Basics
Benchmarking, in the classical sense,
requires significantly more effort. It requires
the development of detailed scripts, Defining the problem
extensive set-up of tables and stand-alone The first step in a successful benchmarking
hardware for execution. Results are effort is to clearly define the rationale for the
extremely reliable but success depends upon
a commitment of resources (time, money, task and the expected data. Why are you
people, & computing) from management. contemplating a benchmark and what do you
To benchmark, the following are required: expect to achieve by it. What data metrics
are you trying to gather and at what level.
Design and Build tables Try to limit the scope of the exercise so that
Design and write prototype programs you don’t end up with reams of inconclusive
Tailor the on-line environment
Set up a special operating environment results.
Load the data
Design the test scenarios
Set up the scripts and resources needed
Requires systems, DBA, programmers, Designing the test bed
testers etc.
The design of the benchmark follows
naturally from the definition of the
4. Benchmarking, Prototypes and objectives. Remember that not all answered
Modeling: A Combination Approach. can be obtained in a single benchmark
execution. In fact, benchmarks should
By combining benchmarking and prototyping follow the rule of experimental design that
for code which can be executed, we can suggests the manipulation of only one
obtain fairly accurate measurements of variable at a time to avoid the effects of
system requirements without modeling. The compounded changes. For example, if we
accuracy should be better than modeling and wish to measure the performance
only one programmer and tester should be characteristics of DB2 buffer pools, it is
required. In addition, the use of prototyping unwise to vary both size and BP mapping at
and benchmarking to supply data to be the same time. If we do, it may be
utilized in a model yields a very accurate, impossible to determine which change
effective and flexible mechanism for produced how much result. In fact, if is
predicting the performance of very complex possible to have changes which cancel each
environments. Clearly, the facility of this other giving the impression of no change at
approach depends on the extent to which all.
code exists. Path length analysis can be
useful when code is not yet ready, but a There are some conditions under which it is
detailed design specification along with difficult or impossible to measure one

Find a CMG regional meeting near you at [Link]/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - [Link]/conference

variable without another. For example, it is data. What we are contemplating is a

difficult to effect DB2 data sharing without a representation, as close as possible, of the

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at [Link]
SYSPLEX environment. How do you then projected production environment so as to
assess the overhead of one versus the other ? afford pristine measures of performance that
In cases like this it is necessary to design do not require manipulation or adjustment.
phased benchmarks that record discrete This is often not practical and the nature of
measurements that then become part of a creating this environment and allowing for
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at [Link]/subscribe

composite scenario. In this case, we would disparity will be covered in more detail in the
first devise benchmarks to measure the presentation.
overhead of SYSPLEX (in a non-DB2
environment) and back that measurement out Careful planning and control must be
of the SYSPLEX/DB2 data sharing test to imposed on the benchmark environment. A
approximate the overhead of data sharing set of control procedures is strongly
uniquely. recommended to insure that accurate data is
collected and preserved. A sample of a
control procedure follows:
Required metrics
The nature, quantity and archiving of data
captured during the benchmark must be Benchmark Control
carefully planned prior to execution. After Procedures
all, this is why we ran the benchmark. The
data is key to solving the questions that
The purpose of this document is to provide a
drove us to develop the benchmark. As
checklist to be reviewed prior to executing a
such, the nature of the initial questions
benchmark. It describes the documentation,
should be the key factors in determining the
implementation, verification, and reporting
nature of metrics that will be gathered.
mechanisms that the benchmarks will follow
to provide a more consistent and
On the other hand, try not to fall into the
standardized pattern that can be repeated at
classical analysis trap - data overload. You
any time. A benchmark checklist is also
can have too much data. It seriously
provided to aid the individual that is running
encumbers your ability to process
the benchmark.
information and wastes time. Just as you
need to focus your benchmark goals, you
To provide a successful benchmarking
must limit the data that you capture to the
environment, several control factors need to
necessities. Remember, do not try to create
be put in place. The first of these factors is
a single benchmark to answer all your
to maintain standardization in software
questions.
version and release control, not only for
vendor products, but also for the application
programs that are being tested. Without
Constructing the test version and release control, it is very difficult
environment to implement repeatability. Next, it is felt
that a stand-alone benchmarking
Part of the classical definition of a environment is essential for maintaining
benchmark is that it is actual programs consistency from one run to another. This
running on actual hardware using actual means that all Address Spaces that are not

Find a CMG regional meeting near you at [Link]/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - [Link]/conference

pertinent to the benchmark should be Operating System:

quiesced. A scheduling mechanism should • - [Link](IEAIPS)

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at [Link]
be in place to coordinate this activity with • - [Link](IEAICS)
the other Central Segment subsystems. • - Auxiliary paging devices(Number and
location)
Documentation: • - Versions of operating system software
Hardware: CICS:
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at [Link]/subscribe

• - Processor type • - Run a CICS “STAT” before the

• - Memory configuration and cache sizes benchmark run
• - Number of channel and device adapters • - CICS startup JCL and parameters
• - Disk configuration • - RCT parameters
• - Number of CICS regions used in
benchmark run
• - Version of application software being
run
• - location of CICS disk files
DB2:
• - DSNZPARM
• - Bind parameters
• - locations of application
table(data/index) allocations/partitions on
disks
• - table(data/index) sizes
TPNS:
• - Data(UTBL) used in this benchmark
run
• - Network used (especially the number of
terminals and UTI)
• - Script that was used to drive the
transactions

Implementation:
CICS:
1. Copy the load modules from the
development environment to the
benchmark environment.
2. Copy the DBRM’s from the development
environment to the benchmark
environment.
3. Bind the plans with their corresponding
transactions.

Find a CMG regional meeting near you at [Link]/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - [Link]/conference

4. Re-initialize CICS? If CICS is not re- 2. Environment

initialized, then the new programs must

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at [Link]
be NEWCOPY’d. Is test down?
5. Verify that the transactions that are being Yes ___ No ____
run for this benchmark are included in How much memory is allocated to
the CICS PCT. Production?
6. Verify that the programs that are being Main______ Expanded _______
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at [Link]/subscribe

run for this benchmark are included in What none essential Address Spaces are
the CICS PPT. executing?
7. Verify that the proper RCT is being used ____________________________________
by CICS (“DSNC DISP STAT”).
8. Verify that the appropriate copies of the 3. CICS Prod Region
application tables are loaded.
What is the status of [Link]
DB2: members for the CICS and DB2 A.S. we are
1. Reload the necessary DB2 tables to benchmarking?
restore them to their initial state. IEAICS00 - What Performance Group is
2. Re-initialize DB2? CICS using?
_________________
TPNS: IEAIPS00 - What is the CICS Dispatching
1. Verify the scripts that are going to be Priority?
used for this particular benchmark run. ________________
• UTBL (types and quantity of - What are the PPGRT & PWSS settings?
transactions) _________________
• NETWORK (especially the number - What is the DB2 Dispatching Priority?
of terminals) _________________
• SCRIPT (how many transactions) Have the SMF data sets been dumped prior
2. Primer? to this run?
Yes ____ No ____
What is the status of RMF? Up ____ Down
_____ Interval _____
What is the status of OMEGAMON/MVS?
Checklist: Up ____ Down _____

1. Description 4. TPNS

What is the purpose of this benchmark? What scripts are we using?

____________________________________ Network _________
____________________________________ Program __________ Table ___________
____________________________________ How many and which CICS are we testing?
____________________________________ Number ____ CICS _______________
____________________________________ How many terminals per CICS?
____________________________________ _____________________
What is the UTI setting? __________

Find a CMG regional meeting near you at [Link]/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - [Link]/conference

What type of logging is in use? A sample of a checklist that might serve to

__________________ control the actual benchmark execution is

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at [Link]
presented here:

5. CICS
BENCHMARK CHECKLIST
What is the setting(s) for AMXT?
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at [Link]/subscribe

_________________ TPNS (or other driver) Check-out

What is the setting(s) for MXT? • Verify the scripts that are going to be
_________________ used for this particular benchmark run.
What load modules by version & release are • UTBL (types and quantity of
we testing? transactions)
What is the status of OMEGAMON/CICS? • NETWORK (especially the number of
Up ____ Down _____ terminals)
What is the status of CICS trace?
• SCRIPT (how many transactions, type,
On ____ Off ______
mix, etc.)
What is the status of STROBE?
On ________ Off _________
Quiesce system activity
What version/release of the RCT are we
using? _______________ • Stop non-Benchmark DB2 systems
• Bring down all non-benchmark CICS
6. DB2 regions
• Limit TSO users
Which DB2 A.S. are we testing? _________
What is the status of OMEGAMON/DB2? Application Issues
Up _____ Down ______ • Identify Application programs and
What Buffer Pool numbers are in use? libraries for BM Group to copy to BM
____________________________________ environment.

Initialize Benchmark Environment

• Run ‘BEFORE JOB’ to INITIALIZE
7. Required Signatures: ENVIRONMENT

Verify MVS Environment

• Set IPS/ICS to BM
• RMF/SMF - Set RMF=BM, Set
SMF=BM

Verify CICS Environment

• Copy the load modules from the
development environment to the
benchmark environment.
• Copy the DBRM’s from the development
environment to the benchmark
environment.

Find a CMG regional meeting near you at [Link]/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - [Link]/conference

• Bind the plans with appropriate Stop Benchmark

parameters.

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at [Link]
• NEWCOPY Programs.
• Check PCT, PPT. Post-Benchmark Procedures
• Verify that the appropriate copies of the
application specific tables are loaded.
• Set RCT = BM. Restore Production MVS Environment
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at [Link]/subscribe

• Verify that no CICS traces are active • Back-out all changes and restore
Production MVS Environment
• Reset IPS/ICS
Verify DB2 Environment • Reset DB2
• Modify DB2 buffer pools • Restore all TSO and Batch Initiators
• Verify that only DB2 performance class
1,2,3 monitors are active Restore VSAM and DB2 Database
Environment
Restore Benchmark Environment • Restore DB2 database and all required
• Restore required DB2 databases, VSAM VSAM and flat files
and flat files
Run Jobs to Save Data
• Run ‘AFTER JOB’ to SAVE:

Run Test SMF data

• Run primer to open data sets and Produce CICS, DB2, MVS Reports
initialize via 5 minutes of transactions Produce TPNS response time and
• Run Script 1 throughput report
• Wait 5 minutes Save CICS Journal
• Run Script 2, Wait Save TPNS MSG LOG
• Run Script 3 Create MXG PDB
• Log all Times.
• VERIFY TEST RESULTS and analyze
Monitor Test during test execution MSG LOG Output
Monitors- Omegamon/DB2, CICS, and
MVS
• Monitor CICS MXT
• Monitor CPU, DASD and Storage
• Monitor throughput
• If test appears to be running normally,
Capturing Data & Presenting
then continue. Otherwise if there are Results
problems, fix them and continue with
test. The collection and analysis of data was the
Note: Need to run with minimal purpose of the benchmark activity so it is not
monitoring set, e.g., No DEXAN, Strobe surprising to learn that ultimate consideration
etc. must be given to insuring that this precious
information is carefully secured. A sample

Find a CMG regional meeting near you at [Link]/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - [Link]/conference

data capture checklist is presented below. ____________________________________

Please note that type and location of stored ____________________________________

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at [Link]
data are detailed is insure that all relevant ____________________________________
information is protected. ____________________________________
____________________________________
TEST LOG AND RESULTS FORM ____________________________________
Benchmark:____________________
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at [Link]/subscribe

Date:_________________
MVS Environment changes (e.g. storage
Identify, Describe &/or Attach the fencing parms)
following: ____________________________________
____________________________________
Application/Program Library ____________________________________
Changes/Compile Options ____________________________________
____________________________________ ____________________________________
____________________________________ ____________________________________
____________________________________
____________________________________
____________________________________ CICS Changes (i.e., RCT, region
____________________________________ configuration, MXT, etc.)
____________________________________
____________________________________
TPNS Setup (must include: Network, ____________________________________
Scripts & UTBL) ____________________________________
____________________________________ ____________________________________
____________________________________ ____________________________________
____________________________________
____________________________________
____________________________________ DB2 Changes (i.e., DSNZPARM, Bind
____________________________________ parameters, Buffer Pools, etc.)
____________________________________
____________________________________
Workload Mix Executed (must include: ____________________________________
Transaction Volume & Test Duration) ____________________________________
____________________________________ ____________________________________
____________________________________ ____________________________________
____________________________________
____________________________________ Location of SAVED Benchmark Output
____________________________________ TPNS Log
____________________________________ ____________________________________
RMF/SMF Data
____________________________________
Hardware Environment Changes (include Copy of IPS/ICS members
CPU, DASD, Memory & MDF/LPAR ____________________________________
changes)

Find a CMG regional meeting near you at [Link]/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - [Link]/conference

DSNZPARM
____________________________________

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at [Link]
Copy of modified tables, e.g. RCT, PPT, etc.
__________________________________
CICS Log data set
____________________________________
MXG PDB
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at [Link]/subscribe

____________________________________
Application Table allocations & DASD
assignments
_________________________________

Analyzing the collected data is subject to the

requirements of the benchmark. The
presentation will focus on some complex
analysis of benchmark data (e.g. the merging
of SMF, CICS and DB2 data to develop a
consolidated view of transaction activity).
We will also cover the requirements of data
presentation and summary reporting.

Summary

Benchmarks remain valuable tools in the

arsenal of capacity planners. They can be
invaluable in determining the performance
characteristics of unique or leading edge
hardware, software and applications. The
data obtained can be combined with models
and other analytic techniques to supply a
very accurate assessment of resource
requirements. As always, this process must
be well understood and carefully managed or
it will drain resources and yield erroneous
results.

Find a CMG regional meeting near you at [Link]/regions