WORKLOAD CHARACTERIZATION AND BENCHMARK

Workload

Workload characterization dates back to the early 70’s. Since then, this discipline has evolved
following the evolution of computer architecture. In the early days, computers were mainframes,
and their workloads were basically composed of batch jobs and transactions. The advent of time-
sharing systems and computer networks has changed the approach of the users toward the systems.
This advent, which has been coupled with an increased processing power of the systems and with
the introduction of graphical user interfaces, has opened the systems to new processing
requirements. All this has also led to the development of distributed systems and of client/server
applications. The new services provided on top of Internet, such as the World Wide Web, have
introduced the concept of multimedia workloads. These workloads consist of a mix of different
types of application (e.g., file transfers, real time audio applications) characterized by different
performance requirements on the resources of servers and clients as well as of the networks

The performance of any type of system cannot be determined without knowing the workload, that
is, the requests being processed. Workload characterization consists of a description of the
workload by means of quantitative parameters and functions; the objective is to derive a model
able to show, capture, and reproduce the behavior of the workload and its most important features.

The performance of a computer system depends not only on its design and implementation, but
also on the workload to which it is subjected. Different workloads may lead to different absolute
performance numbers and in some cases to different relative ranking of systems or designs.
Traditionally, the term workload is defined as the set of all inputs (programs, commands, and data)
from the users to the system. Depending on the objective of the performance evaluation study, the
workload may include programs, commands, and data submitted by the users, and also system
programs (compilers, editors, etc) requested by the users.

In computer performance evaluation, a workload is simply: The set of tasks, programs, jobs, or
requests that a computer system processes over a period of time. It represents what the system is
asked to do. A workload is the collection of tasks and requests imposed on a computer system,
defining how system resources are exercised and forming the basis for performance evaluation. A
workload may consist of: Programs or applications, User requests or transactions, Jobs submitted
to the system, Data sizes and access patterns or/ and Arrival rates and execution times.

Workload types categorize computing tasks by their demands, including Transactional High-
volume, short, quick database operations (e.g., banking, e-commerce checkouts), Batch Large,
scheduled data processing jobs, often overnight (e.g., billing, payroll), Analytical Complex queries
on large datasets for business intelligence (e.g., data warehousing, reporting), High-
Performance Computing (HPC) (scientific simulations), Streaming Continuous data ingestion and
processing (e.g., IoT, real-time monitoring), AI/ML Training complex models and serving
predictions (inference), and Web/Microservices (APIs, web apps), plus deployment-specific types
like VMs, Containers, and Serverless, each requiring different resources and optimization
strategies.

Workload Characteristics refers to the demands placed by the requests on the various system
resources and can be described by:

i. Arrival rate of jobs/requests

ii. Job size or execution time

iii. Resource demands (CPU, memory, I/O)

iv. Concurrency level (number of users/processes)

v. Variability and burstiness

Workload is very important as it determines: System performance, resource utilization, response

time, throughput and scalability. Performance results are only meaningfully relative to a given
workload. Each request, that is, each workload component, is described by a set of parameters
which explain these demands. There are static parameters related to hardware and software
resource consumptions and dynamic parameters related to the behavior of the requests. The
magnitudes of these parameters depend on the nature of the single request and can be conveniently
used to quantify the workload. The approach commonly adopted for workload characterization is
experimental, that is, based on the analysis of measurements collected on the system while the
workload is being processed. Due to the complexity of the systems and of their workloads, such
an experimental approach becomes quite challenging. Appropriate instrumentation has to be
developed in order to ensure the quality of the measurements which have to adjust to the
characteristics of the systems and of their workloads. To capture detailed information about the
behavior of the workload, it might be necessary to insert into the system probes, such as event
counters. Another issue to be addressed deals with the large amount of collected measurements. A
good tradeoff between what has to be measured and the amount of collected data has to be
achieved. Moreover, the degree of intrusiveness and the overhead introduced by the
instrumentation system have to be as low as possible in order not to perturb the behavior of the
system and of its workload. Once the measurements are available, appropriate techniques have to
be applied for their analysis in order to ensure the accuracy and the representativeness of the
workload model which will be derived.

Workload characterization is the process of analyzing, describing, and modeling the workload
imposed on a computer system in order to understand its performance behavior and to support
accurate performance evaluation, capacity planning, and system design. Workload characterization
aims to:

1. Understand how a system is used in practice

 2. Identify performance bottlenecks

3. Support fair comparison of different systems

4. Enable realistic benchmarking and simulation

5. Guide system tuning and capacity planning

6. Predict future performance under changing workloads

Without proper workload characterization, performance results may be misleading or irrelevant.

Workload Model

When it comes to load testing, choosing the right workload model is crucial for accurately
simulating real-world user behavior and understanding how your application will perform under
different conditions.

Workload models are a strategy that defines how users interact with your application during a load
test. They help simulate the conditions your application will face in production, allowing you to
identify potential bottlenecks and optimize performance. The choice of workload can significantly
influence the behavior of your tests and the insights you gain from them.

Workload models in computer performance evaluation represent real-world system demands (like
user actions, data volume, request types) as quantifiable patterns to predict performance, ensuring
systems handle expected loads, peak traffic, and user behaviors accurately before deployment,
using key metrics like concurrent users, request frequency, and task mix. These models are built
by collecting real data and applying statistical methods to capture features like burstiness,
correlations, and different user types, moving beyond simple counts to structural models of
workload generators. Examples of workload models includes instruction mix models (early CPU
focus), synthetic workloads (standardized like SPEC benchmarks), kernel benchmarks, and real-
application benchmarks

Purpose of workload model

Workload models are simplified representations of real system workloads. Their purpose is to
make performance analysis realistic, manageable, and repeatable.

Key Purposes of Workload Models

i. Workload Modelling helps to generate the expected production-like conditions in the
performance test environment to check the true performance of the application.

ii. Another purpose is that when there are some production performance issues detected in the
live environment then workload modelling helps to replicate the same condition to
reproduce the issue in the performance test environment.
 iii. Represent Real System Usage: Workload models capture how users and applications
actually use a system, including: Job arrival rates, execution times, resource demands etc.
ensuring performance studies reflect real-world behavior.

iv. Enable Performance Evaluation: They provide input to: Analytical models (e.g., queueing
theory), simulation models, benchmarking experiments etc. Without workload models,
performance results lack meaning.

v. Support System Comparison: Workload models allows fair comparison of different

systems and also supports the evaluation of hardware or software alternatives under the
same workload

vi. Facilitate Capacity Planning: They help answer questions such as:

i. How many users can the system support?

ii. When should resources be upgraded?
iii. What happens under peak load?

vii. Predict Future Performance: By varying workload parameters, models help predict:
i. System behavior under increased load

ii. Effects of new applications or users

viii. Reduce Complexity: Real workloads are complex and unpredictable. Workload models:
i. Abstract essential characteristics

ii. Remove unnecessary details

iii. Make analysis mathematically and computationally feasible

[Link] Repeatable Experiments: Workload models ensure: Experiments can be repeated

and results can be verified and validated
x. Assist in System Design and Tuning: Designers use workload models to: Identify
bottlenecks, optimize scheduling and resource allocation and improve response time and
throughput.

Types of workload

i. Real Workload
A real workload is derived directly from actual usage of a system. It represents real users, real
applications, and real operating conditions, hence, if performance evaluations are made on a
system when executing the actual programs, commands, and data submitted by the user(s) during
a particular time interval, then the performance measures are said to be obtained under real
workload. Consequently, real test workloads can only be used in measurement experiments on real
systems. Examples are Apache web server access logs, E-commerce transaction histories, Mobile
app usage traces and Production ML inference request logs.

ii. Artificial Workload

An artificial workload is manually constructed to test specific components or behaviors of a system
using simplified patterns. An artificial workload mimics the behaviors of real workloads and does
not usually produce any useful computational results. It consists of a set of operations that are
either typical for a given set of applications or that are chosen in order to stress the system in ways
not possible with an actual application. Within the class of artificial models, we can distinguish
further between executable and non-executable models. Examples include CPU stress tests (e.g.,
tight loops), Disk benchmarks (sequential read/write), Network ping or throughput tests and
Microbenchmarks like SPECint subtests.

iii. Synthetic Workload

A synthetic workload is automatically generated using models derived from real workloads,
without directly copying real data. A synthetic workload is a constructed workload designed from
the definition of the system profile. It is scalable in form thereby allowing for adjustment to meet
different profile and to wider scope of system evaluation. A synthetic workload is a mix of real and
artificial workload components. If only a set of real programs is used, the model is called natural
workload or benchmark, if the synthetic workload consists of real programs and artificial
components, then it is called a hybrid model. Examples includes synthetic web traffic generators,
cloud workload simulators, Database benchmark generators (TPC benchmarks) and Synthetic ML
datasets.

Steps Involved in Workload Model

Workloads appear in many contexts, and therefore have many different types. To construct a
workload model, the following steps are involved:
1. Understand or define the specific workload to be modeled as well as the objectives of the
workload. For instance, web server workload, java workload, database workloads,
transaction processing workload etc.
2. Choice of suitable instrumentation, that is, use of existing performance/monitoring tools or
construction of ad-hoc ones.
3. Experimental measurement
4. Analysis of workload data
(a) Preliminary analysis
(b) Analysis of the parameter distributions
(c) Sampling
5. Construction of static/dynamic workload model
(a) Static analysis
(b) Dynamic analysis
 6. Representative (customize phase). The matching of the model with the type of system
under test.

Workload Model Development Cycle

Model Formulation refers to the process of making judicious choices regarding the assumptions
and structure of simplified representations used to model real-world systems in Earth and Planetary
Sciences. It is the step where our knowledge of a natural system is translated in mathematical form.
It involves two steps: the construction of a conceptual model and the formulation of this conceptual
model into mathematical equations

We take two approaches to formulating models: 1) Template approach,

2) Constructive approach.

The constructive approach is the more fundamental and general. However, readers with less
analytic skill may prefer the template approach. The template approach is essentially a “model in
a can” approach. In this approach, examples of standard applications are illustrated in substantial
detail. If you have a problem that closely resembles one of these “template” models, you may be
able to adjust it to your situation by making modest changes to the template model. The advantage
of this approach is that the user may not need much technical background if there is a template
model that closely fits the real situation.

Parameterization is the process of assigning values to variables within a model in order to ensure
that it is as accurate as feasible or appropriate. In a simulation model as well as the actual values
to be used, there will be some consideration of the appropriate distribution to be used for generating
the values. It is important that we consider the availability of data to assist in parameterizing our
models from the outset of model design and construction.

BENCHMARK
In computing, a benchmark is the act of running a computer program, a set of programs, or other
operations, in order to assess the relative performance of an object, normally by running a number
of standard tests and trials against it. It could also be used absolutely to determine if the system
will perform a given function or not. Benchmarking is usually associated with assessing
performance characteristics of computer hardware, for example, the floating-point operation
performance of a CPU, but there are circumstances when the technique is also applicable to
software. Software benchmarks are, for example, run against compilers or database management
systems and so on. Benchmarks provide a method of comparing the performance of various
subsystems across different chip/system architectures. Benchmark programs are specific programs
written and implemented by the user to test the performance of a given system. Benchmarking is
one part of computer performance evaluation process, which includes qualitative as well as
quantitative considerations.

Qualitative evaluation focuses on design characteristics or potential flaws while quantitative

evaluation focuses on the significant number of components/hardware that must be present in the
system/architecture. Quantitative evaluation comes after the qualitative issue has been settled. In
quantitative considerations the issues are those of cost, productivity, reliability, and performance.

Cost includes:
(a) Acquisition cost
(b) Installation cost
(c) Operation cost
(d) Training cost
(e) Application development cost
(f) Maintenance cost

Productivity includes:
(a) Cost-performance ratio
(b) Quality of life (i.e interactive access, high speed graphics, powerful editors, debuggers and
on line documentation)

Reliability: This involves how reliable the system is, especially when applications need to run for
several hours and even days.

Performance: Performance measurement helps in pointing out the strength and weaknesses of
current system architecture.
Types of Benchmark

Livermore Loops Benchmark

Livermore loops (also known as Livermore Fortran kernels or LFK) is a benchmark for parallel
computers. It was created by Francis H. McMahon and published in 1986. It consists of 24 samples
of floating-point computational routines which are largely vectorizable extracted from scientific
programs at Lawrence Livermore National Laboratory, Calif. Each routine/loop carries out a
different mathematical kernel. Those kernels include:
1. Hydrodynamics fragment
2. Banded linear systems solution
3. Tridiagonal linear systems solution
4. Inner product
5. General linear recurrence equations
6. Equation of state fragment
7. Alternating direction implicit integration
8. Integrate predictors
9. Difference predictors
10. Fist sum
11. First difference
12. Implicit conditional computation
13. Matrix-matrix transport
14. Planckian distribution; etc

Linpack Benchmark
Linpack uses linear algebra algorithms to solve a set of equations with or without floating-point
operations, using compilers such as Fortran. This implies that it can be used to test both scalar and
vector floating-point performance. This will result in test metrics such as MIPS (million
instructions per second), MFPPS (million floating points per second). The tests represent the
equations by an n x n matrix, i.e 100 x 100, 300 x 300 or 1000 x 1000 with 100 x 100 array being
the standard case. The main routine may be configured for half, single (32 bits), double (64 bits)
or full precision tests. At the centre of LINPACK benchmark is the Double precision AX + Y.
That is, a scalar A is multiplied by a vector X and the vector result is also vectorially added to
vector Y. This procedure which is referred to by the acronym DAXPY is the same procedure for
performing Gaussian elimination in system of linear equation.

Whetstones Benchmark
This is a synthetic benchmark for basic arithmetic operations; used to test performance of mid-size
and small computers.

Dhrystones Benchmark
Developed in 1984 by Reinhold Weicker at Siemens. This kernel contains many procedure calls
and is considered to represent systems programming environments. This is a synthetic benchmark
for system-programming operations; used to test non-numeric performance of mid-size and small
computers.
AIM Benchmark
This program is written in C and is used to test Unix system performance. It is also called AIM
Benchmark Suite VII or AIM7 and was developed by AIM Technology Inc. The tests carried out
by AIM benchmark include disk/file system, database management system, compilers, word
processing, spreadsheet and accounting packages.

SPEC benchmark
Standard Performance Evaluation Corporation (SPEC) developed two benchmarks for testing the
performance of CPU when performing integer and floating-point operations. These are SPECint
and SPECfp.
1. SPECint: This is a computer benchmark specification for CPU’s integer processing power.
It is maintained by the Standard Performance Evaluation Corporation (SPEC). The files
for SPECint are written in a standard programming language, which is then compiled for
each particular CPU architecture and operating system. Thus, the performance measured
is that of the CPU, RAM, and compiler, and does not test I/O networking, or graphics.
2. SPECfp: This is a computer benchmark designed to test the floating-point performance of
a CPU. To compute the SPECfp score, benchmark applications are run on a reference
machine and the time each application requires for completion is recorded as the reference
time. When evaluating the performance of another machine, the benchmark applications
are run on that system and the time the applications required for completion is recorded.
Then the ratio between the recoded time and the reference time is computed.

For example, [Link] application takes 1280 seconds to complete on the AlphaStation
machine, while it takes 1700 seconds on the reference machine. Then, the ratio is:
1700/1280 = 1.328. This implies that AlphaStation machine is 32.8% faster than the
reference machine in running [Link] application.

Multiple Benchmark
We want to summarize the performance of a computer; the easiest way is to use a single number
that can be compared against the numbers of other machines. This typically involves running tests
on the machine and taking some sort of mean; the mean of a set of numbers is the central value
when the set represents fluctuations about that value. There are a number of different ways to
define a mean value; among them the arithmetic mean, the geometric mean, and the harmonic
mean.
An acceptable practice is to evaluate the performance of more than one benchmark program on
more than one computer. From the result obtained, a unit metric characterizing performance can
be obtained. An example from smith, 1988 of three computers computational speed on two
benchmark programs of about 100 floating point operations is shown in the table below. The
computational rates are given in second.
Execution time of two programs on three different computers

Computer1 Computer2 Computer3

(Sec) (Sec) (Sec)
Benchmark
Prog1 100 10 5
Prog2 0.1 1 5

Arithmetic mean 50.05 5.5 5

Geometric mean 3.2 3.2 5
Harmonic mean 0.2 1.8 5

Explanations
1. Arithmetic mean: This is the sum of the rates divided by the number of programs.
E.g for Computer1: (100 + 0.1) /2 = 50.05, Computer2: (10 + 1)/2 = 5.5, Computer3: (5 +
5 / 2) = 5
2. Geometric mean: This is the nth root of the product of the rates. Where “n” is the total
number of benchmark programs. E.g for Computer1: √100 ∗ 0.1 = 3.2 , Computer2:
√10 ∗ 1 = 3.2, Computer3: √5 ∗ 5 = 5
3. Harmonic mean: This is the sum of the reciprocals of the rates and the result divided by
the number of programs. Then, the numerator and denominator are interchanged to get the
final result.

E.g for Computer1: 1/100 + 1/0.1 = 1001/100 * 1/2 = 1001/200 = 200/1001 = 0.2 sec
Computer2: 1/10 + 1/1 = 11/10 * 1/2 = 11/20 = 20/11 = 1.8 sec
Computer3: 1/5 + 1/5 = 2/5 * 1/2 = 2/10 = 1/5 = 5/1 = 5 sec

The result of this computation is used to summarize performance expressed as a rate. It is expected
to correspond to the execution time of real programs. One can write a matrix inversion in say
Pascal, C, or BASIC and then test the three programs on three hardware and configurationally
different computers. Using the statistical metrics of arithmetic, geometric, and harmonic mean to
determine the best of the three.

CPU Benchmarks

SPEC CPU2000 is the industry-standardized CPU-intensive benchmark suite. The System

Performance Evaluation Cooperative (SPEC) was founded in 1988 by a small number of
workstation vendors who realized that the marketplace was in desperate need of realistic,
standardized performance tests. The basic SPEC methodology is to provide the benchmarker with
a standardized suite of source code based upon existing applications that have already been ported
to a wide variety of platforms by its membership. The benchmarker then takes this source code,
compiling it for the system in question. The use of already accepted and ported source code
greatly reduces the problem of making apples-to-oranges comparisons SPEC designed CPU2000
to provide a comparative measure of computer intensive performance across the widest practical
range of hardware. The implementation resulted in source code benchmarks developed from
real user applications. These benchmarks measure the performance of the processor, memory
and compiler on the tested system. The suite contains 14 floating point programs written in
C/Fortran and 11 integer programs (10 written in C and 1 in C++). The SPEC CPU2000
benchmarks replace the SPEC89, SPEC92 and SPEC95 benchmarks.
The Java Grande Forum Benchmark suite consists of three groups of benchmarks, micro-
benchmarks that test individual low-level operations (eg: arithmetic, cast, create), Kernel
benchmarks which are the heart of the algorithms of commonly used applications (eg: heapsort,
encryption/decryption, FFT, Sparse matrix multiplication, etc), and applications (eg: Raytracer,
MonteCarlo simulation, Euler equation solution, Molecular dynamics, etc). These are computing
intensive benchmarks available in Java.

SciMark is a composite Java benchmark measuring the performance of numerical codes

occurring in scientific and engineering applications. It consists of five computational kernels:
FFT, Gauss-Seidel relaxation, Sparse matrix-multiply, Monte Carlo integration, and dense LU
factorization. These kernels are chosen to provide an indication of how well the underlying Java
Virtual Machines perform on applications utilizing these types of algorithms. The problems sizes
are purposely chosen to be small in order to isolate the effects of memory hierarchy and focus
on internal JVM/JIT and CPU issues. A larger version of the benchmark (SciMark 2.0 LARGE)
addresses performance of the memory subsystem with out-of-cache problem sizes.

Embedded and Media Benchmarks

EEMBC Benchmarks

The EDN Embedded Microprocessor Benchmark Consortium (EEMBC - pronounced embassy)

was formed in April 1997 to develop meaningful performance benchmarks for processors in
embedded applications. EEMBC is backed by the majority of the processor industry and has
therefore established itself as the industry-standard, embedded processor benchmarking forum.
EEMBC establishes benchmark standards and provides certified benchmarking results through
EEMBC Certification Labs (ECL) in Texas and California. The EEMBC's benchmarks comprise
a suite of benchmarks designed to reflect real- world applications, while it also includes some
synthetic benchmarks. These benchmarks target the automotive/industrial, consumer, networking,
office automation, and telecommunications markets. More specifically, these benchmarks target
specific applications that include engine control, digital cameras, printers, cellular phones,
modems, and similar devices with embedded microprocessors. The EEMBC consortium
dissected these applications and derived 37 individual algorithms that constitutes the EEMBC's
Version 1.0 suite of benchmarks.
BDTI Benchmarks

Berkeley Design Technology, Inc. (BDTI) is a technical services company that has focused
exclusively on Digital Signal Processing since 1991. BDTI provides the industry standard BDTI
Benchmarks™, a proprietary suite of DSP benchmarks. BDTI also develops custom benchmarks
to determine performance on specific applications The benchmarks contain DSP routines such as
FIR filter, IIR filter, FFT, dot- product, and Viterbi decoder.
MediaBench
The MediaBench benchmark suite consists of several applications belonging to image processing,
communications and DSP applications. Examples of applications that are included are JPEG,
MPEG, GSM, G.721 Voice compression, Ghostscript, ADPCM, etc. JPEG is the compression
program for images, MPEG involves encoding/decoding for video transmission, Ghostscript is
an interpreter for the Postscript language, and ADPCM is Adaptive differential pulse code
modulation. The Media Bench is an academic effort to assemble several media processing related
benchmarks.

Java Benchmarks

SPECjvm98 The SPECjvm98 suite consists of a set of programs intended to evaluate

performance for the combined hardware (CPU, cache, memory, and other platform-specific
performance) and software aspects (efficiency of JVM, the JIT compiler, and OS
implementations) of the JVM client platform [31]. SPECjvm98 uses common computing features
such as integer and floating-point operations, library calls and I/O, but does not include AWT
(window), networking, and graphics. Each benchmark can be run with three different input sizes
referred to as S1, S10 and S100. The 7 programs are compression/decompression (compress),
expert system (jess), database (db), Java compiler (javac), mpeg3 decoder (mpegaudio), raytracer
(mtrt) and a parser (jack).

SPECjbb2000 (Java Business Benchmark) is SPEC's first benchmark for evaluating the
performance of server-side Java. The benchmark emulates an electronic commerce workload in
a 3-tier system. The benchmark contains business logic and object manipulation, primarily
representing the activities of the middle tier in an actual business server. It models a wholesale
company with warehouses serving a number of districts. Customers initiate a set of operations
such as placing new orders and checking the status of existing orders. It is written in Java,
adapting a portable business-oriented benchmark called pBOB written by IBM. Although it is
a benchmark that emulates business transactions, it is very different from the Transaction
Processing Council (TPC) benchmarks. There are no actual clients, but they are replaced by
driver threads. Similarly, there is no actual database access. Data is stored as binary trees of
objects.
The CaffeineMark 2.5 is the latest in the series of CaffeineMark benchmarks. The
benchmark suite analyses Java system performance in eleven different areas, nine of which can
be run directly over the internet. It is almost the industry standard Java benchmark. The
CaffeineMark can be used for comparing appletviewers, interpreters and JIT compilers from
different vendors. The CaffeineMark benchmarks can also be used as a measure of Java
applet/application performance across platforms.

VolanoMark is a pure Java server benchmark with long-lasting network connections and
high thread counts. It can be divided into two parts: server and client, although they are provided
in one package. It is based on a commercial chat server application, the VolanoChat which is
used in several countries world-wide. The server accepts connections from the chat client. The
chat client simulates many chat rooms and many users in each chat room. The client
continuously sends messages to the server and waits for the server to broadcast the messages to
the users in the same chat room. VolanoMark creates two threads for each client connection.
VolanoMark can be used to test both speed and scalability of a system. In speed test, it is run in
an iterative fashion on a single machine. In scalability test, the server and client are run on
separate machines with high-speed network connection.

Transaction Processing Benchmarks

The Transaction Processing Council (TPC) is a non-profit corporation founded in 1988 to define
transaction processing and database benchmarks and to disseminate objective, verifiable TPC
performance data to the industry. The term transaction is often applied to a wide variety of
business and computer functions. Look at it as a computer function, a transaction could refer to
a set of operations including disk read/writes, operating system calls, or some form of data
transfer from one subsystem to another. TPC regards a transaction as commonly understood in
the business world: a commercial exchange of goods, services, or money. A typical transaction,
as defined by TPC, would include the updating to a database system for such things as inventory
control (goods), airline reservations (services), or banking (money). In these environments, a
number of customers or service representatives input and manage their transactions via a
terminal or desktop computer connected to a database. Typically, the TPC produces benchmarks
that measure transaction processing (TP) and database (DB) performance in terms of how many
transactions a given system and database can perform per unit of time, e.g., transactions per
second or transactions per minute. The TPC benchmarks can be classified into 2 categories, On-
Line Transaction Processing (OLTP) and Decision Support Systems (DSS). OLTP systems are
used in day-to-day business operations (airline reservations, banks), and are characterized by
 large number of clients who continually access and update small portions of the database
through short running transactions. Decision support systems are primarily used for business
analysis purposes, to understand business trends, and for guiding future business directions.
Information from the OLTP side of the business is periodically fed into the DSS database and
analyzed. DSS workloads are characterized by long running queries that are primarily read-only
and may span a large fraction of the database. There are four benchmarks that are active, TPC-
C, TPC-W, TPC-R and TPC-H. These benchmarks can be run with different data sizes, or scale
factors. In the smallest case (or scale factor =1), the data size is approximately 1 GB. The earlier
TPC benchmarks, namely TPC-A, TPC-B, and TPC-D have become obsolete.

TPC-C
TPC-C is an OLTP benchmark. It simulates a complete computing environment where a
population of users executes transactions against a database. The benchmark is centered around
the principal activities (transactions) of a business similar to that of a worldwide wholesale
supplier. The transactions include entering and delivering orders, recording payments, checking
the status of orders, and monitoring the level of stock at the warehouses. While the benchmark
portrays the activity of a wholesale supplier, TPC-C is not limited to the activity of any
particular business segment, but rather represents any industry that manages, sells, or distributes
a product or service. TPC-C involves a mix of five concurrent transactions of different types
and complexities, either executed on-line or queued for deferred execution. There are multiple
on-line terminal sessions. The benchmark can be configured to use any commercial database
system such as Oracle, DB2 (IBM) or Informix. Significant disk input and output are involved.
The databases consist of many tables with a wide variety of sizes, attributes, and relationships.
The queries result in contention on data accesses and updates. TPC-C performance is measured
in new-order transactions per minute. The primary metrics are the transaction rate (tpmC) and
price per transaction ($/tpmC).

TPC-H
The TPC Benchmark (TPC-H) is a decision support system (DSS) benchmark. It consists of a
suite of business-oriented ad-hoc queries and concurrent data modifications. The queries and the
data populating the database have been chosen to have broad industry-wide relevance. This
benchmark is modeled after decision support systems that examine large volumes of data, execute
queries with a high degree of complexity, and give answers to critical business questions. There
are 22 queries in the benchmark. The performance metric reported by TPC-H is called the TPC-
H Composite Query-per-Hour Performance Metric and the TPC-H Price/Performance metric.
One may not perform optimizations based on apriori knowledge of queries in TPC-H.
TPC-R

The TPC Benchmark R (TPC-R) is a decision support benchmark similar to TPC-H, but which
allows additional optimizations based on advance knowledge of the queries. It consists of a suite
of business-oriented queries and concurrent data modifications. As in TPC-H, there are 22
queries. The performance metric reported by TPC-R is called the TPC-R Composite Query-
per-Hour Performance Metric and the TPC-R Price/Performance metric.
TPC-W

TPC Benchmark W (TPC-W) is a transactional web benchmark. The workload simulates the
activities of a business oriented transactional web server in an electronic commerce environment.
It supports many of the features of the TPC-C benchmark and has several additional features
related to dynamic page generation with database access and updates. Multiple on-line browser
sessions and on-line transaction processing are supported. Contention on data accesses and updates
are modeled. The performance metric reported by TPC-W is the number of web interactions
processed per second (WIPS). Multiple web interactions are used to simulate the activity of a retail
store, and each interaction is subject to a response time constraint. Different profiles can be
simulated by varying the ratio of browsing and buying i.e. simulating customers who are primarily
browsing and those who are primarily shopping.

Web server Benchmarks

SPECweb99 is the SPEC benchmark for evaluating the performance of World Wide Web
Servers. It measures a system's ability to act as a web server. The initial effort from SPEC in
this direction was SPECweb96, but it contained only static workloads, meaning that the requests
were for simply downloading web pages that do not involve any computation. But if one examines
the use of the web, it is clear that many downloads involve computation to generate the
information the client is requesting. Such web pages are referred to as dynamic web pages.
SPECweb99 includes dynamic web pages. The file accesses are made to closely match today’s
real-world web server access patterns. The pages also contain dynamic ad rotation using cookies
and table lookups.

Mail server benchmarks

SPECmail2001 is a standardized mail server benchmark designed to measure a system’s ability
to act as a mail server servicing email requests. The benchmark characterizes throughput and
response time of a mail server system under test with realistic network connections, disk storage,
and client workloads. The benchmark focuses on the ISP as opposed to Enterprise class of mail
servers, with an overall user count in the range of approximately 10,000 to 1,000,000 users. The
goal is to enable objective comparisons of mail server products.
File Server Benchmarks

System File Server Version 2.0 (SFS 2.0) is SPEC’s benchmark for measuring NFS (Network
File System) file server performance across different vendor platforms. It contains a workload that
was developed based on a survey of more than 1,000 file servers in different application
environments.

PC Benchmarks

A variety of benchmarks are available, primarily from Ziff Davis, and Bapco to benchmark the
Windows based personal computer. Table 6 lists the most common PC benchmarks. Ziff Davis
Winstone and Bapco SYSMARK are benchmarks that measure overall performance while the
other benchmarks are intended to measure performance of one subsystem such as video or audio
or one aspect such as power.
Popular personal computer benchmarks
Benchmark Description
Business A system-level, application-based benchmark that measures a PC's
Winstone overall performance when running today's top-selling Windows-based
32-bit applications. It runs real 32-bit business applications through a
series of scripted activities and uses the time a PC takes to complete
those activities to produce its performance scores. The suite includes
five Microsoft Office 2000 applications (Access, Excel, FrontPage,
PowerPoint, and Word), Microsoft Project 98, Lotus Notes R5,
NicoMak WinZip, Norton AntiVirus, and Netscape Communicator.
WinBench99 A subsystem-level benchmark that measures the performance of a
PC’s graphics, disk, and video subsystems in a Windows environment.
3DwinBench Tests the bus used to carry information between the graphics
adapter and the processor subsystem. Hardware graphics adapters,
drivers, and enhancing technologies such as MMX/SSE are tested.
CD WinBench99 Measures the performance of a PC’s CD-ROM subsystem, which
includes the CD drive, controller, and driver, and the system processor.
Audio WinBench Measures the performance of a PC’s audio subsystem, which includes
99 the sound card and its driver, the processor, the DirectSound and
DirectSound 3D software, and the speakers.
Battery Mark Measures battery life on notebook computers.
I-bench A comprehensive, c r o s s -platform benchmark that tests the
performance and capability of Web clients. The benchmark provides a
series of tests that measure both how well the client handles features
and the degree to which network access speed affects performance.

Web Bench Measures Web server software performance by running different

Web server packages on the same server hardware or by running a
given W e b server package on different hardware platforms.
NetBench A portable benchmark program that measures how well a file server
handles file I/O requests from clients. NetBench reports throughput
and client response time measurements.
3Dmark MAX 99 From Futuremark Corporation. Is a nice 3D Benchmark which
measures 3D gaming performance. Results are dependent on CPU,
memory architecture, and the 3D Accelerator employed.
SYSMARK Measures a system’s real-world performance when running typical
business applications. This benchmark suite comprises the retail
versions of eight application programs and measures the speed with
which the system under test executes pre-determined scripts of user
tasks typically performed when using these applications. The
performance times of the individual applications are weighed and
combined into both category-based performance scores as well as a
single overall score. The application programs employed by SYSmark
32 are: Microsoft Word 7.0 and Lotus WordPro 96 for word
processing, Microsoft Excel 7.0 (for spreadsheet), Borland Paradox
7.0 (for database), CorelDraw 6.0 (for desktop graphics), Lotus
Freelance Graphics 96 and Microsoft Powerpoint 7.0 (for desktop
presentation) and Adobe Pagemaker 6.0 (for desktop publishing).

Techniques and tools for performance evaluation improve year by year. For instance, performance
monitoring counters were not available to the public until 1997. Benchmarks get updated
almost every year. Those interested in experimental performance evaluation should
continuously monitor the state-of-the-art.