Chapter 3 Study Guide
Selection of Techniques and Metrics
A simple and professional explanation of the whole chapter
What this file gives you
This guide reorganizes the chapter into clear ideas, short explanations, practical
examples, and end-of-chapter revision points. It is written for studying, not just reading.
How to choose the right evaluation
Main chapter focus technique and the right performance
metrics
Selection criteria, validation rules, common
Core outputs you should remember metrics, utility classes, and SMART
requirements
1. Chapter overview
The chapter explains that good performance evaluation is not only about calculating
numbers. First, you must choose a suitable evaluation technique. Then you must choose
metrics that truly describe the service, the errors, and the resource cost. Finally, you must
write performance requirements in a measurable way.
How to choose between analytical modeling, simulation, and measurement.
Why validation is essential before trusting any result.
How to select complete, non-redundant, low-variability metrics.
The meaning of common metrics such as response time, throughput, efficiency, utilization,
reliability, and availability.
How to convert vague statements into SMART requirements.
2. Criteria for selecting an evaluation technique
The chapter compares three major techniques: analytical modeling, simulation, and
measurement. The best choice depends mainly on the life-cycle stage of the system, then
on practical limits such as time, tools, cost, and required accuracy.
Analytical
Criterion Simulation Measurement
modeling
Mostly after a
Stage Any stage Any stage
prototype exists
Time required Small Medium Varies
Computer
Analysts and Instrumentation and
Tools languages /
mathematics real test setup
simulators
Accuracy Low Moderate Varies
Trade-off
Easy Moderate Difficult
evaluation
Page 1
Cost Small Medium High
Saleability of
Low Medium High
results
How to choose quickly
Use analytical modeling when you need a fast early estimate. Use simulation when the
system is too complex for a clean formula but not yet available for real measurement.
Use measurement when a real implementation or prototype exists and you want
realistic evidence.
3. The three rules of validation
One of the strongest ideas in the chapter is that no single technique should be trusted
alone. Each technique should be checked by another technique whenever possible.
1. Do not trust analytical-model results until they are validated by simulation or
measurement.
2. Do not trust simulation results until they are validated by analytical modeling or
measurement.
3. Do not trust measurement results until they are validated by simulation or analytical
modeling.
In simple words, validation means: compare one view of the system with another
independent view. If the conclusions match, your confidence becomes much stronger.
Simple example
If your simulation says the average delay is low, but real measurements show a much
larger delay, you should not immediately trust the simulation. You should check the
workload assumptions, the model details, and the measurement setup.
4. Selecting performance metrics
A metric is useful only if it truly describes the performance the user or designer cares
about. The chapter recommends starting from the service itself: what does the system do
correctly, what can go wrong, how long it takes, how fast it works, and what resources it
consumes.
Include time-related metrics: response time, delay, turnaround time.
Include rate-related metrics: throughput, bit rate, packets per second, transactions per
second.
Include resource metrics: CPU time, memory use, buffer consumption, bandwidth use.
Include error-related metrics: error rate, loss probability, duplication probability,
disconnect probability.
When relevant, include both the mean and the variability.
Think at both the individual level and the global/system level.
4.1 Metric selection criteria
Criterion Meaning Why it matters
The metric should not
Stable metrics are easier to
Low variability fluctuate wildly for the same
compare and interpret
condition
Do not keep two metrics that This avoids wasting effort
Non-redundancy
tell almost the same story and confusing the analysis
Completeness The final metric set should A good study must not ignore
Page 2
cover all important outcomes major failures or costs
4.2 Case study: two congestion-control algorithms
The chapter uses a communication-system example to show how metrics are chosen from
the service and its failure modes. The service is to send packets from a source to a
destination in order.
Outcome / concern Useful metric
Response time, throughput, and resource
Packets delivered in order
time per packet
Out-of-order arrivals Probability of out-of-order packets
Duplicate packets Probability of duplicate packets
Lost packets Probability of lost packets
Too much loss leading to broken session Probability of disconnect
Shared resource among flows Fairness
The chapter also notes that some metrics can be redundant. For example, if throughput
and delay are already captured together through power, keeping both may not add much
value for the final comparison.
Exam note
When asked to choose metrics, start with the service, then list correct outcomes,
incorrect outcomes, and resource usage. This is the safest way to build a complete
answer.
5. Commonly used performance metrics
5.1 Response time and reaction time
Response time is the time from the user request until the system response is available.
Reaction time is often a shorter interval: from the end of user input until the system starts
responding. Different books may define response time slightly differently, so always state
your definition clearly.
Term Simple meaning
Time from request to response (or to
Response time completed response, depending on the
definition used)
Time until the system starts reacting after
Reaction time
the request is ready
Time from submitting a batch job until its
Turnaround time
output is completed
5.2 Capacity
Capacity is not just one number. The chapter distinguishes between nominal capacity,
usable capacity, and knee capacity. This distinction is important because the largest
throughput is not always the best operating point if response time becomes unacceptable.
Page 3
Figure: Capacity concepts from the chapter slides.
Metric Meaning
Maximum achievable throughput under ideal
Nominal capacity
conditions
Maximum throughput that still respects a
Usable capacity
response-time limit
A balanced operating region with low
Knee capacity
response time and high throughput
A common exam explanation is this: before the knee, load increases and throughput rises
efficiently. Near the knee, the system still performs well. After that point, response time
grows rapidly and the operating point becomes much less attractive.
Page 4
5.3 Throughput, stretch factor, efficiency, utilization, reliability, and availability
Metric Definition / interpretation
Requests, packets, bits, or transactions
Throughput
completed per unit time
Response time with multiprogramming
Stretch factor divided by response time without
multiprogramming
Useful performance compared with ideal or
Efficiency
nominal performance
Utilization Fraction of time a resource is busy
How often errors occur, or how long the
Reliability
system operates without error
Fraction of total time the system is
Availability
operational and usable
For availability, the chapter highlights the standard relationship:
Availability = MTTF / (MTTF + MTTR)
This means that a system becomes more available when it fails less often (high MTTF) and
is repaired faster (low MTTR).
6. Utility classification of metrics
The chapter groups metrics according to how utility changes as the metric changes. This
is a very helpful interpretation tool when you compare designs.
Figure: Three utility classes shown in the chapter.
Class Meaning Examples
Smaller values mean higher Delay, error rate, loss
Lower is better
utility probability, repair time
Larger values mean higher Throughput, capacity,
Higher is better
utility availability
Temperature, utilization
A target range or target point
Nominal is best target, frame size target in
is best
some applications
Page 5
Why this matters
Before comparing alternatives, first ask: should this metric be minimized, maximized, or
kept near a target value? This prevents wrong conclusions.
7. Setting performance requirements
The chapter criticizes vague statements such as “the system should be efficient” or
“errors should be extremely low.” Such statements sound good, but they are not sufficient
for engineering or evaluation.
Bad requirement style Why it is weak Improved direction
“The system should be Not specific and not State a maximum delay or
fast.” measurable minimum throughput
Define a numeric CPU,
“Overhead should be
No threshold is given memory, or bandwidth
low.”
budget
State an acceptable
“Errors should be
The phrase is subjective probability, such as less than
extremely rare.”
1E-7
The chapter says requirements should be SMART:
Specific: clearly state what is being measured.
Measurable: use a metric with a numerical threshold or target.
Acceptable: the requirement should make sense for the stakeholders.
Realizable: it must be achievable with the available technology and constraints.
Thorough: the set of requirements should cover all major aspects, not only one.
7.1 LAN case study from the chapter
The local-area-network example shows how SMART requirements look in practice. The
service is to send a frame to its destination. Then the chapter converts speed, reliability,
and availability into measurable limits.
Requirement area Examples from the chapter
Access delay < 1 second; sustained
Speed
throughput >= 80 Mbits/sec
Bit error probability < 1E-7; duplicate
Reliability probability < 1E-5; frame loss probability <
1%
Mean time to initialize < 15 ms; mean time
Availability / recovery
to repair < 1 hour
This example is important because it shows the correct style of writing requirements: each
statement uses a metric and a clear limit.
8. Solved exercise 3.1
The chapter asks which methodology should be chosen in several situations. The safest
answer is to connect each situation with the stage of the system, the time available, and
the cost of evaluation.
Exercise Best methodology Why
Select a personal Measurement / The products already exist,
computer for yourself benchmarking and direct tests are realistic
Page 6
and practical
You need realistic evidence,
Select 1000 workstations Combination: measurement but also trade-off analysis for
for your company + analytical / cost study cost, manageability, and
scale
Both are finished products;
Compare two spreadsheet
Measurement direct comparison is the most
packages
convincing approach
Compare two data-flow
Speed is the priority, so a
architectures if the
Analytical modeling rough early estimate is
answer was needed
better than no answer
yesterday
Compare them if the You have enough time for a
answer is needed next Simulation more realistic study before
quarter hardware exists
Compare them if the Measurement, if prototypes Longer time allows more
answer is needed next exist; otherwise simulation accurate evidence and
year plus validation validation
9. What to memorize for the exam
Topic Must-remember idea
Life-cycle stage is the key factor; then
Selecting technique consider time, tools, cost, accuracy, trade-off
analysis, and saleability
Never trust only one technique; validate by
Validation
another method
Cover time, rate, resource, errors, and
Metric selection outcomes; prefer low-variability, non-
redundant, complete metrics
Know the difference between nominal,
Capacity
usable, and knee capacity
Decide whether lower, higher, or nominal
Utility class
values are better
Write SMART requirements with real metrics
Requirements
and numeric limits
10. Self-test with short answers
Q: Why is measurement not always the best first choice?
A: Because measurement usually needs a real or prototype system, may cost more, and
can make trade-off studies harder.
Q: Why is validation important?
A: Because one technique alone can be misleading or wrong; validation increases
confidence in the result.
Q: Why is “the system should be efficient” a weak requirement?
A: Because it is not specific or measurable; it does not say which metric or threshold
should be checked.
Q: Give one example of a “lower is better” metric.
A: Delay, packet loss probability, or mean time to repair.
Q: Give one example of a “higher is better” metric.
Page 7
A: Throughput, availability, or usable capacity.
Q: What is the difference between nominal and usable capacity?
A: Nominal capacity is the best throughput under ideal conditions; usable capacity is the
throughput that still satisfies a response-time limit.
11. Final summary
This chapter teaches a systematic way of thinking. First, choose the evaluation technique
that matches the stage and the purpose. Second, validate results using another method.
Third, choose metrics that fully describe correct service, failure cases, and resource
usage. Fourth, interpret each metric correctly through its utility class. Finally, write SMART
performance requirements so the system can be judged clearly and fairly.
Reference
Prepared from Chapter 3: “Selection of Techniques and Metrics” in the uploaded course
material.
Page 8