Performance Evaluation of Computer Systems
Class 1 - Introduction
Pedro Braconnot Velloso
NCA — USEEJ7
Who am I?
• Pedro Braconnot Velloso
• Associate professor
• Bureau 35.1.50
• Interests
• Anomaly Detection
• Wireless networks
• Data science
• Blockchain
• Security
AIML — USEEN6 2
Syllabus
• Part I — Queueing Theory
• Introduction
• Basic concepts of probability
• Discrete Random Variables
• Continuous Random Variables
• Expectation
• Conditional distribution and expectation
• Stochastic Process
• Discrete-Time Markov Chains
• Continuous-Time Markov Chains
• Network of queues
AIML — USEEN6 3
Syllabus
• Part II — Simulations
• Random number generation
• Generating Discrete random variables
• Generating Continuous random variables
• Discrete event simulation
• Statistical Analysis of simulated data
AIML — USEEN6 4
Bibliography
• Kishor S. Trivedi, "Probability and Statistics with Reliability, Queuing,
and Computer Science Applications,” Wiley, 2016
• Raj Jain, ”The Art of Computer Systems Performance Analysis:
Techniques for Experimental Design, Measurement, Simulation, and
Modeling," Wiley-Interscience, 1991
• Sheldon M. Ross, ”Introduction to Probability Models," 11th ed.,
Elsevier, 2014
AIML — USEEN6 5
Grades
• Components
• 2 Lab exercises (L1, L2,)
• ICIN paper presentation (P)
• Final exam (E)
• Final grade (FG)
AIML — USEEN6 6
Introduction
Performance evaluation
• Obtain or provide the highest performance at the lowest cost
• Computer systems users
• Designers
• Administrators
• Helps in determining how well a system is performing
• Finding any possible improvement
NCA — USEEJ7 8
Performance evaluation
• Due to the large number of different computer systems and applications
• There is no standard
• Measure of performance
• Measurement environment
• Technique
NCA — USEEJ7 9
Solving problems
• Specifying performance requirements
• Evaluating design alternatives
• Comparing two or more systems
• System tuning
• Determining the optimal values of a parameter
• Bottleneck identification
• Workload characterization
• Capacity planing
• Forecasting
NCA — USEEJ7 10
Compute system
• Any collection of
• Hardware
• CPU
• Software Focus on connected systems !
• Firmware
NCA — USEEJ7 11
Performance evaluation techniques
• Measurement
• Simulation
• Analytical modeling
NCA — USEEJ7 12
Basic terminolgy
• Metrics
• The criteria used to evaluate the performance of the system
• Response time
• Throughput
• Packet delivery ratio
• Workload
• Requests made by the user of the system
NCA — USEEJ7 13
Basic terminolgy
• Parameters
• An extensive list of system and workload characteristics
• Affect the performance of the system
• System parameter examples
• Network topology
• Mobility
• Workload parameter examples
• Number of users
• Request arrival patterns
NCA — USEEJ7 14
Basic terminolgy
• Factors
• Parameters that are varied in the study
• Some of the parameters might be fixed
• At their typical values
• Outliers
• Values that are too high or too low compared to a majority of values
NCA — USEEJ7 15
Goals
• Select appropriate
• Evaluation techniques
• Performance metrics
• Workloads
• Conduct performance measures correctly
• Load generator
• Monitor
NCA — USEEJ7 16
Goals
• Use proper statistic techniques to compare alternatives
• Simply comparing the average result of a number of repeated trials
• Does not lead to correct conclusions
• Design measurement and simulation experiments
• Provide the most information with the least effort
• Separate out the effect of individual factors
• Perform simulations correctly
NCA — USEEJ7 17
Goals
• Use simple queueing models to analyse the performance of systems
• System utilisation
• Average service time per query
• Number of queries completed during the observation interval
• Average number of jobs in the system
• Probability of number of jobs in the system being greater than 10
• 90 percentile response time
• 90 percentile waiting time
NCA — USEEJ7 18
The art of performance evaluation
• Successful evaluation cannot be performed mechanically
• Requires
• Intimate knowledge of the system being modeled
• Careful selection
• The methodology, workload and tools
NCA — USEEJ7 19
Common mistakes in performance evaluation
• Usually due to
• Simple oversights
• Misconceptions
• Lack of knowledge about performance evaluation techniques
NCA — USEEJ7 20
Common mistakes in performance evaluation
• No goal
• A flexible model
• Easily modified to solve different problems — does not exist!
• Solution
• Define a specific goal
• Metrics, workload, and methodology all depend upon
• Biased goals
• Implicit or explicit bias in stating the goals
• “Show that our system is better than the others”
• Comparing two systems in different conditions
NCA — USEEJ7 21
Common mistakes in performance evaluation
• Unsystematic approach
• Use a systematic approach to Identify a complete list of
• Goals, system parameters, factors, metrics, and workloads
• Analysis without understanding the problem
• “A problem well stated is half solved”
• Incorrect performance metrics
• Different metrics give different results
• Do not compare apples and bananas
NCA — USEEJ7 22
Common mistakes in performance evaluation
• Unrepresentative workload
• Workload should be representative of the actual usage of the system
• Wrong evaluation technique
• Adapting the problem to the technique one can solve better
• Instead of using the most appropriate technique for the problem
• Overlooking important parameters
• Make a complete list of parameters
NCA — USEEJ7 23
Common mistakes in performance evaluation
• Ignoring signi cant factors
• Not all parameters have an equal effect on the performance
• It is important to identify the one that when varied
• Make a significant impact on the results
• The choice of factors should be based on their relevance
• Not on the knowledge of the analyst
• Every attempt should be made
• Get realistic values for all relevant parameters
NCA — USEEJ7 24
fi
Common mistakes in performance evaluation
• Inappropriate experimental design
• Number of measurements and simulations
• Parameters values
• Inappropriate level of detail
• Avoid too narrow or too broad
• No analysis
• Avoid just presenting raw data and a bunch of graphs
• Erroneous analysis
• Too short simulations
NCA — USEEJ7 25
Common mistakes in performance evaluation
• Ignoring sensitivity analysis
• Ignoring error in the input
• Often the parameters of interest cannot be measured
• Measure another parameter to estimate the other one
• Introduce additional uncertainties
• Improper treatment of outliers
• If the outlier was not caused by a real system phenomenon
• It should be ignored
• Assuming no change in the future
NCA — USEEJ7 26
Common mistakes in performance evaluation
• Ignoring variability
• If variability is high
• The mean may be misleading
• Too complex analysis
• To achieve a specific result
• It is always better to have a simple model
• Avoid starting with a too complex model
• First get some results and insights
• Then introduce complications
NCA — USEEJ7 27
Common mistakes in performance evaluation
• Improper presentation of results
• Results must be explained and understandable
• Ignoring social aspects
• Social skills are very important to present the results
• Two different audiences
• Decision makers and academic peers
• Omitting assumptions and limitations
NCA — USEEJ7 28
A systematic approach
• State goals and define de system
• Choosing the system boundaries
• List services and outcomes
• Select metrics
• List parameters
• System and workload
• Select factors to study
• Start with a small number of factors
• Small number of levels
NCA — USEEJ7 29
A systematic approach
• Select evaluation technique
• Select workload
• A list of service request to the system
• Design experiments
• Offer maximum information with minimal effort
• Analyze and interpret data
• Present results
• Start over if necessary
NCA — USEEJ7 30
Exercise
• Choose a system for performance study. Briefly describe the system and list
• Services
• Performances metrics
• System parameters
• Workload parameters
• Factors and there ranges
• Evaluation technique
• Workload
• Justify your choices
NCA — USEEJ7 31
Selecting an evaluation technique
Criterion Analytical model Simulation Measurement
Stage Any Any Postprototype
Time required Small Medium Varies
Tools Analysts Computer languages Instrumentation
Accuracy Low Moderate Varies
Trade-off evaluation Easy Moderate Difficult
Cost Small Medium High
Saleabitlity Low Medium High
Source: Raj Jain “The art of computer systems performance analysis
NCA — USEEJ7 32
Selecting an evaluation technique
Criterion Analytical model Simulation Measurement
Stage Any Any Postprototype
Time required Small Medium Varies
Tools Analystis Computer languages Instrumentation
Accuracy Low Moderate Varies
Trade-off evaluation Easy Moderate Difficult
Cost Small Medium High
Saleabitlity Low Medium High
Source: Raj Jain “The art of computer systems performance analysis
NCA — USEEJ7 33
Selecting an evaluation technique
Criterion Analytical model Simulation Measurement
Stage Any Any Postprototype
Time required Small Medium Varies
Tools Analystis Computer languages Instrumentation
Accuracy Low Moderate Varies
Trade-off evaluation Easy Moderate Difficult
Cost Small Medium High
Saleabitlity Low Medium High
Source: Raj Jain “The art of computer systems performance analysis
NCA — USEEJ7 34
Selecting an evaluation technique
Criterion Analytical model Simulation Measurement
Stage Any Any Postprototype
Time required Small Medium Varies
Tools Analystis Computer languages Instrumentation
Accuracy Low Moderate Varies
Trade-off evaluation Easy Moderate Difficult
Cost Small Medium High
Saleabitlity Low Medium High
Source: Raj Jain “The art of computer systems performance analysis
NCA — USEEJ7 35
Model validation
• “Until validated, all evaluation results are suspect”
• It is important to validate the results of a model
• Using at least one of the two other techniques
NCA — USEEJ7 36
Combining different techniques
• Two or more techniques might be used sequentially
• A simple analytical model can be used
• Find the appropriate range for system parameters
• Simulations can be used later
• Study the performance in that range
• It reduces the number of simulation runs considerably
• A more productive use of resources
NCA — USEEJ7 37
Selecting performance metrics
• The system might provides different services
• List all the services
• For each service request
• Several outcomes
• Perform the service correctly
• Perform the service incorrectly
• Refuse to perform the service
NCA — USEEJ7 38
Selecting performance metrics
• Responsiveness
• Productivity
• Utilization
Speed
• Percentage of time the
resource is busy
Reliability
• The resource with the highest
utilisation is the bottleneck
Availability
Source: Raj Jain “The art of computer systems performance analysis
NCA — USEEJ7 39
Individual and global metrics
• Computer systems shared by many users
• Two types of performance metrics should be considered
• Individual
• Reflect the metric for each user
• Response time, throughput, and delay (can also be global)
• Global
• For the whole system
• Resource utilization, reliability, and availability
NCA — USEEJ7 40
Setting performance requirements
“The system should be both processing and memory ef cient. It should not create
excessive overhead.”
“There should be an extremely low probability that the network will duplicate a
packet, deliver a packet to the wrong destination, or change the data in a packet “
• SMART
• Specific
• Measurable
• Acceptable
• Realizable
• Thorough
NCA — USEEJ7 41 fi
Example
• Specifying the performance of a high-speed LAN
• Sending a frame from a source to a destination —> Outcomes
• Correctly deliver the frame
• Incorrectly deliver the frame
• Do not deliver the frame
NCA — USEEJ7 42
Example — requirements
• For packets that were correctly delivered
• Speed
• The access delay at any station —> less than 1 second
• Sustained throughput —> at least 100 Mbps
NCA — USEEJ7 43
Example — requirements
• 6 errors modes considered
• Reliability
• The probability of any bit being in error —> less than 10-7
• The probability of any frame being in error —> less than 1%
• The probability of a frame in error being delivered
• Without error indication —> less than 10-15
• The probability of a frame being misdelivered
• Due to an undetected error
• In the destination address —> less than 10-18
• The probability of duplicate frames —> less than 10-5
• The probability of losing a frame on the LAN —> less than 1%
NCA — USEEJ7 44
Exercises
• What methodology would you choose?
• To select a personal computer for yourself
• To select 1,000 workstations for your company
• Make a complete list of metrics to compare
• Two personal computers
• Two database systems
• Two disk drivers
NCA — USEEJ7 45
Performance Evaluation of Computer Systems
Class 1 - Introduction
Pedro Braconnot Velloso
NCA — USEEJ7