Measurement System Analysis

Aathreya Consultants-VVI
 General Measurement System Guidelines

Introduction

Measurement data are used more often than before.

Example:

 Decision to adjust a manufacturing process or

not.

 To determine if a significant relationship exists

between two or more variables.
Aathreya Consultants-VVI
 Measurement Data

 Professional Management calls for taking decisions

based on facts, as expressed in terms of data
 The decision to adjust a process or not, is based on the
measured values of the output characteristic
 Benefits of using data based procedures largely depends
on the Quality of the measurement data
 The Quality of the measurement data is related to the
Statistical properties of multiple measurements from a
measurement system
 If all the measurements are close to a master value, the
Quality of data is said to be high
 If, instead, measurements are away from the master
value, the Quality of the data is low.
Aathreya Consultants-VVI
General Measurement System Guidelines

Quality of Measurement Data

The quality of measurement data is related to the

statistical properties of multiple measurements
obtained from a measurement system operating
under stable conditions.
If the measurements are all ‘close’ to the master
value for the characteristic, then the quality of
data is said to be ‘high’.
If some or all measurements are ‘far’ away from
master value, then the quality of data is said to
be ‘low’. Aathreya Consultants-VVI
General Measurement System Guidelines

The statistical properties most commonly used to

characterize the quality of data are

Bias

Variance

Aathreya Consultants-VVI
General Measurement System Guidelines

The Measurement Process

The term ‘measurement’ is defined as “the

assignment of numbers to material things to
represent the relations among them with respect
to particular properties”. – Eisenhart (1963)
The process of assigning the numbers is called
the measurement process, and the value
assigned is defined as the measurement value.
Thus, measurement process should be viewed as
a manufacturing process that produces
measurements (data) as their output.
Aathreya Consultants-VVI
 Measurement System

A measurement system consists of

 Measuring instrument(s)
 Accessories
 Inspector
 Environmental conditions
 Measured part

Measurement Process

Measurement can be considered as a process, similar to

a manufacturing process
Measurement is the assignment of numbers to material
things to represent the relations among them with
respect to particular Aathreya Consultants-VVI
properties.
 The parameters of the measurement process are
 Input : Component to be measured
 Process factors : Equipment, Inspector, etc.
 Output : Measured value

Thus, we can compare the measurement process to a

manufacturing process.

Manufacturing process Measurement Process

Manufacturing equipment / machine Inspection equipment
Machine accessories Inspection accessories
Operator Inspector
Raw material Component to be measured
Product output Measured value of the
characteristic
Aathreya Consultants-VVI
General Measurement System Guidelines
The Measurement Process
In order to effectively manage variation of any
process, there needs to be knowledge of:
What the process should be doing
What can go wrong
What the process is doing
Specifications and engineering requirements
define what the process should be doing.
The purpose of a Process Failure Model Analysis
(PFMEA) is to define the risk associated with
potential process failures and to propose
corrective action before these failures can occur.
The outcome of the PFMEA is transferred to the
Aathreya Consultants-VVI
control plan.
 General Measurement System Guidelines
The Measurement Process

General Process

Input Operation Output

Measurement Process

Process
to be Measurement Analysis Decision
Managed
Value

Aathreya Consultants-VVI
 Process Control
A Typical Process

People
Methods Identifying
Material PROCESS Product Improvement
Equipment Opportunities
Environment

Input Process / System Output

Accurate Measurement
using Measurement System Analysis
 New Process Acceptance

• Observed Process  Production Gauge

Variation Variation

 Actual Process
Variation
General Measurement System Guidelines

Terminology

Gage: Any device used to obtain measurements;

frequently used to refer specifically to the
devices used on the shop floor; including go/no-
go devices.
Measurement System: The collection of
operations, procedures, gages and other
equipment, software and personnel used to
assign a number to the characteristic being
measured, the complete process used to obtain
measurements.
Aathreya Consultants-VVI
 General Measurement System Guidelines
Statistical Properties of Measurement Systems
An ideal measurement system would produce only
‘correct’ measurement each time it was used.
Such a system is said to have zero bias, and zero variance.
All measurement systems must have the following
properties:
 The Measurement System must be in Statistical Control.
 Variability of the Measurement System must be small
compared with the manufacturing process variability.
 Variability must be small compared with the
specifications.
 The increments of measure must be small relative to the
smaller of eitherAathreya
the process variability or the
Consultants-VVI
specification limits.
 General Measurement System Guidelines
Standards

National Standard (Maintained by Government)

Primary Standard (Maintained by Government agency,

private company, an academic
institution)

Secondary Standard (Company standard maintained by a

company’s metrology department)

Working Standards (Production standards maintained

by production personnel)

Traceable standards is useful for minimizing conflicts

when there is lack of agreement between the
measurements of the producer and customer.
Aathreya Consultants-VVI
 General Measurement System Guidelines
Measurement System in Quality Cycle

Management Responsibility

Improvement Quality Planning

Measurement Performance

Measurement System Analysis: Appropriate statistical

studies shall be conducted to analyze the variation pattern
present in the results of each type of measuring and test
equipment system. This requirement shall apply to
measurement systems referred in the control plan. The
analytical methods and acceptance criteria used should
conform to those in the MSA
Aathreya reference manual (eg: bias,
Consultants-VVI

stability, linearity, repeatability & reproducibility studies.).

Procedures for assessing the Measurement Systems

Measurement Issues

Issues that need to be addressed:

Does the measurement system have adequate

discrimination?

Is the measurement system statistically stable

over time?
Are the statistical properties consistent over
the expected range and acceptable for process
analysis or control?
Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems

Philosophy Interest

Product Control Is the part in a specific

category ?

Process Control Is the process variation

stable and acceptable ?

Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems

Effects of Product Decisions

A wrong decision will sometimes be made
whenever any part of the above measurement
distribution overlaps a specification limit. For
example, a good part will sometimes be called
“bad” (type I error, producer’s risk or false alarm)
if:
LSL USL

or

Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems

And a bad part will sometimes be called “good”

(type II error, customer’s risk or miss rate) if:

LSL USL

or

Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems
That is, with respect to the specification limits,
the potential to make the wrong decision about
the part exists only when the measurement
system error intersects the specification limits.
This gives three distinct areas:
LSL USL

I II III II I

Target
Where:
I Bad parts will always be called bad
II Potential wrong decision can be made
III Good parts will always be called good
Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems

Measurement system studies can be used to

provide the following:

A criteria to accept new measuring equipment

A comparison of one measuring device against

another

A basis for evaluating a gage suspected of

being deficient

A comparison for measuring equipment before

and after repair
Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems
Useful Definitions:
Bias: Bias is the difference between the observed
average of measurements and the reference value.

Reference Observed
value average value
Bias

Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems

Useful Definitions:

Reference value: Also known as the accepted

reference value or master value is a value that
serves as an agreed upon reference for the
measured values. It can be determined by
averaging several measurements with a higher
level (eg., metrology or layout equipment) of
measuring equipment.

Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems
Useful Definitions:

Stability: Stability or drift is the total variation in

the measurements obtained with a measurement
system on the same master when measuring a
single characteristic over an extended time period.

Stability
Time 2

Time 1

Aathreya Consultants-VVI
Determining and Assessing
Stability Time
• Stability (or drift) is the
total variation in the
measurements obtained
with a measurement system
• on the same master or part when
measuring a single characteristic
• over an extended
time period

Reference Value
Procedures for assessing the Measurement Systems
Useful Definitions:
Linearity: Linearity is the difference in the bias values
through the expected operating range of the gage.
Reference Bias Reference Larger
value value bias

Observed average Observed average

Lower part of range Higher part of range
Bias
Observed No bias
average value

Reference value
Aathreya Consultants-VVI
Varying linear bias
Procedures for assessing the Measurement Systems
Useful Definitions:
Repeatability: It is the variation in measurements
obtained with one measurement instrument when
used several times by an appraiser while
measuring the identical characteristic on the same
part.

Repeatability
Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems
Useful Definitions:
Reproducibility: It is the variation in the average
of the measurements made by different appraisers
using the same measurement instrument when
measuring the identical characteristic on the same
part.
Operator B

Operator C
Operator A

Reproducibility
Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems
Useful Definitions:
GRR or Gage R&R: The combined estimate of
measurement system repeatability and
reproducibility. Measurement system capability;
depending on the method used, may or may not
include the effects of time.
Reference Value

A C B

Aathreya Consultants-VVI
GRR
Procedures for assessing the Measurement Systems
Useful Definitions:

Consistency: The degree of change of

repeatability over time. A consistent measurement
process is in statistical control with respect to
width (variability).
UCL

Average
Range

LCL
Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems

Useful Definitions:

Uniformity: The change in repeatability over the

normal operating range. Homogeneity of
repeatability.

Size 1 Size N

Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems
Useful Definitions:

System Variation: Measurement system variation

can be characterized as:

Capability: Variability in readings taken over a

short period of time.

Performance: Variability in readings taken over a

long period of time. Based on total variation.

Uncertainty: An estimated range of values about

the measured value in which the true value is
believed to be contained.
Aathreya Consultants-VVI
Procedures for assessing the Measurement Systems
Analysis of a Measurement System

The purpose is to better understand the sources

of variation that can influence the results
produced by the system.
The distribution to describe the Measurement
System Variation can be characterized by
1) Location
Bias
Stability
Linearity
2) Width or Spread
Repeatability Aathreya Consultants-VVI
Reproducibility
Procedures for assessing the Measurement Systems
Bias Study
To determine the measurement system’s BIAS at a
specific location in process range, it is necessary to
obtain an accepted reference value of a part.
This can usually be done with tool room or layout
inspection equipment.
Steps
1. Measure one of the sample parts precisely on tool
room or layout inspection equipment.
2. Have an appraiser measure the same part a
minimum of 10 times, using the gage being
evaluated.
3. Calculate the average of readings. The difference
between the reference value and the observed
Aathreya Consultants-VVI
average represents the measurement system’s bias.
Procedures for assessing the Measurement Systems
Bias Example
A sample of one part is measured ten times by one
appraiser. The values are listed below. The reference
value determined by layout inspection equipment is 0.80
mm and the process variation for the part is 0.70 mm.
X1 = 0.75 X6 = 0.80
X2 = 0.75 X7 = 0.75
X3 = 0.80 X8 = 0.75
X4 = 0.80 X9 = 0.75
X5 = 0.65 X10 = 0.70
The observed average is 7.5 / 10 = 0.75.
Bias = observed average – reference value
= 0.75 – 0.80
Aathreya Consultants-VVI
= - 0.05
 Analysis of Results – Numerical
n

x i
1) Compute the average of n readings X  i 1
n
2) Compute the repeatability standard deviation.
max  xi   min  xi 
 repeatability 
d 2*
where d2* is taken from table with g = 1 and m = n.

3) Determine the ‘t’ statistic for the bias

bias = observed average measurement – ref. value
b = r / n
t = bias / b
4) Bias is acceptable at the  level if zero falls within (1-
) confidence bounds around the bias value.
 d2 b
    d2 b
 
Bias   * tv ,1   zero  Bias   * tv ,1 
 d2 2
  d2 2


where d2, d2* are found from the table with g=1 and m=n
Aathreya Consultants-VVI
and t,1-/2) is found using t table.
If the bias is statistically non-zero, look for

error in master or reference value

worn instrument

instrument measuring the wrong characteristic

instrument not calibrated properly. Review

calibration procedure.

instrument used improperly by appraiser.

Review measurement instructions.
Aathreya Consultants-VVI
 Linearity Study
Conducting the Study

Linearity can be evaluated using the following

guidelines.

1) Select g  5 parts whose measurements cover

the operating range of the gage.
2) Have each part measured by layout inspection
to determine its reference value and to confirm
that the operating range of the subject gage is
encompassed.
3) Have each part measured m  10 times on the
subject gage by one of the operators who
normally use the gage.
√ Select the parts at random to minimize
Aathreya Consultants-VVI
appraiser ‘recall’ bias in the measurements.
 Analysis of Results – Graphical

1) Calculate the part bias for each measurement and the

bias average for each part.
biasi,j = xi,j – (reference value)
biasi = (mj=1biasi,j) / m
2) Plot the individual biases and the bias average with
respect to the reference values as a linear graph.
3) Calculate and plot the best fit line and the confidence
band of the line using the equations.
For the best fit line, use yt = axi + b
where xi – reference value
yi – bias average
and Aathreya Consultants-VVI
 Analysis of Results – Graphical (Contd…)
 1 
 x y   gm   
x y
a     s lo p e b  y  ax  intercept
1
 x 2  g m  x 
2

For a given x0, the  level confidence bands are

 1


x  x 
2 2
 1 
lower: b ax 0   t gm  2 ,1  
0
 s

 x  x 
2
 2
 gm 
  i  
 1


  
2 2
 1 x0  x 
upper: b ax 0   t gm  2 ,1    s
2  gm
 x  
2
  x 
  i  

where s  y i2  b  y i  a  xi y i
gm  2
4) Plot the “bias = 0” line and review the graph. For the
measurement system linearity to be acceptable, the
“bias = 0” line must be
Aathreya entirely within the confidence
Consultants-VVI
bands of the fitted line.
 Analysis of Results – Numerical

Here, the following two hypotheses are tested

1) H0 : a = 0 Slope = 0
a
Do not reject H0 if t   t gm2,1
  2
 s 
 2 

  xj  x 
 
2) H0 : b = 0 intercept (bias) = 0
b
Do not reject H0 if t   t gm2,1
 2  2
 1  x 
 gm
  
2
Aathreya  xj  x
 Consultants-VVI 
Gage Repeatability & Reproducibility
Study (Gage R&R Study)

For variable measuring equipment

Initial considerations

 Use the engineering judgement to assess the

variation factors
 Critical dimensions require more parts and/or
trials (repeat measurements)
 Bulky/heavy parts may dictate fewer samples and
more trials
 Appraisers should be selected from those who
normally operate the instrument
Aathreya Consultants-VVI
 Gage Repeatability & Reproducibility
Study (Gage R&R Study)

For variable measuring equipment

Initial considerations (Contd…)

 Sample parts must be selected from the process

and they should represent its entire operating
range
 Instrument must have a discrimination (resolution)
that allows at least one-tenth of the expected
process variation of the characteristic
 The measurement method is well defined
 Measurements should be made in a random order.
Aathreya Consultants-VVI
 Conducting the R&R Study

 Select ten parts from the process at random,

preferably one part from every day’s production

 Identify them with a serial number, than cannot be

observed by the appraisers

 Select two or three appraisers, who normally

operate the measuring system

 Measure the parts randomly so that all the parts

are measured by all the appraisers for two or three
times (trials)

 Record the observations

Aathreya Consultants-VVI
 Conducting the R&R Study

 Calculate the variations due to different factors,

according to the formulae given

 A measurement system with a R&R% of 10% or

less is considered good

 A measurement system with R&R% more than 30%

requires review and is considered not acceptable

 A measurement system with R&R% between 10%

to 30% may have to be evaluated for acceptance,
considering the criticality of the part.
Aathreya Consultants-VVI
 Table for recording R&R study observations
Operator Trial Parts Avera
ge
1 2 3 4 5 6 7 8 9 10
A 1
2
3

Avg Xa-bar
R Ra-bar
B 1
2
3

Avg Xbbar
R Rbbar
C 1
2
3

Avg Xc-bar
R Rc-bar
Part
Avg Aathreya Consultants-VVI Rp
Xp-bar
 Formulae

Xa-bar, Xb-bar, Xc-bar = Average of all the measurements

made by A, B & C, respectively
Ra-bar, Rb-bar, Rc-bar = Average of the range of the trials
made on 10 parts by A, B & C, respectively
Rp = Maximum part average – Minimum part average
R-bar = (Ra-bar + Rb-bar + Rc-bar) / No. of operators
X-bar Diff = Max X-bar – Min X-bar
UCLR = R-bar * D4 D4 = 3.27 for 2 trials; 2.58 for 3 trials
LCLR = R-bar * D3 D3 = 0 for both 2 & 3 trials

Homogenize the data for Range

Calculate R-bar, X-bar Aathreya

Diff and Rp only after homogenizing.
Consultants-VVI
 Formulae

Equipment Variation (EV) = R-bar * K1

(K1 = 0.8862 for 2 trials; 0.5908 for 3 trials)

Appraiser Variation (AV) = Sqrt{(X-bar Diff *K2)2 – [EV2/(nr)]}

(K2 = 0.7071 for 2 operators; 0.5231 for 3 operators)
(n : No. of parts; r : No. of trials)

Repeatability & Reproducibility (R&R) = Sqrt{EV2 + AV2}

Aathreya Consultants-VVI
 Formulae

Part Variation (PV) = Rp * K3

n K3 n K3 n K3

2 0.7071 5 0.403 8 0.3375

3 0.5231 6 0.3742 9 0.3249

4 0.4467 7 0.3534 10 0.3146

Total Variation (TV) = SQRT{R&R2 + PV2}

Aathreya Consultants-VVI
 Formulae

% EV = (EV / TV) * 100

% AV = (AV / TV) * 100

% R&R = (R&R / TV) * 100

% PV = (PV / TV) * 100

Aathreya Consultants-VVI
Measurement System Analysis
For attribute
data
?

Bad
Good
 Objectives

• Introduce Measurement Systems Analysis for

attribute data
• Discuss procedure for performing a
Measurement Systems Analysis for attribute
data
• Perform an exercise to practice the
methodology
Attribute (Discrete) Data Analysis
• Attribute data contains less information than
variables data, but sometimes it is all that is
available
• Therefore, we must be even more demanding
about the integrity of attribute measurement
systems
• The issue is the same – Can I rely on the data
coming from my measurement system?
 Attribute MSA Goals

• The goals are similar to MSA for variables data systems

• In an attribute MSA, we are getting similar information,
but in a different approach:
– %Agreement Within Appraisers (Repeatability)
– %Agreement Between Appraisers (Reproducibility)
– %Agreement with known standard (Accuracy)
– %Overall Screening Effectiveness (Overall reliability)
– Kappa score is an indicator of how much better the
measurement system is than random chance
 Attribute MSA Set-up
• Guidelines:
– 2-3 Trained Operators
– Proper Method
– Representative Samples
• Generally at least 30 units to measure
• Each unit is measured 2-3 times by each
operator
• Randomize the trials
 Study Sample Selection

• Need 30-50 samples which span the normal extremes of the process
– Approximately 80% of the samples should be from the ‘gray’ areas
– Remaining samples should be clearly good or clearly bad

Example: Visual classification of cookies as “good” or “bad”

– We collect a sample of 30 cookies
– 5 units will be clearly defective (broken, burned, with holes)
– 5 units will be clearly acceptable (everything correct)
– The remaining samples vary in the quantity and type of defects
(e.g. dark but not clearly burned, small dents at surface)
 Procedure for Performing an MSA
• Create your MSA worksheet
• Have operator 1 rate all samples once in random order
• Have operator 2 rate all samples once in random order
• Continue until all operators have measured all samples once
(this is Trial 1)
• Repeat measurement steps for the required number of trials
• Analyze results and determine follow-up action
 Creating a MSA Worksheet
Setup
• For setting up the table to have the correct Trial /
Operator / Sample
combinations you can use the Create Gage R&R
Worksheet
procedure as well for attribute data!
Creating a MSA Worksheet
Setup
 Attribute Data Example
• Open file MSA_attribute.MPJ, worksheet Attribute
R&R
 Minitab Attribute MSA
• Minitab has an Attribute GR&R Study that
statistically analyzes attribute measurement
systems.
 Minitab Attribute MSA

• We will only use the percentage

agreement data this time
 Graphical Output

• Operator 1 only agreed from Trial 1 to Trial 2 in 90% of the time

• Operator 2 agreed 100% of the time
• This is only Within Appraiser, not between
 Analytical Output
• We need to look in the Session Window for the Between Appraiser data
• Here we see that of all 20 appraisals completed, only 17 matched (85%)
• Notice the confidence intervals provided as well

Between Appraisers
Assessment Agreement
# Inspected # Matched Percent (%) 95.0% CI
20 17 85.0 ( 62.1, 96.8)
# Matched: All appraisers' assessments agree with each other.
Comparison to a Standard
• So far we have compared appraisers to themselves (repeatability)
and to one another
• If we do not compare the appraisers to a standard, we could give
ourselves a false sense of security in our measurement system
• We simply need to add a column to our data sheet indicating the
‘Known’ values for comparative purposes
• The output will change slightly – let’s take a look
Comparison to a Standard
• Open worksheet Known Response
• Design:
– 3 Appraisers
– 3 Trials
– 10 Samples
• Ratings are Good and Bad in column
Assess
• The Known Standard values are in
column Actual
Comparison to a Standard

• We will add the known

levels represented in the • Make sure that the graphs
‘Actual’ column to be included are
checked
 Graphical Analysis

• Within Appraiser agreement is like • Appraiser vs. Standard graph

repeatability. shows percent agreement
between
• Graph shows the 95% confidence the inspectors and the master
interval for the percent (%)
value or expert opinion.
agreement.
Analytical Output

Within Appraiser

Assessment Agreement

Appraiser # Inspected # Matched Percent (%) 95.0% CI

1 10 7 70.0 ( 34.8, 93.3)
2 10 5 50.0 ( 18.7, 81.3)
3 10 8 80.0 ( 44.4, 97.5)

# Matched: Appraiser agrees with him/herself across trials.

• Within Appraiser calculations are like repeatability

metrics
• Notice the confidence intervals for each appraiser!
Analytical Output
Each Appraiser vs Standard

Assessment Agreement

Appraiser # Inspected # Matched Percent (%) 95.0% CI

1 10 5 50.0 ( 18.7, 81.3)
2 10 4 40.0 ( 12.2, 73.8)
3 10 2 20.0 ( 2.5, 55.6)

# Matched: Appraiser's assessment across trials

agrees with standard.
• This indicates how often the inspector is
giving the correct answer
• We want 90% or better agreement
 Analytical Output

Between Appraisers

Assessment Agreement

# Inspected # Matched Percent (%) 95.0% CI

10 3 30.0 ( 6.7, 65.2)

# Matched: All appraisers' assessments agree with each other.

• Between Appraisers agreement is like

Reproducibility
 Analytical Output
All Appraisers vs Standard

Assessment Agreement

# Inspected # Matched Percent (%) 95.0% CI

10 2 20.0 ( 2.5, 55.6)

# Matched: All appraisers' assessments agree with the

known standard.
• All Appraisers vs Standard gives an indication
of the effectiveness of the Inspection Standards
 Kappa Techniques
• Used when a measurement system is needed to classify items in a
non-quantitative manner
• Kappa techniques treat all misclassifications equally
• Kappa techniques do not assume ratings are equally distributed
across the possible range

• Requirements for usage:

– Units to be measured are independent from one another
– Raters inspect and classify independently
– The rating categories are mutually exclusive and exhaustive
 Kappa
• Kappa (K) is defined as the proportion of agreement
between raters after agreement by chance has been
removed:

K= Pobserved - Pchance
1 - Pchance

where:
Pobserved = Proportion of units classified in which the raters
agreed
Pchance = Proportion of units for which one would expect
agreement by chance
Kappa Techniques- Roadmap
Select Raters

Select Parts

Measure Parts

Collect Data

Calculate Kappa

Make Decision
 Kappa Techniques
Example
• A Green Belt was tasked to reduce the number of defects created during the
labelling process
• After implementing some “low hanging fruit” improvements, the Green
Belt tracked (at Final Inspection) the proportion defective produced each
day over a months period
• During this data collection period, the data showed no significant
improvements to defect reduction
• During his project review, the Green Belt realized that he might be making
improvements, but his measurement system (Visual Inspection) may be
inadequate!
• In order to get a better understanding of the inspection process, the Black
Belt had two different inspectors independently perform an inspection of 10
samples
– The inspectors were asked to pass or fail each sample based on the
documented acceptance criteria
Kappa Techniques
Example

• Here are the results of Sample # Inspector 1 Inspector2

that study: 1 pass fail
2 fail pass
3 pass pass
4 fail fail
5 pass pass
What conclusion would 6 fail fail
you draw? 7 pass pass
8 fail pass
9 pass fail
10 fail fail
Kappa Techniques
Example
• Step 1: Convert the inspection data into
proportions and
put into a contingency table
Inspector 1
Pass Fail
Pass 3 (.30) 2 (.20) 5 (.50)
Inspector 2
Fail 2 (.20) 3 (.30) 5 (.50)
5 (.50) 5 (.50)

• Step 2: Calculate Pobserved

Pobserved = Sum of the probabilities on the diagonal
X1Inspector 1
Pobserved = X1 + X2 = 0.6
Pass X2
Fail

Pass
Inspector 2
Fail
Kappa Techniques
Example

• Step 3: Calculate Pchance

The probabilities for each classification for each
inspector multiplied, thenInspector
summed 1
Pass Fail

Pass X1 Y1 Z1 (.50)
Inspector 2
Fail Y2 X2 Z2 (.50)
Z3 (.50) Z4 (.50)

Pchance = (Z1 x Z3) + (Z2 x Z4) = 0.50

Step 4: Calculate K
K = (Pobserved - Pchance) / (1- Pchance) = 0.20
Kappa Techniques
Example

What might be said about his measurement

methods?
Kappa Value Suggested Interpretation
-1 to 0.0 Agreement expected by chance
=/> 0.60 Marginal - Significant effort required
=/> 0.70 Good - Improvement Warranted
=/> 0.90 Excellent

A Kappa value of +1 means Perfect Agreement

General Rule:
If K<0.70, the measurement system needs attention
 Minitab Approach

• Minitab can do this calculation for us

automatically
• We can identify that a column has Ordered
data in it, identifying a classification is
being applied
• Open worksheet Kappa Study
• What is shown are the original results and
the stacked results
 Minitab Approach
• We identify that the data in the ‘Results’ column are categorical and run
the analysis
• Make sure that the Kappa option is checked in the Results window
Minitab Kappa Output
Between Appraisers

Assessment Agreement

# Inspected # Matched Percent (%) 95.0% CI

10 6 60.0 ( 26.2, 87.8)

# Matched: All appraisers' assessments agree with each other.

Kappa Statistics

Response Kappa SE Kappa Z P(vs > 0)

fail 0.2000 0.3162 0.6325 0.264
pass 0.2000 0.3162 0.6325 0.264

* NOTE * Single trial within each appraiser. No percentage of

assessment agreement within appraiser is plotted.
Improving Attribute Measurement
Systems
• Sense multipliers (devices to improve
human senses)
• Masks / templates (block out unimportant
information)
• Checklists
• Automation
• Reorganization of work area
• Visual aids
 Summary
• Sometimes attribute data is all we have available
• Even measurement systems for attribute data need to
be evaluated before starting process improvements
• MSA techniques for attribute data are similar to
those for variable data
• Target is to evaluate the applicability of respective
measurement systems
• Kappa Techniques provide a method to quantify the
quality of attribute measurement systems
 Measurement Systems
Questions to Ask

• Have you picked the right measurement system?

• Is this measurement system associated with either critical
inputs or outputs?
• What is the measurement error?
• What are the sources of variation?
• What needs to be done to improve this system?
• Have we informed the right people of our results?
• Who owns this measurement system?
• Who owns troubleshooting?
 Measurement Systems
Questions to Ask

• Does this system have a control plan in

place?
• What’s the training frequency? Is that
frequent enough?
• Do identical systems match?
THANKS

Aathreya Consultants-VVI