Measurement Systems Analysis

If you can’t measure it, you can’t reliably improve it

Fully Crossed Designs, Youden Plot

Continuous Data, Stable Characteristics

 Experimental Procedure: Continuous Data
30 units that span the range of variation are selected.
Each unit is tested twice by each operator or by each
operator/gage combination. The test order must be random.
• The appraiser/equipment repeatability is calculated/plotted
• The appraiser reproducibility is calculated/plotted
• The system bias is calculated/plotted.

3 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Operator Repeatability
Each operator’s individual repeatability is calculated
first.

Estimates of reproducibility and bias are not reliable

until repeatability is established.

4 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 The Total Measurement Repeatability, e
Once each operator has passed individual repeatability,
a combined repeatability can be calculated.

e is calculated in the same way as with one operator.

The average range is calculated from the individual
ranges, Rij for each unit/operator set.

 Rij R
Rij = | XAij - XBij | R e 
nopk d2

5 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Reproducibility Error, O

𝑋𝐴𝑖𝑗 + 𝑋𝐵𝑖𝑗
The averages for each unit: 𝑋ത𝑖𝑗 =
2
σ 𝑋ത𝑖𝑗
The product average for each operator: 𝑋ധ𝑗 =
𝑘

The Range of Operator Product Averages 𝑅𝑂 = 𝑚𝑎𝑥 𝑋ധ𝑗 − 𝑚𝑖𝑛𝑋ധ𝑗

2
𝑅𝑂 𝜎𝑒2
The Reproducibility, o 𝜎𝑂 = −
𝑑2 𝑛𝑂

6 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Statistical Significance of Reproducibility
Since every operator will have some inherent
differences from other operators, it is essential to
determine if the differences are significant and
important.

In order to do this a small reproducibility decision chart

can be constructed. The product averages for each
operator are plotted against the following decision
limits:

7 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Operator to Operator Reproducibility
The Center Line = Grand Average of all of the part readings
X
X j

n op
The Upper & Lower Decision Limits, UDL:
 e 
X  3 
 nk 
These decision limits are derived from the hypothesis that
there is no difference between operators.
ˆ
x 
ˆ
n
If any values fall outside the decision limits, that operator(s)
is significantly different from the other operators.
8 © 2020 IDEXX Laboratories, Inc. All rights reserved.
 Example of Operator Reproducibility
An example of a reproducibility decision chart with a significant
difference between operators is:
Operator Reproducibility
4.9
4.8
Product Average

4.7
4.6
4.5
4.4
4.3
1 2 3 4
Operator

Remember, a statistically significant difference can exist between

operators but there may be no practical importance if the
reproducibility is a very small portion of the total variation.
9 © 2020 IDEXX Laboratories, Inc. All rights reserved.
 Repeatability & Reproducibility Statistics

Repeatability & Reproducibility, R&R: 𝜎𝑅&𝑅 = 𝜎𝑒2 + 𝜎02

The R&R Intraclass Correlation 𝜎𝑒2 + 𝜎02

Coefficient, rR&R: 𝜌𝑅&𝑅 =1− 2
𝜎𝑇 + 𝜎02

The percentage of the total variation which is due to

product variation is rR&R x 100.

The percentage of the total variation that is due to

R&R variation is (1-rR&R) x 100.

10 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 The Discrimination Ratio

The Discrimination Ratio, DR R&R

1 + 𝜌𝑅&𝑅
𝐷𝑅 𝑅&𝑅 =
1 − 𝜌𝑅&𝑅

11 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Caution on Reproducibility Calculations
System repeatability, e has meaning even when
multiple operators are used.
Operator reproducibility, o and the Reproducibility
R&R number has no real meaning.
Reproducibility errors occur only in conjunction with
repeatability errors. So we only calculate a total R&R
number…We need to use the “Control Chart” approach
to determine if there is a significant and important
difference between operators...

12 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Reproducibility: More Fake Math

The use of 3 operators is essentially a fixed effect.

If there is a difference between the operators it will be
systemic by definition.

Standard deviations can only be calculated for random

variation.

13 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Reproducibility Example
Repeatability & Reproducibility
All Operators
The first check is to ensure
25
that the individual operator
20 repeatability is acceptable.
Measurement B

15

10
The Youden plot at left shows
all operators.
5

0
0 5 10
Measurement A
15 20 25
The operator repeatability is
Operator 1
Operator 3
Operator 2
Operator 4
visually similar.
1:1 Line Specification

14 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Operator to Operator “Stability”
A quick analysis of the consistency of the measurement error of
each operator (Operator repeatability) is done using a Range
Control Chart.

Repeatability & Reproducibility

All Operators Measurement Variation: Stability
25
Range Control Chart
1.6
20
1.4
1.2
Average Range
Measurement B

15
1

10
0.8
0.6

5
0.4
0.2
0
0
0 5 10 15 20 25
Measurement A
Operator 1 Operator 2 Operator 3 Operator 4
Operator 1 Operator 2
Operator 3 Operator 4
1:1 Line Specification
There is no statistically significant
15 © 2020 IDEXX Laboratories, Inc. All rights reserved. difference in repeatability.
 Operator to Operator “Stability”
It is possible to fix both repeatability and any
systemic bias at the same time.

The mathematical rule is that any reproducibility

number is meaningless unless the operator
repeatability is consistent and acceptable.

16 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Operator to Operator Reproducibility
An ANOM chart is used to
determine the statistical
significance of the difference Operator Reproducibility: Bias
between operators. 11.4
11.2

Product Average
11.0
10.8

The mean of all measurements 10.6

10.4

for each operator is plotted 10.2

10.0

against decision limits. Points Operator 1 Operator 2 Operator 3 Operator 4

outside of the decision limits

indicate statistically different While the differences are statistically
significant, we must also assess the
operators. importance of the differences.

17 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Individual Operator Graphs
Always look at the individual Operator 1 Operator 2

operator Youden Plots to confirm 25 25

a practical difference. 20 20

15 15

Trial B

Trial B
10 10
Repeatability e = 0.55
5 5
Reproducibility o = 0.41
0 0
R&R R&R = 0.68 6.43% of the total variation
0 5 10 15 20 25 0 5 10 15 20 25

Total Variation T  2.66 Trial A Trial A

e = 1 e = 0
R&R
Dre == 6.4%
6.8 of the total Dr = 7.3 Dr = 7.7

observed
Dr R&R= variation
5.5
Operator 3 Operator 4
Dre = 6.8 25 25

Dr R&R= 5.5 20 20

15 15
Trial B

In this example, checking the

Trial B
10 10
individual operator Youden Plots, 5
5

the statistically significant 0

0 5 10 15 20 25 0

differences are not substantial and Trial A

0 5 10
Trial A
15 20 25

are not practically different. e = 1 e = 1

Dr = 6.2 Dr = 6.4
18 © 2020 IDEXX Laboratories, Inc. All rights reserved.
 Example: Supplier
vs. IDEXX IQA

Groovy VetTest Tip

A CTQ is the distance from
the ‘fins’ to the end of the
tip.
Fin to Tip
Length 3 supplier inspectors and 1
IQA inspector are involved in
the initial R&R study.

30 parts are measured twice

by each inspector.

19 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Results: Youden Plot
Repeatability & Reproducibility
All Operators
1.177

1.176

1.175
Measurement B

1.174

1.173

1.172

1.171

1.17

1.169
1.169 1.17 1.171 1.172 1.173 1.174 1.175 1.176 1.177
Measurement A

Inspector 1 Inspector 2
Inspector 3 IQA
1:1 Line Specification

20 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 Results: Measurement Error Stability
Repeatability & Reproducibility Measurement Variation: Stability
All Operators Range Control Chart
0.0012
1.177
0.001

Total Supplier

Average Range
1.176 0.0008

Measurement
1.175 Error 0.0006

0.0004
Measurement B

1.174 0.0002

0
1.173
Inspector 1 Inspector 2 Inspector 3 IQA

1.172

1.171 In this case, the statistical test is not

IQA Measurement
helpful: Visually it is obvious that the
1.17
Error
IQA inspector has much better
1.169
1.169 1.17 1.171 1.172 1.173 1.174 1.175 1.176 1.177 measurement error than the 3 supplier
Measurement A
inspectors. The Range chart is limited by
Inspector 1 Inspector 2 the small sample size which drives a zero
Inspector 3 IQA
1:1 Line Specification lower limit. The upper limit is driven by
the 3 supplier inspectors.
21 © 2020 IDEXX Laboratories, Inc. All rights reserved.
 Results: Reproducibility Bias
Operator Reproducibility: Bias
1.175
Repeatability & Reproducibility
All Operators 1.174

Product Average
1.177
Reproducibility 1.173

between
1.176
IQA and 1.172

the
1.175
Supplier 1.171
Inspector 1 Inspector 2 Inspector 3 IQA
Measurement B

1.174

1.173
The ANOM chart shows that the bias is
1.172
statistically significant with IQA reading
1.171 substantially lower than the supplier
1.17 inspectors.
1.169
1.169 1.17 1.171 1.172 1.173 1.174 1.175 1.176 1.177 The statistical analysis of this bias is
Measurement A
redundant to the visually obvious bias
Inspector 1 Inspector 2
Inspector 3 IQA
on the Youden plot.
1:1 Line Specification

22 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 “Plot your data
and think about
your data”

Ellis Ott

23 © 2020 IDEXX Laboratories, Inc. All rights reserved.

 The System Performance is not Acceptable
Repeatability & Reproducibility
All Operators
1.177

A this point the MSA obviously 1.176

1.175

fails. There is no need for

Measurement B
1.174

further quantification. 1.173

1.172

1.171

1.17

The graph points us to the best 1.169

1.169 1.17 1.171 1.172 1.173 1.174 1.175 1.176 1.177

clue to understanding what is Inspector 1

Measurement A

Inspector 2

wrong. Inspector 3
1:1 Line
IQA
Specification

What is different between

IQA and the supplier?
24 © 2020 IDEXX Laboratories, Inc. All rights reserved.
 Fixing the System
Observations were made of the
two inspector groups.

The supplier was measuring from a

knit line just above the fin edge Bottom of Knit line
Fin to Tip to Tip
with a visual comparator.

IQA used a fixture that located the

fins on a flat surface to measure How the tip
the length with a height gauge. functions
25 © 2020 IDEXX Laboratories, Inc. All rights reserved.
 After Fixing the System: The Youden Plot
Repeatability e = 0.00017
Repeatability & Reproducibility
All Operators Reproducibility o = 0.00016
1.177 R&R R&R = 0.00024 78.27% of the total variation
Total Variation T  0.00021
1.176

1.175 Dre = 1.4

Dr R&R= 1.2
Measurement B

1.174

1.173 The DR values are low because the product

1.172
variation is low. The tips only span about
30% of the tolerance range.
1.171

1.17 This is typical for injection molded parts or

1.169 any part fabricated with a tool that wears.
1.1690 1.1700 1.1710 1.1720 1.1730 1.1740 1.1750 1.1760 1.1770
Measurement A
If we extrapolate the measurement error
Inspector 1 Inspector 2
Inspector 3 IQA
across the tolerance we can see that the
1:1 Line Specification error is small relative to the tolerance and is
acceptable for use.
26 © 2020 IDEXX Laboratories, Inc. All rights reserved.
 After Fixing the System: Stability & Bias
Measurement Variation: Stability
Range Control Chart
0.0005

The stability chart demonstrates 0.0004

Average Range
0.0003
that the repeatability of each 0.0002

operator is nearly identical. 0.0001

0
Inspector 1 Inspector 2 Inspector 3 IQA

The bias chart demonstrates that Operator Reproducibility: Bias

1.1725
while there is a statistically

Product Average
significant bias between operators 1.1720

is very small: 0.0003” vs. a

tolerance range of 0.006”. 1.1715
Inspector 1 Inspector 2 Inspector 3 IQA

27 © 2020 IDEXX Laboratories, Inc. All rights reserved.