See discussions, stats, and author profiles for this publication at: [Link]

net/publication/346734912

Test Method Validation Risk Assessment and Mitigation Tools

Conference Paper · December 2020

CITATIONS READS

0 1,181

1 author:

Ron Snee
Snee Associates, LLC
447 PUBLICATIONS 9,535 CITATIONS

SEE PROFILE

All content following this page was uploaded by Ron Snee on 08 December 2020.

The user has requested enhancement of the downloaded file.

 Test Method Validation
Risk Assessment and Mitigation Tools
Ronald D. Snee, PhD
Snee Associates, LLC
Newark, Delaware USA

Laboratory University 2020

Sponsored by Knowledge Exchange Network
October 5 - 7, 2020
1
 About the Speaker He is also an Adjunct Professor in the
pharmaceutical programs at Temple and
Rutgers Universities. He worked at DuPont
for 24 years prior to starting his consulting
career.
Ron received his BA from Washington and
Jefferson College and MS and PhD degrees
from Rutgers University. He is an
academician in the International Academy for
Quality and Fellow of the American Society
of Quality (ASQ), American Statistical
Ron Snee, PhD is Founder of Snee
Association, and American Association for
Associates, LLC, a firm dedicated to the
the Advancement of Science.
successful implementation of process and
organizational improvement initiatives. He He is an Honorary Member of ASQ and has
provides guidance to pharmaceutical and been awarded ASQ’s Shewhart, Grant and
biotech senior executives in their pursuit Distinguished Service Medals, and ASA’s
of improved business performance using Deming Lecture and Dixon Consulting
Quality by Design, Lean Six Sigma and Awards. He is a frequent speaker and has
other improvement approaches that published 7 books and more than 330
produce bottom line results. papers in the fields of quality, performance
improvement, management, and statistics.
He has authored several articles on how to
He recently received the Institute of
successfully implement QbD, coauthored 3
Validation Technology’s Speaker of the Year
books on QbD tools and speaks regularly
Award.
at pharmaceutical and biotech conferences.

2
 Abstract
Pharmaceutical industry is experiencing a growing need to improve
performance driven by global competition and the increasing impact of
information technology. Central to this need is good data, a characteristic of
good science. Quality data are arguably more important today than ever
before. It is becoming widely accepted that data are a corporate asset. The
industry is also focusing on risk assessment and reduction as evidenced by
FDA’s focus on data integrity.
Methods and tools widely used in test method development are discussed
with emphasis on how these tools reduce risk in method development,
validation and use. Methods discussed include method repeatability and
reproducibility, method robustness and test method performance verification.
Methods for assessing that amount of product variation that can be attributed
to the manufacturing process, sampling procedures and test method are also
presented. A system for effectively using management review to increase the
sustainability of the risk-based approaches is also presented and illustrated
with pharmaceutical and biotech case studies and examples.

Using
3
 Session Outline
Part 1 - A Look at Today’s Realities
 Importance and need to improve measurement integrity and quality
 Need to Reduce Risk in method validation
 The Promise of QbD – Building Quality into Test Methods
Part 2 – Approaches for Reducing Test Method Risk
 Test method development using designed experiments
 Measurement quality - Repeatability and reproducibility
 Improving test method robustness
 Measurement process control – Continued Method Performance Verification
Part 3 - How to Troubleshoot Measurement Test Methods
 Is the variation due to sampling method or test method?
 Using designed experiments to quantify variation
 Using data to identify opportunities for improvement
 Graphical methods to visualize and understand variation

4
 Lab Results
Tell Us How Our Processes Are Performing
 Test labs produce the data that are used to:
 Develop products and processes
 Control and improve processes
 Assess product quality
 Improving test methods improves the quality of the
data used to design, control and improve products and
processes:
 Risk of poor quality product getting to patients and
ineffective and inefficient process performance is
reduced in the process

Improved Test Method Performance Reduces Risk

5
 Risks in Test Method Development

 Missing an important variable in method design

 Method produces poor quality measurements
 High variation; poor repeatability and reproducibility
 Method is not robust
 Results sensitive to variation in procedures, analysts,
reagents, equipment, etc.
 Method performance deteriorates over time
 Lack of management attention

6
 Identifying and Reducing Method Risk

Risk Tools to Assess and Reduce Risk

Missing Important Method Experimentation Strategy
Design Factors
Poor Quality Measurements Gage R&R Studies
Repeatability and Reproducibility
Method not Robust to Use Method Robustness Studies using
Factors Designed Experiments
Method Performance Continued Method Performance
Deterioration over Time Verification (CMPV) System –
Improve Method Stability
Lack of Management Make Management Review part of
Attention The CMPV System

7
 Variation Drives Risk,
Quality, Cost and Customer Satisfaction

It is All About Variation

“If I had to reduce my message for management to just a few
words, I'd say it all had to do with reducing variation”
W. Edwards Deming
8
 Quality by Design
Generic Definition

“A Systematic Process for

Building Quality into a Product
from the Inception to Final Output”
“Product” of Analytical Laboratory Processes:
Test results that have the desired precision and
accuracy delivered in a cost-effective and timely
manner compliant with GLPs

9
 Analytical Method QbD Building Blocks

Method
R&R Process
Control
Critical Quality Method
Attributes (Ys) Design Design
Space Risk
Critical Method Method Level
Parameters (Xs) Robustness

Raw
Materials
(Xs)
Failure Modes and Effects Analysis

Life Cycle Management and Continuous Improvement

Continued Method Performance Verification
10
Use of QbD in Test Method Development and Use
 Method Design
 Risk assessment using Failure Modes and Effects Analysis
 Create method design using design of experiments
 Meet method design performance criteria
 Method Evaluation
 Assess method repeatability and reproducibility (Gage R&R)
 Assess robustness of method design space (DOE)
 Assess sampling variation
 Method Control and Continued Method Performance Verification
 Monitor test method stability - repeatability and reproducibility
using control (reference) samples
 Continued Method Performance Verification using an integrated
system of method control and improvement techniques

11
 Measurement Is a Process

Y = f(X) Controlled Variables (Xs)

Cause (X) and • Test Time and Temperature
• Mixing Speed and Time
Effect (Y) • Reagent Concentrations
• Apparatus
Process Inputs (Xs) Process Outputs (Y)
• Raw Materials Test • Quality Measurements
• Instruments • Test Results on Time
Method
• Personnel • Cost Effective Operations

Environmental Variables (Xs)

• Ambient Temperature and Humidity
• Reagent Quality
• Analyst
• Day of Week
• Season of Year
• Shift 12
 Tablet Content Variation Study
Variation Due to:
Time Series Plot of % ACTIVE INGREDIENT
Raw Material Lot? No
5.3
Tablet Press? No
Analyst?
5.2
% ACTIVE INGREDIENT

5.1

5.0

4.9

4.8

4.7

4.6
1 12 24 36 48 60 72 84 96 108
Batch

13
Tablet Content Variation Study
Results Reported by Analysts A, B, C and D
Time Series Plot of % ACTIVE INGREDIENT
5.3
C
C

5.2
A
B
% ACTIVE INGREDIENT

A B
5.1 CC
B B B
A B B
A BA B B BB B B BBB BBB BB BB B B B
A A BB A B BBB B
5.0 B B BB BB B B B B B B B B B
AA B B B B D
B B C A B B
A B BB B
C B
A B A B B B A A D DD B
4.9 D
B D
A B B BB C
B B
4.8 A
A B CC
A A D
A
A
4.7
A A
A
A
Work of
A

4.6
Analyst B
1 12 24 36 48 60 72 84 96 108
Batch

14
 Tablet Content Variation Study
Analyst B Has the Least Amount of Variation
Dotplot of % ACTIVE INGREDIENT vs ANALYST

A
Work of
ANALYST

Analyst B
B
C
D
4.68 4.77 4.86 4.95 5.04 5.13 5.22
% ACTIVE INGREDIENT

15
Test Method Design
Experimentation Strategy

16
Test Method Design - Experimentation Strategy

Screening Optimization Robustness

Experiment Experiment Experiment

 Screening Experiment for 6 or more method variables

 Identify variables with largest effects
 Study selected variables in an optimization experiment
 Are often sufficient when the method variables and ranges
have been determined
 Optimization experiment – Identify method Design Space
 Variable settings that will give the best method performance
 Robustness Experiment – Assure that small variations in method
use do not affect the performance of the method

17
 Example – Test Method Development
 Screening Experiment
 11 Design Variables evaluated – Each at 2 levels
 Experiment Design
 24 run Plackett-Burman design in 4 blocks
 6 runs in each of 4 weeks
 4 critical variables were identified
 Optimization Experiment
 4 variables from screening experiment were optimized
 Experiment Design
 28-Run subset of a 3x3x2x3 factorial design
 Runs identified by computer-aided selection procedure enabling
estimation of main effects and two-factor interactions

18
 Example – Test Method Development
Results Summary
 Assay Method is robust to variation in the four principal
control parameters.
 Assay precision is 2-4% depending on the number of samples
tested and the number of tests made on each sample.
 Reaction time and reaction temperature had the largest effects
 Increasing reaction temperature significantly decreases
method variation
 Method design space can be expressed as a function of the
four principal control parameters
 Next Step: Method raw material optimization

19
 Reducing Test Method Risk
Assessing Measurement Quality
Repeatability and Reproducibility

20
 Sources of Process Variation
 Excessive variation in
process output results may Measurement
be due to the performance
of the:
 Manufacturing Process Manufacturing
 Sampling Method Sampling
 Measurement Method
 A combination of these
sources of variation

Gage R&R Studies Assess Measurement Variation

Test Method Repeatability and Reproducibility

21
 Reducing Test Method Risk
Improving Data Quality
 Pharmaceutical R&D and Operations are data driven
 Critical that measurements be of high quality
 Assess Test Method Repeatability and Reproducibility Using
Gage R&R Studies
 “Gage” is any device used to make measurements
 Study Variation Due To:
 Analysts and Instruments (Reproducibility)
 Repeat Tests (Repeatability)
 Gage R&R Studies Can Be Done:
 During Test Method Development
 Prior to release of new measurement device for use
 After Test Method Has been in Use
22
 Measurement Data
Sources of Variation

Total
Variation

Process or Measurement
Product System
Variation Variation

Accuracy Precision

23
 Measurement Data
Sources of Variation
Total
Variation
Gage R&R Studies
Process or Measurement Focus on
Product System Repeatability and
Variation Variation Reproducibility

Accuracy Precision

Repeatability Reproducibility

Measurement Variation (Precision)

= Repeatability + Reproducibility
24
 Repeatability and Reproducibility
Gage Repeatability Gage Reproducibility

Operator B

Operator C

Repeatability

Operator A

Reproducibility

Gage Repeatability is the Gage Reproducibility is the

variation in measurements variation in the average of
obtained when one operator measurements made by
uses the same gage several different operators using the
times to measure a sample same gage when measuring a
sample

25
 Gage R&R Study Design?
 Select N Samples that represent the full range of long-term
variation for your process
 Select K Analysts that customarily test the samples
 Each Analyst will measure each sample R Times
Analyst 1 Analyst 2 Analyst 3
Sample T1 T2 T1 T2 T1 T2
1
2
3 Example Design
4 • 10 Samples
• 3 Analysts
5
• 2 Tests by Each Analyst
6 on Each Sample
7 • 10x3x2=60 Test Results
8
9
10
26
 HPLC Gage R&R Study
Sampling Design
- 6 Samples
- 4 Analysts per Sample – Total of 24 sub-samples
- 4 Tests per sub-sample – Total of 96 tests

Sample Sample 1 …… Sample 6

Analyst A1 A2 A3 A4 …………..
A21 A22 A23 A24

Tests T1 T5 T9 T13 …………..

T81 T85 T89 T93
T2 T6 T10 T14 …………..
T82 T86 T90 T94
T3 T7 T11 T15 T83 T87 T91 T95
T4 T8 T12 T16 T84 T88 T92 T96

Adapted from Ahuga and Dong (2005) Handbook of Pharmaceutical Analysis by HPLC

27
 HPLC Gage R&R Study
Partial Data List - Samples 1 and 6
Sample Analyst Test Assay% Sample Analyst Test Assay%
1 1 1 99.3 6 1 1 98.1
1 1 2 99.4 6 1 2 98.4
1 1 3 99.9 6 1 3 97.2
1 1 4 99.4 6 1 4 97.1
1 2 1 99.5 6 2 1 96.4
1 2 2 99.2 6 2 2 97.9
1 2 3 98.6 6 2 3 94.4
1 2 4 101.6 6 2 4 97.3
1 3 1 97.8 6 3 1 95.4
1 3 2 97.4 6 3 2 96.1
1 3 3 98.1 6 3 3 96.9
1 3 4 99.1 6 3 4 96.4
1 4 1 97.9 6 4 1 95.9
1 4 2 98.3 6 4 2 95.2
1 4 3 99.1 6 4 3 97.2
1 4 4 98.9 6 4 4 96.8

28
 HPLC Gage R&R Study
Two-Way ANOVA Table

Source DF SS MS F P
Sample 5 187.501 37.5001 39.9815 0.000
Analyst 3 15.623 5.2076 5.5522 0.009
Sample*Analyst 15 14.069 0.9379 1.4339 0.155
Repeatability 72 47.097 0.6541
Total 95 264.290

P< .05 indicates significant effects

• Significant differences between
• Samples (as expected)
• Analysts
• Analyst differences are the same for all Samples
• Sample-Analyst interaction is not significant

29
 HPLC Gage R&R Study
Variance Components
Source VarComp % of Total
Total Gage R&R 0.90299 28.32
Repeatability 0.65413 20.52
Reproducibility 0.24885 7.81
Analyst 0.17790 5.58
Analyst*Sample 0.07095 2.23
Part-To-Part 2.28514 71.68
Total Variation 3.18812 100.00
• Total Gage R&R should be <30%
• Majority of Gage Variation is due to Test-to-Test

Number of Distinct Categories = 2

• Number of categories should be >4)

30
 80
% Study Var 100.0

Percent
HPLC Gage
40 R&R Study 97.5

Repeatability – Test-to-Test Variation 95.0

0
Gage R&R Repeat Reprod Part-to-Part 1

R Chart by Analyst
1 2 3 4
4 UCL=3.802
Sample Range 100.0

2 _ 97.5
R=1.667
95.0

0 LCL=0

Xbar Chart by Analyst

1 2 3 4
100.0
Control Chart for Range 100.0

• Repeatability
Sample Mean

is stable across Analysts

UCL=97.782

Average
97.5 _
_ 97.5
X=96.568
LCL=95.354 95.0
95.0

31
 HPLC Gage R&R Study
Reproducibility: Analyst–to–Analyst Variation

Analyst Assay %
1 97.19
2 96.66
3 96.25
4 96.18
Analyst Differences
Are Small
Range = 1.01

32
 Number of Distinct Categories
 Reflects measurement resolution
 Ability to detect differences between samples
 Number of distinct categories

>4 distinct data

 part to part  categories is
  1.41
  measurement 
desired

1/16 inch 1/10 inch

1/8 inch

33
Number of Distinct Categories
The number of distinct levels that can be measured by
the measuring device – also known as resolution
1 Distinct Category –
unacceptable for estimating
process parameters; only
indicates whether the process is
producing conforming or
nonconforming samples

2-4 Distinct Categories –

generally unacceptable for
estimating process parameter;
provides only coarse estimates

5 or more Distinct Categories –

acceptable for estimating
process parameters

34
 HPLC Gage R&R Study
Measurement Variation vs Total Variation

Measurement
Variation

Measurement Variation Takes up 63% of the Total Variation

Method Not Useful for Distinguishing Between Different Samples

35
 Thickness Measurement System
Number of Categories = 5 (Acceptable Resolution)

Measurement
Variation

Measurement Variation Takes up 23% of the Total Variation

Method Useful for Distinguishing Between Different Samples

36
Types of Operator Differences - Examples
Case A
Operator 2
Test-to-Test
Variation Is
Higher than
Other Operators

Case B
Operators
Do Not Agree
on Length of
Samples 6 and 10
Shortest and
Longest Length

37
 Improving Measurement Methods

 My measurement method is not adequate. The Gage R&R

statistics for repeatability and reproducibility are too large.
 Now what do I do?
 One possibility is to evaluate the measurement process /
procedure for ruggedness
 Effects of small variations in the how the method is used.
 There are other sources of variation in a measurement
method in addition to instruments and analysts which are
typically the subject of Gage R&R studies.

38
 Reducing Test Method Risk
Assess Test Method Robustness

39
 Robustness - An Underused Concept

 Key aspect of Statistical Thinking

 Reduce the effects of uncontrollable variation in:
 Product design
 Process design
 Management practices
 Anticipate variation and reduce its effects
 Design process/product to be insensitive to variation

Robust Processes and Products are

More Likely to Perform as Expected

40
 Robustness/Ruggedness of a Test Method

 Measurement procedure is “rugged,” if it is immune to

modest (and inevitable) departures from the conditions
specified in the method (Youden 1961)
 Youden demonstrated that a surprisingly small amount of
work is required to evaluate ruggedness when the
experiments are properly designed. The basic design is a
fractional factorial design typically involving 8, 12 or 16 tests
 Testing for Robustness
 Make small changes to the procedure
 Use two-level fractional factorial designs and Plackett-Burman
designs to measure the factor effects

41
 Method Robustness Example
Measurement of Water(%) in a Chemical Compound
Objective: How robust (insensitive) is the measurement
procedure to small changes in its execution
Procedure: Make a large batch of the compound carefully
mixed up to assure uniformity. Create 8 subsamples.
Change seven procedure factors and measure the % water.
Factor Base Condition Alternative Condition
– +
A – Distillation Rate 2 drops/second 6 drops/second
B – Reaction Time 0 minutes 15 minutes
C – Amount of Water 2 mL 5 mL
D – Aniline 12 mL 8 mL
E – n-Heptane 190 mL 210 mL
F – Distillation Time 45 minutes 90 minutes
G – Reagent new used

42
 2**(7-4) Experimental Design for
Testing Robustness
A B C D E F G % Water
– – – + + + – 18.80
+ – – – – + + 20.58
– + – – + – + 19.90
+ + – + – – – 18.03
– – + + – – + 19.50
+ – + – + – – 19.16
– + + – – + – 19.88
+ + + + + + + 19.85

An Equivalent Design is an
Eight Run Plackett-Burman Design

43
Moisture Test Method - Factor Effects
Effect = High - Low

Factor Effect %*
A -0.1150 -0.6
B -0.0950 -0.5
C 0.2700 1.4
D 0.8350 4.3
E -0.0700 -0.4
F 0.6300 3.2
G 0.9900 5.1
* Percent of Average=19.46
Factor Effects Are Small
Within 5.5% of Average Measurement
44
Moisture Test Method - Factor Effects

G=Reagent

D=Aniline

F=Distillation Time

45
 0.73 Moisture Test Method
Factor Effects
1.08
Aniline
Effect=0.84 Reagent
0.40 Effect=0.93

1.05 0.70
1.13

1.47 0.74
0.35 0.68
0.72 0.72
Distillation Time
Effect=0.63

46
Moisture Test Method - Factor Effects(%)

G=Reagent
D=Aniline
F=Distillation
Time

Factors D, F and G:
Have Largest Effects
Effects are < 5.5%

47
 Conclusion:
Moisture Test Method Robustness Experiment

 Factor effects are all less than 5.5 % of the average response
 Test Method
 Standard Deviation = 0.79
 Two Standard Deviation Limits: +/- 1.78
 Coefficient of Variation(RSD) = 4.0%
 Measurement process is relatively robust or rugged

48
 Interpreting Robustness Studies
Statistical Significance
 Metric: p-value
 Interpretation: p< 0.05 implies the effect is statistically
significant
Practical Significance
 Depends on the situation – Context and Objectives
 Is the effect large enough to warrant action – Change
your behavior?
Metric Calculation Typical Goal
% Effect Factor Effect/Average > 5-10% Depending
Measurement on Situation
Coefficient of Standard Deviation/Average < 5-10% Depending
Variation (Relative Measurement on the Situation
Standard Deviation)

49
 Test Method Ruggedness
Example – Dissolution Method Validation
 Objective: How robust (insensitive) is the dissolution
method to small changes in its execution.
 Procedure:
 Make a large batch of tablets to assure uniformity
 Vary 8 factors of the method using a 12-run Plackett-
Burman design
 Measure the dissolution time for each run

50
 Dissolution Method Validation
Ruggedness Test Variables

Variable (X) Low Level (-) High Level (+)

X1=Acid Conc (x1000) 9 11
X2=Polysorb Conc (x100) 4 6
X3=Stirring Speed (rpm) 45 55
X4=Temperature (deg C) 35 39
X5=Degasssing No Yes
X6=Filter Position Low High
X7=Operator A B
X8=Apparatus V W

Mixture of Quantitative and Qualitative Variables

51
 Dissolution Method Validation
Ruggedness Test Design (Plackett- Burman)

Poly- De- Oper- Appa- Disso

Run Acid sorb Stir Temp gas Filter ator ratus D1 D2 D3 Time
1 11 6 45 39 Yes High A V -1 1 -1 20.5
2 9 6 55 35 Yes High B V -1 -1 1 16.5
3 11 4 55 39 No High B W -1 -1 -1 16.7
4 9 6 45 39 Yes Low B W 1 -1 -1 19.3
5 9 4 55 35 Yes High A W 1 1 -1 14.7
6 9 4 45 39 No High B V 1 1 1 18.1
7 11 4 45 35 Yes Low B W -1 1 1 15.5
8 11 6 45 35 No High A W 1 -1 1 17.9
9 11 6 55 35 No Low B V 1 1 -1 16.5
10 9 6 55 39 No Low A W -1 1 1 19.1
11 11 4 55 39 Yes Low A V 1 -1 1 16.7
12 9 4 45 35 No Low A V -1 -1 -1 20.5

Adapted from Lewis, Mathieu and Phan-Tan-Luu (1999)

52
Dissolution Method Validation
Pareto Analysis of Method Variable Effects

Pareto Chart of the Effects

(response is Dissolution Time, Alpha = .05)
4.030
Stirring Speed
Temp
Polysorb Conc
Operator
Dummy 1
Term

Apparatus
Degassing
Acid Conc
Dummy 3
No Significant
Filter Position
Dummy 2
Effects
0 1 2 3 4
Effect
Lenth's PSE = 1.4

53
 Dissolution Method Validation
Analysis of Test Method Variable Effects
Low High
Variable (X) Level Level Effect %
Acid Conc 9 11 -0.73 -4.1
Polysorb Conc 4 6 1.27 7.2
Stirring Speed 45 55 -1.93 -10.9
Temperature 35 39 1.47 8.3
Degasssing No Yes -0.93 -5.3
Filter Position Low High -0.53 -3.0
Operator A B -1.13 -6.4
Apparatus V W -0.93 -5.3
Dummy 1 -1 1 -0.93 -5.3
Dummy 2 -1 1 -0.53 -3.0
Dummy 3 -1 1 -0.73 -4.1

Effect = Change in dissolution time when variable moves low to high

% = Effect relative to average dissolution time=17.7 min.

54
 Dissolution Test Method Robustness
Conclusion
 Method is rugged (robust)
 None of the factor effects is significant
 Observed method dissolution time
 Standard deviation is 1.9 min
 2 standard deviation limits +/- 3.8 min
 Coefficient of variation (RSD) 10.7%
 If variation is a concern consider tighter control on the
variables with the largest observed effects
Factor Low High Effect Pct
Polysorb Conc 4 6 1.27 7.2
Stirring Speed 45 55 -1.93 -10.9
Temperature 35 39 1.47 8.3

55
Method Robustness – Underlying Theory
 Departures from test method standard operating
procedures happen frequently
 Human variation is a significant cause
 Small variations in operating parameters have no
practical effect on performance
 Interactions are small or non-existent
 Additive main effect models are adequate
 Plackett-Burman and two-level fractional factorial
designs work well
 Small run size provides adequate sensitivity
 Typical experiment sizes are 8, 12 and 16 runs

56
 Reducing Test Method Risk
Process Variation is too High
Where is the Variation Coming From?

Measurement

Process?
Sampling Method?
Process
Analytical Lab? Sampling
Some or All of the Above?

57
 Chemical Analysis Test Variation

Runs were made by 3 operators to locate a source of

variation in a chemical analysis. A Run consisted of:
 Heat treating a specimen
 Performing a chemical analysis.
In the study:
 3 operators
 Each made 6 combustion runs and
 Titrated each run in duplicate.
 Total of 36 test results were obtained

Specimen is Consumed in the Analysis

“Nested” Gage R&R Study Will Be Used

58
 Chemical Analysis Test Variation
Operator 1 Operator 2 Operator 3
O R A Y O R A Y O R A Y
1 1 1 156 2 1 1 125 3 1 1 183
1 1 2 154 2 1 2 127 3 1 2 185
1 2 1 151 2 2 1 94 3 2 1 172
1 2 2 154 2 2 2 96 3 2 2 186
1 3 1 154 2 3 1 98 3 3 1 181
1 3 2 160 2 3 2 102 3 3 2 191
1 4 1 148 2 4 1 118 3 4 1 172
1 4 2 150 2 4 2 124 3 4 2 176
1 5 1 154 2 5 1 112 3 5 1 181
1 5 2 157 2 5 2 117 3 5 2 184
1 6 1 147 2 6 1 98 3 6 1 175
1 6 2 149 2 6 2 110 3 6 2 177

O = Operator
R = Run
A = Analysis
Y = Test Result
59
 Chemical Analysis Test Variation
Plot of Individual Test Results
Individual Value Plot of TEST RESULT
200

1 80
Operator 1
1 60 Operator 3
TEST RESULT

1 40
Operator 2
Large Operator Effects
1 20 Operator 2 Has:
• Lower Average
1 00 • More Run-to-Run Variation
Than Other 2 Operators
Run 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6
OPERATOR 1 2 3

60
 Data Structure Chemical Test
Analysis Analysis Analysis Variation
Operator Run 1 2
O1 R1 A1 A2
R2 A1 A2 Operator
R3 A1 A2
R4 A1 A2
R5 A1 A2
R6 A1 A2
O2 R1 A1 A2 R1 R2 R3 R4 R5 R6
R2 A1 A2
R3 A1 A2
R4 A1 A2 A1 A1 A1 A1 A1 A1
R5 A1 A2
A2 A2 A2 A2 A2 A2
R6 A1 A2
O2 R1 A1 A2
R2 A1 A2
Nested ANOVA
R3 A1 A2 Runs Nested in Operators
R4 A1 A2 Analyses Nested in Runs
R5 A1 A2
R6 A1 A2
61
 Test Method Control
Reduces Test Method Risk
Use Continued Method
Performance Verification

62
Use of Blind Controls (Reference Samples)
 Blind Control – Samples from a common source are
regularly submitted for analysis along with routine
production samples in a way that the analyst can not
determine the difference between the production samples
and the control samples
 “ The use of blind controls is rare in the pharmaceutical
industry. The roots of this probably lie in the compliance
aspects of the study. However there is no better way to
understand the true variability of the analytical method”

B. K Nunnally and J. S McConnell (2007)

Six Sigma in the Pharmaceutical Industry
Understanding, Reducing and Controlling Variation in Pharma and Biologics

63
Measurement of Potency (%) on a QC
Control Material

Potency (%) on a QC
Control Material

84 Rows of Data
42 QC Runs
2 Measurements / Run

VII-64
64
 Measurement of Potency (%) on a QC
Control Material (Continued)
Variables Control Chart
XBar of Potency
96.0
95.5
Poor Reproducibility
Mean of Potency

95.0 UCL=95.101

94.5
Avg=94.178
• Out-of-Control Runs
94.0
93.5
Detected.
93.0
LCL=93.254
• Process Is Not Stable;
92.5
• Long-Term Variation = 58%
3
6
9
12
15
18
21
24
27
30
33
36
39
42
Run
Note: The sigma was calculated using the range.

R of Potency

UCL=1.604
1.5 Good Repeatability
Range of Potency

1.0 • Within-Run Variation in

0.5
Control
Avg=0.491

0.0 LCL=0.000
3
6
9
12
15
18
21
24
27
30
33
36
39
42

Run VII-65
65
 Measurement of Potency (%) on a
QC Control Material
Nested ANOVA: Potency versus Run – 42 Runs, 2 Tests per Run

Analysis of Variance for Potency

Source DF SS MS F P
Run 41 27.7756 0.6775 3.795 0.000
Error 42 7.4978 0.1785
Total 83 35.2734
Long-Term Variation = 58%
Method Reproducibility is Poor
Variance Components
% of
Source Var Comp. Total StDev
Run 0.249 58.29 0.499
Error 0.179 41.71 0.423
Total 0.428 0.654

66
Assessing Measurement Process Stability
 Q: When Should I worry about measurement stability?
 A: When Long-Term Variation represents more than 20% of
the total variation
 Total Variation = Long-Term Variation (Reproducibility)
+ Short-Term Variation (Repeatability)

Measurement Process Stability Long-Term Variation

Not a Problem < 20%
May be A Problem 20 – 30%
Corrective Action May Needed? > 30%

 This guideline is based on the assumption that

 Well-controlled process will detect a shift of 1.5 short-
term standard deviations

Ref: Snee and Hoerl, Quality Progress, May 2012, 39-41

67
Assessing Measurement Process Stability
Example – Stable and Unstable Processes

Stable Process
• Good Reproducibility
• Long-Term
Variance = 21%

Un-Stable Process
• Poor Reproducibility
• Long-Term
Variance = 58%

68
Assessing Test Method Performance Over Time
Another Approach

Use Data from Product Stability Studies

 One or more batches of the product is tested at
various time points to assess the stability of the
product over time. In such studies the:
 Product is the same over time
 Method is the same over time
 After the “Time Trend” has been accounted for:
 The variation remaining is due to Test Method
“Repeatability” and “Reproducibility”

69
 Product Stability Data
Repeatability and Reproducibility
Scatterplot of Y vs Time(months)
53

52

51
Time
Trend Line
50

49
Y

Group
48 Mean
47 REPEATABILITY
Variation Around the
46 Group Mean REPRODUCIBILITY
Difference Between
45
Group Mean and
0 5 10 15
Trend Line 20
Time(months)

70
Use a Systematic Approach
Make Management Part of the
Approach

71
 Getting Management’s Attention
Developing A Systematic Approach for
Continued Method Performance Verification
Guiding Principle
If you want something to happen on a regular and
sustained basis you need to have a management
system in place to guide and sustain the effort
Management System
Framework of processes and procedures used to
ensure that an organization can fulfill all tasks
required to achieve its objectives. (Wikipedia 2011)

“What Get’s Measured,

and Paid Attention To,
Gets Done”

72
 Production and Test Method Monitoring System
Batch
Production
Over Time

Batch Process Process

Quality Data Adjustment Models
Y=f(X)

Monitoring Tools Periodic

• Control Charts Data Analysis Process
• Capability Analysis and Improvement
• Variance
Components Mgt Review
• Histograms
• Pareto Charts
Test Method Management Reviews Test
Performance Method Performance During
Data Production Monitoring Review
73
Periodic Reviews of Test Method Performance
A Team Sport

Review Team Timing

Process Operators Daily/Hourly
Area Managers and Staff Weekly
Site Managers and Staff Monthly
Business Manager and Staff Quarterly

Periodic Reviews Are

Critical to Success

74
 Identifying and Reducing Method Risk
Risk Tools to Assess and Reduce Risk
Missing Important Method
Design Factors Experimentation Strategy
Poor Quality Gage R&R Studies
Measurements Repeatability and Reproducibility
Method not Robust to Use Method Robustness Studies using
Factors Designed Experiments
Method Performance Continued Method Performance
Deterioration over Time Verification (CMPV) System –
Improve Method Stability
Lack of Management Make Management Review part of The
Attention CMPV System

75
 Test Method Development and Validation
Tips and Traps
 Think broadly about what variables can affect test method
performance – Consider using screening experiments
 Test Samples in Gage R&R studies should cover the range of
production
 Assess method in-use performance
 Define method resolution
 Use control (reference) samples to monitor method
performance over time – Consider use of product stability data
 Report test method coefficient of variation (RSD)

Plot the Data in All Aspects of

Exploration, Analysis and Communication of
Test Method Results
76
 QbD Techniques Reduce Risk
Improve Test Method Performance
 Use Screening, Optimization and Robustness studies
 An useful way to design effective test methods
 Measurement quality can be improved using Gage
Repeatability and Reproducibility studies
 Creating robust measurement systems using statistical
design of experiments
 Measurement stability can be controlled using Control
samples and Statistical Process Control techniques
 Product variation: Be sure to separate sampling and
process variation from test method variation

Risk is Reduced when

Test Method Performance is
Stable, Capable and Robust
77
References – Improving Measurement Systems
Borman, P., M., etal (2007) “Application of Quality by Design to Analytical
Methods”, Pharmaceutical Technology, October 2007, 142-152.
Montgomery, D. C. (2009), Design and Analysis of Experiments,7th Edition,
John Wiley and Sons, New York, NY, Chapter 13.
Schweitzer, M., etal (2010) “Implications and Opportunities of Applying QbD
Principles to Analytical Measurements”, Pharma Tech, Feb 2010, 52-59.
Snee, R. D. (1983), “Graphical Analysis of Process Variation Studies”, Journal
Quality Technology, 15, 76-88.
Snee, R. D. (2005) “Are We Making Decisions in a Fog? The Measurement
Process Must Be Continually Measured, Monitored and Improved”, Quality
Progress, December 2005, 75-77
Snee, R. D. & R. W. Hoerl (2012) “Going on Feel: Monitor and Improve Process
Stability to Make Customers Happy”, Quality Progress, May 2012, 39-41
Snee, R. D. (2015) “Management Holds the Key to Continued Process
Verification”, Pharmaceutical Manufacturing, January/February 2015, 33-35.
Snee, R. D. (2019) “Risk-Based Test Method Development, Validation and Life
Cycle”, American Pharmaceutical Review, July/August 2019, 64-67.
Snee, R. D. (2020) Risk-Based Continued Method Performance Verification
System”, Pharmaceutical Engineering, to appear.
Weitzel, J., R. A. Forbes and R. D. Snee (2015) “Use of the Analytical Target
Profile in the Lifecycle of an Analytical Procedure: with an example for an
HPLC Procedure”, J. of Validation Technology, Vol. 20, Issue 4, Jan 2015.

78
 For Further Information,
Please Contact:
Ronald D. Snee, PhD
Snee Associates, LLC
Newark, DE
(610) 213-5595
Ron@[Link]

Please visit our website at:

[Link]

View publication stats 79