Quality Today
Are you controlling quality,
or just running controls?
James O. Westgard
University of Wisconsin Medical School
Westgard QC, Inc.
Madison, WI
[Link]
What’s the quality of quality
control?
Do the right QC right
Definition of quality for QC itself
1st right means right control rules and right
number of control measurements (N)
Need to selection appropriate QC procedures
2nd right means implementing QC correctly
Need to apply QC correctly
Calculations, interpretation, corrective actions
1
Doing right QC right!
Individualize QC for each test
Select appropriate rules and Ns
Use general selection guidelines
Use specific QC planning process
Computerize the implementation of rules and
data interpretation
Monitor bias via peer comparison and/or
proficiency testing programs
Update QC design when necessary
How implement right QC right?
Select appropriate control materials
Good practice to have at least one control material
independent of manufacturer of test system
Define when controls are analyzed
What is an analytical run?
Establish your own control limits
Calculate your own means & SDs from your own
laboratory data
Interpret control data correctly
Take action for “out-of-control” situations
2
Where find guidance?
General quality management guidelines
ISO 15189
Medical laboratories – Particular requirements
for quality and competence
Specific QC guidelines
CLSI C24-A3
Statistical Quality Control for Quantitative
Measurement Procedures: Principles and
Definitions – 3rd edition
ISO 15189, Section 5.5
Examination procedures
Performance specifications for each
procedure used in an examination shall
relate to the intended use of that
procedure
What quality is needed for the intended
clinical use of the test?
What precision and accuracy are required for
the measurement procedure to achieve the
quality needed for clinical use?
3
ISO 15189, Section 5.6
Assuring the quality of examination
procedures
The laboratory shall design internal quality
control systems that verify the attainment
of the intended quality of results.
It is important that the control system
provide staff members with clear and
easily understood information on which to
base technical and medical decisions.
What’s the point?
Must define quality required for intended use of
a test!
How do that?
What formats are used in defining quality goals or
requirements?
What information is available for guidance?
Must plan analytic process to achieve the quality
required for intended use of a test!
How relate precision and accuracy of a measurement
procedure to quality required?
How select the QC needed to verify the attainment of
the intended quality of results.
4
How define quality required?
Medically
Diagnostic Important Proficiency
Classification Changes Testing
Goals Criteria
Total
Individual Clinical Outcome Biologic
Biologic Criteria (DInt) Goals
Goals
Analytical Outcome
Performance Criteria Criteria (TEa)
SDMax, BiasMax
State
of the Operating Specifications Arbitrary
Art (smeas, biasmeas, control rules, N) Control
Bottom line is the “Operating Specifications” for
precision, accuracy, and QC (rules and N)
Cholesterol Example
US Guidelines for Quality
1988 National Cholesterol Education Program (NCEP)
Intended clinical use
clinical decision interval
Less 5.17 mmol/L
mmol/L good, over 7.23 mmol/L
mmol/L requires followup
Specify method performance
maximum allowable imprecision,
imprecision, maximum allowable
inaccuracy
CV 3.0% or less, Bias 3.0% or less
Recommend running QC
Include controls each run
1992 US CLIA requires test be correct within 10%
when assessed by proficiency testing
Allowable total error
Specifies minimum QC of 2 materials per 24 hours
5
What’s the point?
There are different ways to define the quality
requirement for a test
Highest form is “clinical decision interval”
Followed by “allowable total error”
Followed by “maximum performance criteria”
“maximum allowable bias” and “maximum allowable SD”
Easiest form to use is “allowable total error”
Most readily available information for “biologic goals”
Must consider the form of the requirement to
translate quality into performance specifications
How relate quality requirement to
performance specifications and QC?
Derive “operating specifications”
Allowable precision
Allowable bias
Necessary QC
Derived from quality-planning models
Analytical model expands Total Error to include QC
Clinical model expand analytical model to include
pre-
pre-analytic variables such as within-
within-subject
biologic variation
Methodology published in 1991
Clin Chem 1991;37:656-
1991;37:656-661; 37:1938-
37:1938-1944
6
Quality-Planning Models
Error Budgets
Conventional Analytical Quality Clinical Quality
Total Error Budget Planning Budget Planning Budget
Preanalytic
Stable Imprecision
Stable Imprecision factors
Stable Inaccuracy Stable Imprecision
Stable Inaccuracy
QC Safety Margin Stable Inaccuracy
Same bottom line, but different expenses QC Safety Margin
Different bottom line and different expenses
What’s the point?
Need “safety margin” for QC
1.0
0.9
0.8 N R
Probability of Rejection (P)
0.7 13s/2of32s/R4s/31s/6x
6 1
0.6 13s/22s/R4s/41s
4 1
0.5
12.5s
0.4 4 1
12.5s
0.3 2 1
13s/22s/R4s
0.2
2 1
0.1 13s
2 1
0.0
0.0 1.0 2.0 3.0 4.0
Systematic Error (multiples of s)
Safety Margin of
1.7s to 3.7s Needed
7
How plan QC?
CLSI C24-A3, Section 6
Step-by-step process
Define quality specifications
Select control materials
Determine method performance
Identify candidate QC strategies
Predict QC performance
Set goals for QC performance
Select appropriate QC rules
CLSI C24-A3, Appendix
Practical tools and examples
Sigma-metrics QC selection tool
Define quality in format of allowable total
error (TEa)
Calculate sigma-metric to characterize
performance of measurement procedure
Relate sigma-metric to error detection
characteristics of QC procedures
8
Sigma-Metric
Sigma = (TEa – Bias)/CV
Where TEa represents the “tolerance
limits” or quality requirement
Bias represents method inaccuracy
CV represents method imprecision
Relationship of Sigma to QC
Critical Systematic Error (ΔSEcrit)
Index used to describe size of error that
needs to be detected by QC procedure
ΔSEcrit = [(TEa – Bias)/CV] – 1.65
Sigma
ΔSEcrit + 1.65 = Sigma
Can relate ΔSE to rejection characteristics of
QC rules and numbers of QC measurements
using known power curves
9
Relationship of Quality and QC
Sigma Scale 3σ 4σ 5σ 6σ
1.65 2.65 3.65 4.65 5.65 Pfr Ped N R
Probability for Rejection (P) 1.0
1 /2of3 /R /3 /6
3s 2s 4s 1s x
0.9
0.07 0.07 6 1
0.8 13s /22s /R4s /41s
Desirable 0.03 0.03 4 1
0.7 Error 1
2.5s
Detection 0.04 0.04 4 1
0.6
1
2.5s
Desirable 0.03 0.03 2 1
0.5 False
1 /2 /R
Rejection 3s 2s 4s
0.4 0.01 0.01 2 1
1
0.3 3s
0.00 0.00 2 1
0.2 13.5s
0.00 0.00 2 1
0.1 1
3s
0.00 0.00 1 1
0.0
0.0 1.0 2.0 3.0 4.0
Systematic Error (SE, multiples of s)
What’s the point?
QC needed depends on sigma
6 sigma process - any common single rule QC
will do!
5 sigma process - single rule QC with 2.5s
limits and N of 2-3
4-sigma process – single rule QC with 2.5s
limits or multirule QC with Ns of 3-6
3-sigma process – do all QC possible Ns of 6-8
Can’t afford to run enough controls to detect
medically important errors!
10
How often should controls be run?
NCCLS C24-A3, Section 7
Analytical run should be based on:
Stability. An analytical run is an interval
(i.e., a period of time or series of
measurements) within which the accuracy
and precision of the measuring system is
expected to be stable;
Susceptibility. Between which, events
may occur causing the measurement
process to be more susceptible (i.e.,
greater risk) to errors important to detect.
Length of analytical run
Run length is specific for analytical system
and measurement procedure
Manufacturer may recommend maximum run
length
User should confine (or shorten) run length
based on operating conditions in the
laboratory, which may differ from the
conditions evaluated by the manufacturer
11
What’s the point?
Default guidelines, such as US CLIA regulations,
may specify run length to be one day, one week,
or even one month
CLIA default is 2 levels control per 24 hours
Proposed “equivalent QC procedures” may reduce QC
to 2 levels per week or even 2 levels per month
User needs to carefully consider susceptibility to
problems and limit run length on basis of
practical considerations
How set control limits?
NCCLS C24-A2: Section 8.6
Should be calculated from mean and SD
that describe the variation in the lab
Desirable to have at least 20 measurements
on 20 different days
“If assayed materials are used, the values
stated on the assay sheets provided by the
manufacturer should be used only as guides.
Actual values for the mean and standard
deviation must be established by serial
testing in the laboratory.”
12
How set control limits?
NCCLS C24-A2: Section 8.6
Recommends calculation of cumulative
means and SDs to provide better (more
stable) values for control limits
Desirable to have cumulatives that represent
approximately 100 measurements
At 30 measurements per month, may use
cumulatives based on 3 or 4 months data
At 20 measurements per month, may use
cumulatives based on 5 or 6 months data
What’s the point?
Lab must determine own mean and SD to
set proper control limits
Wrong SD will misrepresent a control rule
If assigned SD is twice as large as real SD, then
the control limits will be too wide, error detection
and false rejections will be lower than expected
Wrong mean will also change a control rule
May be too large in one direction, too small in the
other, therefore error detection and false rejection
may be different from expected (higher or lower,
and unbalanced)
13
What to do when “out of control”?
CLSI C24-A3: Section 8.7
Recommends changing current practice
“Responses such as repeating control
measurements or reanalyzing new
control materials are not productive
when QC strategies have been
carefully planned and control rules
selected to minimize the false
rejections of analytical runs…”
Should troubleshoot and take
appropriate corrective action
Problem of repeating controls
often due to use of 2SD limits
Mean
- 2.00s + 2.00s
How How
many many
outside? outside?
-3s -2s -1s 0 1s 2s 3s
False rejections expected to be 5% for N=1;
9% for N=2; 14% for N=3;
14
What’s the point?
Laboratories should avoid the use of 2SD
control limits!
Levey-Jennings chart with 2 SD limits is still
the most common QC procedure in the world!
Leads to “repeat, repeat, until get lucky!”
Corrupts the QC process because problems
never identified and never fixed
Are you just running controls or do you want
to control quality?
Points of Care in Managing QC
DO select control materials with known stability
Generally means acquiring commercial materials
whose stability has been tested
DO determine your own mean and SD
Start with 20 measurements
Verify your results are within expected or labeled
values
Use cumulative means and SD until get approximately
100 measurements
15
Points of Care in Managing QC
DO overlap new lot of control material
with old lot
Can transfer SD from old lot to get started,
but must determine new mean with own data
Then update SD and mean after collecting
more of your own data
DO monitor stability of control materials
with peer data
Points of Care in Managing QC
DON’T misuse control range from package insert
Will likely provide limits that are too wide
Cause low error detection
DON’T misuse SD from peer group
Will likely provide limits that are too wide
DON’T misuse mean from peer group
OK to get started, important to monitor, but need to
use mean determined from own laboratory data
16
Points of Care in Managing QC
DON’T calculate means and SDs from
control data that is “out-of-control”
Want control data that represents good
performance, not bad performance
Old practice that is predicated on use of 2 SD
control limits, where elimination of out-of-
control results leads to narrowing of control
ranges
Points of Care in Managing QC
Improvements to make
DO define quality required for each test
DO select QC procedures to minimize false
rejections
DO select QC procedures to detect
medically important errors
DO adopt modern QC planning tools
DO standardize QC operations
17
Do Right QC Right!
DO implement right QC designs
Start with general selection guidelines, then
advance to quantitative QC planning process
DO implement QC right using computerized
charting and data analysis tools
Need flexibility to implement different QC designs
and support data calculations
DO participate in EQC/peer programs
Monitor bias against peer groups
DO provide education and training
18