1

Principles of EQA
Annette Thomas
Chair IFCC C-AQ

Director Weqas
[Link]

Advancing excellence in laboratory medicine for better healthcare worldwide

 What is External Quality Assessment? 2

= proficiency testing scheme

System designed to objectively assess the

quality of results by an external agency.

Advancing excellence in laboratory medicine for better healthcare worldwide

 IQC vs EQA 3

Internal Quality Control External Quality Assurance

Results Known Unknown
Results Available Immediately Later, when report received
Frequency of Testing Minimum daily, Periodically
per batch, eg monthly/
per shift Bimonthly
quarterly
Concentrations Normal, abnormal Pathological range covered over
multiple samples
Assess Imprecision Accuracy & imprecision
Comparison Your lab only Your lab to all labs & other labs
using your method

Advancing excellence in laboratory medicine for better healthcare worldwide

 Types of EQA Schemes – which one for you ? 4

• Commercial
PT, EQA or • Not for profit
• Linked to professional body
EQAP? • Regulatory

• Broad Scope - All disciplines

Scope • Narrow Scope - Discipline specific
• Limited scope – Specialist

Accredited • Yes
• No?
to ISO 17043
Advancing excellence in laboratory medicine for better healthcare worldwide
 Expectations of EQA Provider 5

Scheme designed
•Clinical Scientist or medically qualified
Accreditation and overseen by
• If not accredited, labs •Independent Scientific or Medical
appropriately
status – 17043 should justify why competent
Advisory group.
•Statistical expertise
professionals

•Variable across Schemes (EQALM study

Clinically relevant 2009) * Reporting to
•Where EQA used to assess IVDs,
•Mechanism for identification and
Distribution minimum of 6 distributions p.a. (BS EN Professional body / reporting of Persistent Poor performance
frequency 14136:2004) Regulatory body. issues
•For core tests - monthly

Clinically relevant •Evidence of reproducibility

•Training
•Cover clinically appropriate range
Range and •“Blinded” Education •Helpline
number of •Commutable materials •Pre analytical
samples •Challenging samples •Post Analytical

• Based on clinical •Alerts manufacturers

Clinically relevant
outcomes Post-marketing •Alerts competent authority
performance
• Based on biological surveillance •Alerts laboratories
criteria
variation •Alerts professional bodies

* A Thomas, Accred Qual Assur (2009) 14: 439-444

Advancing excellence in laboratory medicine

* A Thomas, for better
Accred healthcare
Qual Assur worldwide
(2009) 14: 439-444
 Objectives of EQA 6

• Provide a measure of the quality of a test

• To supplement internal quality control procedures
• Provide a measure of the “state of the art” of a test
• To obtain consensus values when true values are unknown
• To investigate factors in performance (methods, staff etc)
• To act as an educational stimulus to improvement in
performance
• To provide a Post market vigilance service
• To provide evidence and monitoring of harmonisation
strategies
IFCC 1977

Advancing excellence in laboratory medicine for better healthcare worldwide

 General Cycle of EQAS 7

EQA samples
received

Corrective action User analyses EQA

taken if required samples

EQA report Results sent to EQA

reviewed organiser

EQA report received

Advancing excellence in laboratory medicine for better healthcare worldwide

 Choosing an EQA - considerations 8

✓Design for clinical need

✓Material type & frequency
Identify performance specification
Data analysis
Reports
Education element
Troubleshooting support
Advancing excellence in laboratory medicine for better healthcare worldwide
 Material & Frequency 9

• Appropriate matrix
• Stable, homogeneous material
• Appropriate concentration range
• Challenging samples
• Appropriate frequency of testing

Advancing excellence in laboratory medicine for better healthcare worldwide

 Sample Matrix 10

Lyophilized material – serum / urine / whole blood

Pro Con
– Good long term stability – Commutability issues
– Very easy to transport – Expensive
– Difficult to prepare -
reconstitution errors
– Denaturation of
proteins/lipoproteins
– Issues with assigning
reference targets

Advancing excellence in laboratory medicine for better healthcare worldwide

 Sample Matrix 11

Fresh/ Frozen clinical material - from individual donors

or pool of donors
Pro Con
– Gold standard – Difficult to obtain
– Commutable – performs different concentration
as patient samples levels
– Easy to use – Unstable analytes
– Can set reference targets – Storing and transport
expensive
– Risk of bacterial
contamination

Advancing excellence in laboratory medicine for better healthcare worldwide

 Appropriate Concentration Levels 12

• Clinically relevant
• Range of concentration levels
• Not just around the reference interval
• Cover the analytical and pathological range
• Challenging samples

Advancing excellence in laboratory medicine for better healthcare worldwide

 Choosing an EQA? 13

✓Design for clinical need

✓Material type & frequency
✓Identify performance specification
Data analysis
Reports
Education element
Troubleshooting support
Advancing excellence in laboratory medicine for better healthcare worldwide
 14
What is the Target Value?
• Determining the ‘right’ , ‘true’, ‘correct’
value?

Advancing excellence in laboratory medicine for better healthcare worldwide

 15
Target Value
• Reference value
– “True” value with meteorological traceability.
– Commutable EQA material assayed using:
– Reference methods eg HbA1c
– By an Accredited reference laboratory
• Using higher order methods eg IDGC-MS glucose,
creatinine.
• Certified Reference material

Advancing excellence in laboratory medicine for better healthcare worldwide

 Advantage of Reference Measurement Targets 16

• Traceable to higher order

• Establishes method traceability for the lab–
requirement of ISO 15189
• Independent assessment of manufacturer
traceability claims.
• Highlights the pitfalls of using the trimmed overall
mean as an accuracy target in EQA Schemes
• Overall mean and method mean may not be
traceable, may not be stable, may be influenced by
large numbers from one manufacturer.
• Useful in the post market vigilance of the IVD -
Directive

Advancing excellence in laboratory medicine for better healthcare worldwide

 Traceability 17

• Reference
measurement values
shown on report (and
reference value
uncertainty). Full
traceability chain to SI
units available.
• Lab results compared
directly to reference
values
• SDI scores, Sigma
scores and bias plot
based on reference
values
Advancing excellence in laboratory medicine for better healthcare worldwide
 Use of gravimetric data 18

Analyte Gravimetric LC-MS/MS Recovery % Traceability

Amphetamine (ug/L) 3000 2936 97.9
Good agreement was
observed between the
Cannabis (ug/L) 300 227 75.7
‘gravimetric’ weighed in
Opiate (Morphine) (ug/L) 5000 5483 109.7
target and the LC-
Benzodiazepine (ug/L) 800 749 93.6 MS/MS data for the
Cocaine (ug/L) 800 842 105.3 majority of analytes,
Methadone (EDDP) (ug/L) 600 720 120.0 however a decreased
Methamphetamine (ug/L) 3000 3193 106.4
recovery of 76% and
83% was observed for
6-Acetylmorphine (ug/L) 30 25 83.3
cannabis and
Buprenorphine (ug/L) 30 31 103.3
acetylmorphine
Ketamine (ug/L) 3000 2786 92.9 respectively.
Barbiturates (ug/L) 800 820 102.5

MDMA (ug/L) 2500 2766 110.6

Phencyclidine (ug/L) 73.4 78* 106.3

Advancing excellence in laboratory medicine for better healthcare worldwide

 19
Target Value

• Statistical
– Overall mean, median
– Method, instrument mean
– Shows the ‘state of art’ of method performance
but gives no indication of the “true” value

Advancing excellence in laboratory medicine for better healthcare worldwide

 Using Statistical Comparison 20

• Mean ± 2 SD = 95% returned results are acceptable

• This gives information on whether your result is
acceptable compared with all results and your method
• But does it give information on whether your result is
acceptable compared to the ‘true’ patient result?
• Would your result alter the clinical management of the
patient?
• Does not provide context to clinical utility
Advancing excellence in laboratory medicine for better healthcare worldwide
 External Quality Assurance Programs 21

• Use quality standards to allow labs to assess

their performance and respond accordingly
• the quality standard is the allowable difference
from a target
• A tool for review of QAP results
• Can be based on
– statistical comparison
– expert opinion
– clinical need
– other criteria
Advancing excellence in laboratory medicine for better healthcare worldwide
 Target values used in Quantitative EQA 22

Loss of information for Assessment of accuracy

Reference Analyser mean

Overall mean / Method mean /
values median median Peer group
Gold standard assessment
Used if no ref Peer group only.
and data assessment
Gaussian. only
Gravimetric

Advancing excellence in laboratory medicine for better healthcare worldwide

 What is the best approach? 23

Fresh Patient / commutable samples using reference method as target

Fresh patient / commutable samples using a secondary reference method as

target

Fresh patient / commutable samples using data from peer group of expert laboratories as
target

• Mean / median of method group with acceptable performance using fresh

patient / commutable samples

Commercial material. - It is essential to understand how the matrix affects the performance of
methods if non commutable material is used. Only peer review within instrument / method group
can be undertaken in these instances.

Advancing excellence in laboratory medicine for better healthcare worldwide

 24

Determining Performance
specifications.

Advancing excellence in laboratory medicine for better healthcare worldwide

 Acceptable Performance Specification 25

• A range of values around the target value that is considered

acceptable laboratory performance
• A tool for review of EQA results
• It provides a simple tool to allow a rapid, standardised
assessment of EQA results in both numerical and graphical
report formats.
• A result outside the acceptable range should alert the
laboratory that that their assay may produce results that are
at risk of detrimentally affecting clinical decision making.

Advancing excellence in laboratory medicine for better healthcare worldwide

 Using Statistical Comparison 26

TEe = 10% TEa = 5%

Method 1 Method 2

• Mean ± 2 SD = 95% returned results are acceptable

• This gives information on whether your result is acceptable
compared with all results and your method
• May not be clinically relevant

Advancing excellence in laboratory medicine for better healthcare worldwide

 Which Performance Criteria to use? 27

Regulatory Acceptable performance from

requirement, TE EQA, TE %
Test CLIA Rilibak RCPA WEQAS Labquality
(US) (Germany) (Aus) (UK) (Finland)
Glucose ± 0.33 ± 15% ± 0.5 ± 7.5%@4 ± 6.0%
mmol/l or mmol/l or mmol/l
± 10% ± 10%

HbA1c n/a ± 18% ± 0.5 or ± ± 7%

5%

Advancing excellence in laboratory medicine for better healthcare worldwide

 Specification Hierarchy 28

Model Model Model

1 2 3

Analytical performance Analytical performance “State of the art” -

specification based on specification based on Interlaboratory variation
clinical outcomes biological variation

• What we need but • Data available but not • What we can achieve
data not readily always achievable but may not be “fit for
available purpose”

Data from
outcome Improvements in
studies methods / technology

Advancing excellence in laboratory medicine for better healthcare worldwide

 From Biological goals 29

Test I (%) B (%) TE (%)

(0.01)
Glucose 2.2 1.9 7.0
HbA1c 1.7 1.5 5.5

Desirable quality specification can be calculated from:

I < 0.5CVw
B< 0.25 (CVw 2 + CVb2)½
TE = 2.33 I + B (a<0.01) for EQA

Advancing excellence in laboratory medicine for better healthcare worldwide

 Analytical goals 30

Biological goals Weqas TE criteria

Analyte Conc. I (%) B (%) TE (0.01) SD 2 SD TE
Albumin 40 1.6 1.3 4.9 1.3 2.6 6.5
Bicarb 20 2.4 1.6 7.2 1.3 2.6 13.0
Highlighted
Ca 2.3 1 0.8 3.1 0.05 0.1 4.3
TE are
Cl 100 0.6 0.5 1.9 1.4 2.8 2.8 those
Creat 80 2.2 3.4 8.4 8 16 20.0 where
Glucose 4.2 2.2 1.9 7.0 0.16 0.32 7.6 Biological
Mg 0.8 1.8 1.8 6.0 0.03 0.06 7.5 goals not
Osmo 245 0.7 0.4 2.0 3.4 6.8 2.8 achievable
Phos 0.8 4.3 3.2 13.1 0.03 0.06 7.5
K 4 2.4 1.8 7.4 0.08 0.16 4.0
Na 135 0.4 0.3 1.2 1.5 3 2.2
TP 70 1.4 1.2 4.4 1.6 3.2 4.6
Urate 0.34 4.3 4.8 14.8 0.02 0.04 11.8
Urea 8 6.2 5.5 19.8 0.35 0.7 8.8

Advancing excellence in laboratory medicine for better healthcare worldwide

 45.00
Creatinine Precision Profile (CV %) 31

40.00

35.00
1999-2000

30.00
2004-2010
25.00
cv

2011
20.00

15.00 2012

10.00

5.00 Biological
Goal Te =
8.4 %
0.00
0 100 200 300 400 500 600 700

Overall Mean Creatinine (umol/L)

Advancing excellence in laboratory medicine for better healthcare worldwide

 Choosing an EQA - considerations 32

✓Design for clinical need

✓Material type & frequency
✓Identify performance specification
✓Data analysis
✓Reports
Education element
Troubleshooting support
Advancing excellence in laboratory medicine for better healthcare worldwide
 What to Look for in the EQA Reports 33

• Design should be such that the user can:

– Understand the report
– take appropriate action
– provide help and improvement
– Is education

Advancing excellence in laboratory medicine for better healthcare worldwide

 What to Look for in EQA Reports? 34

• What is the turn around time?

– how long after the due date will you receive your
report?
– the earlier the better
– can remember what was happening at that time
– can take corrective action sooner
• Is the report electronic or paper?
– Electronic quicker but not all sites may have
printer access.

Advancing excellence in laboratory medicine for better healthcare worldwide

 What to Look for in EQA Reports? 35

• Report should
– Clearly identify performance
• Assess accuracy to target value
• Assess imprecision
• Assess linearity
• Assess performance over time
• Identify errors and poor performance
– Clearly identify method / analyser performance

Advancing excellence in laboratory medicine for better healthcare worldwide

 Assessment of Accuracy 36

• From the linear regression

analysis equation, y=mx+c,
the trueness (bias) is
calculated at the critical
level (x) , which for
Cholesterol is 5.0 mmol/L.
• when x=5 then y=
0.9777(*5)+0.042 = 4.9305
• Bias = (y-x)/x*100 =
(4.9305-5.0)/5.0*100 = -
1.39%
• Imprecision (CV) = Sy.x/ x =
0.026/5.0*100 = 0.5%

Advancing excellence in laboratory medicine for better healthcare worldwide

 Uncertainty From 37
EQA Data

• Laboratory within
run Imprecision:
• Sy.x = 0.06 mmol/L
• CV% = (Sy.x/
x)*100 =
0.06/7*100 =
0.86%

Advancing excellence in laboratory medicine for better healthcare worldwide

 Uncertainty From EQA Data 38

• Between batch CV% provided on End of Batch reports (12 month review)
• Pool M891a - CV% of reported results: 4.51%
• Top-down approach – CV% is method uncertainty (relative standard
uncertainty)

Advancing excellence in laboratory medicine for better healthcare worldwide

 Specificity and Sensitivity Studies 39

Pregnancy testing
Bile Acids

Advancing excellence in laboratory medicine for better healthcare worldwide

 Post Market vigilance – INR thromboplastin 40

INR results classified into pre and post recalibration. participants using strips calibrated to WHO reference
thromboplastin rTF/09

The pre calibration strips compared well with the results from Distribution 0517 (Median 2.8) however much
higher results and a wider distribution of results was observed for the post calibration strips. Weqas immediately
contacted the manufacturer and sent them the data.

Aug 2018 – Urgent field safety notice issued to inform users that the manufacturer was reverting back to
previous WHO reference standard.

Advancing excellence in laboratory medicine for better healthcare worldwide

 Choosing an EQA - considerations 41

✓Design for clinical need

✓Material type & frequency
✓Identify performance specification
✓Data analysis
✓Reports
✓Education element
Troubleshooting support
Advancing excellence in laboratory medicine for better healthcare worldwide
 Educational role 42

• Pre-analytical effects
• Number and type of methods used
• Performance of methods used
• accuracy
• precision
• Susceptibility of methods to interference
• including other analytes and matrix
• Interpretation of results

Advancing excellence in laboratory medicine for better healthcare worldwide

 Educational role 43

• Education on methods and interferences

• eg
– Concentration near diagnostic “cut off”
– Samples with known variants for HbA1c
– Samples with added interferances

Advancing excellence in laboratory medicine for better healthcare worldwide

 Intralaboratory variation – Roche TnT 44

30

25

AF
LoQ AJ
20
AQ 1

AQ 2
CV %

15
AH 1

AH 2

10 AI 1

AI 2

ULN JU 1
5
JU 2

0
0 2 4 6 8 10 12 14 16 18 20
Troponin T (ng/L)
Advancing excellence in laboratory medicine for better healthcare worldwide
 Interferences - Effect of icterus on assays 45
Sample 2 - Creatinine - Non Icteric Pool Sample 3 - Creatinine - Icteric Pool
214.7 214.7

+2
199.5 199.5 WEQAS
+2
WEQAS SD
SD

Creatinine (umol/L)
Creatinine (umol/L)

184.3 Reference
184.3 Reference
Value
Value

-2 -2
169.1 169.1 WEQAS
WEQAS
SD SD

153.9
Method Method Method
Method Method
Method DAY 153.9 Method Method
C n=4 AD n=3 OL n=10 AU n=10 A n=7 C n=25 VIDMS VI n=8 Enz AD n=12 M n=7 C n=32 AD n=18 OL n=5 AU n=7 DX n=26 C n=34 M n=14 A n=3 Method Method Method Method DAY
JR n=4 KJ n=53 Vit n=4 JI n=120 n=13 C n=4 AD n=3 OL n=10 AU n=10 A n=7 C n=25 VIDMS VI n=8 Enz AD n=12 M n=7 C n=32 AD n=18 OL n=5 AU n=7 DX n=26 C n=34 M n=14 A n=3
n=9 n=53 JR n=4 KJ n=53 Vit n=4 JI n=120 n=13
n=9 n=53
Mean 183.2 185.3 185.7 188.7 179.1 178.2 180.3 189.4 189.3 184.9 185.3 190.9 186.4 189.1 193.0 180.3 177.1 176.2 180.2 182.7 180 185.3 172.3 182.3
Mean 164.4 167.5 173.1 175.0 178.2 180.8 171.0 168.1 172.6 168.7 169.0 170.3 173.8 170.9 168.6 161.3 180.7 172.8 179.9 153.0 160.7 169.6 128.7 170
+2SD 197.2 198.1 201.5 193.9 197.3 200.2 192.5 202.8 196.3 188.5 188.3 198.9 192.2 194.9 198.8 191.1 184.5 187.4 185.2 188.9 190 192.5 190.9 186.5
+2SD 181 179.7 188.5 179.4 192.6 196.8 175.2 177.9 179.6 174.9 175.4 177.1 180.6 177.5 173.2 198.1 194.3 180 192.5 188 170.9 181.4 142.1 171.6
-2SD 169.2 172.5 169.9 183.5 160.9 156.2 168.1 176 182.3 181.3 182.3 182.9 180.6 183.3 187.2 169.5 169.7 165 175.2 176.5 170 178.1 153.7 178.1
-2SD 147.8 155.3 157.7 170.6 163.8 164.8 166.8 158.3 165.6 162.5 162.6 163.5 167 164.3 164 124.5 167.1 165.6 167.3 118 150.5 157.8 115.3 168.4

Sample 2 - Glucose - Non Icteric Pool

Sample 3 - Glucose - Icteric Pool
9.16
9.16

+2
WEQAS
8.64 SD
8.64 +2
WEQAS
SD
Glucose (mmol/L)

Glucose (mmol/L)
Reference
Value
8.12 Reference
8.12 Value

-2
7.6 WEQAS -2
SD 7.6 WEQAS
SD

7.08
Vitros n=13
Abaxis Method GOD
AD n=5 DAY n=8 M n=13 C n=5
Method HEX
AD n=23 OL n=17 AU n=14 DX n=16 A n=11 M n=8 C n=84
Method O2 7.08 Abaxis Method GOD Method HEX Method O2
Piccolo n=3 n=32 n=175 n=11 Vitros n=13 AD n=5 DAY n=8 M n=13 C n=5 AD n=23 OL n=17 AU n=14 DX n=16 A n=11 M n=8 C n=84
Piccolo n=3 n=32 n=175 n=11
Mean 7.97 8.53 8.42 8.26 8.4 8.51 8.51 8.43 8.3 8.51 8.54 8.34 8.35 8.53 8.44 8.26
Mean 7.96 8.5 7.63 7.18 7.39 7.87 8.42 8.42 8.29 8.47 8.5 7.92 8.36 8.51 8.45 8.29
+2SD 8.15 8.71 8.88 8.54 8.92 8.75 8.97 8.77 8.64 8.93 8.8 8.74 8.55 8.99 8.72 8.48
+2SD 8.14 8.66 8.53 7.5 7.85 8.11 8.82 8.78 8.67 8.73 8.76 8.6 8.54 8.93 8.71 8.43
-2SD 7.79 8.35 7.96 7.98 7.88 8.27 8.05 8.09 7.96 8.09 8.28 7.94 8.15 8.07 8.16 8.04
-2SD 7.78 8.34 6.73 6.86 6.93 7.63 8.02 8.06 7.91 8.21 8.24 7.24 8.18 8.09 8.19 8.15

Advancing excellence in laboratory medicine for better healthcare worldwide

 Choosing an EQA - considerations 46

✓Design for clinical need

✓Material type & frequency
✓Identify performance specification
✓Data analysis
✓Reports
✓Education element
✓Troubleshooting support
Advancing excellence in laboratory medicine for better healthcare worldwide
 Troubleshooting Support 47

• Can they provide a consultation service?

• Available by phone, fax, e-mail
• Provide help with
– method classification queries
– analyte problems
– report interpretation
– troubleshooting

Advancing excellence in laboratory medicine for better healthcare worldwide

 Problem Solving Flow Chart 48

START HERE

IMPRECISION INACCURACY

[1] [2]
Are you satisfied with YES Are you satisfied with YES
your imprecision values? slope and intercept? ☺
(Sy.x, r) (m, c)

Advancing excellence
Pages in laboratory
18-19 medicine for better healthcare worldwide
of SP-QL1-IntLabEQA
 49

[1] Are you satisfied with

your imprecision values? Problem Solving Flow
(Sy.x, r)
NO
[3] Check whether the
Chart
cause is curvilinear data
(m,c,Sy.x,r)
NO
[4] Then the error is random, IMPRECISION
check whether there is
clerical error. YES
NO
Eliminate
[5] blunder
Check for causes go to [2]
of imprecision
e.g. inexperienced
operators, faulty
equipment, inappropriate go to [2]
methods
Advancing excellence in laboratory medicine for better healthcare worldwide Pages 18-19 of SP-QL1-IntLabEQA
 Problem Solving Flow Chart 50

START HERE

IMPRECISION INACCURACY

[1] [2]
Are you satisfied with YES Are you satisfied with YES
your imprecision values? slope and intercept? ☺
(Sy.x, r) (m, c)

NO

[6]
Identify type of error

curvilinear proportional mixed constant

(m, c, Sy.x, r) (m) (m,c) (c)

Advancing excellence
Pages in laboratory
18-19 medicine for better healthcare worldwide
of SP-QL1-IntLabEQA
 Problem Solving Flow Chart 51

Curvilinear proportional mixed constant

(m, c, Sy.x, r) (m) (m, c) (c)

Check for time Is it a “one

point” cal?
INACCURACY expired reagents
NO
NO YES
YES Check all
Check zero
Calibrators
Replace (reagent blank,
inc. zero
reagents serum blank,
instrument zero).
Recalibrate (try different lot)

Run linearity check

Are you satisfied
with slope (m)? Are you satisfied
with intercept (c)?
YES NO NO YES
Check slope Check method
Adjust & Recalibrate
value of instrument specificity
Advancing excellence in laboratory medicine for better healthcare worldwide
Pages 18-19 of SP-QL1-IntLabEQA
 EQA and User in partnership 52

• Should not be viewed as a

pass/fail exercise
Part of • Educational – troubleshooting,
Quality recommendations of best practice
Improvement • Identify poor methods
• Provide training and help

Advancing excellence in laboratory medicine for better healthcare worldwide

 Key elements of National EQA 53

Laboratory

Manufac
turer
EQA
Competent Provider
Authority

Professional /
Regulatory body

Method performance Participant Performance evaluation

evaluation Quality improvement
Post market vigilance Continuous Education
Harmonisation of results
Harmonisation of processes
Audit and recommendations
Advancing excellence in laboratory medicine for better healthcare worldwide
 54

Advancing excellence in laboratory medicine for better healthcare worldwide

 55

Advancing excellence in laboratory medicine for better healthcare worldwide