Guidance Document on Measurement

Uncertainty for GMO Testing Laboratories

S. Trapman, M. Burns, H. Broll, R. Macarthur,

R. Wood, J. Zel

EUR 22756 EN/2 - 2009

The mission of the JRC-IRMM is to promote a common and reliable European measurement
system in support of EU policies.

European Commission
Joint Research Centre
Institute for Reference Materials and Measurements

Contact information
Address: S. Trapmann
E-mail: [Link]@[Link]
Tel.: 014/571 714
Fax: 014/571 548

[Link]
[Link]

Legal Notice
Neither the European Commission nor any person acting on behalf of the Commission is
responsible for the use which might be made of this publication.

Europe Direct is a service to help you find answers

to your questions about the European Union

Freephone number (*):

00 800 6 7 8 9 10 11

(*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls may be billed.

A great deal of additional information on the European Union is available on the Internet.
It can be accessed through the Europa server [Link]

JRC 50052

EUR 22756 EN/2

ISBN 978-92-79-11228-7
ISSN 1018-5593
DOI 10.2787/18988

Version 2

Luxembourg: Office for Official Publications of the European Communities

© European Communities, 2009

Reproduction is authorised provided the source is acknowledged

Printed in Belgium
 Guidance Document on Measurement
Uncertainty for GMO Testing Laboratories

S. Trapmann, M. Burns, H. Broll, R. Macarthur, R. Wood, J. Zel

This guidance document has been elaborated on request of the European Network of GMO Laboratories
(ENGL) to further harmonise the estimation of measurement uncertainty of real-time PCR
 Disclaimer

Certain commercial equipment, instruments, and materials are identified in

this report to specify adequately the experimental procedure. In no case does such identification
imply recommendation or endorsement by the European Commission, nor does it imply that the
material or equipment is necessarily the best available for the purpose.
Summary
This technical report outlines the technical issues related to the estimation of measurement
uncertainty (MU) involved in the GMO sector. In particular it gives guidance to GMO testing
laboratories how to estimate the analytical variability of quantitative analytical results
obtained by real-time PCR. This guidance document has been written on request of the
European Network of GMO Laboratories (ENGL) as a follow-up of a workshop on
Measurement Uncertainty in the GMO sector organised by the Institute for Reference
Materials and Measurements (IRMM), Geel, Belgium and held on 05.07.2005.

It is recognised that in order to be able to judge if an analytical results exceeds a threshold; the
MU must be estimated and reported together with the measurement result. Enforcement
Authorities shall therefore estimate the MU associated with an analytical result and use it to
decide whether an analytical result falls within the specification of food and feed control. The
value obtained by subtracting the expanded uncertainty from the reported concentration is
used to assess compliance. Only if this value is greater than the legal threshold, it is sure
'beyond reasonable doubt' that the sample concentration of the analyte is beyond what is
permissible.

Two selected approaches for the estimation of MU are presented in detail; references to
alternative approaches are given. The first approach presented in detail is using data from
collaborative trial in combination with in-house quality control data for the estimation of MU.
Prerequisites for the use of such collaborative trial data are outlined. In case no suitable
collaborative trial data are available, an alternative approach for the estimation of MU, using
data obtained on within-laboratory samples, is presented.

1
2
Table of Content

SUMMARY ........................................................................................................................................................... 1

TABLE OF CONTENT........................................................................................................................................ 3

GLOSSARY........................................................................................................................................................... 4

1 INTRODUCTION.............................................................................................................................................. 5
1.1 PURPOSE....................................................................................................................................................... 6
1.2 PROCEDURES FOR THE ESTIMATION OF MEASUREMENT UNCERTAINTY ................................................... 6
1.3 STATE OF THE ART OF COLLABORATIVE TRIALS IN GM QUANTIFICATION ............................................. 10
2 THE RELATIONSHIP BETWEEN THE ANALYTICAL RESULT AND MEASUREMENT
UNCERTAINTY, WITH PARTICULAR REFERENCE TO REGULATION (EC) NO 1829/2003 AND
(EC) NO 1830/2003 ............................................................................................................................................. 11
2.1 GENERAL DESCRIPTION OF THE ESTIMATION AND USE OF MEASUREMENT UNCERTAINTY .................... 11
2.2 CALCULATION OF MEASUREMENT UNCERTAINTY FROM COLLABORATIVE TRIAL DATA AND CONFIRMED
BY INTERNAL QUALITY CONTROL DATA ......................................................................................................... 12
2.3 CALCULATION OF MEASUREMENT UNCERTAINTY FROM INTERNAL QUALITY CONTROL DATA ............. 14
2.3.1 Within-laboratory reproducibility ...................................................................................................... 15
2.3.2 Method and laboratory bias control................................................................................................... 17
2.3.3 Estimation of the uncertainty component associated with bias .......................................................... 18
2.3.4 Calculation of the relative standard uncertainty ................................................................................ 18
2.4 EVALUATION OF MEASUREMENT UNCERTAINTY ...................................................................................... 19
2.5 CALCULATION OF THE CRITICAL LEVEL................................................................................................... 20
2.6 CALCULATION OF THE LIMIT OF DETECTION AND QUANTIFICATION ...................................................... 20
2.7 REPORTING RESULTS ................................................................................................................................. 22
2.8 ESTIMATION OF THE UNCERTAINTY ASSOCIATED WITH THE MEAN ........................................................ 22
2.9 HOW TO USE MEASUREMENT UNCERTAINTY ............................................................................................ 23
3 FURTHER CONSIDERATIONS................................................................................................................... 24

ACKNOWLEDGEMENTS................................................................................................................................ 26

ANNEX I: DEFINITIONS APPLICABLE TO GMO ANALYSIS ................................................................ 27

ANNEX II: EXAMPLE FOR AN INTERNATIONAL COLLABORATIVE TRIAL TO DETERMINE

ACCURACY OF A REAL-TIME PCR PROCEDURE.................................................................................. 30

ANNEX III: EXAMPLE FOR THE EVALUATION OF THE MEASUREMENT UNCERTAINTY

BASED ON SINGLE LABORATORY RESULTS.......................................................................................... 34

ANNEX IV: EXAMPLE OF DATA HANDLING OPERATIONS POTENTIALLY LEADING TO

UNCERTAINTY................................................................................................................................................. 38

ANNEX V: DATA CALCULATED ON THE BASES OF CRL COLLABORATIVE TRIALS ................ 40

3
Glossary
AOAC Association of Official Analytical Chemists
c sample concentration
CRL Community Reference Laboratory
CRM Certified Reference Material
CTAB cetyltrimethylammonium bromide
Ct-value number of PCR cycles to pass a set threshold
DG SANCO Directorate-General Health and Consumer Protection
DNA deoxyribonucleic acid
ENGL European Network of GMO laboratories
ERM® European Reference Material®
EU European Union
EURACHEM Network of analytical chemistry organisations in Europe
GM genetically modified
GMO genetically modified organism
GUM Guide to the Expression of Uncertainty in Measurement
IQC internal quality control
IRMM Institute for Reference Materials and Measurements
ISO International Standards Organisation
IEC International Electrotechnical Commission
IUPAC International Union of Pure and Applied Chemistry
LC critical level
LOD limit of detection
LOQ limit of quantification
m/m mass fraction
MU measurement uncertainty
n number of independent measurements
NIB National Institute of Biology
NMKL Nordic Committee on Food Analysis
PCR polymerase chain reaction
RSU relative standard uncertainty
RRS Roundup Ready® soya
TVP texturised vegetable protein
U expanded uncertainty
u standard uncertainty

4
1 Introduction
It has been recognised that there are a number of actions that may be taken by those
responsible for the enforcement of EU legislation which directly affect decisions as to
whether a sample, or a lot from which a sample is taken, is in compliance with EU legislation.

It must be appreciated that the enforcement of any requirement in EU legislation which is

based on measurement results depends on the analytical methods and their measurement
uncertainties (MUs). Without common and uniform criteria for evaluation and interpretation
of the MU, different Member States will make different judgements as to whether any
particular lot is in compliance with its EU specification.

After a workshop on MU in the GMO sector organised and held on 05.07.2005 at the Institute
for Reference Materials and Measurements (IRMM) in Geel, Belgium, a task force of the
European Network of GMO Laboratories (ENGL) has been formed to outline the related
scientific and technical issues. This resulting technical report provides recommendations and
gives guidance to the Enforcement Authorities in Member States on procedures to be adopted
that reduce the possibility of Member States taking differing views as to whether a particular
sample is in compliance with any particular EU specification.

This report focuses on analytical issues only. In particular it considers the treatment of
analytical uncertainty (normally known as the MU) in the interpretation of a specification. It
is mainly concerned with quantitative analytical results. This report is written in a form such
that the complex issues involved can be readily appreciated by all. In particular, it:

• sets out the issues;

• gives recommendations for consideration by the Enforcement Authorities of the Member
States; and
• gives a series of technical Annexes to aid practitioners in estimating their MUs.

This report further develops the general DG SANCO report 'On the relationship between
analytical results, measurement uncertainty, recovery factors and the provisions of EU food
and feed legislation, with particular reference to community legislation concerning
contaminants in food and undesirable substances in feed' [1] for application in the GMO
sector.

It should be noted that the MU estimation as such is independent from the unit of
measurement. Independent if mass fractions (m/m) or copy number ratios are used, the
estimation is carried out in the same way. Attention needs to be paid, that the unit of
measurement is not changed during its uncertainty estimation. However, the user should pay
extra attention when using percent as measurements unit as these absolute values can easily
be mixed-up with relative values given as well in percent.

[1] DG Health and Consumer Protection: Report to the Standing Committee on the Food Chain and Animal
Health on the relationship between analytical results, the measurement uncertainty, recovery factors and the
provisions in EU food and feed legislation with particular focus on the community legislation,
[Link]

5
1.1 Purpose

The requirements and recommendations made in this document are aimed at providing help to
the practitioner dealing with samples under Regulation (EC) No 882/2004 'on official controls
performed to ensure the verification of compliance with feed and food law' [2] and
specifically under Regulation (EC) No 1829/2003 on 'genetically modified food and feed' [3]
and Regulation (EC) No 1830/2003 concerning the traceability and labelling of genetically
modified organisms and the traceability of food and feed products produced from genetically
modified organisms' [4]. In addition these recommendations should be applied for seed. Only
detection procedures which are intended for an event-specific quantification (e.g. by real-time
PCR approach) of material derived from a GMO are covered.

1.2 Procedures for the estimation of measurement uncertainty

MU is generally thought of as applying to quantitative measurements. Thus, in the GMO

sector it will apply to the quantitative estimation of the 'concentration' (however expressed) of
an authorised GMO. The concept will also apply qualitatively in the case of both authorised
and non-authorised GMOs (i.e. confirmation of presence/absence). This latter aspect is
increasingly being recognised as being of importance, but work on 'Qualitative Measurement
Uncertainty' is only just commencing at the international level [5].

MU, which should take account of all effects on a measurement process, is the most important
single parameter that describes the quality of measurement. The MU is linked to the
individual measurement performed but not to a real-time PCR method as such. Therefore each
laboratory has to evaluate the specific MU for a measurement results obtained under defined
conditions. The uncertainty arises both from sampling and analysis, unless it could be proven
that the sampling carried out is representative and that sampling uncertainty can therefore be
neglected. It should be noted that generally a large amount of MU can arise from the upstream
sampling stage, this is however not the rationale of this document and this guidance document
addresses only the MU arising downstream the sampling stage.
Significant progress has been made in devising procedures to estimate the uncertainty that
originates in the analytical portion of the measurement, and guidance on these procedures is
available. It has, however, become increasingly apparent that sampling is often an important
contribution to uncertainty and requires equally careful management and control. Thus, the
uncertainty arising from the sampling process should be evaluated. While existing guidance
identifies sampling as a possible contribution to the uncertainty in a result, procedures for
estimating the resulting uncertainty are not well developed and further, specific, guidance is
required. At this time this guidance document will only consider analytical MU, but will be
revised as further information, most notably that from the EURACHEM Working Group

[2] Regulation (EC) No 882/2004 of the European Parliament and of the Council of 29 April 2004 on official
controls performed to ensure the verification of compliance with feed and food law, animal health and animal
welfare rules, Official Journal of the European Union, L 165
[3] Regulation (EC) No 1829/2003 of the European Parliament and of the Council of 22 September 2003 on
genetically modified food and feed, Official Journal of the European Union, L 268/1
[4] Regulation (EC) No 1830/2003 of the European Parliament and of the Council of 22 September 2003
concerning the traceability and labelling of genetically modified organisms and the traceability of food and feed
products produced from genetically modified organisms and amending Directive 2001/18/EC, Official Journal of
the European Union, L 268/24
[5] IUPAC Project 2005-024-2-600 (Guidelines for validation (interlaboratory) of qualitative methods)

6
dealing with sampling uncertainty, becomes available. The issue has also been discussed in
the Codex Committee on Methods of Analysis and Sampling [6].
It is important to recognise that there is always a MU associated with an analytical result
reported by a control laboratory, whether or not the MU is reported. It should be noted that
control laboratories testing for compliance with regulations (EC) No 1829/2003 and
1830/2003 must report MU uncertainty together with the measurement result.

The first recognised approach to MU, the 'Guide to Expression of Uncertainty in

Measurement' (GUM) [7] lays down general rules for the expression and estimation of MU.
GUM introduces the idea of uncertainty and distinguishes it from errors. It furthermore
describes the steps involved in the estimation of uncertainty. GUM places emphasis on the
component-by-component approach, in which the method is dissected and incremental
calculations of uncertainty are made and eventually added up to provide a combined
uncertainty.
The evaluation of the measurement uncertainty for a method requires the analyst to look
closely at all the possible sources of uncertainty in the method concerned; this approach is
called 'bottom-up approach'. As a visualisation tool cause and effect diagrams also named
'fishbone diagrams' are used and practical studies are carried out to help identifying the major
source of uncertainty associated with the measurement. Figure 1 is giving examples of
possible sources of MU for DNA quantification by real-time PCR. By concentrating on the
major sources of uncertainty a good estimate of the uncertainty as a whole can be achieved,
for further details the reader is referred to other documents exploiting this approach [8].
Once the measurement uncertainty has been estimated for a certain method in a particular
laboratory, this estimate can be applied to subsequent results, provided that they are carried
out in the same laboratory under the same conditions and that quality control data justify the
correctness of this approach. In practice collaborative trail data are used to verify that the
'bottom-up approach' chosen covers sufficiently all potential variations, if this is not the case
major sources of uncertainty were incorrectly identified.

[6] CCMAS (April 2006): Uncertainty of Sampling, CX/MAS 06/27/10

[7] ISO (1995): Guide to Expression of Uncertainty in Measurement, ISBN-9267-101889
[8] EURACHEM / CITAC (2000): Quantifying Uncertainty in Analytical Measurement, second edition,
EURACHEM Secretariat, BAM, Berlin ([Link]

7
 PCR
measurement
result Storage Data analysis
DNA of
quantity extracts

PCR
DNA quality conditions
and purity
PCR
inhibition
PCR
instrument

GM
Equal concentration
DNA PCR
extraction inhibition
of targeted
sequences PCR
instrument Dilution
Particle
size
Storage of
Sample
PCR calibrants
homogeneity
conditions
Type of
DNA calibrant, Repeatability
extraction commutability
Calibration

Figure 1: Cause and effect diagram ('fishbone diagram') visualising examples for possible MU
contributions for DNA quantification by real-time PCR. Contributions to the reproducibility are shaded
grey.

It is desirable that information available as a result of accreditation requirements is used by

the laboratories when estimating MU in order to avoid duplicate work being carried out.
Additionally there has been some criticism of the practicability of the 'bottom-up approach'
for the estimation of MU and two alternative ways are presented in this document.
In the GMO food and feed sectors, where there is a high emphasis being placed on the use of
methods of analysis validated through collaborative trials, information obtained from such
trials can be used in many situations to estimate MU. However, it is important to note that
data from collaborative trials can only be used if the findings are confirmed by internal quality
control procedures.

It is recommended that GMO laboratories use information derived from the following
procedures to aid their estimation of the uncertainty of measurement results:
• The ISO Guide to expression of uncertainty in measurement [7]
• The EURACHEM Guide to quantifying uncertainty in analytical measurement [8]:
A. component-by-component approach
B. use of collaborative trial and/or internal quality control data
These documents recommend procedures based on a component-by-component approach,
method validation data, internal quality control data and proficiency test data.
In many cases the overall uncertainty may be evaluated by an inter-laboratory (collaborative)
trial by a number of laboratories and a number of matrices:

8
• IUPAC/ISO/AOAC International protocol for the design, conduct and interpretation of
method performance studies [9]
• ISO 21748 guidance document for the use of repeatability, reproducibility and trueness
estimates in measurement uncertainty estimation [10]
• ISO/IEC 17025 outlining the internal quality control approach [11]
• Nordic Committee on Food Analysis (NMKL) suggesting the use of experimental data
generated within the individual laboratory [12]
• Nordtest report outlining the use of data obtained on routine samples for the estimation of
MU [13]

The above, and others, notably:

• The concept set by Commission Decision 2002/657/EC implementing Council Directive
96/23/EC concerning the performance of analytical methods and the interpretation of
results [14]
• An AOAC international approach [15] is outlined in the DG SANCO MU report [1].

It is recognised that further procedures for the estimation of MU may exist and are being
developed, and that, in this evolving situation, further recommendations will be made as to
acceptable procedures. As an example, it is anticipated that procedures based on results
obtained from participation in proficiency testing schemes will be developed.

For the implementation of decision thresholds an acceptable maximum MU needs to be

defined, if most of the results are expected to be close to the decision threshold. Each
laboratory has to prove that its own estimated MU is smaller than the accepted maximum
MU. A MU threshold has so far not been agreed in the area of GMO detection.

[9] Horwitz W (1995): Protocol for the Design, Conduct and Interpretation of Method Performance Studies, Pure
Appl. Chem., 67, 331-343
[10] ISO/TS 21748 (2004): Guidance for the use of repeatability, reproducibility and trueness estimates in
measurement uncertainty estimation
[11] ISO/IEC 17025 (2005): General requirements for the competence of testing and calibration laboratories
[12] NMKL (1997): Estimation and expression of measurement analysis in chemical analysis, procedure No5
[13] Magnusson B, Näykki T., Hovind H, Krysell M (2004): Handbook for Calculation of Measurement
Uncertainty in Environmental Laboratories, TR 537 of 2004-02
[14] Commission Decision 2002/657/EC of 12 August 2002 implementing Council Directive 96/23/EC
concerning the performance of analytical methods and the interpretation of results, Official Journal of the
European Union L221 (Text with EEA relevance) (notified under document number C (2002) 3044)
[15] Horwitz W (2003): The Certainty of Uncertainty Journal of AOAC INTERNATIONAL, 86, 109-111

9
1.3 State of the art of collaborative trials in GM quantification

Currently, more than 23 collaboratively tested quantitative real-time PCR-based methods,

with at least 6 participants per trial, have been published. The majority of collaboratively
tested methods were conducted with sample and calibration material such as ground seed as it
is available from IRMM. Material from the same source was used as unknown sample and as
material for the establishment of the calibration curve. Hence, any matrix effect which might
have an influence on the MU will not be identified by this approach. Some trials were
conducted using DNA that was simply provided by the organising institute and do not cover
the DNA extraction step. Several trials use a common source of primer, probes and
polymerase, which leads as well to a lower value for the reproducibility. In case only one type
of PCR instrument is used the reproducibility investigated is only valid for this type of
instrument. Additionally it should be noted that collaborative trials are never carried out under
routine conditions.

The potential bias associated with results may not reflect real measurements, where the
calibration matrix may not exactly match the analytical sample matrix. In order to estimate the
size of such effects a reconciliation procedure can be used. For an example see Annex II.

In general, a list of collaborative trials including results can be found in ISO standard 21570
[16]. Additionally method validations carried out by the Community Reference Laboratory
(CRL) for GMO food and feed can be found on the CRL homepage [17]. Codex Alimentarius
Commission is currently developing a document called 'Consideration of the methods for the
detection and identification of foods derived from biotechnology', which will list as well
methods validated through collaborative trial.

Whenever possible, if MU estimations are derived from collaborative trial data alone, then the
methodology used within the collaborative trial should include all experimental stages from
laboratory sample receipt to reporting the end result. For instance, if using collaborative trial
data from a trial that did not include the DNA extraction step the uncertainty arising from this
missing step has to be estimated on the basis of other data and added to the uncertainty
budget.

[16] ISO 21570 (2005): Foodstuffs – Methods of analysis for the detection of genetically modified organisms
and derived products – Quantitative nucleic acid based methods
[17] Homepage of the Community Reference Laboratory for GMO food and feed: [Link]

10
2 The relationship between the analytical result and
measurement uncertainty, with particular reference to
regulation (EC) No 1829/2003 and (EC) No 1830/2003

2.1 General description of the estimation and use of measurement uncertainty

The general process of estimating and using MU entails:

− the evaluation of all possible sources of uncertainty in the method concerned and
identification and combination of the major sources of uncertainty ('bottom-up approach');
or alternatively
− the use of collaborative trial and/or single-laboratory validation data and/or in-house
quality control data to estimate relation between sample concentration (c) and standard
uncertainty (u); and
− the calculation of the Critical Level (LC), Limit of Detection (LOD) and Limit of
Quantification (LOQ) from u

The estimation of MU is achieved by the estimation of two values u0, the standard uncertainty
associated with results of the measurement of low concentrations of GM, and RSU the relative
standard uncertainty associated with the results of the measurement of high concentrations of
GM (Section 2.4). The value for u0 can be estimated by extrapolation, provided that
measurement results for samples close to the LOQ are available. In the other case the analysis
of diluted samples should be considered. RSU can be estimated using collaborative trial data,
in-house quality assurance data or single laboratory validation data. Examples based on
collaborative trial and in-house quality assurance data are given below.

The estimates of the standard uncertainty can be based on collaborative trial data, provided
the laboratory can confirm, via internal quality control (IQC) data, that it is able to achieve the
same performance of the laboratories that took part in the trial. If the estimate of the standard
uncertainty derived from collaborative trial does not cover the complete reproducibility due to
the set-up of the trial, a reconciliation procedure should be used to estimate the contribution of
effects not investigated in the trial (Annex II). Each significant source of uncertainty not
covered by the collaborative trial data should be evaluated in the form of a standard
uncertainty and combined with the reproducibility standard deviation [8]. During the
'reconciliation' stage, it is necessary to identify any sources of uncertainty that are not covered
by the collaborative trial data. The sources which may need particular consideration are [8]:

• Sampling: Collaborative trials rarely include a sampling step. If the method used in-
house involves sub-sampling, or the measurand is estimating a bulk property from a small
sample, then the effects of sampling should be investigated and their effects included.
• Pre-treatment: In most trials, samples are homogenised, and may additionally be
stabilised, before distribution. It may be necessary to investigate and add the effects of the
particular pre-treatment procedures applied in-house.
• Method bias: Method bias is often examined prior to or during interlaboratory study,
where possible by comparison with reference methods or reference materials. Where the bias
itself, the uncertainty in the reference values used, and the precision associated with the bias
check, are all small compared to the repeatability standard deviation (sr), no additional

11
allowance need be made for bias uncertainty. Otherwise, it will be necessary to make
additional allowances.
• Variation in conditions: Laboratories participating in a trial may tend towards the
means of allowed ranges of experimental conditions, resulting in an underestimate of the
range of results possible within the method definition. Where such effects have been
investigated and shown to be insignificant across their full permitted range, however, no
further allowance is required.
• Changes in sample matrix: The uncertainty arising from matrix compositions or levels
of interferents outside the range covered by the trial will need to be considered.

In cases were no collaborative trial data are available, the MU can be estimated from
analytical results measured within the laboratory concerned. Preference should, however, be
given to data obtained through collaborative trials.

2.2 Calculation of measurement uncertainty from collaborative trial data and

confirmed by internal quality control data

A collaborative trial is usually carried out once as a part of method validation, or at relatively
infrequent intervals during the working lifetime of a method; it tells the analyst what
performance they can expect for the method applied. However, attention needs to be paid that
the MU estimated from a method validation carried out by collaborative trial covers all
measurement steps contributing to the MU (Figure 2).

In order to use MU estimates from collaborative trials properly, it is important that the same
replication (e.g. two PCRs of one DNA extract), more or less the same Ct values and design
of the calibration curve are used to produce a single measurement result in the laboratory.

12
 Measurement principle

Analytical method

Analytical procedure

Independent
calibration

Food/feed Measurement
sample Sample DNA extraction / Real-time PCR Data evaluation result
preparation purification

Measurement Uncertainty contributions

Repeatability of sample Repeatability of DNA Repeatability of real-time Repeatability of

preparation: extraction: PCR: calibration:
- Extraction behaviour - resulting DNA purity - inhibition - calibrator value
- homogeneity - extracted amounts - instrument effects - commutability of
- etc. - etc. - etc. calibrator
- etc.

Method validation
using gDNA solutions

Method validation using matrix CRMs

Method validation using samples from various sources

Figure 2: Contributions to the estimation of the MU within one laboratory

The purpose of IQC is to ensure that the performance of the analytical system remains
effectively unchanged during its application. IQC is required for accreditation according to
ISO/IEC 17025.

13
In general, sR determined in a collaborative trial can be used as an estimate of MU if the
following conditions are met:

• the collaborative trial studied the whole method using sample types similar to those
being measured in the laboratory;
• repeatability in the laboratory is similar to the repeatability reported in the
collaborative trial;
• within-laboratory reproducibility is not larger than the reproducibility reported in the
trial;
• results from the analysis of Certified Reference Materials (CRMs) in the laboratory do
not display significant bias, provided the relevant CRM properties correspond fully to
those of the test sample;
• a true value has been assigned to the test material independently of the test results.

If these conditions are not met then estimates of the additional uncertainty components must
be combined with the collaborative trial sR to provide an estimate of MU.

Within this sector, collaborative trials are often undertaken using samples containing a range
of concentrations of GM material, including the concentrations close to LOQ. The
uncertainties u0 and RSU can be estimated by plotting sR of collaborative trials against the
assigned value (or mean) of the concentration of GM and calculating the linear regression.
The intercept of a linear regression (or sR associated with the lowest concentration) can be
used as an estimate of u0 and the slope of the linear regression can be used as an estimate of
RSU. If the intercept is negative or the lowest concentration is close to the LOQ, sR associated
with the lowest concentration can be used as an estimate of u0.

CRMs are well suited for the estimation of bias and its control in IQC. Measurement variation
is best estimated and controlled by the repeated independent analysis of samples. If a CRM is
employed as an 'unknown sample' for IQC, material from a different source should be used
for calibration.

2.3 Calculation of measurement uncertainty from internal quality control data

MU can be estimated using the value of within-laboratory reproducibility; provided it could

be proven that no bias exists. The general approach is to estimate the value of within-
laboratory reproducibility (measurement variation) using the repeated independent analyses of
a range of real samples, and to carry out a bias control using CRMs. The use of normal quality
assurance procedures and measurements results obtained on routine samples should mean that
suitable data are readily available [12, 13]. For new methods, where quality assurance data is
not available, suitable analytical results can be obtained from a within-laboratory method
validation [13]. The approach is presented in Section 2.3.1.

In order to ensure that the analytical result and its uncertainty cover the true concentration of
GM in a sample, it is important to prove that no bias exists.
An example for the calculation of MU based on internal quality control data is given in Annex
III.

14
2.3.1 Within-laboratory reproducibility

The within-laboratory reproducibility can be estimated using measurements results obtained

on routine samples as outlined in the NMKL procedure No. 5 [12] or the Nordtest report [13].
The following approach is based on the Nordtest report [13], a calculation example can be
found in Annex III.

It is important that the estimation of MU covers all steps of the analytical procedure. Hence,
within-laboratory reproducibility should be determined by the repeated independent analyses
of samples in analytical runs that represent the long-term variation of analytical components
within the laboratory, e.g. different operators, stock solutions, new batches of critical
reagents, recalibrations of equipment. Also, samples should represent the different matrices
and concentrations to which the estimates of MU will be applied. In particular, samples with a
GMO content close to legal or contractual thresholds against which results will be compared
should be included (e.g. 0.9 % for approved events in unlabelled food products).

Uncertainty estimates should be updated by the addition of new results as they become
available. Once a database is established then it may be advisable to remove results that are
older than one year, from the estimation of measurement uncertainty, if new results are
generated frequently. In regular intervals revalidation should be considered.

Repeated independent results produced by at least 15 samples should be used. Usually there is
a limited resource that can be devoted to measurements that are used to estimate uncertainty.
Therefore, in order to maximise the matrices and concentrations studied it is recommended
that the smallest replication (i.e. two independent measurements (n = 2) per sample,) are
applied to the largest number of samples possible. In the context of the Nordtest [13]
'duplicate measurements' mean two independent examples of the measurement as it is usually
applied in the laboratory. For example, if it is the usual practice to report results as the mean
of two analyses then the two independent measurements will consist of two results each given
by the mean of two analyses.

The mean ( ci ) of two independent analytical results is calculated as:

c i ,1 + c i ,2
ci = Equation 1
2
ci ......... mean of two analytical result
ci ,1 ....... result of first analysis of sample i
ci ,2 ...... result of second analysis of sample i

The absolute difference ( d i ) between the first and the second analysis is calculated as:

d i = c i ,1 − c i ,2 Equation 2

d i ........ absolute difference between two analytical result

15
ci ,1 ....... result of first analysis of sample i
ci ,2 ...... result of second analysis of sample i

The relative difference between analyses (radi) is calculated as:

di
rad i = 100 Equation 3
ci
radi ........... relative difference
d i ........ absolute difference between two analytical result
ci ......... mean of two analytical result

Given a set of differences and relative differences calculated from the analysis of a number of
samples the average difference ( d ) and average relative differences ( rad ) can be calculated.

The within-laboratory sR is in the case of two independent measurements results (n = 2) given

by:

d d
sR = = Equation 4
d 2 1.13
sR .......... within-laboratory reproducibility standard deviation
d2 .......... constant depending on the number or independent measurements (n) [13]:

Table 1: Factor d2 depending on the number of independent measurement results carried out on one
sample
n d2
2 1.128
3 1.693
4 2.059
5 2.326
6 2.534
7 2.704
8 2.847
9 2.970
10 3.078

Note: In case more than two independent measurement results are measured, di reflects the
range between the lowest and the highest results measured.

The within-laboratory reproducibility relative standard deviation (RSDR) is given by

rad
RSD R = Equation 5
1.13
RSDR .... relative within-laboratory reproducibility
rad ....... average relative differences

16
2.3.2 Method and laboratory bias control

After the measurement of a CRM the absolute difference between the mean measured value
and the certified value can be calculated as:
Δm = c m − c CRM Equation 6
Δm .............. absolute difference between mean measured value and certified value
cm ............... mean measured value
cCRM ......... certified value

The uncertainty of Δm is uΔ, that is calculated from the uncertainty of the certified value and
the uncertainty of the measurement result according to:
u Δ = u m2 + u CRM
2
Equation 7
uΔ .......... combined uncertainty of result and certified value (= uncertainty of Δm)
um.......... uncertainty of the measurement result
uCRM ...... uncertainty of the certified value

The uncertainty of the measurement result (um) can be estimated with the help of the relative
standard deviation of the repeatability (sr) and d2 (Section 2.3.1). Alternatively the uncertainty
of the measurement results (um) can be estimated by dividing the standard deviation by the
square root of the number of measurements carried out:
sr
um = Equation 8
n
um.......... uncertainty of the measurement result
sr ........... standard deviation of the repeatability
n ........... number of independent measurement results

The expanded uncertainty UΔ, corresponding to a confidence level of approximately 95 %, is

obtained by multiplication of uΔ by a coverage factor k = 2.

U Δ = 2 *u Δ Equation 9
UΔ .............. expanded uncertainty of difference between result and certified value
uΔ ............... combined uncertainty of result and certified value

If Δm ≤ UΔ then there is no significant difference between the measurement result and the
certified value, meaning that the method does not have a bias. In case a bias was found the
cause has to be investigated and eliminated. Approaches to calculate a bias can be found in
GUM [2], but have to be considered carefully as a bias may be a constant or a factor.
The expanded uncertainties UCRM of each certified value are given on the certificate of each
individual reference material. The standard uncertainty, uCRM, of the certified value is
obtained by dividing the stated expanded uncertainty by the coverage factor given on the
certificate.
Note: The Nordtest report [13] mentions a factor of 2.8 which should not be mixed-up with
the coverage factor k = 2, leading to a confidence of about 95 %. Factor 2.8 can be used to

17
check if the MU estimated before is also applicable for measurements on a new sample. In
case of inhomogeneity of the sample or the method being out of control, the before calculated
MU might not be applicable any more. If the absolute difference between the two
measurements is higher than 2.8 times the standard deviation, the MU calculation has to be
reconsidered and /or the sample homogeneity questioned.
Furthermore, it should be noted that the CRM used for the bias control should not at the same
time be used for calibration. In case this can not be avoided the analysis of a CRM with a low
GM concentration calibrated with the diluted extracts of a CRM with a higher GM
concentration should be considered.

2.3.3 Estimation of the uncertainty component associated with bias

Uncertainty associated with bias should, where possible, be estimated by the measurement of
CRMs. The general approach is to measure the concentration of GM in a CRM in a number of
analytical runs. An estimate of the uncertainty associated with bias is gained by combining the
uncertainty associated with the mean measurement result with the uncertainty associated with
the certified value of the concentration of GM in the CRM.
Note: In case of a bias the experimental set-up needs to be changed until no bias is found.

The relative standard uncertainty associated with the bias (ubiasr) is given by:
2
RSD R 2 ⎛ u CRM ⎞
u biasr = + ⎜⎜ 100 ⎟⎟ Equation 10
n ⎝ c CRM ⎠
RSDR..... relative within-laboratory reproducibility
n ........... number of measurements
uCRM ..... standard uncertainty associated with the certified value of the CRM
cCRM ..... certified value of the CRM

Note: UCRM of each certified value is given on the certificate of each individual reference
material. The standard uncertainty, uCRM, of the certified value is obtained by dividing the
stated expanded uncertainty by the coverage factor given on the certificate.

2.3.4 Calculation of the relative standard uncertainty

The relative standard uncertainty (RSU) is calculated by combining relative variation (RSDR )
and relative uncertainty associated with bias (ubiasr) using:

RSU = RSD R2 + u biasr

2
Equation 11
RSDR .............within-laboratory reproducibility
ubiasr ...............standard uncertainty associated with biasr

Note: The individual standard uncertainties need to have the format of a standard deviation in
order to allow summing up. Independent uncertainties can be combined by taking the square
root of the sum of the individual squares. RSU and ubiasr are not completely independent from
each as all measurements are influenced by the RSDR (Equation 10 and 11). The effect is
considered to be negligible. Additionally it should be noted that in some cases either RSU, or
u0 can be zero.

18
The laboratory should generally explain how the MU has been calculated. Additionally the
coverage factor applied and the corresponding uncertainty levels should be stated. A general
explanatory note can be prepared to ease communication.

2.4 Evaluation of measurement uncertainty

Given of absolute standard uncertainty (u0) and relative standard uncertainty (RSU) estimated
from collaborative trial results (Section 2.2) or in-house quality control data (Section 2.3) then
the standard uncertainty u associated with a measurement result c is given by:

u = u 02 + (c × RSU )2 Equation 12
uo ...................absolute standard uncertainty
c.....................measurement result
RSU ...............relative standard uncertainty

Note: It has to be stressed that Equation 12 is valid under the assumption that u0 is constant
and RSU proportional to the concentration c. This assumption should be checked using in-
house validation data. A situation can occur, that u0 is so small that it can be neglected (see
example in Annex III).

Regulations (EC) No 1829/2003 and (EC) No 1930/2003 set a labelling threshold for the total
authorised GMO presence on an ingredient basis. As such, the GMO contents for various
events of one ingredient (e.g. MON 810 maize and 1507 maize) must be added together and
the uncertainties associated with each individual GMO measurement combined. The MU of
various methods can be combined by adding the squares and taking the square root of the
sum:

uc = ∑i =1 ,n
u meth
2
,i Equation 13
uc ...................combined standard uncertainty associated with the measurement result for one ingredient
n ....................number of method applied
umeth,i ..............absolute standard uncertainty of individual method

The expanded uncertainty U (giving a confidence level of approximately 95 %) is given by:

U = 2 ×u c Equation 14
uc ...................combined standard uncertainty associated with the measurement result for one ingredient

19
2.5 Calculation of the critical level

The critical level (LC) is the lowest measurement result that demonstrates with sufficient
confidence that the analyte is present (Figure 3). LC is given by

LC = 2 × u 0 ` Equation 15
uo ...................absolute standard uncertainty

k×u0 α

0 LC
Figure 3: The LC is the lowest measurement result that demonstrates with sufficient confidence that the
analyte is present (concentration > 0). If a value of k = 2 is used to estimate LC then the probability (α) of
a sample containing none of the analyte giving a result above LC is 2.5 %.

2.6 Calculation of the limit of detection and quantification

Figure 4 illustrates the connection between LC and LOD. The LC is a statistically calculated
number. If the result of the measurement is less than LC, then we can explain (with 95 %
confidence) that the current sample does not contain any analyte. We use a t-statistic to decide
if the expected value of GM concentration is 0 or not. If our measurement result is smaller
than LC then we can report with 95 % confidence, that the sample does not contain any GM.
If the measurement result is above LC, we are sure that GM is present. It should be noted that
the probability to take a result for a blank is 50 % for measurement results below the LC.
While LC refers to the measurement result, the LOD and LOQ refer to the measured GMO
concentration. Above the LOD, the probability to wrongly take a blank for a result is the same
as for the LC while the probability to incorrectly interpret a signal for the presence of a GM is
5 %. It will be sure to detect a GM if the measurement result is above the LOD. The LOQ is
the measured GM concentration at which the measurement uncertainty is below a certain
(arbitrarily chosen; usually 30 %) value.

20
 β

0 LC LOD
Figure 4: The LOD is the lowest true concentration that we can be sufficiently confident of detecting. This
means it is the lowest true concentration that is unlikely to give a result below the critical level. If it is
estimated as the lowest concentration where the lower limit of the expanded (k = 2) uncertainty is greater
then the LC then the probability (β) of not detecting the analyte present at a concentration equal to the
LOD is 2.5 %

The LOD is given by:

4u 0
LOD = Equation 16
1 − 4RSU 2
uo ...................absolute standard uncertainty
RSU ...............relative standard uncertainty

The LOQ is given by:

u 02
LOQ = Equation 17
RSU MAX
2
− RSU 2
uo ...................absolute standard uncertainty
RSU ...............relative standard uncertainty
RSUMAX .........largest acceptable relative standard uncertainty

Note: Where the value of LOQ calculated using Equation 17 is less than the value of LOD
then LOQ = LOD.
For the calculation of uo the user needs to investigate whether the lowest concentration
analysed is close to the LOQ of the method (Section 2.2).

Where RSUMAX is the largest acceptable relative standard uncertainty that can be associated
with results. By collaborative trial we can consider it as the greatest acceptable gradient of the
regression line.

Note: LOD and LOQ are frequently estimated as multiples of u0 with the LOD equal to 4-
times u0 and the LOQ 9-times u0. This approach would require a constant u0, as this is not
given for values measured below the LOQ, the approach given here was favoured. However,
estimating LOD and LOQ as multiples of u0 could be still used, when the necessary
precautions are taken.

21
2.7 Reporting results

Measurement results should be reported as follows:

Where c > LOQ

' Concentration = (c ± U)
The uncertainty reported with this result is an expanded uncertainty calculated from a
standard uncertainty using a coverage factor of 2. It is equivalent to a confidence level of
approximately 95 %.'

Where LC < c < LOQ

Report ' Concentration = (c ± U) 'unless it is not possible to estimate concentrations below

LOQ (e.g. where LOQ is also the lowest point on the calibration curve).
If this is the case report the result as ' Concentration ≤ (LOQ + ULOQ ) ' where ULOQ is the
expanded uncertainty at a concentration equal to the LOQ.

Where c < LC

' Concentrat ion < LC

The target GM has not been detected in this sample.'

2.8 Estimation of the uncertainty associated with the mean

Where the RSU associated with an individual measurement result is estimated from the results
of a collaborative trial ( RSU = RSDR ) then the RSU associated with the mean of n
independent replicates is given by:

RSD rep
2
(n − 1)
RSU n = RSD R − 2
Equation 18
n
RSUn ..............RSU associated with the mean result
RSDR .............relative standard uncertainty
RSDrep ............relative standard deviation describing between-replicate variation
n .................... number of independent measurements

Where RSU n is the relative standard uncertainty associated with the mean result and RSDrep
is the relative standard deviation describing between-replicate variation. The value of RSDrep
can be estimated using the method described in Section 2.3.1.

22
2.9 How to use measurement uncertainty

The MU report of DG SANCO [1] states that Enforcement Authorities shall use the
measurement uncertainty associated with an analytical result when deciding whether an
analytical result falls within the specification for food and feed control purposes. The way that
MU is to be used by Enforcement Authorities must be taken into account when analytical
specifications are discussed. In practice, the analyst will determine the analytical result and
estimate the MU at that level. The value obtained by subtracting the uncertainty from the
reported concentration, is used to assess compliance. Only if that value is greater than the
maximum concentration stipulated in legislation, it is sure 'beyond reasonable doubt' that the
sample concentration of the analyte is greater than that prescribed by legislation.

The estimated MU must be reported together with the measurement result. The uncertainty is
of special importance, when the range of the expanded uncertainty encompasses the legal
limit [11].

23
3 Further considerations
The authors consider that the two approaches given are efficient ones. However, other
approaches to estimate the MU may be used. Furthermore, the authors wish to communicate
further considerations which should be made when estimating the uncertainty linked to GMO
quantification measurements by real-time PCR. In general these issues are currently
recognised and discussed and their implications are evaluated at present.

Regulation (EC) No 1829/2003 and (EC) No 1930/2003 set a labelling threshold for the total
authorised GMO presence on an ingredient basis. The MU and LODs associated with a
sample are influenced by the level to which the ingredient itself is present. In compound food
and feed the ingredient under investigation resembles less than 100 %. Although the GMO
concentration of interest from a legislative perspective is the same, the number of DNA
targets can be much lower in compound stuff, which can lead to quantifications close to the
LOD.
Recent years have witnessed major advances with respect to technology and methods used to
identify ingredients derived from GM materials, such as real-time PCR platforms and
microarrays. However, this rapid advance in technological expertise has not been mirrored by
a concurrent progression in the bio-analytical measurement community. Until now data
handling of data obtained during GM detection has not be standardised [18].

The estimation of MU in any analytical approach attempts to identify all components that add
significant variability to the end result. As such, MU estimation must take into account the
whole analytical process from the initial generation of the result through to its final reporting.
Whilst most MU estimation is focused on the practical experiments associated with the
analytical process, little work has been done to explore the analysis of the data associated with
such experiments. Example approaches to data handling that can contribute to MU are shown
in Annex IV.

Statistical consideration of the amplification process used in quantitative PCR determinations

suggests that the variation displayed by measurement results are described by a mixture of
normal, binomial, and lognormal distributions, dominated by the latter two. A recent trial [19]
has shown that GMO proficiency test results consistently follow a positively skewed
distribution. Log-transformation prior to calculating z-scores is effective in establishing near-
symmetric distributions that are sufficiently close to normal to justify interpretation on the
basis of the normal distribution. It is therefore recommended that data from real-time PCR
analyses should be log-transformed before calculation of the MU.

The estimation of the uncertainty associated with qualitative analysis of unauthorised GMOs
is more difficult to determine than that associated with quantitative analysis of authorised
GMOs because of the lack of validated methods, calibrants and CRMs. Generally acceptable

[18] Burns M, Valdivia H, Harris N (2004): Analysis and interpretation of data from real-time PCR trace
detection methods using quantitation of GM soya as a model system, Analytical and Bioanalytical Chemistry,
378(6), 1616-1623
[19] Thompson M, Ellison SLR, Owen L, Mathieson K, Powell J, Key P, Wood R, Damant A (2006): Scoring
in Genetically Modified Organisms Proficiency Tests Based on Log-Transformed Results, JAOAC, 89(1), 232-
239

24
guidance on this topic is still under development. However, suggested guidance and an
example of the estimation of LC and LOD for qualitative analysis is given in Annex IV.

25
Acknowledgements
The authors would like to thank Patricia Bonner, Andrew Damant, Emile Laurensse, Roberta
Onori and Boldiszar Vadja from the ENGL, as well as Anja Bråthen Kristoffersen from the
National Veterinary Institute (Olso, NO) for their valuable comments. At the time the authors
would like to thank Andrej Blejec from National Institute of Biology (Ljubljana, SI), Hendrik
Emons and Thomas Linsinger from the IRMM (Geel, BE) for their important contributions to
this document.

26
ANNEX I: Definitions applicable to GMO analysis
Measurement Uncertainty (MU)
Parameter, associated with the result of a measurement, that characterises the dispersion of the
values that could reasonably be attributed to the measurand [20].

Notes:
1. The parameter may be, for example, a standard deviation (or a given multiple of it), or the
half-width of an interval having a stated level of confidence.
2. Uncertainty of measurement comprises, in general, many components. Some of these
components may be evaluated from the statistical distribution of results of a series of
measurements and can be characterised by experimental standard deviations. The other
components, which can also be characterised by standard deviations, are evaluated from
assumed probability distributions based on experience or other information.
3. It is understood that the result of a measurement is the best estimate of the value of a
measurand, and that all components of uncertainty, including those arising from systematic
effects, such as components associated with corrections and reference standards, contribute to
the dispersion.

Accuracy
The closeness of agreement between a test result and the accepted reference value (adopted
from [21]).

Applicability
The description of analytes, matrices, and concentrations to which a method can be applied.
(modified from [22]).

Bias
Difference between mean measured value from a large series of test results and an accepted
reference value (a certified or nominal value). The measure of trueness is normally expressed
in term of bias [12].

Critical level (LC)

The lowest measurement result that demonstrates with confidence that the analyte is present
[modified from 23].

Dynamic range - Range of quantification

The range of concentrations over which the method performs in a linear manner with an
acceptable level of accuracy and precision.

Expanded measurement uncertainty

The expanded uncertainty (U) allows to calculate an interval within which the value if the
measurand is believed to lie with a higher level of confidence. U is obtained by multiplying
the combined standard uncertainties by a coverage factor k. The choice of the factor k is based

[20] ISO (1993): International vocabulary of basic and general terms in metrology, second Edition
[21] ISO/FDIS 3534-1 (2006): Statistics - Vocabulary and symbols - Part 1: General statistical terms and terms
used in probability
[22] Codex CX/MAS 02/4 (2002): Proposed draft guidelines for evaluating acceptable methods of analysis
[23] IUPAC recommendation (1995): Nomenclature in evaluation of analytical methods including detection and
quantification capabilities, Pure & Appl. Chem., 67(10), 1699-1723

27
on the level of confidence desired. For an approximate level of confidence of 95 %, k is
always 2 if the degree of freedom is > 2 [adopted from 8].

Limit of Detection (LOD)

Limit of detection is the lowest concentration or content of the analytes that can be detected
reliably, but not necessarily quantified (slightly modified from [24]). LOD is generally
expressed as the amount of analyte at which the analytical method detects the presence of the
analyte at least 95% of the time (< 5% false negative results).

Limit of Quantification (LOQ)

The limit of quantification of an analytical procedure is the lowest amount or concentration of
analyte in a sample, which can be quantitatively determined with an acceptable level of
precision and accuracy (modified from [24]).

Practicability
The ease of operations, in terms of sample throughput and costs, to achieve the required
performance criteria and thereby meet the specified purpose (modified from [24]). Generally,
the method should be practical for its intended purposes.

Repeatability standard deviation (RSDr)

The standard deviation of test results obtained under repeatability conditions. Repeatability
conditions are conditions where test results are obtained with the same method on identical
test items in the same laboratory by the same operator using the same equipment within short
intervals of time (adopted from [21]).

Reproducibility standard deviation (RSDR)

The standard deviation of test results obtained under reproducibility conditions.
Reproducibility conditions are conditions where test results are obtained with the same
method on identical test items in different laboratories with different operators using different
equipment [21].

Recovery
Proportion of the amount of analyte, present in or added to the analytical portion of the test
material, which is extracted and presented for measurement.

Robustness
The robustness of an analytical method is a measure of its capacity to remain unaffected by
small, but deliberate variations in method parameters and provides an indication of its
reliability during normal usage (adopted from [25]).

Sensitivity
The sensitivity of a method is a measure of the magnitude of the response caused by a certain
amount of analyte.
The method should be sensitive enough in order to be able to detect/quantify with respect to
the thresholds as provided in the relevant legislation.

[24] ISO/FDIS 24276 (2005): International Standard, Foodstuffs – Methods of analysis for the detection of
genetically modified organisms and derived products – General requirements and definitions
[25] EMEA (1995): Validation of analytical methods: definitions and methology, CPMP/ICH/381/95,
[Link]

28
Since sensitivity is method- and purpose-dependent it should be specified in the protocol. A
reasonable goal for sensitivity is that required to meet levels specified in contracts, with a
reasonable certainty that the level does not exceed the required limit.
Sensitivity as a term is used in two different ways - LOD and the slope of a curve. The use of
the LOD is the preferred term to use as a measure of the ability of a method to detect a small
amount of analyte.

Specificity
Property of a method to respond exclusively to the characteristic or analyte of interest.

Standard uncertainty
Uncertainty of the result of measurement expressed as a standard deviation [7].

Trueness
The closeness of agreement between the average value obtained from a large series of test
results and an accepted reference value (adopted from [21]).

29
ANNEX II: Example for an international collaborative trial to
determine accuracy of a real-time PCR procedure

This Annex gives a detailed description of the concept and execution of an international
collaborative trial in order to validate a real time PCR based method to quantify Roundup
Ready® soya (RRS).

The collaborative trial was carried out in the year 2000. In total 31 laboratories took part in
the trial, either using the ABI Prism® 7700 SDS (14 participants), ABI Prism® 5700 SDS
(5 participants) or the LightCycler® system (12 participants).

In total five unknown samples containing between 0.1% and 5 % (m/m) of CRM IRMM-
410R [26] and one sample consisting of texturized vegetable protein (TVP) containing 2 %
(m/m) of the soya bean line GTS 40-3-2 were used in this trial.

The trial encompassed the whole procedure as needed for analysing a test sample, including
the DNA extraction carried out according to a pre-scribed protocol using the classical CTAB
solution and the PCR step. For each sample two DNA extractions have been analysed in
parallel. Each test sample was analysed in 3 independent replicates. The mean value together
with the standard deviation was reported.

For the relative quantification, separate standard curves were established for the reference
target sequence and for the GMO target sequence. Therefore, DNA from 5 % (m/m) CRM
IRMM-410R was extracted and used as a calibrant for both standard curves. At each of the
four calibration points, duplicate (ABI PRISM® 7700 SDS and GeneAmp®5700) or single
determinations (LightCycler System) were performed using a series of 1:4 dilution intervals.
Finally, the copy number measured for the unknown samples were obtained by interpolation
from individual standard curves. Subsequently the percentages were calculated by dividing
the copy number of the RRS copy number by the reference target sequence copy number and
by multiplication with 100.

For primer/probe systems and PCR conditions, respectively, see details in ISO 21570 [16].

Data on specificity, linearity, LOD and LOQ have been determined prior to the collaborative
trial.

Due to the concept of the trial, material from the same source was used as unknown samples
(0.1, 0.5, 1, 2 and 5 % (m/m) CRM IRMM-410R) and as calibrant (5 % (m/m) CRM IRMM-
410R). Therefore the set-up of the study is not well suited for the evaluation of the trueness of
the applied real-time PCR method. However, the results could be used for assessing the
precision of the applied method, whereby the described issues reflect a worst-case scenario
resulting in an underestimation of the precision of the applied real-time PCR method.

[26] Trapmann S, Le Guern L, Kramer GN, Schimmel H, Pauwels J, Anklam E, Van den Eede G, Brodmann P
(2000): The Certification of a new set of Reference Materials of Soya Powder with different Mass Fractions of
TM
Roundup Ready Soya, EC certification report EUR 19573 EN, ISBN 92-828-9639-0

30
 As an example of the statistical evaluation of the results, Table 2 summarises the reported
individual participant results using either the ABI Prism® 7700 SDS or ABI Prism® 5700
SDS. For checking significant outliers a Cochran and Grubs test was applied to the results
reported by the participants. Both independent results for each sample were averaged and
tested. After elimination of identified outliers for the 0.5, 1 and 5 % samples the mean
quantities and standard deviation were calculated.

Table 2 Validation data for ABI PRISM ™ 7700 SDS and GeneAmp® 5700 SDS

Sample Sample 1 Sample 2 Sample 3 Sample 4 Sample 5 Sample 6

0.1 (m/m) % 0.5 (m/m) % 1 (m/m) % 2 (m/m) % 5 (m/m) % 2 (m/m) % TVP
Number of laboratories 19 19 19 19 19 19
Number of outliers 0 2 1 0 1 0
Number of laboratories retained
19 17 18 19 18 19
after eliminating outliers
Mean value [m/m %] 0.11 0.49 1.00 2.27 5.11 1.71
Bias [%] 9 -1 0 13 2 -15
Repeatability standard deviation
0.04 0.12 0.21 0.25 0.53 0.48
sr
Repeatability relative standard
33 24 21 11 10 28
deviation RSDr [m/m %]
Repeatability limit r
0.10 0.33 0.59 0.71 1.48 1.34
(r = 2.8 x sr)
Reproducibility standard
0.05 0.3 0.28 0.71 1.38 0.55
deviation sR
Reproducibility relative standard
44 27 28 32 27 32
deviation RSDR [m/m %]
Reproducibility limit R
0.13 0.37 0.77 2.00 3.87 1.54
(R = 2.8 x sR)

31
 1.6
Reproducibility standard deviation (%)

1.4

1.2

0.8

0.6
y = 0.26x + 0.09
2
0.4 R = 0.98
0.2

0
0 1 2 3 4 5 6
Mean concentration (%)

Figure 5: Mean concentration (c) plotted against sR

In this case, the lowest analysed concentration is presumably lower than the LOQ. The value
of u0 is estimated from the value of sR associated with the lowest concentration sample:

u0 = 0.09

The value of RSU is estimated from the gradient of a linear regression line.

RSU = 0.26

Hence the standard uncertainty associated with a measurement result c is given by (from
Equation 14):

u = 0.09 2 + (0.26 × c )2

Hence, LC is equal to a measurement result of (from Equation 15) 0.18 % (m/m) GM

The LOD for the method is equal to a concentration of (from Equation 16) 0.49 % (m/m) GM.

If a relative standard uncertainty of less than 30 % is required for fit for purpose quantitative
results then the limit of quantification is given by (from Equation 17):

LOQ = 0.60 % (m/m)

Given a measurement result 2.3 % (m/m) GM the expanded uncertainty is given by (from
Equation 15). As generally applied for measurement uncertainties the value calculated for the
expanded uncertainty is rounded up:

32
U = 2 × 0.09 2 + (0.26 × 2.3)2 =1.3 % (m/m)

Hence, the result should be reported as:

'(2.3 ± 1.3) % (m/m) GM

The uncertainty associated with the result is an expanded uncertainty calculated from a
standard uncertainty using a coverage factor of 2. It is equivalent to a confidence level of
approximately 95 %.'

33
ANNEX III: Example for the evaluation of the measurement
uncertainty based on single laboratory results
Evaluation of within-laboratory reproducibility

36 routine samples containing different concentration levels of RRS are analysed twice
independently (Table 3) and are used to calculate the reproducibility standard deviation and
relative standard deviation.

Table 3: Measurement results obtained on routine samples (n = 2) and calculation of the relative
difference

Analysis GM concentration GM concentration mean difference relative difference

number ci,1 ci,2 ci di radi [%]
(Equation 1) (Equation 2) (Equation 3)
[m/m %] [m/m %]

32 0.104 0.101 0.102 0.003 2.9

12 0.155 0.147 0.151 0.008 5.3
30 0.142 0.170 0.156 0.028 17.9
26 0.177 0.174 0.176 0.003 1.7
29 0.220 0.320 0.270 0.100 37.0
34 0.295 0.254 0.274 0.041 14.9
17 0.328 0.262 0.295 0.066 22.4
6 0.280 0.340 0.310 0.060 19.4
35 0.303 0.331 0.317 0.028 8.8
15 0.347 0.414 0.381 0.067 17.6
22 0.360 0.485 0.423 0.125 29.6
4 0.400 0.600 0.500 0.200 40.0
16 0.691 0.614 0.652 0.077 11.8
20 0.698 0.700 0.699 0.002 0.3
10 0.641 0.823 0.732 0.182 24.8
23 0.998 0.931 0.965 0.067 6.9
8 1.352 1.349 1.351 0.003 0.2
21 1.493 1.671 1.582 0.178 11.3
14 1.948 1.250 1.599 0.698 43.7
9 2.086 1.733 1.909 0.353 18.5
18 2.017 1.879 1.948 0.138 7.1
33 1.750 2.202 1.976 0.452 22.9
13 2.331 2.466 2.398 0.135 5.6
36 2.800 2.500 2.650 0.300 11.3
1 4.300 3.900 4.100 0.400 9.8
2 4.900 4.800 4.850 0.100 2.1
28 5.600 5.700 5.650 0.100 1.8
25 15.270 15.240 15.255 0.030 0.2
11 23.224 20.197 21.710 3.027 13.9
5 24.000 23.000 23.500 1.000 4.3
3 36.000 41.000 38.500 5.000 13.0
27 45.000 46.000 45.500 1.000 2.2
19 53.985 56.187 55.086 2.202 4.0
24 50.091 60.808 55.449 10.717 19.3
7 71.000 82.000 76.500 11.000 14.4
31 94.000 91.000 92.500 3.000 3.2

34
Here sR is estimated from the six results with the lowest mean concentration and RSDR is
estimated from the remaining 30 results.

Hence,

d = 0.0304

0.0304
sR = = 0.0269 % GM
1.13

rad = 13.1 %

13.1
RSDR = = 11.5 %
1.13

Method and Laboratory bias control

For the bias control a CRM with a certified RRS concentration of (10.0 ± 1.6) g/kg was used
(ERM-BF410d [27]). The certificate states that a coverage factor of k = 2 was applied. uCRM is
therefore 0.8 g/kg.
Six independent measurements (n = 6) were carried out (Table 4).

Table 4: Measurement results obtained on a CRM

Analysis GM concentration Mean GM Standard

number c concentration deviation
[g/kg] cm s
[g/kg] [g/kg]
1 11.0
2 10.9
3 12.1 11.1 0.5
4 11.2
5 10.7
6 10.9

The standard deviation (s) is divided by the square root of the number of measurements (n), as
the average of the results is compared with the certified value. um is therefore estimated as
0.5/√6 g/kg = 0.2 g/kg.
Δm = c m − c CRM = 11.1 − 10.0 g/kg = 1.1 g/kg
uΔ = um2 + uCRM
2
= 0.22 + 0.82 g/kg = 0.82 g/kg

[27] Trapmann S, Catalani P, Conneely P, Corbisier P, Gancberg D, Hannes E, Le Guern L, Kramer GN,
Prokisch J, Robouch P, Schimmel H, Zeleny R, Pauwels J, Van den Eede G, Weighardt F, Mazzara M, Anklam
E.(2002): The Certification of Reference Materials of Dry-Mixed Soya Powder with different Mass Fractions of
Roundup ReadyTM Soya, EC certification report EUR 20273 EN, ISBN 92-894-3725-1

35
The expanded uncertainty UΔ is 2 uΔ = 1.64 g/kg. This is larger than the difference Δm
between the certified and the measured value. The measured mean value is therefore not
significantly different from the certified value and it can be concluded that the method has no
bias.

Uncertainty associated to the bias

Any bias that may be associated with results is less than the uncertainty associated with bias.
The standard uncertainty associated with this (ubiasr) is calculated as (Equation 10):
⎛ 0.8 x 100 ⎞
2 2
11.5
u biasr = +⎜ ⎟ = 9.3 %
6 ⎝ 10.0 ⎠
For
n ........... 6
RSDR .... 11.5 %
uCRM ..... 0.8 g/kg (U (k=2) = 1.6 g/kg)
cCRM ..... 10.0 g/kg

Calculation of the relative standard uncertainty

The relative standard uncertainty (RSU) is calculated by combining RSDR with the ubiasr
(Equation 11):

RSU = 11.5% 2 + 9.3% 2 = 14.79%

Evaluation of the measurement uncertainty

Given that for the sample under investigation a RRS concentration of 15 g/kg has been
measured, the standard measurement uncertainty is calculated according to Equation 12. As
outlined in Section 2.2, sR associated with the lowest concentration can be used as an estimate
of u0:

u = 0.0269 g / kg 2 + (15g / kg × 14.79% )2 = 2.22 g/kg

The value for u0 indicates already that it in this specific case it can be neglected. Hence, a
direct calculation as outlined in the Nordtest report [13], following Equation 11 would be also
possible:
RSU = 9.3% 2 + 11.5% 2 = 14.79 %
u = 14.79% x 15g / kg = 2.22 g/kg

The expanded uncertainty is calculated according to Equation 14 using a coverage factor of 2,

which gives a level of confidence of approximately 95%:

U = 2 × 2.22g / kg = 4.44 g/kg

36
Final step and reporting

Given that for the sample under investigation a RRS concentration of 15 g/kg has been
measured, the analytical result is reported as:

'Roundup Ready® soybean = (15 ± 5) g/kg

The expanded MU of relative 30 % is estimated from two independent analyses of real

samples. A coverage factor of 2 was used, corresponding to a confidence level of
approximately 95 %. No measurement bias has been detected.'

37
ANNEX IV: Example of data handling operations potentially
leading to uncertainty
This list is by no means exhaustive, but it provides examples of which areas of data handling
would benefit from increased standardisation, in order to reduce MU associated with a
reported result. This list should raise awareness regarding the number of data handling aspects
that can potentially give rise to MU. Additionally, although a number of current projects (e.g.
the European sixth framework programme: Co-Extra [28]) are attempting to address the area
of standardised data handling associated with GM results, there is little published literature
relating to standardised guidelines on the subject [18, 29]. Thus, the following areas of data
handling have been emphasised as potential factors that can account for MU in a reported
result, but additional work needs to be conducted in the GM and real-time PCR fields in order
to provide a standardised way to approach data handling.

• A calibration curve is often constructed by measuring an instrument response according to

a range of reference standards of known analyte concentration. The concentration of the
analyte in the sample unknown is then evaluated based on this calibration curve. The
construction of the calibration curve can affect the reliability of the result, and it needs to
be considered if the curve should be produced using average values, or values from
individual independent replicates.
• The model used to construct the calibration curve is another important consideration.
Typically, simple linear regression is used to produce the 'best fitting straight line' by
minimising the sum of the squares of the deviations between the observed and expected
values, given that a linear relationship exists. However, simple linear regression does not
take into account the variability associated with each standard on the curve. Whilst in
some experiments, linear regression is sufficient to give an acceptable response, it can bias
results particularly when the variability of standards at the extreme ends of the calibration
curve have different precision. This often occurs in routine GM analysis, and statistically,
alternative models that take into account this heterogeneity of variance would be more
suitable, for example weighted regression or censored regression.
• The majority of reported results associated with GM quantification are based on using
singleplex reactions within the laboratory. Singleplex reactions occur where the
endogenous target analyte and the transgenic target analyte are assessed in separate wells
or reaction vessels. However, an alternative approach is to use duplex reactions, where
both endogenous and transgenic target analytes are assessed in the same reaction vessel.
These two approaches have different characteristics associated with them in terms of use
of resources and costs, and competition between reactions. However, the different reaction
kinetics of the two approaches also dictate that the data can be evaluated according to a
variety of methods, and one method may not be the most appropriate way to handle both
sets of data.
• The analyte concentration of sample unknowns are typically estimated based on the
regression Equation associated with the calibration curve. Because of the nature of the

[28] Homepage of the European FP6 project Co-Extra – GM and non-GM supply chains: their co-existence and
traceability: [Link]
[29] Burns MJ, Nixon GJ, Foy CA, Harris N (2005): Standardisation of data from real-time quantitative PCR
methods - evaluation of outliers and comparison of calibration curves, BMC Biotechnology, 5:31 doi:
10.1186/1472-6750-5-31

38
 PCR data set and the target analyte concentrations involved, this normally involves log
transformation of data. A guidance document relating to how to evaluate sample
unknowns would be beneficial, as taking individual independent replicates and then log
transforming to give an average value, can give a very different result to averaging
individual replicate values before log transformation. Additionally, the confidence level
with which the final result is estimated will also differ according to what approach to
transforming data has been adopted.
• To help ensure that reaction conditions are similar between calibration curves, the slope of
calibration curves are often compared. This gradient of the calibration curve is indicative
of the efficiency of the PCR reaction, and for quality purposes it is also important that the
PCR efficiency of standards used to produce a calibration curve is similar to the PCR
efficiency associated with sample unknowns. This comparison has typically been
conducted using a subjective visual assessment, but a statistical approach based on
analysis of co-variance has been published that facilitates a more objective evaluation.
The production of guidelines on how to facilitate this could impact upon the assessment of
uncertainty in the reported result.
• Individual PCR measurements in a data set that are thought to be outliers have the
potential to introduce bias into a result. Standardised approaches to aid in successful
identification of outlying values given the typical non-normal distribution of some values
from real-time PCR, is only part of the issue associated with this problem. The second half
of the problem involves recommending an approach to facilitate an objective decision on
whether to include or exclude suspected outlying values, and this can impact greatly upon
the final measurement. An objective approach using a combination of graphical and
statistical techniques followed by identification and handling of outlying values using ISO
guidelines, has been published [29].

The limited list of data handling approaches described above requires careful thought
regarding the best statistical approaches to help interpret data. More work needs to be done at
an international level in order to provide a standardised guidance document that would
address all data handling approaches in this field. Production of such a standardised 'best
practice guidelines' document will facilitate more accurate analysis of data arising from GM
detection methods, and will contribute towards a greater understanding of the MU inherent
within different data handling approaches. This will help provide a more meaningful and
clearer interpretation of data arising from GM detection methods than is currently available.

39
ANNEX V: Data calculated on the bases of CRL collaborative trials
Table 5: LOD, LOQ, U and estimated enforcement level based on collaborative trial data

Matrix used for

validation (DNA Measurement unit Expanded Uncertainty U Estimated
extraction used during (k = 2) at labelling enforcement level
GM event method) validation LOD [%] LOQ [%] threshold (0.9 %) [%] [%]
Ring trial Roundup Ready (from Annex II)
RUR flour (CTAB) mass fraction GM 0.30 0.65 0.53 1.92

CRL ring trails including DNA extraction step

NK603 maize flour (CTAB) mass fraction GM 0.15 * 0.66 3.33
GA 21 maize flour (CTAB) mass fraction GM 0.36 * 0.58 2.46
MON 863 maize flour (CTAB) mass fraction GM 0.004 0.004 0.34 1.46

CRL ring trials without DNA extraction step

Bt 11 maize DNA copy number ratio 0.11 0.09 0.31 1.38
TC1507 maize DNA copy number ratio 0.10 0.10 0.42 1.67
T25 maize DNA copy number ratio 0.01 0.01 0.43 1.72
DAS 59122 maize DNA copy number ratio 0.25 0.22 0.25 1.21
RUR H7-1 sugar beet DNA copy number ratio 0.12 0.10 0.26 1.27
281-24-236 cotton DNA copy number ratio 0.14 0.13 0.31 1.35
3006-210-23 cotton DNA copy number ratio 0.15 0.15 0.40 1.62
LL62 rice DNA copy number ratio 0.13 0.11 0.25 1.24

* RSU is bigger than RSUmax. The method is not good enough to fulfil the set criteria. Either the criteria have to be set less strict or the methods have to be improved.

Furthermore it should be noted that reliable estimation of the enforcement levels requires the inclusion of the DNA extraction. Therefore
estimated enforcement levels based on ring trials using extracted DNA may underestimate the uncertainty and hence the estimated enforcement
level.
The data provided here can be used if the method used in the laboratory is exactly the same. Details related to the method validation can be found
on the homepage of the CRL for GMO food and feed [17].

40
 GM concentration predicted [%] 7.00

6.00

5.00

4.00

3.00

2.00

1.00

0.00
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
GM concentration measured [%]

Figure 6: Enforcement level calculated for RRS collaborative trial given in Annex II, expressed in mass/mass %

Using the data obtained during the method validation of RRS by collaborative trial (Annex II), it can be concluded that samples for which a GM
concentration above 1.8 % (dashed line) is measured (using the RRS method) contain more that than the legal threshold of 0.9 % GM (red line).

41
European Commission

EUR 22756 EN/2 – Joint Research Centre – Institute for Reference Materials and Measurements
Title: Guidance Document on Measurement Uncertainty for GMO Testing Laboratories
Author(s): S. Trapmann, M. Burns, H. Broll, R. Macarthur, R. Wood, J. Zel
Luxembourg: Office for Official Publications of the European Communities
2009 – 41 pp. – 21.0 x 29.7 cm
EUR – Scientific and Technical Research series – ISSN 1018-5593
ISBN 978-92-79-11228-7
DOI 10.2787/18988

Abstract
This technical report outlines the technical issues related to the estimation of measurement uncertainty (MU)
involved in the GMO sector. In particular it gives guidance to GMO testing laboratories how to estimate the
analytical variability of quantitative analytical results obtained by real-time PCR. This guidance document has
been written on request of the European Network of GMO Laboratories (ENGL) as a follow-up of a workshop
on Measurement Uncertainty in the GMO sector organised by the Institute for Reference Materials and
Measurements (IRMM), Geel, Belgium and held on 05.07.2005.
It is recognised that in order to be able to judge if an analytical results exceeds a threshold; the MU must be
estimated and reported together with the measurement result. Enforcement Authorities shall therefore estimate
the MU associated with an analytical result and use it to decide whether an analytical result falls within the
specification of food and feed control. The value obtained by subtracting the expanded uncertainty from the
reported concentration is used to assess compliance. Only if this value is greater than the legal threshold, it is
sure 'beyond reasonable doubt' that the sample concentration of the analyte is beyond what is permissible.
Two selected approaches for the estimation of MU are presented in detail; references to alternative
approaches are given. The first approach presented in detail is using data from collaborative trial in
combination with in-house quality control data for the estimation of MU. Prerequisites for the use of such
collaborative trial data are outlined. In case no suitable collaborative trial data are available, an alternative
approach using data from within-laboratory samples for the estimation of MU is presented.
 How to obtain EU publications

Our priced publications are available from EU Bookshop ([Link] where you can place
an order with the sales agent of your choice.

The Publications Office has a worldwide network of sales agents. You can obtain their contact details by
sending a fax to (352) 29 29-42758.
 LA-N1-22756 EN-C
The mission of the JRC is to provide customer-driven scientific and technical support
for the conception, development, implementation and monitoring of EU policies. As a
service of the European Commission, the JRC functions as a reference centre of
science and technology for the Union. Close to the policy-making process, it serves
the common interest of the Member States, while being independent of special
interests, whether private or national.