See discussions, stats, and author profiles for this publication at: [Link]

net/publication/50196781

Uncertainty of measurement and conformity assessment: A review

Article  in  Analytical and Bioanalytical Chemistry · February 2011

DOI: 10.1007/s00216-011-4776-y · Source: PubMed

CITATIONS READS

26 1,313

2 authors:

Desimoni Elio Barbara Brunetti

University of Milan University of Milan
117 PUBLICATIONS   2,239 CITATIONS    40 PUBLICATIONS   703 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Characterization of new materials View project

Development of nano-based sensors for rapid food physical-chemical analysis View project

All content following this page was uploaded by Desimoni Elio on 16 July 2014.

The user has requested enhancement of the downloaded file.

Anal Bioanal Chem (2011) 400:1729–1741
DOI 10.1007/s00216-011-4776-y

REVIEW

Uncertainty of measurement and conformity assessment:

a review
Elio Desimoni & Barbara Brunetti

Received: 27 October 2010 / Revised: 1 February 2011 / Accepted: 4 February 2011 / Published online: 25 February 2011
# Springer-Verlag 2011

Abstract The uncertainty of measurement is the key indica- a quantitative indication of its quality, so that the user of the
tor of the quality of any experimental result. Proper result can assess its reliability [1]. Such an indication is
consideration of this uncertainty is imperative when testing a represented at best by the measurement uncertainty (MU),
sample against legal/compositional limits. This task can be the value associated with the result of a measurement that
quite challenging when the entity measured in the investigated characterises the dispersion of the values that could
sample is so close to the limit that its uncertainty, however reasonably (e.g. with a given probability/confidence level)
estimated, critically affects decision-making. This explains be attributed to the measurand [1]. As emphasized by the
the many literature contributions discussing the problem. ISO Guide to the Expression of Uncertainty in Measure-
Even though some of the most authoritative organisations ment (GUM), without a clear indication of their uncertainty,
have issued specific guidelines aimed at assisting the staff measurement results cannot be compared either among
involved in such measurements, several aspects of conformity themselves or with reference values given in a specification
testing are still debated in the literature. In this review, after a or standard [1].
short outline of existing information, somewhat more detailed Unfortunately, when dealing with measurements aimed
insight is given into the guidelines of ASME, ISO, and at evaluating conformity with some specification, the
Eurachem/CITAC, because they are the most useful tools for matter becomes quite complex when the measured entity
operators of testing and calibration laboratories. Some aspects in the sample under investigation (e.g. the concentration
of Council Directive 96/23/EC are also discussed. Insight into in chemical analyses) is so close to the specification that
the contents of the mentioned documents enables emphasis of the MU, anyhow estimated, critically affects decision-
analogies and discrepancies. making. This explains the uninterrupted appearance of
contributions devoted to discussion of the multi-faceted
Keywords Conformity testing . Decision rules . Limiting aspects of considering the MU when assessing confor-
values . Specification limits . Guard band . Uncertainty of mity to legal or compositional limits [2–32]. As can be
measurement seen, these papers, listed in chronological order, span the
last fifteen years. Noticeably, a few of them appeared even
after some of the most authoritative organisations issued
Introduction specific guidelines aimed at assisting the staff involved in
such measurements [33–36]. This is probably indicative of
It is well known that, when reporting the result of a a still ongoing debate.
measurement of a physical quantity, it is mandatory to give In this review, after a short outline of existing literature
information a somewhat more detailed insight is given into
the guidelines of ASME [33], ISO [34], and Eurachem/
E. Desimoni (*) : B. Brunetti CITAC [35], because they are the most useful tools for
DiSTAM, Università degli Studi di Milano,
operators of testing and calibration laboratories. Of course,
Via Celoria 2,
20133 Milano, Italy this paper is not aimed exhaustively at presenting the three
e-mail: [Link]@[Link] standards, a task that would obviously require much more
1730 E. Desimoni, B. Brunetti

extensive types of presentation. Its only objective is to

assist a reader in approaching those guidelines and Measurement result
comparing the basic concepts presented therein. Council with its uncertainty
Directive 96/23/EC [36] is also mentioned and partly A interval
discussed. Insight into some of the basic contents of these
four last documents [33–36] enables emphasis of a few B
analogies and discrepancies. Upper
limiting
C value
Interpreting analytical results affected by measurement
uncertainty against limiting values D

Conformity1 testing is the systematic examination of the

extent to which an entity conforms to a specified criterion E
[34]. A specification for a measurable characteristic (for
example, the concentration in chemical analysis) is usually
F
formulated as a single limiting value, e.g. an upper or a Lower
lower limiting value, LVU or LVL, respectively, or as a set limiting
of limiting values, e.g. both an upper and a lower limiting G value
value. The term specification limit, SL, is also used in place
of limiting value.
H
Most frequently, when dealing with a set of limiting
values, permitted values of the characteristic are those
falling within the LVL–LVU interval. But, in some cases,
permitted values are those falling outside that interval. An Fig. 1 Assessing conformity when the measurement result is more or
example of this last situation is that relevant to some less close to a higher or lower limit. In these figures, as usual, the
inflammable compounds [13]: if their concentration in air is permitted values of the characteristic are those falling within the
LVL–LVU interval
below a given LVL, the gaseous mixture cannot burn or
explode whereas if it is above the LVU, the mixture can
burn but it cannot explode. Within the two limits the does not comply with the specification, because the whole
mixture explodes. uncertainty interval is above the limit, whereas in case D
Several papers have presented the very basic aspects of the product complies with the specification, because the
interpreting how experimental results, being affected by whole uncertainty interval is below the limit. Of course,
MU, should be interpreted against some specification these two cases do not pose any problem of decision
limits. The problem is schematised by more or less detailed making at the selected confidence level.
figures in which different measurement results, with their In the two remaining cases, B and C, the uncertainty
MU interval, are compared with one LV or with a set of LVs interval encompasses the LVU, so knowledge of the
[5, 7, 10, 16, 20, 26, 27, 31, 37–39]. The uncertainty measurement result does not enable any decision making:
interval is estimated according to a given confidence level, the result lies in the so-called uncertainty range. Case B does
usually 95% (see the next section). Most frequently, the not allow statement of conformity at the chosen level of
problem is presented as in Fig. 1, or as in its top half. Four confidence (for example, 95%) even if non-conformity is
possible experimental situations are recognisable at each more probable than conformity. The opposite applies in case
LV. Occasionally, an additional situation is added in which C, in which conformity is more probable that non-
the measurement result coincides with a limit [7, 10, 37]. In conformity. Then, cases B and C are those requiring further
one case, eight different situations are considered [16]. But investigation. The first possibility is that allowed by using a
the four situations A–D of Fig. 1 allow any possible measurement method precise enough to reduce the MU
reasoning. By limiting the attention at the upper limiting interval at the level necessary to move from case B to case A
value only, one can easily argue that in case A the product or from case C to case D. This solution is not always
possible, and usually implies a substantial rise of analysis
cost and time. Alternatively, one can apply the two-stage
1
In many of the references cited in this paper, the word “compliance”
procedure suggested by ISO 10576–1 [34] (see the section
is used as a synonym of conformity. Strictly speaking, compliance
indicates the action of making something conform or fulfilling a dealing with that standard). Again, additional measurements
regulatory requirement. are necessary so that the cost and time of the analysis are
Uncertainty of measurement and conformity assessment: a review 1731

accordingly increased. Some authors have suggested that, in The coverage factor is a multiplier chosen on the basis of
cases such as B and C, stating conformity or non-conformity the desired level of confidence to be associated with the
with a level of confidence lower than 95% is better than interval defined by U=k·uc. Most frequently, k is in the
nothing [7, 10, 37]. However, such a possibility does not range 2 to 3 [1]. When the normal distribution applies and u
always appear realistic, as in the case of court cases in which is a reliable estimate of the standard deviation of the
the conformity or non-conformity statement must be measurand, U=2·uc defines an interval having a level of
“beyond reasonable doubt” [3]. confidence of approximately 95% (more exactly, a level of
Of course, the discussion about a lower limiting value confidence of 95.45%), and U=3·uc defines an interval
(half bottom of Fig. 1) mirrors that detailed above. having a level of confidence of approximately 99% (more
It has also been reported that, even at present, it is exactly, a level of confidence of 99.73%). However, some
possible that some specification makes no reference to aspects relevant to the concept of uncertainty still deserve
properly considering the effects of MU on the assessment specific comments.
of conformity. In these cases “… it may be appropriate for Notwithstanding the detailed and authoritative docu-
the user to make a judgement of conformity, based on ments intended to explain the meaning of uncertainty, it
whether the test result is within the specified limits with no was noticed that surprisingly invalid MUs were sometimes
account taken of the uncertainty. This is often referred to as provided, often labelled as “standard deviation” [18].
a shared risk, since the end-user takes some of the risk that Examples of such erroneous estimates are an uncertainty
the product may not meet the specification after being resulting from calibration only, a repeatability standard
tested with an agreed measurement method” [37]. deviation and a linearity of some calibration curve.
Finally, recent papers, when examining uncertain cases Moreover, using k=2 or 3 can no longer be accepted if
such as B and C in Fig. 1, also had the objective of the combined uncertainty has too few degrees of freedom
evaluating the effect of MU on producer’s and user’s risk [1, 19, 26]. If the effective number of degrees of freedom,
(usually associated to type I and, type II errors, respectively) νeff, is too low (for example, fewer then six according to
in classification and conformity assessments [32]. Ref. [43], fewer than 30 according to Ref. [19]) the
Student-t distribution is the most appropriate (approximate)
choice for associating a level of confidence with U. It is
Which uncertainty? known that νeff can be estimated by use of the Welch–
Satterthwaite formula [1]. As underlined by the GUM, the
Nowadays, the term “uncertainty of measurement” is experimental standard deviation of the mean of as many as
definitely used to mean the expanded uncertainty, U, thirty repeated observations of a measurand described by a
obtained by multiplying the combined standard uncer- normal distribution has itself an uncertainty of approxi-
tainty, uc (sometimes reported as uc(y), where y is the mately 13% [1].
estimate of the measurand Y), by the coverage factor, k [1, It is also possible that the reported uncertainty data look
2, 7, 16, 22, 33–35, 37–39]. The intended purpose of U is questionable. In these situations, the Horwitz equation [44–
to provide an interval around the result of a measurement 46] can sometimes provide a more realistic view [18, 19].
that may be expected to encompass a large fraction of the The equation is usually presented as:
distribution of values that could reasonably be attributed
to the measurand [1]. RSD% ¼ 2ð10:5logðmf ÞÞ ð1Þ
The combined standard uncertainty is the total uncer-
tainty of a measurement result estimated by properly Where RSD% is the among-laboratory relative standard
combining all the uncertainty components affecting the deviation and mf is the analyte mass fraction (e.g. mf=10−6
whole experimental procedure. Whenever the procedure means mg kg−1). The equation describes the precision
includes the sampling of the material under examination, it performances of a measurement method without regard to
is mandatory considering the uncertainty of sampling the nature of the analyte, the type of test material, the
among all the other uncertainty components (see for applied analytical technique, and the complexity of the
example Refs. [13–15, 34, 35, 39–42]). However, it should procedure. Equation 1 is presented in Fig. 2. Acceptable
be also noted that the combined standard uncertainty is an performances usually provide variability values within one-
estimated standard deviation relying on the assumption that half to twice the predicted RSD% [45]. Within-laboratory
no source of uncertainty has been neglected or overlooked values are expected to be one-half to two-thirds of the
and that, consequently, is itself affected by a more or less among-laboratory values [46]. Even if significant devia-
significant uncertainty. Also the GUM (section G 1.2) tions from the estimates obtained by the Horwitz equation
emphasizes that the value of the expanded uncertainty is at are possible (Ref. [47] and references cited therein)
best only approximate [1]. nevertheless, the equation is still an acceptable basis for
1732 E. Desimoni, B. Brunetti

100 Test for conformity versus tests for non-conformity

90
Traces- Minor- Main constituents A usually neglected aspect when dealing with some
80 limiting value is that, before performing a test, one should
decide whether it has to be a test for conformity or a test for
70 non-conformity [12, 13]. In particular, it was emphasized
60 that, if a declaration of conformity with the specifications
cannot be stated, it does not mean that the sample under
RSD%

50 examination is in non-conformity. It can only be stated that

the test failed to demonstrate conformity. Likewise, if non-
40 conformity cannot be stated, it does not mean that the
30
sample under examination is in conformity with the
specifications [13]. The distinction between the two tests
20 is also considered by the ISO standard (see below) [34] and
was already emphasized by Currie [47]: ”Acceptance of a
10 hypothesis, based on statistical testing, must not be taken
0
literally. More correctly, one simply fails to reject the
12 11 10 9 8 7 6 5 4 3 2 1 0 hypothesis in question. For example, non-detection of an
log(mf) analyte does not prove its absence.” and “….Assumption
testing, itself, rests upon assumptions. The vast majority of
Fig. 2 Graphical illustration of the Horwitz equation. RSD% is the
statistical tests performed on the chemical measurement
among-laboratory relative standard deviation and mf is the analyte
mass fraction process and its results, for example rely upon the
assumptions of randomness and normality”.
So, after a test for conformity, statement A can be
reviewing doping cases [18], especially when no credible claimed: “The measurements have demonstrated, beyond
MU and no proficiency testing data are available, or when any reasonable doubt, that the value of the measurand is in
no performance requirements are defined [19]. But it must conformity with the requirements”. On the contrary, after a
be stressed that the Horwitz equation should never be used test for non-conformity, statement B can be claimed: “The
as a substitute for the experimental uncertainty estimate of a measurements have demonstrated, beyond any reasonable
result. doubt, that the value of the measurand is not in conformity
It was also worthwhile mentioning the need to avoid with the requirements”. If the result of the selected test is
confusion between the uncertainty associated with the inconclusive, statement C can be claimed “The measure-
experimental measurements and that associated with setting ments have not been able to demonstrate, beyond any
the conformity limiting values [17]. The latter only reflects reasonable doubt, if the value of the measurand is or is not
uncertainties associated with evaluating the potential in conformity with the requirements” [13]. These state-
deleterious effects of a particular value of the characteristic ments perfectly correspond to those reported in paragraphs
under examination when the entity subject to conformity 7.2–7.4 of the ISO standard [34].
assessment is used in a certain context [17]. Situations in
which an uncertainty is associated both with the limiting
value and with the analytical result were also discussed [7, Signal and concentration domains
10]. But according to the ISO 10576–1 international
standard (see the relevant section) the MU should neither In general, conformity or non-conformity tests may deal
explicitly nor implicitly be referred to in the designation of with any type of target variable or measurand. When
the LVs [34]. dealing with most chemical analyses, the measurand is a
It was also emphasized that MU cannot be evaluated concentration, however expressed. In this case, tests are
without metrological traceability. This is particularly relevant to comparison of the concentration of a given
mandatory in forensic contexts, in which it is important analyte in a sample under investigation with a concentration
that MU contains all relevant factors, including all limiting value (or a set of concentration LVs). In these
traceability chains [19]. cases, decision making is usually performed in the
Finally, it was also suggested that, if possible, much concentration domain (CD): measurement results with their
more reliable conformity tests can be performed by using MU intervals (whatever evaluated) are compared with the
uncertainty estimates from interlaboratory comparisons in a proper legal/compositional concentration limiting value, as
learning process [14]. done in Fig. 1.
Uncertainty of measurement and conformity assessment: a review 1733

When confronting the problem of testing for conformity as “undetected” results falling below the limit of
with a concentration LV, some authors tried to develop detection [9].
alternative approaches enabling decision-making to be The rules for stating when a limiting value is exceeded
performed directly in the signal domain (SD), that is by were introduced in a paper where, after discussing the one-
comparing the signal of the sample under investigation with stage and two-stage procedures detailed in Ref. [12] and
the signal relevant to a sample containing exactly the subsequently adopted by the ISO guidelines [34] (see the
specified LV (e.g. to a suitable certified reference material, section about the ISO guidelines, below), the authors
CRM). Then, the conclusion of the comparison had simply suggested two procedures for estimating the number of
to be translated to the CD by a proper calibration constant. measurements necessary for appropriate reliability of the
It is well known that the physical quantities of interest results [12, 13].
(concentration in this case) cannot be measured directly but Another paper was intended to highlight some experi-
are connected to the measured signals through a calibration mental problems preventing uniform implementation of
constant [48]. This view is at the basis of some accepted legislative standards in the EU and Codex [22]. The authors
approaches suitable for estimating the limit of detection emphasized that often there is no common interpretation of
(LOD), where proper statistical tests must enable evaluation analytical results across the EU in the food sector, so that
of whether the concentration of the analyte in the sample significantly different decisions may be taken after analy-
under investigation is higher than zero (see, for example, sing the same sample. Particular attention directed at the
among the most authoritative, Refs. [49, 50]). Of course, consequences of reporting and using the experimental
estimating the LOD is a problem quite similar to that of results in different ways, considering, or not, the recovery
assessing conformity, where proper statistical test must and using results including a different number of significant
enable evaluation of whether the concentration of the figures [22].
analyte in the sample under investigation is higher (or Economic aspects affecting conformity assessment were
lower) than a limiting value. also considered by some authors. One paper discussed
But comparing signals requires proper consideration economic terms of common rules in conformity assessment
of both false-positive (type I) and false-negative (type II) based on measurement by extending tools of sampling
errors: see for example the ISO approach to the LOD when using inspection by variable and inspection by
[50]. Approaches have been tentatively proposed for attribute [24]. Another paper discussed decision-making
assessing conformity to some limiting values by working in conformity assessment in terms of effective cost
in the SD and considering both types of errors [4, 6, 8, associated with measurement, testing and incorrect
11]. Unfortunately, they suffer from disadvantages, name- decision-making [31].
ly the actual availability of the CRM containing exactly Attention was also drawn to the need to provide an
the concentration of the analyte specified by the limiting unambiguous and simple procedure for assessing confor-
value, [6, 8, 11] (quite an improbable case) and, if mity by designing really appropriate decision rules for
available, the uncertainty of the analyte concentration in conformity tests (see the following sections). This should
the analysis certificate of the CRM [20]. In subsequent require the knowledge of the acceptable level of the
papers, approaches were proposed for performing a test probability of making a wrong decision. The author
for conformity, or a test for non-conformity, by working in emphasized that, at least in principle, the acceptable level
the concentration domain but, at least, by taking into of the probability of making a wrong decision can be
account both false-positive and false-negative errors [20, determined if the cost/consequence of taking a wrong
23]. In particular, an approach was proposed based on an decision is known [25]. Unfortunately, as in the case of
existing model of the limit of detection [28]. measurements concerning contaminants in foods, little or
no information is usually available [25].
Finally, when introducing the methods used by
Interpretative problems accredited calibration laboratories, for example within the
Deutscher Kalibrierdienst (e.g. the German accreditation
Several papers were intended to deal with some peculiar body for calibration laboratories, DKD), it was also
aspects of conformity tests. Attention was focused at reported that when measuring a characteristic for confor-
cases in which legislative limits were set below the mity with a tolerance zone, e.g. in the case of a set of LVs, a
detection capability of the existing analytical techniques statement of conformity should only be made if the ratio of
[9]. The examined case study showed that legislation the width of the tolerance zone to the standard uncertainty
based on limiting values may sometimes be beyond associated with the estimate of the characteristic is
analytical capability, and that the limited analytical sufficiently large [27]. Such a condition can be quantified
capability can be exacerbated by the practice of reporting by the measurement capability index, Cm, e.g. the ratio of
1734 E. Desimoni, B. Brunetti

the width of permissible values to some multiple of the when a measurement is within this zone. A transition zone
standard uncertainty associated with the estimate of the is the set of values of a characteristic, for a specified
characteristic [27] or, analogously, to some multiple of the measurement process and decision rule, that is neither in
standard deviation representing the variability of the process the acceptance nor rejection zone.
or product [32]. This aspect is also managed in the ASME A guard band is the magnitude of the offset from the
document [33] (see the N:1 decision rule in the relevant specification limit to the acceptance or rejection zone
section). boundary.
Simple acceptance means a situation when the accep-
tance zone equals and is identical with the specification
International standards zone. Simple rejection means a situation when the rejection
zone consists of all values of the characteristic outside the
As anticipated in the Introduction, at present the guidelines specification zone. Relaxed acceptance means a situation
of ASME [33], ISO [34], and Eurachem/CITAC [35] are when the acceptance zone is increased beyond the
among the most useful tools for the operators of testing and specification zone by a guard band (Fig. 3a). In a binary
calibration laboratories involved in assessing conformity or decision rule, relaxed acceptance is accompanied by
non-conformity with given specification. This because they stringent rejection. Stringent acceptance, in contrast ,
describe procedures sufficiently simplified to be widely means a situation when the acceptance zone is reduced
interpretable and managed. from the specification zone by a guard band(s) (Fig. 3b). In
The ASME document providing guidance for assessment a binary decision rule, stringent acceptance is accompanied
of electrical and mechanical products, is briefly discussed by relaxed rejection.
here below, because: It should be kept in mind that conformity tests including
the choice of the guard band are based on limitations
1. its principles inspired the Eurachem/CITAC guide [25,
stemming from economic, health, or other fields of
26, 35]; and
interest. The tests performed rely on scientific criteria
2. it is a convenient introduction to the concept of guard
and limitations, but the final decision is from the outside
bands and decision rules.
world.
The terminology adopted below can differ from the original The ASME guidelines identify different cases of
terminology used in the three documents because of the acceptance and rejection zones by decision rules. In
need to use the same symbols for the same object/ particular, it reports that the most common form of
quantities. acceptance and rejection in industry is that performing
simple acceptance and rejection using an N:1 decision rule.
The ASME B89.7.3.1-2001 guidelines N:1 means that the measurement interval, result±U, cannot
be larger than the fraction 1/N of the specification zone.
The objective of the ASME B89.7.3.1-2001 standard Usually N is taken equal to 3 or 4. Of course, using this
“Guidelines for decision rules: considering measurement decision rule can lead to decision making problems when
uncertainty in determining conformance to specifications” the measurement result is too close to the specification limit
(the ASME guidelines from now on) is to facilitate the (s). Because the N:1 decision rule is not applied by the ISO
development of understanding between suppliers and and Eurachem/CITAC guidelines [34, 35], it is not
customers regarding proper consideration of MU in considered in the following paragraphs.
conformity tests [33]. It was prepared by the American To increase confidence that a rejected product is
Society of Mechanical Engineers, but it can greatly help in actually out of specification, e.g. by choosing a low risk
understanding the other guidelines because it lists some for the producer, ASME guidelines apply a stringent
basic definitions. rejection and relaxed acceptance. This means that the
According to the ASME guidelines [33], a decision rule relaxed acceptance zone is obtained by increasing the
is a documented rule that describes how MU will be specification zone by a Z% guard band at the specifica-
allocated with regard to accepting or rejecting a product tion limit or at both specification limits. Z% is the size
according to its specification and the results of a measure- of the guard band expressed as a percentage of the
ment. An acceptance zone is the set of values of a expanded uncertainty (a 100% guard band has the
characteristic, for a specified measurement process and magnitude of U) (Fig. 3a).
decision rule, that results in product acceptance when a Similarly, to increase confidence in product quality by
measurement is within this zone. A rejection zone is the set reducing the probability of accepting an out-of-specification
of values of a characteristic, for a specified measurement product, e.g. by choosing a low risk for the consumer,
process and decision rule, that results in product rejection ASME guidelines apply a stringent acceptance and relaxed
Uncertainty of measurement and conformity assessment: a review 1735

Fig. 3 a Symmetric two-sided Lower Upper

stringent rejection and relaxed specification specification
acceptance according to ASME; limit limit
b Symmetric two-sided relaxed
rejection and stringent
gOut g Out
acceptance according to
ASME
Specification zone

Stringent rejection Relaxed acceptance zone Stringent rejection

zone zone
SYMMETRIC TWO-SIDED RELAXED ACCEPTANCE AND STRINGENT REJECTION
ZONES (ASME B89.7.3.1-2001)

Lower Upper
specification specification
limit limit

Specification zone

g In g In

Relaxed rejection zone Stringent acceptance zone Relaxed rejection zone

EXAMPLE OF GUARD BANDS USED FOR CREATING A BINARY DECISION RULE WITH
STRINGENT ACCEPTANCE AND RELAXED REJECTION ZONES (ASME B89.7.3.1-2001)

rejection. Here the acceptance zone is obtained by reducing are not permissible (Fig. 4). The intervals are based on
the specification zone by the guard band(s) amount. Again accepted and required probabilities.
the size of the guard band is expressed as a percentage of U The ISO standard also details the requirements for defining
(Fig. 3b). limiting values. The entity and the quantifiable characteristic
of the entity shall be clearly and unambiguously specified, the
The ISO 10576–1 international standard test procedure should be a standardised one and, as already
cited in the section Which uncertainty?, the MU shall neither
The objective of the ISO 10576–1 international standard explicitly nor implicitly be referred to in the designation of
(the ISO standard from now on) is to provide assurance of the LVs. The ISO standard reports examples of single and
conformity or assurance of non-conformity, either in the double LVs and specifies that the uncertainty interval shall be
form of supplier’s declaration, or of a third party certifica- determined according to the GUM [1].
tion. In its introduction, the ISO standard, also, provides The principal feature of the ISO standard is the
some definitions. Conformity testing is defined as a recommendation that the conformity test be performed as
systematic examination of the extent to which an entity a two-stage procedure, in agreement with a previous
conforms to a specified criterion [34]. The limiting values suggestion [12, 13]. The advantage of the two-stage
(LV) or specification limits (SL) are the specified values of procedure is a substantially higher probability of declaring
the characteristic giving upper and/or lower bounds of the conformity for entities with permissible values of the
permissible values. The region of permissible values is the quantity of interest (the concentration, in chemical analy-
interval or intervals of all permissible values of the ses) which are closer to the LV. The two-stage procedure is
characteristic. The region of non-permissible values is the represented in Fig. 5. By the wording “appropriate
interval or intervals of all values of the characteristic that combination of the two (set of) measurement results”
1736 E. Desimoni, B. Brunetti

The Annex B of the ISO standard reports few illustrative

examples [34].

The Eurachem/CITAC guide

The Eurachem/CITAC guide (the Guide from now on) [35]

was issued more recently than the ASME and ISO stand-
ards and, as already reported, it follows the principles
outlined in ASME B89.7.3.1-2001 [33]. The principles of
the Guide were also detailed in a recent paper [26]. As
already reported, the Guide describes typical scenarios
arising when some measurement result is used for assessing
compliance with an upper LV according to Fig. 1. As with
the ASME guidelines [33] and ISO standards [34], the
Fig. 4 Division of the domain of the characteristic in regions of Guide assumes that uncertainty has been evaluated by the
permissible and non-permissible values according to ISO 10576–1. method provided by ISO [1] and Eurachem [43] and
RPV, region of permissible values; RNPV, region of non-permissible includes the uncertainty of sampling. Most of definitions
values. a Case of an upper LV; b case of a lower LV; c First case of
double limits (the region of permissible values is within the limits); d are equivalent to those given by ASME and ISO. As done
second case of double limits (the region of permissible values is by ASME, the Guide emphasizes that the key to the
outside the limits) assessment of conformity is the concept of decision rules.
Decision rules enable determination of Acceptance and
Rejection zones. The zones are determined in such a way
(Fig. 5) it is meant that, in the second stage, the decision
that if the measurement result lies in the acceptance zone
can be taken by computing the average estimate of the
the product is in conformity with the requirements while, if
quantity of interest and its uncertainty interval by using the
it lies in the rejection zone, it is in non-conformity with the
n2 results obtained in the second stage only or those
specification. In mentioning the different zones, the Guide
obtained in the second stage plus the n1 results obtained in
does not mention simple, stringent, and relaxed zones as
the first stage. The one-stage procedure is applied when the
ASME does.
two-stage procedure is not necessary, because the first stage
In addition, the Guide presents cases of more or less
enables the necessary decision making, or when it cannot
simple decision rules. In particular, it gives details of a
be performed by some reason. Of course, the one-stage
decision rule set up by the Article 6 - Interpretation of
procedure stops at the end of the first stage of Fig. 5.
results of Directive 96/23/EC [36] (see the next section):
Conformity/non-conformity may be assured if, after
performing the measurement procedure and calculating the 1. The result of an analysis shall be considered non-
MU, the estimated uncertainty interval of the measurement compliant if the decision limit of the confirmatory
result is inside the region of permissible/non-permissible method for the analyte is exceeded.
values. 2. If a permitted limit has been established for a substance, the
The ISO standard introduces both tests for conformity decision limit is the concentration above which it can be
and for non-conformity, by specifying the following decidedwithastatisticalcertaintyof1−α that the permitted
possibilities of reporting the results of the conformity limit has been truly exceeded.
assessment: 3. If no permitted limit has been established for a
substance, the decision limit is the lowest concentration
& Assurance of conformity: the conformity test has
level at which a method can discriminate with a
demonstrated beyond any reasonable doubt that the
statistical certainty of 1−α that the particular analyte
value of the characteristic is in conformity with the
is present.
requirements.
4. For substances listed in Group A of Annex I to
& Assurance of non-conformity: the conformity test has
Directive 96/23/EC, the α error shall be 1% or lower.
demonstrated beyond any reasonable doubt that the
For all other substances, the α error shall be 5% or
value of the characteristic is not in conformity with the
lower.
requirements.
& Inconclusive result: the conformity test has not been As emphasized by the Guide, such statements correspond to
able to demonstrate beyond any reasonable doubt that a decision of non-conformity or rejection with low
the value of the characteristic is or is not in conformity probability of false rejection (high confidence of correct
with the requirements. rejection) (Fig. 6a). It is easily observed that, in practice,
Uncertainty of measurement and conformity assessment: a review 1737

Fig. 5 Flow diagram for the

two-stage procedure. IMU: un- Stage 1
certainty interval Perform the measurement procedure
Calculate the uncertainty interval, IMU

YES
Is IMU inside RPV?

NO

Does I MU include NO
a SL?

YES

Stage 2
Perform the measurement procedure once more
Determine an appropriate combination of the two
measurement results and calculate the new IMU

YES
Is IMU inside RPV?

NO

NO
Does I MU include
a SL?

YES IMU is entirely included

Conformity may be
assured in the R NPV

Inconclusive test
Non-conformity
may be assured

the acceptance zone in Fig. 5a corresponds to the relaxed Appendix A of the Guide reports some examples of how
acceptance zone according to ASME (reported in Fig. 3a). the guard bands can be determined. In general the size of
In the case presented above, the value of the guard band,g, the guard band is k·u (see the section Which uncertainty?).
is chosen so that, for a measurement result greater than or A point deserving some comment is relevant to Case 1a, in
equal to LV+g, the probability of false rejection is less than which only the standard uncertainty, u, is available [35]. It
or equal to α. A typical value of α (the probability of false is reported that in many cases, current practice is to use k=
positive errors) is 5%. 2. As stated by the Guide, on the assumption that the
Analogously, in Fig. 6b the acceptance zone corre- distribution is approximately normal, this choice gives a
sponds to the stringent acceptance zone according to level of confidence of approximately 95% that the value of
ASME guidelines (reported in Fig. 3b). Some potentially the measurand lies in the interval y±2·u. On this basis, the
unclear aspects of the Eurachem/CITAC definition of Guide states that “the probability that the value of the
acceptance and rejection zones were recently discussed measurand is less than y+2·u is approximately 97.5%.” It is
[29, 30]. likely that this last sentence can be quite perplexing to
1738 E. Desimoni, B. Brunetti

LV
g: guard band

f(c) Upper limiting

value
y
2uc 2uc
a) 47,7% 47,7%

Acceptance zone Rejection zone

2.3% 2.3%

g: guard band A

c
g
b)

f (c) Upper limiting

Acceptance zone Rejection zone value
y
uc uc
Fig. 6 Acceptance and rejection zones for an upper LV according to
Eurachem/CITAC
34.1% 34.1%
readers not well trained in probability distributions.
Figure 7a enables elucidation of the difference between
95%, e.g. the percentage area of the distribution included in
the y±2uc interval, and 97.5%, e.g. the area at the left of the
y+2uc value. Figure 7b displays the situation for a guard 15.9% 15.9%
band equal to one uc. The reported values are those
obtained by the following equations for Fig. 7a: B
Z c
yþ2u
c
f ðcÞdc ffi 0:954 ð2Þ g
y2uc

Fig. 7 a Explanation of the sentence “If the size of the guard band is
2·u, then the probability that y is less than y+2·u is approximately
Z c
y2u Z1 97.5%”. b same as a but the size of guard band is u
f ðcÞdc ffi 0:023 ffi f ðcÞdc ð3Þ
0 yþ2uc
About Directive 96/23/EC
and by the following equations for Fig. 7b:
The Commission Decision of 12 August 2002 (the Decision
Z
yþuc
from now on) was aimed at implementing Council
f ðcÞdc ffi 0:683 ð4Þ Directive 96/23/EC concerning the performance of analyt-
yuc ical methods and the interpretation of results obtained in the
monitoring of specific substances and residues thereof in
live animals and animal products, when they affect public
Z
yuc Z1
health (available online) [36]. The Decision provides rules
f ðcÞdc ffi 0:159 ffi f ðcÞdc ð5Þ for the analytical methods to be used in the testing of
0 yþuc official samples and specifies common criteria for the
Uncertainty of measurement and conformity assessment: a review 1739

interpretation of analytical results of official control Example 1

laboratories for such samples. The Article 6 - Interpretation
of results is reported here in the section dealing with the The first example deals with the ISO approach [34] according to
Eurachem/CITAC guide. the flow diagram reported in Fig. 5 (the two-stage procedure).
In the Annex “Performance Criteria, Other Requirements The 98/83/EC directive on the quality of water
and Procedures for analytical methods” the Decision gives intended for human consumption specifies the upper
the following definitions: limit value LVu =10 μg L−1 for the concentration of
arsenic in drinking water [52]. When using a two-stage
1.11. Decision limit (CCα) means the limit at and above
procedure, the sample is divided into two subsamples, and
which it can be concluded with an error probability
the second is only used if the uncertainty interval, IMU,
of α that a sample is non-compliant.
contains the limiting value. The ISO approach accepts
1.12. Detection capability (CCβ) means the smallest
uncertainty intervals given in the form of a confidence
content of the substance that may be detected,
interval (subclause 6.4). Suppose that the measurements
identified and/or quantified in a sample with an
are performed with a standard measurement procedure which
error probability of β. In the case of substances for
operates with a combined standard uncertainty of uc =
which no permitted limit has been established, the
1.485 μg L−1 at concentration levels around the LVu.
detection capability is the lowest concentration at
According to the ISO approach (Annex B, Example 2), if
which a method is able to detect truly contaminated
n independent measurements, each with uncertainty σY, are
samples with a statistical certainty of 1−β. In the
performed and the arithmetic mean of the measurements is
case of substances with an established permitted
Y , then the confidence interval is given as:
limit, this means that the detection capability is the
concentration at which the method is able to detect z1 a ; s Y
permitted limit concentrations with a statistical Y p2
ffiffiffi ð6Þ
n
certainty of 1−β.
where z1 a is the 1  a2 quantile of the standard normal
CCα and CCβ are concentration values. It follows that 2
distribution.
CCα is the upper limit of the region of permissible
A first series of three independent analyses of the arsenic
concentration values. The Decision should represent a
concentration in the first water subsample gives the
qualified reference for operators of laboratories accredited
concentration C As;1 ¼ 9:09mgL1 . Using z1a ¼ 1:96 (often
for official residues control. Unfortunately, it was shown 2
approximated to 2.0) to choose α=0.05, one can obtain the
that some statements of the Decision can generate misun-
uncertainty interval:
derstanding and/or confusion [21]. For example, Articles
[Link] and [Link] of the Decision recommend estimation 1:96  1:485
of CCα and CCβ according to ISO 11843 [51]. But ISO IMU ¼ 9:09  pffiffiffi ¼ 9:09  1:68 mgL1 ð7Þ
3
11843 uses different symbols (xC and xD in place of CCα
and CCβ), considers sample statistics (t-distributions) in Because the upper limit value, LVu =10 μg L−1, is within the
place of population statistics (normal distributions) and uncertainty interval, the test is inconclusive at the given
estimates the two limits by use of somewhat different confidence level.
approaches [21]. Moreover, the Annex of the Decision A second series of four independent analyses of the
explains the meaning of CCα in the Fig. 3.2 of the arsenic concentration is then performed with the second
Decision. But that figure can mislead the reader, because water subsample. This gives the result C As;2 ¼ 8:66mgL1 .
it reports CCα in a frequency versus response diagram [21] The uncertainty interval is now:
whereas CCα is a concentration. In contrast, the subsequent
1:96  1:485
Fig. 3 of the Decision correctly shows CCβ in a frequency IMU ¼ 8:66  pffiffiffi ¼ 8:66  1:46 mgL1 ð8Þ
versus concentration diagram. 4
Again the upper limit value, LVu =10 μg L−1, is within the
uncertainty interval and the test is inconclusive.
Worked examples The results of both set of measurements are the
combined. The concentration of arsenic resulting from the
C 3þC 4
Worked examples in such a tricky matter as conformity testing seven measurements is:C As ¼ As;1 7 As;2 ¼ 8:84 mgL1 .
are hardly representative of the plethora of possible experi- The new IMU is:
mental situations. Nevertheless, some examples representa-
tive of basic experimental situations are presented here to help 1:96  1:485
IMU ¼ 8:84  pffiffiffi ¼ 8:84  1:10 mgL1 ð9Þ
readers evaluate ISO and Eurachem/CITAC approaches. 7
1740 E. Desimoni, B. Brunetti

This uncertainty interval is all below the LVu. This last References
result enables the statement of conformity with the
specified limit at the given confidence level. 1. Guide to the expression of Uncertainty in Measurement (1995) 1st
Ed., and corrected and reprinted Ed., ISO Secretariat, Switzerland
Example 2 2. Analytical Methods Committee (1995) Uncertainty of measure-
ment: implications of its use in analytical science. Analyst
120:2303–2308
The second example deals with the Eurachem/CITAC 3. Barwick V (1996) Setting and enforcing legislative limits. Anal
approach [35]. In this case, one should define the size of Commun 33:123–124
the guard band and choose to perform a test enabling high 4. de Ruig WG (1996) Characterizing and optimizing analytical
confidence of correct rejection (as in Fig. 6a) or high method quality. Trends Food Sci Technol 7:234–240
5. Ellison S, Wegscheider W, Williams A (1997) Measurement
confidence of correct acceptance (as in Fig. 6b). uncertainty. Anal Chem 69:607A–613A
Commission regulation (EU) No 105/2010 of 5 February 6. Desimoni E, Mannino S (1998) Uncertainty of measurement and
2010 amending Regulation (EC) No 1881/2006 setting legislative limits. Accredit Qual Assur 3:335–337
maximum levels for specific contaminants in foodstuffs 7. King B (1999) Assessment of compliance of analytical results
with regulatory or specification limits. Accredit Qual Assur 4:27–
with regard to ochratoxin A specifies the upper limit value 30
LVu =80 μg kg−1 for the concentration of ochratoxin A in 8. Buzoianu M (2000) Measurement uncertainty and legal limits in
liquorice extract [53]. Suppose that the combined uncer- analytical measurements. OIML Bull XLI:5–12
tainty of a measurement method, evaluated before 9. Ellison SRL, Barwick VJ, Williams A (2000) Legislative limits
below detection capability. Accredit Qual Assur 5:308–313
performing the necessary measurements, is 3.5 μg kg−1. 10. Hibbert DB (2001) Compliance of analytical results with
By using the above data, choosing α=0.05 and performing regulatory or specification limits: a probabilistic approach.
the test enabling high confidence of correct acceptance, the Accredit Qual Assur 6:346–351
guard band is equal to g=1.65·3.5=5.775 μg kg−1. The 11. Desimoni E, Mannino S, Brunetti B (2001) On the assessment of
compliance with legal limits: Part 1: signal and concentration
guard band is subtracted from the upper limit value. The domains. Accredit Qual Assur 6:452–458
acceptance zone then extends to 80–5.775 = 12. Holst E, Thyregod P, Wilrich P-Th (2001) On conformity testing
74.225 μg kg−1. Suppose that the concentration of and the use of two stage procedures. Int Stat Rev 69:419–432
ochratoxin A in two samples are 86.07 μg kg−1 and 13. Christensen JM, Holst E, Olsen E, Wilrich PT (2002) Rules for
stating when a limiting value is exceeded. Accredit Qual Assur
72.33 μg kg−1. The first sample is rejected and the second 7:28–35
is accepted. In contrast, when performing the test is to 14. Andersson H (2002) The use of uncertainty estimates of test
enable high confidence of correct rejection, the guard band results in comparison with acceptance limits. Accredit Qual Assur
is added to the upper limit value. The acceptance zone then 7:228–233
15. Glaeser H (2002) Compliance with legal requirements: definition
extends to 80+5.775=85.775 μg kg−1. In this second case, of limits, sampling and measurement uncertainty. Accredit Qual
both samples are accepted. Assur 7:388–391
16. Kessel W (2002) Measurement uncertainty according to ISO/
BIPM-GUM. Thermochim Acta 382:1–16
17. Källgren H, Lauwaars M, Magnusson B, Pendrill L, Taylor P
Conclusions (2003) Role of measurement uncertainty in conformity assessment
in legal metrology and trade. Accredit Qual Assur 8:541–547
This literature information confirms that some aspects of 18. van der Veen AMH (2003) Measurement uncertainty and doping
the assessment of conformity with legal or compositional control in sport. Accredit Qual Assur 8:334–339
19. van der Veen AMH (2004) Measurement uncertainty and doping
limiting values deserve further developments. A general control in sport. Part 2: Metrological traceability aspects. Accredit
agreement exists about the need to properly take into Qual Assur 9:311–316
account the uncertainty of measurement in decision 20. Desimoni E, Brunetti B, Cattaneo R (2004) The assessment of
making, and use of the correct MU (including the sampling compliance with legal limits. Part 2. How to consider both type I
and type II errors when working in the concentration domain.
uncertainty component and estimated according to GUM). Accredit Qual Assur 9:445–449
But problems still exist, especially concerning the need for 21. Desimoni E, Brunetti B (2004) About CCα and CCβ as
unification and/or unequivocal formulation of the wording introduced by the commission decision of 12 August 2002
of prescriptions by the regulatory Authorities. Finally, implementing council directive 96/23/EC. Accredit Qual Assur
9:724–725
decision making could be greatly facilitated by issuing 22. Wood R (2006) Compliance decision criteria—problems encoun-
really unified and, consequently, generally agreeable and tered and actions taken in the EU and Codex. Accredit Qual Assur
usable guidelines. 11:83–88
23. Desimoni E, Brunetti B (2006) Considering uncertainty of
measurement when assessing compliance or non-compliance
Acknowledgements Financial support by the Fondo Interno per la with reference values given in compositional specifications
Ricerca Scientifica e Tecnologica 2008 (Università degli Studi di and statutory limits: a proposal. Accredit Qual Assur 11:363–
Milano) is acknowledged. 366
Uncertainty of measurement and conformity assessment: a review 1741

24. Pendrill LR (2008) Operating ‘cost’ characteristics in sampling by 37. ILAC-G8 (1996) Guidelines on assessment and reporting of
variable and attribute. Accredit Qual Assur 13:619–631 conformity with specification, International Laboratory Accredi-
25. Williams A (2008) Compliance with specifications. Accredit Qual tation Cooperation, ILAC, Australia
Assur 13:617–618 38. NMKL Procedure No. 5 (1997) Estimation and expression of
26. Williams A (2008) Principles of the EURACHEM/CITAC guide measurement analysis in chemical analysis, NMKL, Finland
“Use of uncertainty information in compliance assessment”. 39. UKAS LAB 12 (2000) The expression of uncertainty in testing,
Accredit Qual Assur 13:633–638 UKAS, UK
27. Czaske M (2008) Use of uncertainty measurement by accredited 40. Wunderli S (2003) Uncertainty and sampling. Accredit Qual
calibration laboratories when stating compliance. Accredit Qual Assur 8:90
Assur 13:645–651 41. Heydorn K, Esbensen K (2004) Sampling and metrology. Accredit
28. Desimoni E, Brunetti B (2008) About considering both false Qual Assur 9:391–396
negative and false positive errors when assessing compliance and 42. Ramsey MH, Thompson M (2007) Uncertainty from sampling, in
non-compliance with reference values given in compositional the context of fitness for purpose. Accredit Qual Assur 12:503–
specifications and statutory limits. Accredit Qual Assur 13:653– 513
662 43. Ellison SRL, Rosslein M, Williams A (eds.) (2000) EURACHEM/
29. Desimoni E, Brunetti B (2010) About Acceptance and Rejection CITAC Guide “Quantifying uncertainty in analytical measure-
zones as defined in the EURACHEM/CITAC Guide (2007) “Use ments” QUAM: 2000.P1. [Link] Accessed 17
of uncertainty information in compliance assessment”. Accredit September 2010
Qual Assur 15:45–47 44. Horwitz W, Kamps LR, Boyer KW (1980) J Assoc Off Anal
30. Ellison SLR, Williams A (2010) Response to “About acceptance Chem 63:1344–1354
and rejection zones”. Accredit Qual Assur 15:49–51 45. Horwitz W (1982) Evaluation of analytical methods used for
31. Beges G, Drnovsek J, Pendrill LR (2010) Optimising calibration regulation of foods and drugs. Anal Chem 54:67A–76A
and measurement capabilities in terms of economics in conformity 46. Albert R, Horwitz W (1997) A Heuristic derivation of the horwitz
assessment. Accredit Qual Assur 15:147–154 curve. Anal Chem 69:789–790
32. Hinrichs W (2010) The impact of measurement uncertainty on the 47. Linsinger TPJ, Josephs RD (2006) Limitations of the application
producer's and user's risks, on classification and conformity of the Horwitz equation. Trends Anal Chem 25:1125–1130
assessment: An example based on tests on some construction 48. Currie LA (1995) Nomenclature in evaluation of analytical
products. Accredit Qual Assur 15:289–296 methods including detection and quantification capabilities
33. ASME B89.7.3.1-2001 (2002) Guidelines for decision rules: (IUPAC Recommendations 1995). Pure Appl Chem 67:1699–
considering measurement uncertainty in determining conformance 1723
to specifications, ASME, New York 49. Currie LA (1968) Limits for qualitative detection and quantitative
34. ISO 10576–1:2003 (2003) Statistical methods – Guidelines for the determination. Application to radiochemistry. Anal Chem 40:586–
evaluation of conformity with specified requirements, Part 1: 593
General principles, ISO, Geneva 50. International Organization for Standardization (ISO) ISO (1997)
35. Ellison SRL, Williams A (eds.) (2007) EURACHEM/CITAC 11843-1 Capability of detection – Part 1:L Terms and definitions
Guide “Use of uncertainty information in compliance assessment” 51. ISO 11843: 1997 (1997) Capability of detection-part 1: terms and
first edn 2007. [Link] Accessed 17 September definitions, part 2: methodology in the linear calibration case,
2010 ISO, Geneva
36. Commission Decision of 12 August 2002 implementing Council 52. Council Directive 98/83/EC of 3 November 1998 on the quality of
Directive 96/23/EC concerning the performance of analytical water intended for human consumption, OJ C, 5.12.98, p. 32
methods and the interpretation of results (2002/657/EC). [Link] 53. Commission regulation (EU) No 105/2010 of 5 February 2010
[Link]/smartapi/cgi/sga_doc?smartapi!celexplus!prod!Doc amending Regulation (EC) No 1881/2006 setting maximum levels
Number&lg=en&type_doc=Decision&an_doc=2002&nu_doc=657. for certain contaminants in foodstuffs as regard ochratoxin A OJ
Accessed 17 September 2010 L, 6.2.2010, p. 7

View publication stats