LEARNING GUIDE

SIX SIGMA-
BASED QUALIT Y
CONTROL

COREL A BOR ATORY. A BBOT T

CHAPTER 2

DETERMINING QUALITY ON
THE SIGMA SCALE
INTRODUCTION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
METHOD VALIDATION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
DATA FOR RISK ASSESSMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
SIGMA AS AN INDICATOR OF RISK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
CALCULATION OF SIGMA METRIC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
GRAPHICAL DETERMINATION OF SIGMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
DETERMINATION OF SIGMA-METRIC FROM RESULTS
OF METHOD VALIDATION STUDIES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
DETERMINING SIGMA-METRIC FROM PT AND SQC DATA. . . . . . . . . . . . . . . . . . . . . . 27
IMPORTANCE OF DETERMINING QUALITY ON
THE SIGMA SCALE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

15 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

INTRODUCTION
The initial steps for implementing a Six Sigma quality management system (6ΣQMS) are to define the
quality goal/requirement for intended use, select an analytical measurement procedure and determine
method performance from lab data.

Figure 2-1 illustrates the initial steps to calculate a Sigma-metric from the quality goal in the form of
an allowable total error (ATE or TEa) and the accuracy (bias) and precision (SD or %CV) observed
for the method. The metric reflects quality on the Sigma scale and provides guidance for selecting the
appropriate SQC procedure (that is, the control rules and number of controls needed), which can
be easily identified with a variety of SQC planning tools. Some of those tools are illustrated in the
next chapter.

Define quality specifications for test

Calculate
Sigma
Select control materials
%TEa-%Bias
%CV
Determine mean, SD

Calculate limits, prepare chart

Utilize
QC Planning Analyze controls each run
Tools, e.g.
OPSpecs,
Westgard Interpret control results
Sigma Rules

Take appropriate action

Figure 2-1: Laboratory process for “doing the right SQC right.”

METHOD VALIDATION
Validating safety characteristics is important for risk-based thinking. In ISO terminology, safety
characteristics for medical devices are reportable range, precision, trueness or bias, detection limit,
interference, and recovery — also commonly called performance characteristics. The key ISO guideline
for risk management of medical devices1 emphasizes design for safety:

IVD medical devices have performance characteristics that determine the accuracy of
examination results. Failure to meet the performance characteristics required for a specific
medical use could result in a hazardous situation that should be evaluated for risk to
the patients.
Manufacturers address safety as part of design and validation of test systems. If precision and bias are
not acceptable, manufacturers redesign the system until they achieve the required performance. The
performance is documented as claims that the FDA reviews as part of the 510(k) approval to market a
new test system.

16 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

CLIA requires labs to validate performance of new test systems and verify that they achieve a
manufacturer’s claims2:
§493.1253 Establishment and verification of performance specifications.
(a) Applicability. Laboratories are not required to verify or establish performance claims for any test
system used by the laboratory before April 24, 2003.
(b) ( 1) Verification of performance specifications. Each laboratory that introduces an unmodified,
FDA-cleared or approved test system must do the following before reporting patient test results.
(i) D
 emonstrate that it can obtain performance specifications comparable to those established by
the manufacturer for the following characteristics: accuracy, precision, reportable range of the
test system.
(ii) Verify that the manufacturer’s reference intervals (normal values) are appropriate for the
laboratory’s patient population.

DATA FOR RISK ASSESSMENT

U.S. CLIA requirements for a QC program call for in-house data to conduct a risk assessment of the
testing process3.

To conduct a risk assessment, the laboratory must identify the sources of potential failures and
errors for a testing process, and evaluate the frequency and impact of those failures and sources
of error.

In-house data, established by the laboratory in its own environment and by its own personnel,
must be included to demonstrate that the stability of the test system supports the number and
frequency of QC documented in the QCP. Data from verification or establishment of performance
specifications and historical (existing) QC data can be included. Published data or data from
manufacturers (e.g., package inserts) may be taken into consideration, but may not be used as
the sole criteria for decision-making.

Specific recommendations are to use data from performance verification/validation studies and existing
QC records. Those data are supposed to demonstrate that the stability of the test system supports the
number and frequency of the QC documented in the QCP.

In-house data can be used to determine test system quality on the Sigma scale (the Sigma-metric) and if
observed bias and precision are suitable for clinical use. Sigma is inherently risk-based and predicts the
expected number of defective test results for a test system in terms of precision and bias and the quality
required for intended use of the test.4 Sigma SQC planning tools guide the selection of appropriate
SQC procedures. In-house data are used to right-size SQC procedures for the quality required for the
intended use of the test.

17 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

SIGMA AS AN INDICATOR OF RISK
Six Sigma quality management assesses the quality of any process on the Sigma scale. Sigma provides
a measure of observed quality relative to the quality required. In manufacturing, the quality required
for intended use is defined as a tolerance specification. The quality produced typically shows variation
around a target or ideal specification.

Part A of Figure 2-2 applies the Sigma model to the clinical lab, with tolerance limits replacing
allowable error requirement for the intended clinical use (ATE or TEa). Precision is represented by
the SD or %CV, characterizing the width of the distribution. The effect of bias is shown by the location
of the distribution relative to the target or true value. Bias shifts the distribution toward one of the
tolerance specifications, thus reducing the amount of variation allowable.

The goal for world-class quality is process variation (i.e., test performance) that fits well within the
tolerance specifications.

Example:
± 6 SD for an assay, as shown in part B of Figure 2-2.

Six Sigma quality ensures essentially no errors exceed the defined quality requirement.

In most industries, minimum acceptable quality is defined as three Sigma, shown in part C of Figure
2-2. For three Sigma, the tolerance limits are completely consumed by 3 SD of variation, and even under
optimal operating conditions, a few defects are produced. Any change in process performance (e.g.,
decreased precision or increased bias) causes an increased risk of producing poor-quality test results. A
Six Sigma process is considered world class, but it’s possible to attain a Sigma value >6, or less than 3.4
defects per million opportunities, with exceptional precision and/or minimal bias.

18 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

 A. Calculation of Sigma Metric

Lower Tolerance Limit Upper

Tolerance
-TEa
Bias Limit
+TEa

Sigma = TEa – Bias

SD

-6s -5s -4s -3s -2s -1s 0s 1s 2s 3s 4s 5s 6s

B. Goal of Six Sigma for World-Class Quality

- Tolerance +Tolerance
Specification Target Specification

-6 SDs should fit +6 SDs should fit

into specification into specification

-6s -5s -4s -3s -2s -1s 0s 1s 2s 3s 4s 5s 6s

C. Minimum Acceptable Performance of Three Sigma

-3s -2s -1s 0s 1s 2s 3s

Figure 2-2: (A) Illustration of calculation of Sigma-metric from the allowable total error (ATE or TEa), inaccuracy (Bias),
and imprecision (SD). (B) Comparison of Six Sigma goal for world class quality with (C) minimum acceptable quality of
Figure 2-2. (A) Illustration of calculation of Sigma-metric from the allowable total error (ATE or TEa),
Three Sigma. inaccuracy (Bias), and imprecision (SD). (B) Comparison of Six-Sigma goal for world class quality
with (C) minimum acceptable quality of 3-Sigma.

19 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

CALCUL ATION OF SIGMA METRIC
The Sigma-metric is calculated using the equation:

Sigma-metric = (TEa – |Bias|)/SD

where TEa is the allowable total error, bias is the systematic error (inaccuracy) and is treated as
an absolute value (|Bias|), and SD is the random error (imprecision), with all terms expressed in
concentration units.

Percentage units may also be used, as in the equation below:

Sigma-metric = (%TEa – |%Bias|)/%CV

TEa
TEa may be defined by the criteria for acceptable performance for EQA/PT surveys.

Example:
The U.S. CLIA criterion for glucose is “Target Value ± 6 mg/dL or ± 10% (whichever is greater).”

The larger of the two limits should be used, depending on the target value (TV) or concentration of the
PT survey material.

If the TV is: Acceptable performance is... Which is a range of...

50 mg/dL 50 ± 6 mg/dL 44 to 56 mg/dL

125 mg/dL 125 mg/dL ± 10% or 125 ± 12.5 mg/dL 112.5 to 137.5 mg/dL

Example:
CLIA provides a list of acceptable performance limits for 70–80 tests. These are the regulated
analytes for which PT performance is assessed by five samples per survey and three surveys
per year. Non-regulated analytes also require PT but may only be assessed by two surveys per
year, with as few as two samples per survey. Non-regulated analyte acceptance limits may be
based on various goal-setting models (e.g., intended clinical use, biologic variation and expert
group recommendations). HbA1c is a non-regulated analyte for which the College of American
Pathologists (CAP) and the National Glycohemoglobin Standardization Program (NGSP) have
established an TEa of +/- 6.0% in 2014.

20 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

Traditionally, EQA/PT programs used peer group grading - that is, a lab’s performance is compared
to the performance of all labs using the same analytical method (i.e., analyzer, reagent, methodology).
Peer group performance is acceptable if a lab’s results agree with the mean value of its peer group
within the established acceptance limits. Thus, satisfactory performance is relative to the peer group.
Accuracy-based EQA/PT programs are becoming more prevalent. Accuracy-based grading compares a
lab’s performance to a predetermined target value established by an accepted gold standard reference
method. Accuracy-based grading is absolute because it’s based on the best available estimate of
scientific truth, using a reference method.

BIAS
Inaccuracy, trueness or systematic error is determined during method validation studies from a
comparison of methods experiment. Labs perform these experiments to verify a manufacturer’s claim
after installation of new test systems. After initial validation, labs are required to monitor bias using
EQA/PT samples with target values established by reference methods, the mean of a survey group or
the mean of a survey peer group. Results are generally reported as the deviation from the target and
expressed as a multiple of the observed group variation (that is, a z-value that describes the deviation
from the target as a multiple of the group standard deviation). For calculation of a Sigma-metric, %Bias
is calculated as the observed bias divided by the target value, multiplied by 100.

%Bias = (Bias/TV) x 100

An important note about bias: Initially when determining Sigma quality, it may be difficult to obtain
a good bias estimate. It is permissible to assume bias is zero and calculate Sigma simply as the ratio
of TEa/SD or %TEa/%CV. This calculation yields a Sigma-metric that is too high (i.e., an optimistic
estimate of quality). Nonetheless, if Sigma is low (<3 when bias is assumed to be zero), it’s sufficient to
indicate the new test system is high risk! If Sigma is >3, it is still important to get a better estimate of
bias for a more reliable determination of Sigma.

SD
Imprecision (random error) is determined from a replication experiment during method validation
studies or SQC data collected during routine operation. Labs perform replication experiments to verify
precision and then monitor ongoing performance from SQC data collected under conditions of routine
operation. %CV is calculated as the observed SD divided by the mean and then multiplied by 100.
%CV = (SD/Mean) x 100

Example Calculations for HbA1c

Given the importance of HbA1c for diagnosing and managing diabetes, the global agreement on quality
requirements, the availability of accuracy-based EQA/PT programs, and the widespread application
of methods in central labs, as well as point-of-care settings, HbA1c provides a good example for Sigma
calculations.

21 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

These examples illustrate the demanding performance required of current analytical methods for the
test quality needed for clinical use of HbA1c.

TEa Bias CV Sigma

6.0% 0.0% 1.0% (6.0–0.0)/1.0 or 6.0

6.0% 1.0% 1.0% (6.0–1.0)/1.0 or 5.0

6.0% 0.0% 1.5% (6.0–0.0)/1.5 or 4.0

6.0% 1.5% 1.5% (6.0–1.5)/1.5 or 3.0

GRAPHICAL DETERMINATION OF SIGMA

The method decision chart is a plot of allowable bias versus allowable precision constructed once an
TEa quality goal is defined. TEa specifies the size of the error budget, consisting of both random and
systematic errors (precision and bias). Historically, the requirement for acceptable performance was
bias plus 2 SD, still a common way of calculating the total analytic error (TAE). The TEa requirement
became more demanding as assay performance improved. The criterion for Six Sigma quality requires
bias plus 6 SD to fit within the TEa.

The multiplier of the SD is the Sigma of interest, and a graphical tool can be constructed to show
performance limits for two, three, four, five and Six Sigma, as shown by the method decision chart in
Figure 2-3. This tool allows plotting an operating point, where the y-coordinate represents the observed
bias and the x-coordinate represents the observed precision. This point represents the Sigma quality of
any method.

Example:
The operating point in Figure 2-3 represents a bias of 2.0% and a CV of 1.0%. It falls on the line
representing four Sigma quality, which agrees with the calculated Sigma-metric: [(6 –2)/1 = 4].

22 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

 Method Decision Chart TEa=6.0%
6

5 t
poin
i ng
at

Accuracy - Observed %Bias

per
4 u ro
yo
ot
Pl
Region of
3 Unacceptable
Performance

1
World
Class
Quality
0
0 0.5 1 1.5 2 2.5 3

Precision - Observed %CV

2 Sigma 4 Sigma 6 Sigma

3 Sigma 5 Sigma

Figure 2-3: Method Decision chart for TEa equal to 6.0%. The observed %Bias is plotted on the y-axis vs the observed
%CV on the x-axis. The different diagonal lines, from top to bottom, represent 2-Sigma, 3-Sigma, 4-Sigma, 5-Sigma,
and 6-Sigma quality. The operating point represents an examination procedure having an observed bias of 2.0%
and an observed precision of 1.0%.

To construct a Method Decision chart for TEa of 6.0%:

1. Scale the y-axis from 0% to TEa, or 6.0%. Label this axis “observed inaccuracy” in units of %Bias.

2. Scale the x-axis from 0% to half of TEa, which is 3.0%. Label this axis “observed imprecision” in units
of %CV.

3. Draw lines for Sigma quality by determining the y-intercept and x-intercept as described below:

TEa Bias CV Sigma

10% Bias Plus 2s TEa or 6.0% 6.0% ÷ 2, or 3.0%

Bias Plus 3s TEa or 6.0% 6.0% ÷ 3, or 2.0%

Bias Plus 4s TEa or 6.0% 6.0% ÷ 4, or 1.5%

Bias Plus 5s TEa or 6.0% 6.0% ÷ 5, or 1.2%

Bias Plus 2s TEa or 6.0% 6.0% ÷ 6, or 1.0%

23 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

It is also possible to construct a normalized method decision chart that can be used for any specified
TEa. This is done by scaling the y-axis from 0 to 100 and the x-axis from 0 to 50, calculating the x- and
y-intercepts as above, and drawing the lines for Sigma. To apply the normalized chart, it is necessary to
express the observed bias and SD or CV as percentages of the TEa. For the HbA1c example above, the
y-coordinate would be 2/6, or 33%, and the x-coordinate 1/6, or 17%. That method is plotted as point
A on the normalized chart shown in Figure 2-4. Method B represents a central lab glucose method for
which TEa is 10%, and method C represents a point-of-care glucose meter for which TEa is 20%. The
advantage of the normalized chart is that many different methods can be shown on the same chart.

Normalized Method Decision Chart

100.0

90.0
Observed Inaccuracy, (100*Bias/TEa)

80.0

70.0

60.0

50.0

40.0

30.0

20.0

10.0

0.0
0.0 10.0 20.0 30.0 40.0 50.0

Observed Imprecision (100*CV/TEa)

Figure 2-4: “Normalized” Method Decision

Figure [Link] where
“Normalized” Methodobserved inaccuracy
Decision Chart where is iscalculated
observed inaccuracy calculated as as 100*Bias/TEa and
100*Bias/ATE and observed imprecision is calculated as 100*CV/ATE, when original parameters
observed imprecision is calculated as 100*CV/TEa, when
are all in units of %. Example original
A is the parameters
same HbA1c method are
as shown in Figure [Link] in units
Example B is a of %. Example A is the same
laboratory glucose examination procedure where ATE is 10% and C a point of care glucose meter
HbA1c method as shown in Figure 2-3. where
Example ATE is 20%. B is a laboratory glucose examination procedure where TEa is 10% and
C a point of care glucose meter where TEa is 20%.

For a more complete discussion of Six Sigma concepts, metrics and application tools, see reference 5.

DETERMINATION OF SIGMA-METRIC FROM RESULTS OF METHOD

VALIDATION STUDIES
Typically, results from replication experiments are represented by the mean, SD and %CV from 20 or
more replicates. If replicates are performed within one run or one day, they reflect “within-run” or
“within-day” precision. Replicates analyzed over many days (e.g., > 20 days) are preferred and reflect
“between-day” or “total” precision. The SDs or %CVs for short-term precision are typically smaller than
those for long-term precision.

24 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

Comparison of method results are presented by plotting the new test system results on the y-axis versus
the comparative method results on the x-axis. The data are subjected to regression equation analysis to
describe results as the equation for a straight line:

y = ax + b,

where a is the slope and b is the y-intercept.

To determine bias at an important medical decision level, Xc:

1. Calculate Yc (aXc + b).

2. Subtract Yc – Xc to estimate bias.

3. Calculate %Bias as Bias(100)/Xc.

REAL-WORLD EXAMPLE 1
A published report on the performance of HbA1c devices provides the following information6:
• Precision [Table 1, reference 5]: CV = 1.9% at 6.5%Hb; CV = 3.2% at 8.9%Hb
• Accuracy [Table 2, lot #1, vs. Tina-Quant, reference 5]: y = 1.04x – 0.35

To determine Sigma:
Define TEa at a critical medical decision level (Xc): TEa is defined as 6.0% by the U.S. NGSP and CAP
PT. The cut-point for diagnosis of diabetes is 6.5%Hb. Therefore, TEa = +/- 6.0% at 6.5%Hb.

1. S
 elect the appropriate estimate of precision: Precision at 6.5%Hb is represented by the lower
control material (i.e., CV = 1.9% at 6.5%Hb.) Note: It’s not always that easy because the chosen
control materials may not align exactly with the Xc of interest, in which case, it may be necessary to
interpolate between the stated performance claims.

2. Calculate bias at Xc:

a. Yc = (1.04 x 6.5) – 0.35 = 6.76 – 0.35 = 6.41
b. Bias = Yc – Xc = 6.41 – 6.50 = -0.09
c. |Bias| = 0.09%Hb
d. |%Bias| = (0.09 x 100)/6.5 = 1.4%

3. Calculate Sigma:
a. Sigma = (%TEa – |%Bias|)/%CV
i. Sigma = (6.0% – 1.4%)/1.9% = 4.6/1.9 = 2.4

REAL-WORLD EXAMPLE 2
This same report6 provides the following information for a second test system:

• Precision [Table 1, reference 5]: CV = 2.1% at 4.7%Hb; CV = 1.2% at 8.0%Hb; CV = 1.1% at 10.9%Hb

• Accuracy [Table 2, lot #1, vs. Premier, reference 5]: y = 1.08x – 0.41

25 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

To determine Sigma:
1. Define TEa at a critical medical decision level (Xc): TEa = +/- 6.0% at 6.5%Hb.

2. Select the appropriate estimate of precision: Precision at 6.5%Hb is probably best represented by the
middle control material at 8.0%Hb (i.e., CV = 1.1%). Here, judgment is important for interpreting the
results of the experimental studies.

3. Calculate bias at Xc:

a. Yc = (1.08 x 6.5) – 0.41 = 7.02 – 0.41 = 6.61
b. Bias = Yc – Xc = 6.61 – 6.50 = 0.11
c. |Bias| = 0.11%Hb
d. |%Bias| = (0.11 x 100)/6.5 = 1.69%

4. Calculate Sigma:
a. Sigma = (%TEa – |%Bias|)/%CV
i. Sigma = (6.0% – 1.69%)/1.1% = 4.31/1.2 = 3.6

REAL-WORLD EXAMPLE 3
Another report7 in the same issue of Clinical Chemistry examined performance of HbA1c methods used
in central lab testing. This study used the NGSP lab to obtain comparison results by an official U.S.
reference method:

• Precision [Table 1, reference 6]: CV = 1.66% at 5.24%Hb; CV = 1.33% at 7.9%Hb

• Accuracy [Table 1 vs. NGSP, reference 6]: y = 0.998x + 0.016

1. Define TEa at a critical medical decision level (Xc): TEa = +/- 6.0% at 6.5%Hb.

2. Select the appropriate estimate of precision: Precision at 6.5%Hb is probably best represented by
taking the average of the CVs because their means bracket the critical decision level of 6.5%Hb.
The average of 1.66% and 1.33% is 1.50%. Again, here’s an example where judgment is important for
interpreting the results.

3. Calculate bias at Xc:

a. Yc = (0.998 x 6.5) + 0.016 = 6.487 + 0.016 = 6.503
b. Bias = Yc – Xc = 6.503 – 6.50 = 0.003
c. |Bias| = 0.003%Hb
d. |%Bias| = (0.003 x 100)/6.5 = 0.05%

4. Calculate Sigma:
a. Sigma = (%TEa – |%Bias|)/%CV
i. Sigma = (6.0% – 0.05%)/1.5% = 5.95/1.5 = 3.97

26 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

DETERMINING SIGMA-METRIC FROM PT AND SQC DATA
Results from PT surveys may be compared to the target value to determine the difference in the
observed results. Such differences can be expressed in concentration units, in percentage units or in
the form of a z-value that describes a multiple of the group SD or %CV. It is useful to calculate the
differences in concentration units as a percentage of the target value and then average those differences
to obtain an estimate of bias. There usually will be two to five samples in a CAP survey in the United
States. Regulated tests require three survey events per year, with five samples per event; non-regulated
tests (all others except waived tests) require two survey events per year, with a minimum of two
samples per event.

The few PT samples available (only two to five) is a limitation leading to a large uncertainty in the bias
estimate. The minimum number of samples for a comparison of methods experiment is usually 20, and
often 40 or more samples are included. Because of the low number of survey samples, it’s generally
good practice to calculate Sigma both with and without bias.

REAL-WORLD EXAMPLE 4
Most U.S. labs analyze two levels of controls per day to comply with the CLIA QC regulations. Typically,
20 to 30 control observations are available each month. Data are summarized monthly by calculating
the mean, SD and %CV. CAP provides an HbA1c survey widely used in the U.S. and allows monitoring of
the nearly 30 different test systems certified by the NGSP. More than 3,000 labs participate in the CAP
survey, with assay peer groups ranging from 20 to 300 labs. Typically, three samples are provided for
each survey event, and there are only two survey events per year because HbA1c is not a regulated test.
Target values are assigned from analysis by reference methods.

• P
 recision: Routine SQC for two levels of control yielded an SD of 0.105%Hb at 5.58%Hb (1.9%CV) and
an SD of 0.155%Hb at 9.58%Hb (1.6%CV).

• A
 ccuracy: The 2014 CAP GH2 survey event A included three samples for HbA1c (GH2-01 = 6.49%Hb,
GH2-02 = 6.97%Hb and GH2-03 = 9.65%Hb). The lab results were 6.7, 7.3 and 9.9%Hb, respectively.

To determine Sigma:
1. Define TEa at a critical medical decision level (Xc): TEa is +/- 6.0%.

2. Select the appropriate estimate of precision: The CV for the controls brackets the critical Xc of
6.5%Hb; thus, the value should be between 1.9% and 1.6%. Interpolating between the controls, a CV of
1.75% is a good estimate.

3. Calculate bias from the differences between the lab results and the CA-assigned reference values:
a. Calculate the differences between the lab results and the assigned values.
i. 6.80 – 6.49 = 0.21%Hb or 3.24% [(0.21 x 100)/6.49]
ii. 7.30 – 6.97 = 0.33%Hb or 4.73% [(0.33 x 100)/6.97]
iii. 9.90 – 9.65 = 0.25%Hb or 2.59% [(0.25 x 100)/9.60]

b. Average the differences to estimate bias.

i. (3.24% + 4.73% + 2.59%)/3 = 3.52%

27 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

4. Calculate Sigma both with and without bias:
a. Sigma = (%TEa – |%Bias|)/%CV
i. Sigma = (6.0% – 3.52%)/1.75% = 1.42

b. Sigma = %TEa/%CV
i. Sigma = 6.0/1.75 = 3.43

REAL-WORLD EXAMPLE 5
The same lab in example 4 analyzed the CAP survey samples for GH2, event B, in 2014, whose samples
had assigned values of 6.58, 8.39 and 5.65%Hb, respectively. The lab results were 6.7, 8.5 and 5.6%Hb,
respectively.

1. TEa is +/- 6.0%.

2. The long-term CVs were again 1.9% and 1.6% at means of 5.58 and 9.58%Hb. Interpolating between
the controls, a CV of 1.75% is a good estimate.

3. Bias is determined from the differences between the lab results and the CAP-assigned values:
a. Calculate the differences between the lab results and the assigned values.
i. 6.70 – 6.58 = 0.12%Hb or 1.82% [(0.12 x 100)/6.58]
ii. 8.50 – 8.39 = 0.11%Hb or 1.31% [(0.11 x 100)/8.39]
iii. 5.60 – 5.65 = -0.05%Hb or -0.88% [(-0.05 x 100)/5.65]

b. Average the differences to estimate bias.

i. (1.82% + 1.31% – 0.88%)/3 = 0.75%

4. Calculate Sigma both with and without bias:

a. Sigma = (%TEa – |%Bias|)/%CV
i. Sigma = (6.0% – 0.75%)/1.75% = 3.00

b. Sigma = %TEa/%CV
i. Sigma = 6.0/1.75 = 3.43

Note the difference between the estimates of bias (3.52% vs. 0.75%) and Sigma in examples 4 and 5 (1.42
vs. 3.00). Those differences likely represent the limitation of having only three samples analyzed by
single measurements in the lab for comparison. The results would be more reliable with more survey
samples or multiple measurements on each sample, but that’s not permitted by CLIA.

28 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

IMPORTANCE OF DETERMINING QUALIT Y ON THE SIGMA SCALE
The examples above suggest the Sigma quality for current HbA1c methods does not yet approach the
goal of Six Sigma for world-class quality. Keep in mind, HbA1c is one of the most standardized tests in
the world. A global IFCC lab network supports reference methods and materials, and several national
lab networks, such as NGSP, certify the equivalence of virtually all test systems being marketed in
the U.S.

The two studies 6,7 cited above were published in 2014. The data from the study of POC HbA1c test
systems6 report Sigmas that range from 0.44 to 4.23, with three of the seven methods demonstrating
quality > three Sigma. Those results are summarized on the method decision chart in Figure 2-5,
and most lab personnel will find this graphical summary to be much more understandable than the
statistical tables in the paper. In the second paper, the data from the study of central lab HbA1c test
systems7 proves only one in six test systems provides quality > three Sigma.

Method Decision Chart TEa=6.0%

6

5
Accuracy - Observed %Bias

2
2

4
1
3

0
0 0.5 1 1.5 2 2.5 3

Precision - Observed %CV

2 Sigma 4 Sigma 6 Sigma

3 Sigma 5 Sigma

Figure 2-5: Summary of performance data for 7 test methods, each compared with 3 reference methods. Method
Decision chart prepared for TEa=6.0%.

29 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

Additional data from the 2014 CAP HbA1c survey provide further evidence of the low Sigma quality
of many test systems. Figure 2-6 shows the performance of current U.S. test systems by plotting the
bias on the y-axis and SD on the x-axis for each method subgroup. Note that this is simply a two-
sided method decision chart adapted for use with PT and EQA data8. The inner diagonal >-shaped
line represents three Sigma quality, and the outer line represents two Sigma quality. Only one method
subgroup achieves three Sigma quality, six achieve between three and two Sigma, and 19 achieve less
than two Sigma. This shows the importance of PT and EQA surveys for evaluating the comparability of
results from different method subgroups (in this case, 26 different methods approved by the FDA and
certified as equivalent by NGSP).

6.0

5.0 2
4.0
Observed Trueness (%Bias)

3.0
3
2.0

1.0

0.0

-1.0

-2.0
3
-3.0

-4.0

-5.0 2
-6.0
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0

Observed Standard Uncertainty (%CV)

Figure 2-6: Sigma Proficiency Assessment Chart

Figure 2-6. for
Sigma 2014 College
Proficiency AssessmentofChart
American Pathologists
for 2014 College (CAP) survey results for
of American
HbA1c GH2-01 sample with concentration of6.49%Hb.
Pathologists (CAP) surveyTEa=6.0%. Each
results for HbA1c point
GH2-01 represents
sample the observed
with concentration of trueness (%Bias,
6.49 %Hb. TEa=6.0%. Each point represents the observed trueness (%Bias, y-axis)
y-axis) a nd the observed Standard Uncertainty (%CV, x-axis) for one of 26 examination 
s ubgroups.
and the observed Standard Uncertainty (%CV, x-axis) for one of 26 examination Results represent a
total of 3,187 laboratories. subgroups. Results represent a total of 3187 laboratories.

WHAT’S THE POINT?

It is absolutely critical to determine quality on the Sigma scale to assess the risk of any test
or test system. Validation of safety characteristics is the most important first step in risk
assessment. It is essential to use in-house validation and QC data to perform a risk assessment.
Determination of Sigma quality is the single best predictor of the risk. Sigma is also a useful
predictor of the QC needed to minimize the risk of poor-quality test results.

30 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S

References
1. ISO 14971:2007. Medical devices – Application of risk management to medical devices. ISO, Geneva, 2007.

2. U.S. Department of Health and Human Services. Medicare, Medicaid, and CLIA Programs: Laboratory requirements relating
to quality systems and certain personnel qualifications. Final Rule. Federal Register Jan 24 2003;16:3640-3714. See also CMS
website [Link]/clia/ and the CMS State Operations Manual, Appendix C, Regulations and Interpretive Guidelines for
Laboratories and Laboratory Services, [Link]/CLIA/03_Interpretive_Guidelines_for_Laboratories.asp.

3. CMS Memo of August 16, 2013: Individualized Quality Control Plan (IQCP): A new Quality Control (QC) option. [Link]/
Regulations-and-Guidance/Legislation/CLIA/Downloads/[Link].

4. Westgard S. Prioritizing risk analysis quality control plans based on Sigma-metrics. In Quality Control in the Age of Risk Analysis,
eds. Westgard JO, Westgard S. Clinics in Laboratory Medicine 2013;33(Number 1):41-53.

5. Westgard S. Six Sigma-metric analysis for analytical testing processes. White paper available from Abbott Diagnostics.

6. Lenters-Westra E, Slingerland RJ. Three of 7 hemoglobin A1c point-of-care instruments do not meet generally accepted analytical
performance criteria. Clinical Chemistry 2014;60:1062-1072.

7. Woodworth A, Korpi-Steiner N, Miller JJ, Rao LV, Yundt-Pacheco J, Kuchipudi L, Parvin CA, Rhea JM, Molinaro R. Utilization of
assay performance characteristics to estimate Hemoglobin A1c result reliability. Clinical Chemistry 2014;60:1073-1079.

8. Westgard JO, Westgard SA. Assessing quality on the Sigma scale from proficiency testing and external quality assessment surveys.
Clinical Chemistry and Laboratory Medicine 2015;53:1531-1535.

31 | Learning Guide: Six Sigma-based Quality Control B AC K TO T H E TA B L E O F CO N T EN T S