Nonstatistical Sampling

AU 350 does not provide a definition of nonstatistical sampling. It states only that “[t]here are
two approaches to audit sampling: nonstatistical and statistical” (AU 350.03). The AICPA’s
Audit Guide, Audit Sampling, provides the following definition:

Any sampling procedure that does not measure the risk is a nonstatistical sampling procedure.
Even though the auditor rigorously selects a random sample, the sampling procedure is a
nonstatistical sampling application if the auditor does not make a statistical evaluation of the
sample results. (AAG-SAM 2.18)

This statement establishes that an auditor may label a sampling technique “nonstatistical”
without regard to the manner of sample selection. Thus, even though the Audit Guide
acknowledges the well-known ability of statistical sampling to measure sampling risk, it
nevertheless sanctions an auditor’s decision to ignore available statistical theory and rely instead
on judgment or intuition in interpreting the results of a sampling procedure. In short, the guide
gives guesswork equal status with measurability. Such a view is potentially hazardous, because
the auditor is permitted to ignore facts that are readily discernable to any practitioner, or legal
adversary, who is knowledgeable in the application of statistical methodology.

Why would an auditor prefer nonstatistical sampling, knowing of the availability of objective
statistical procedures? Various reasons, restated in the 2001 edition of the Audit Guide, have
been cited as the impediments: the cost of training, the cost of sample selection, the cost of
sample evaluation. With the passage of time, these reasons have become progressively weaker.
Mandatory continuing professional education is now a reality, so there should be little reason for
auditors not to advance their skills in sampling techniques. As to the implementation costs
associated with the selection and evaluation of random samples, the ready availability of
computers and off-the-shelf software has greatly mitigated, if not eliminated, these factors as
relevant considerations.

In short, a nonstatistical sample is selected by the exercise of judgment, and not by chance.
Haphazard, judgmental, and purposive sampling are some of the terms that describe a
nonstatistical sample.

Statistical Sampling

AU 350 and the Audit Guide approach statistical sampling in a roundabout way. The Audit
Guide states:

Statistical sampling helps the auditor (1) design an efficient sample, (2) measure the sufficiency
of the evidential matter obtained, and (3) quantitatively evaluate the sample results.

Statistical sampling uses the laws of probability to measure sampling risk. (AAG-SAM 2.17)

Although the foregoing statements are correct, they do not define statistical sampling per se.
Statistical sampling is probability sampling. In probability sampling, every item in the population
under audit has a known chance of selection. The decision as to which items in the population
are to be selected is left to the laws of chance, not to judgment. The most common probability
sampling methods in auditing are equal probability (such as simple random and systematic
sampling) and sampling with probability proportional to size (such as monetary unit sampling).

The prominent feature of statistical sampling is its ability to measure risk. The measurement
instrument is the confidence interval, which gives a calculated range of values for the estimated
amount of misstatement in a population. The measurability of statistical sampling distinguishes it
from so-called judgment sampling, where the decision as to the items selected for examination is
left to the judgment of the auditor. Statistical sampling is a measurement tool. When applied in a
substantive test of details, it measures misstatement in an account or class of transactions. Its
ability to measure arises from the selection method used, which is probability sampling.
Lawyers, judges, and statisticians have explicitly recognized these features of statistical
sampling. The Special Committee on Empirical Data in Decision Making, Recommendation on
Pretrial Proceeding in Cases with Voluminous Data, made the following statement (see
Appendix F, in Fienberg, S.E., ed., The Evolving Role of Statistical Assessments as Evidence in
the Courts, 1989):

[W]hen a survey is based on probability sampling, the probabilities or risks of sampling

misstatements of various sizes can be calculated. This requires the application of appropriate
statistical formulas. Assessments of sampling misstatement are very often expressed in terms of a
standard misstatement. This is a universally accepted measure of the margin of error in a survey
result that is attributable to sampling.

This illuminating report should serve to alert auditors to the growing use of statistically based
evidence in litigation and, by implication, to the risks they face should they ignore the
information contained in samples.

The implication is clear: Ignore the formulas applicable to the results of a probability sample and
rely instead on intuition at your own risk.

Some auditors believe that they must calculate a sample size beforehand for an audit sample to
be statistical. This is incorrect. Any probability sample can be subjected to evaluation by
application of the laws of probability, however arbitrary the choice of sample size. Failure to
calculate beforehand usually results in samples that are either too large or too small for the
auditor’s objectives. They are, nevertheless, statistical.

Statistical and nonstatistical sampling methods are defined in terms of the method by which a
sample is selected, not in terms of a decision by the auditor not to apply statistical methods, even
to a random sample.

When Is Statistical Sampling Appropriate?

Statistical sampling is appropriate whenever an auditor wishes to draw a conclusion about a

population without performing an examination of all the items composing that population.
Moreover, statistical sampling is appropriate when the auditor has no prior knowledge as to
which specific items in a population are misstated.

An important concern that affects the sampling decision is the practicability of selecting a
probability sample. If files are computerized and 100% verification cannot be performed by
computer-assisted audit techniques, then probability sampling is most likely to be the practical
approach. If files are not computerized and the population is large (as a rough rule of thumb, a
large population has more than 500 items), then probability sampling may still be practicable. If
a population of manual records is maintained in numerical order, a computer application may be
used to select random numbers that identify the items to be selected, even items at multiple
locations. The items are then located by hand. If the population is not maintained in numerical
order, then systematic selection (select every kth item after a random start) may be performed.
Systematic selection is one of the easiest procedures to apply, although proper application
requires counting through the population. Although many caution that systematic selection is
subject to bias because a key characteristic of the population under examination may coincide
with the selection interval, in more than 30 years of practice, the author has never observed this
to be even a remote practical concern.

Statistical sampling is appropriate for both routine and nonroutine accounting processes. In a test
of purchase transactions, for example, the auditor may employ statistical sampling to test for
misstatement in account distribution. An auditor may also apply statistical sampling to a
population of securities positions for a large broker-dealer with thousands of positions, to test
valuation and existence assertions.

Sampling Risk

AU 350 states “[s]ampling risk arises from the possibility that, when a test … is restricted to a
sample, the auditor’s conclusions may be different from the conclusions he would reach if the
test were applied in the same way to all items in the [population].” (AU 350.10) AU 350 also
identified two aspects of sampling risk:

The risk of incorrect acceptance is the risk that the sample supports the conclusion that the
recorded account balance is not materially misstated when it is materially misstated.

The risk of incorrect rejection is the risk that the sample supports the conclusion that the
recorded balance is materially misstated when it is not materially misstated. (AU 350.12)

In practice, it is convenient to think of the foregoing in terms of detection risk and estimation
risk, respectively.

Detection risk is the chance that a sample will fail to detect misstatement that actually exceeds
the auditor’s specified maximum tolerable amount. “Detection” refers to the decision rule that an
auditor applies to decide whether a misstatement is tolerable under the circumstances. A
commonly employed rule is the comparison of the calculated upper confidence limit of
misstatement with the specified maximum tolerable amount. In SAS 39 terms, the upper
confidence limit is the projected misstatement plus the allowance for sampling risk. If the
calculated limit is greater than the maximum tolerable amount, the auditor decides that
misstatement may exceed the tolerable amount. Otherwise, the auditor decides that misstatement,
if it exists, is tolerable. If a properly designed sample discloses no misstatements, the auditor
may then decide that misstatement in the population under audit does not exceed the maximum
tolerable amount.

Detection risk is principally a planning concept. The auditor specifies it beforehand and uses it as
one of the factors that determines the appropriate extent of testing reflected in the sample size.

If misstatements are detected, on the other hand, the estimation risk becomes the key risk under
consideration. Estimation risk is the chance that the actual amount of misstatement will not be
within the calculated confidence interval. SAS 39 is dismissive of this risk, which it labels the
risk of incorrect rejection, as being merely an efficiency issue. AU 350.12 states:

[I]f the auditor’s evaluation leads him to the initial erroneous conclusion that a balance is
materially misstated when it is not, the application of additional audit procedures and
consideration of other audit evidence would ordinarily lead the auditor to the correct conclusion.

This is misleading. An auditor does not know that his conclusion is incorrect; only that the
evidence suggests that the population may be materially misstated. Frequently, this is sufficient
for action, and no further audit evidence is needed, even if it were practicable to extend testing or
to apply alternate procedures. More seriously, AU 350.12 invites the auditor to disregard the
results of an unfavorable sample outcome and subordinate it to other, contradictory evidence
whose reliability may be less than that of the sample.

Moreover, if the results of an audit sample are sufficiently precise, they may provide the basis for
the proposal of an adjusting journal entry by the auditor. In such a case, the appropriate risk
consideration is that the adjustment is materially correct. The calculated confidence interval
provides the basis for that assessment. Estimation risk is the complement of the confidence level.

Statistical Sampling and Audit Decisions

The auditor uses a sample to decide whether misstatement exists and whether it may exceed the
tolerable misstatement. This is the essence of the detection objective of a substantive test of
details. While is it possible to design a sample to control for both the detection and estimation
risk, audit samples often are designed only with the detection objective in mind. Nonetheless, if a
properly selected random sample has disclosed misstatement, that sample can always be used to
obtain a confidence interval on the amount of misstatement, regardless of the planning decisions
and the consequent sample size.

For convenience, interval estimates may be classified into six basic categories, each of which is
informative in its own way as to the extent of misstatement in the population. The possibilities
are discussed below in terms of tolerable misstatement (TM), which is $600,000 in the examples,
the lower confidence limit (LCL) on the estimated misstatement, and the upper confidence limit
(UCL) on the estimated misstatement. The projected misstatement (that is, point estimate) is not
needed, as the following examples will show. More importantly, the projected misstatement
could be misleading. A projection (or point estimate) is merely one outcome in a sample space.
Its principal function is to be locator for the confidence interval. It provides no information as to
its margin of error. For example, 10 missstatements of $100 each will yield the same point
estimate as one $1,000 misstatement, but the latter’s margin of error is greater.

Consistency with Generally Accepted Auditing Standards

Assurance that Bias is Avoided:
The risk of bias in judgmental testing can in some applications be real. There is a natural
human tendency to favor (perhaps unconsciously) easily accessible selection points. If the
less accessible population items happen to be the ones in error (their very inaccessibility
may be related to their being in error), the sample bias could lead to seriously misleading
conclusions. It is also possible that judgmental selection will avoid, say, the first and last
items on any page on the grounds that such items do not seem as random as others. Of
course, a true random sample will select such items some of the time. If there is some
systematic reason for initial or final items on each page being more error-prone, a
judgmental test avoiding them could again lead to misleading conclusions. There may
also be an instinctive tendency in a judgmental test to make proportionately more
selections at the beginning when the auditor is fresh than toward the end when tired.
Assurance that Sample Size is Sufficient:
Far more important, however, is the assurance which statistical sampling provides that
the sample size is sufficient to warrant the conclusions expressed. Whatever the abilities
of human judgment in assessing qualitative factors, such as relative strength of internal
control or reasons for a given error encountered, these abilities are noticeably less in
assessing quantitative factors, such as how much testing is enough or how high an error
frequency might really be. Our common sense seems to be less than perfect when it
comes to assessing odds (a deficiency not unrelated to the popularity of lotteries). For
example, if an average group of people is asked to estimate the chance of obtaining three
heads out of six tosses of a fair coin, the most common (and indeed intuitive) answer is
50 percent, though a wide range of answers within the group can be expected. In fact, the
chance is only 31 percent. If groups of auditors are asked how many receivable accounts
must be confirmed to provide a high degree of assurance of detecting a material error if
present, similar discrepancies are likely. The most important benefit which statistical
sampling offers is the reduction, through the use of mathematical aids to judgment, of the
risk of over-auditing or under-auditing.
Of course, where the incremental benefit of converting testing techniques to a statistical
basis is disproportionately costly, the use of statistical sampling would not be justified
despite these risks. Undoubtedly there are many audit tests where this is the case and
where the use of judgmental testing is thus the only responsible course to follow. But
there are also many other audit tests where there is no additional cost to statistical
sampling or where the cost is slight in comparison with the benefits of the greater
objectivity in determining extent, selection and evaluation by statistical means. Suggested
criteria for making this cost-benefit decision were discussed earlier.
It is a guide to judgment, not a substitute for it. In the end, it is the auditor’s responsibility
to choose those tests, test extents and testing techniques, which in their professional
judgement are sufficient to satisfy generally accepted auditing standards.
Office Standards for Sampling:
Again, in the end, it is the auditor’s responsibility to choose those tests, test extents and
testing techniques, which in their professional judgment are sufficient to satisfy generally
accepted auditing standards. The purpose of this standard is to provide guidelines for the
proper use of audit sampling techniques in University and University-related audits.
Statistical and non-statistical sampling techniques are mutually exclusive tools to be used
as dictated by specific audit conditions. The objective in sampling is to infer conclusions
about certain characteristics of a given population, without examining the entire
population. The selection process (statistical vs. non-statistical) should be the result of
considerations and decisions discussed below and does not affect the audit procedures
performed.
There is no difference between statistical sampling and non-statistical sampling in the
execution of a sampling plan, nor does the approach affect the competence of the
evidence obtained or the auditor’s responses to detected errors. Selection between
statistical and non-statistical sampling should be made after an evaluation of the audit
objective(s) and the advantages and disadvantages of statistical and non-statistical
sampling.
Reporting of Results:
In reporting observations and conclusions based on the results of statistical sampling, the
levels of both precision and confidence for the sample should be reported. Standard
report language for reporting the results of testing using statistical audit sampling
techniques includes phrases such as:
"Based on the results of our statistical testing, it is our opinion that with a xx%
confidence level (reliability), the error rate for the population does not exceed x."  In
reporting observations and conclusions based on the results of non-statistical sampling,
the report language must be clear that a non-statistical or judgmental sampling technique
was used or the report should be silent on the sampling/testing technique as further
addressed in the following section, "Guidelines for Concluding Using Non-Statistical
Sampling."
Guidelines for Concluding Using Non-Statistical Sampling:
Non-statistical sampling should not be used, in the report or formal conclusions, to
estimate the number or value of items in a population that were defective or improperly
processed. Non-statistical or judgment sampling is subjective. If no errors are found, the
auditor may be able to conclude there is no basis for examining the population further or
for suspecting any material error. There is no statistical basis for concluding that the
auditor has adequate assurance that the error rate is acceptable/unacceptable or even
reasonable. What can be concluded, however, may be sufficient for the audit purposes.
Documentation of Audit Sampling:
Workpapers evidencing sampling (statistical and non-statistical) techniques should
include, at a minimum, comments on the following items:
        Identification of the controls or attributes being tested;
        Sampling approach (e.g. attribute, variable, judgmental or other);
        Description of the population from which each sample is selected (e.g., size,
homogeneity, etc.);
        For statistical samples, predetermined confidence level and precision level (The
following criteria ranges for sampling compliance matters are suggested: for
confidence level or reliability - 95% to 99%; and for precision or tolerable error
 rate - 2% to 5%. Note: Lesser confidence and precision levels may be acceptable
depending on the audit objective(s).);
        Calculation/determination of the sample size;
        Sample selection method;
        Record of the tests performed;
        Analysis of errors; and
        Conclusions.

Statistical Sampling

Statistical sampling involves the random selection of a number of items for inspection and is
endorsed by the accountancy bodies. In statistical sampling, each item has a calculable chance of
being selected.

A commonly held misconception about statistical sampling is that it removes the need for the use
of the professional judgement. While it is true that statistical sampling uses statistical methods to
determine the sample size and to select and evaluate audit samples, it is the responsibility of the
auditor to consider and specify in advance factors such as, materiality, the expected error rate or
amount, the risk of over-reliance or the risk of incorrect acceptance, audit risk, inherent risk,
control risk, standard deviation and population size, before the sample size can be determined.
Statistical sampling allows an auditor’s judgement to be concentrated on those areas of the audit
where it is most needed. It allows the quantification of key factors and the risk of errors. This is
not to suggest that statistical sampling methods remove the need for professional judgement, but
rather that they allow elements of the evaluation process to be quantified, measured and
controlled.

The advantages of statistical sampling are numerous:

1. The sample result is objective and defensible. Nearly all phases of the statistical process
are based on demonstrable statistical principles.
2. The method provides a means of advance estimation of sample size on an objective basis.
The sample size is no longer determined by traditional methods of guesswork; it is
determined by a statistical method.
3. The method provides an estimate of error. When probability sampling is used, the results
may be validated in terms of how far the sample projection might deviate from the value that
could be obtained by a 100% check.
4. Statistical samples may be combined and evaluated, even though accomplished by
different auditors. That the entire test operation has an objective and scientific basis makes it
possible for different auditors to participate independently in the same test and for the
results to be combined as though accomplished by one auditor.
5. Objective evaluation of test results is possible. Thus, all auditors performing this audit
would be able to reach the same conclusion about the numerical extent of error in the
population. While the impact of these errors might be interpreted differently, there can be
no question as to the facts obtained, since the method of determining their frequency in the
population is objective.

Disadvantages: statistical
Time consuming?
Audit costs increased?
To be able to use it, each item within a population must be individually
identifiable only really practicable with computerised records
It promotes a mechanical approach - no room for "gut feeling"
It's much too complex? Only of use for large auditors on large clients?