Chauvenet’s Criterion, Peirce’s Criterion, and
n-s Criterion
Gilang Ardi Pratama
1
Civil Engineering Departement,Engineering Faculty, Universitas Indonesia,
Indonesia
Abstract.
In order to answer the research questions, it is doubtful that researcher should be
able to collect data from all cases. Thus, there is a need to select a sample of data.
filtering of the data is a sensitive topic, because we dont know to choose which of the
data has the good or bad value from a sample. The following below are the three
methods of the filtering process of the data: Chauvenet’s Criterion, Peirce’s Criterion,
and n- s Criterion.
1 Chauvenet’s Criterion
Chauvenet’s criterion is a way to identify outliers. The method works by creating an
acceptable band of data around the mean, specifying any values that fall outside that
band should be eliminated[1]. The formula to calculate the band is:
Where n is the sample size.
Chauvenet's criteria is easy to understand, can be quickly computed for a billion
rows of data, and even gentle enough to be used on a tiny set of only ten points.
Chauvenet’s criterion makes no distinction between the case of one or several
suspicious data values Normal distribution probabilities can be used to relate the
formula to a maximum deviation from the mean:
τ = |Xi – x̄ |/ s. This formula is a little more intuitive to use[2].
Figure 1. Identification an Original Data[2]
Figure 1 shown of the identification an original data set. The Identification of the
appropriate data is not simple, which of the data to deleted or thrown away from an
observation. Assume a data set with numeric variable x. Suppose there are n
observations in a dataset. We want to throw away all observations which are "not
good enough". How do you do this?
Step 1: Calculate the sample mean, x̄ .
Step 2: Find the sample standard deviation, s.
Step 3: Use the following formula to find the standardized deviation from the mean
for all suspected outliers, data values “i.”
τ = |Xi – x̄ |/ s
We have thirty items in the set, so you’ll want to run the formula twenty times (one
for each data point, i).
Step 4: Compare the values you got in Step 4 with a table of Chauvenet’s criterion
value bellow or calculate investigation value calculating using band /2+0,5 and
finding the z score in z score table (table.2) of that value to see if you can reject
each data point.
Table.1 Chauvenet Table[2]
Table.2 Z-Score Table
The procedure should only be run once on a data set. Some authors advocate
two runs; any outliers that are revealed on the second run are called shielded
outliers. Note: other authors say never to eliminate outliers, so this is a judgment
call based on your data.
The procedure assumes your data is normally distributed.
2. Peirce’s Criterion
The temptation to ignore this data with rationalisations such as blaming faulty
recording equipment ( the equipment had a power surge, there was dirt in the lens)
should be resisted. Instead of arbitrarily dropping data, Peirce's Criterion [3] may be
applied.
The method is similar to the commonly used Chauvenet's criterion; however,
Peirce's criterion is a more rigorous theoretical development based on the Gaussian
distribution which can be applied to more than one suspect data value. In fact,
Chauvenet refers to the prior work of Peirce, writing, in his original work: "What I
have given may serve the purpose of giving the reader greater confidence in the
correctness and value of Peirce's criterion[2]." How do you do this?
1) Calculate the mean and the sample standar deviation (SD) of the complete
data set.
2) Obtain R corresponding to the number of measurements taken from Peirce’s
table.
3) Calculate the maximum allowable deviation: |xi – xm| max
4) For any suspicious data measurements, obtain |xi – xm|.
5) Eliminate the suspicious measurements if: |xi − xn| > |xi − xm|max (4)
6) If this results in the rejection of one measurement, assume the case of two
doubtful observations, keeping the original values of the mean and standard
deviation, and the original number of measurements. Go to step 8.
7) If more than one measurement is rejected in the above test, assume the next
highest value of doubtful observations.
8) Repeat the above calculations (steps 2 – 5),
9) Now obtain the new value of the mean and sample standard deviation of the
reduced data set.
Table.3 Peirce Table[2]
The method can be applied using a table which lists R values corresponding to the
number of data values. The possibility of more than one suspect experimental data
value is also included.
3. n- s Criterion
In statistics, the 68–95–99.7 rule, also known as the empirical rule, is a shorthand
used to remember the percentage of values that lie within a band around
the mean in a normal distribution with a width of two, four and six standard
deviations, respectively; more accurately, 68.27%, 95.45% and 99.73% of the values
lie within one, two and three standard deviations of the mean, respectively[4].
In mathematical notation, these facts can be expressed as follows, where Χ is an
observation from a normally distributed random variable, μ is the mean of the
distribution, and σ is its standard deviation [5]:
In the empirical sciences the so-called n- s expresses a conventional heuristic
that nearly all values are taken to lie within three standard deviations of the mean
[5],and thus it is empirically useful to treat 99.7% probability as near certainty. [1] The
usefulness of this heuristic depends significantly on the question under
consideration. In the social sciences, a result may be considered "significant" if
its confidence level (table 5) is of the order of a two-sigma effect (95%),
Table.3 Sigma Table[5]
while in particle physics, there is a convention of a five-sigma effect (99.99994%
confidence) being required to qualify as a discovery[6]. How you do this ?
1. Calculate the average of the observed data.
2. Calculate the variance of the set. Variance is the spread between data points
and is calculated as the sum of the squares of the difference between each
data point and the mean divided by the number of observations. The first dif-
ference square will be calculated , the second square of difference will be Cal-
culated and the others data points, sum all data points of the different
squares,
3. Calculate the standard deviation, which is simply the square root of the va-
riance.
4. Calculate three-sigma using z score and table, which is three standard devia-
tions above the mean.
in numerical format, The latter equation means that the values of X can differ from its
expectation aa by a quantity exceeding 3σ This circumstance is sometimes used by
an experimenter in certain problems of probability theory and mathematical statistics,
by assuming that the event{|X−a|>3σ}{|X−a|>3σ} is practically impossible and,
consequently, the event {|X−a|<3σ}{|X−a|<3σ} is practically certain. In this case one
says that the experimenter has applied the "three-sigma" rule[6].
Reference
1. Chavuenet, W., A manual of spherical and practical astronomy.
1871.
2. Ross, S.M., Peirce's criterion for the elimination of suspect
experimental data. Journal of Engineering Technology, 2003. 20(2):
p. 38-41.
3. Peirce, B., Criterion for the rejection of doubtful observations. The
Astronomical Journal, 1852. 2: p. 161-163.
4. Grafarend, E.W., Linear and nonlinear models: fixed effects, random
effects, and mixed models. 2006: de Gruyter.
5. Wheeler, D.J. and D.S. Chambers, Understanding statistical process
control. 1992: SPC press.
6. Pukelsheim, F., The three sigma rule. The American Statistician,
1994. 48(2): p. 88-91.