Scribd
Upload
52 views11 pages

Outlier Detection Techniques in Python

The document discusses 11 different methods for outlier detection in Python, including both statistical and machine learning approaches. It begins by defining outliers and how they occur in datasets. Then it covers simple statistical techniques like sorting, visualization with boxplots and histograms, and calculating z-scores. Later it discusses outlier detection using interquartile range, hypothesis testing, and four machine learning algorithms: Robust Covariance, One-Class SVM, Isolation Forest, and Local Outlier Factor. Code examples are provided for many of the methods.

Uploaded by

Neethu Merlin Alan
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
52 views11 pages

Outlier Detection Techniques in Python

The document discusses 11 different methods for outlier detection in Python, including both statistical and machine learning approaches. It begins by defining outliers and how they occur in datasets. Then it covers simple statistical techniques like sorting, visualization with boxplots and histograms, and calculating z-scores. Later it discusses outlier detection using interquartile range, hypothesis testing, and four machine learning algorithms: Robust Covariance, One-Class SVM, Isolation Forest, and Local Outlier Factor. Code examples are provided for many of the methods.

Uploaded by

Neethu Merlin Alan
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

11 different ways for Outlier Detection in Python

[Link]/outlier-detection-python

Outlier Detection Python is a specialized task which has various use-cases in Machine
Learning. Use-cases would be anomaly detection, fraud detection, outlier detection etc. There
are many ways we can find outliers in your analysis.

So, let us talk about outliers in your datasets and explore various quick ways we can identify
outliers in daily analytics lifecycle.

In this post, you will learn :

What are outliers?


How do outliers occur in your dataset
What are the simple methods to identify outliers in your dataset
Outliers Detection using machine learning algorithms – Robust Covariance, One-Class
SVM, Isolation Forest, Local Outlier Factor

1/11
What are outliers?

Outliers: in simple terms outliers are data points which are significantly different from your
entire datasets. For e.g.

Image source

How do outliers occur in a datasets?

Outliers occur either by chance, or either by measurement error or data population is heavy
tailed distribution as shown above.

Main effects of having outliers are that they can skew your analytics in poor analysis, longer
training time and bad results at the end. Most importantly, this distorts the reality which
exists in the data.

Simple methods to Identify outliers in your datasets.

Sorting – If you have dataset you can quickly just sort ascending or descending.

While it is looks so obvious, but sorting actually works on real world.

Outlier Detection Python – Quick Method in Pandas – Describe( ) API

2/11
import numpy as np
import pandas as pd

url =
'[Link]

df = pd.read_csv(url)
[Link]()

If you see in the pandas dataframe above, we can quick visualize outliers. For e.g. in pm2.5
column maximum value is 994, whereas mean is only 98.613.

Data Visualization using Box plots, Histograms, Scatter plots


If we plot a boxplot for above pm2.5, we can visually identify outliers in the same.

Histograms

BoxPlot to visually identify outliers

3/11
Again similar data but different visualization, we can see that there are some long tail outliers
in the data.

Outlier Detection using Z-Scores


Z-scores can help in finding unusual data points with our datasets when our data is following
normal distribution.

Z score formula is (X – mean)/Standard Deviation

We can see outlier at the bottom of the table has different Z-


Score as compared to others.

Create outlier Fences using Interquartile Range


IQR is basically middle of our dataset which is also know as
Median of a dataset. We can calculate IQR with following
formula ( Q3- Q1).

Now based on IQR we can assign lower outer, lower inner,


upper inner, and upper outer ranges and all the data points
which are outside this range can be considered as outliers.

Hypothesis Testing – Grubb’s Outlier test


Grubb’s outlier test can only detect uni variate outliers,
however there are other tests which are available like Tietjen-
Moore test. However it requires specific number of outliers,
which is difficult to do as we are trying to find out the
outliers in first place.

Outlier Detection Using Machine Learning

In this section , we will discuss four machine learning techniques which you can use for
outlier detection.

4/11
Robust Covariance – Elliptic Envelope
This method is based on premises that outliers in a data leads increase in covariance, making
the range of data larger. Subsequently the determinant of covariance will also increase, this
in theory should reduce by removing the outliers in the datasets. This method assumes that
some of hyper parameters in n samples follow Gaussian distribution. Here is flow on how
this works:

[Link] k times:

1. Sample Points randomly and compute there mean and covariance


1. Repeat it twice:

1.2.1 Compute mahalonobis distances for all points and sort them in ascending order

1.2.2 Use smallest hyper parameter distances to computer new estimates of mean and
covariance

2. Find the determinant of covariance

2.1 Repeat the step again with small subset until convergence which means determinants are
equal

2.2 Repeat all points in 1(a) and 1(b)

3. In all subsets of data, use the estimation of smallest determinant and find
mean and covariance.

More information on theory about Robust covariance. Here is a link

Outlier Detection Python Code – Elliptic Envelope

5/11
import numpy as np
from [Link] import EllipticEnvelope
import [Link] as plt
import matplotlib.font_manager
from [Link] import load_wine
%matplotlib inline

# Define "classifiers" to be used


classifiers = {
"Robust Covariance (Minimum Covariance Determinant)":
EllipticEnvelope(contamination=0.25),
}
colors = ['m', 'g', 'b']
legend1 = {}
legend2 = {}

# Get data
X1 = load_wine()['data'][:, [1, 2]] # two clusters

# Learn a frontier for outlier detection with several classifiers


xx1, yy1 = [Link]([Link](0, 6, 500), [Link](1, 4.5, 500))
for i, (clf_name, clf) in enumerate([Link]()):
[Link](1)
[Link](X1)
Z1 = clf.decision_function(np.c_[[Link](), [Link]()])
Z1 = [Link]([Link])
legend1[clf_name] = [Link](
xx1, yy1, Z1, levels=[0], linewidths=2, colors=colors[i])

legend1_values_list = list([Link]())
legend1_keys_list = list([Link]())

# Plot the results (= shape of the data points cloud)


[Link](1) # two clusters
[Link]("Outlier detection on a real data set (wine recognition)")
[Link](X1[:, 0], X1[:, 1], color='black')
bbox_args = dict(boxstyle="round", fc="0.8")
arrow_args = dict(arrowstyle="->")
[Link]("outliers points", xy=(4, 2),
xycoords="data", textcoords="data",
xytext=(3, 1.25), bbox=bbox_args, arrowprops=arrow_args)
[Link](([Link](), [Link]()))
[Link](([Link](), [Link]()))

[Link]("ash")
[Link]("malic_acid")

[Link]()

6/11
Code Source: Scikit Learn

One-Class SVM
One class Support Vector Machine is a special case in support vector machines which is used
for unsupervised outlier detection. For more information on support vector, please visit this
link.

Let see outlier detection python code using One Class SVM. We will see two different
examples for it.

from [Link] import OneClassSVM


X = [[0], [0.44], [0.45], [0.46], [1]]
clf = OneClassSVM(gamma='auto').fit(X)
[Link](X)

array([-1, 1, 1, 1, -1, -1, -1], dtype=int64)


Here -1 refers to outlier and 1 refers to not an outliers.

Let us see another example

7/11
import numpy as np
import [Link] as plt
import matplotlib.font_manager
from sklearn import svm

xx, yy = [Link]([Link](-5, 5, 500), [Link](-5, 5, 500))


# Generate train data
X = 0.3 * [Link](100, 2)
X_train = np.r_[X + 2, X - 2]
# Generate some regular novel observations
X = 0.3 * [Link](20, 2)
X_test = np.r_[X + 2, X - 2]
# Generate some abnormal novel observations
X_outliers = [Link](low=-4, high=4, size=(20, 2))

# fit the model


clf = [Link](nu=0.1, kernel="rbf", gamma=0.1)
[Link](X_train)
y_pred_train = [Link](X_train)
y_pred_test = [Link](X_test)
y_pred_outliers = [Link](X_outliers)
n_error_train = y_pred_train[y_pred_train == -1].size
n_error_test = y_pred_test[y_pred_test == -1].size
n_error_outliers = y_pred_outliers[y_pred_outliers == 1].size

# plot the line, the points, and the nearest vectors to the plane
Z = clf.decision_function(np.c_[[Link](), [Link]()])
Z = [Link]([Link])

[Link]("Outlier Detection")
[Link](xx, yy, Z, levels=[Link]([Link](), 0, 7), cmap=[Link])
a = [Link](xx, yy, Z, levels=[0], linewidths=2, colors='darkred')
[Link](xx, yy, Z, levels=[0, [Link]()], colors='palevioletred')

s = 40
b1 = [Link](X_train[:, 0], X_train[:, 1], c='white', s=s, edgecolors='k')
b2 = [Link](X_test[:, 0], X_test[:, 1], c='blueviolet', s=s,
edgecolors='k')
c = [Link](X_outliers[:, 0], X_outliers[:, 1], c='gold', s=s,
edgecolors='k')
[Link]('tight')
[Link]((-5, 5))
[Link]((-5, 5))
[Link]([[Link][0], b1, b2, c],
["learned frontier", "training observations",
"new regular observations", "new abnormal observations"],
loc="upper left",
prop=matplotlib.font_manager.FontProperties(size=11))
[Link](
"error train: %d/200 ; errors novel regular: %d/40 ; "
"errors novel abnormal: %d/40"
% (n_error_train, n_error_test, n_error_outliers))
[Link]()

8/11
Image and code source: Scikit Learn

Isolation Forest
Isolation Forest is an ensemble model which isolates observations by randomly selecting a
feature and selecting a split value between maximum and minimum of selected feature.

Since this recursive partitioning is represented by a tree structure, and number of splittings is
equivalent to path length from root node to terminating node. For more information, use this
link.

See Isolation Forest in code.

from [Link] import IsolationForest


X = [[-1.1], [0.3], [0.5], [100]]
clf = IsolationForest(random_state=0).fit(X)
[Link]([[0.1], [0], [90]])

array([ 1, 1, -1])
Here -1 refers to outlier and 1 refers to not an outliers.

Local Outlier Factor (LOF)


LOF computes local density deviation of a certain point as compared to its neighbors. It is
different variant of k Nearest neighbors. Simply, in LOF outliers is considered to be points
which have lower density than its neighbors.

Local Outlier Factor in Code

9/11
import numpy as np
import [Link] as plt
from [Link] import LocalOutlierFactor

print(__doc__)

[Link](42)

# Generate train data


X_inliers = 0.3 * [Link](100, 2)
X_inliers = np.r_[X_inliers + 2, X_inliers - 2]

# Generate some outliers


X_outliers = [Link](low=-4, high=4, size=(20, 2))
X = np.r_[X_inliers, X_outliers]

n_outliers = len(X_outliers)
ground_truth = [Link](len(X), dtype=int)
ground_truth[-n_outliers:] = -1

# fit the model for outlier detection (default)


clf = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
# use fit_predict to compute the predicted labels of the training samples
# (when LOF is used for outlier detection, the estimator has no predict,
# decision_function and score_samples methods).
y_pred = clf.fit_predict(X)
n_errors = (y_pred != ground_truth).sum()
X_scores = clf.negative_outlier_factor_

[Link]("Local Outlier Factor (LOF)")


[Link](X[:, 0], X[:, 1], color='k', s=3., label='Data points')
# plot circles with radius proportional to the outlier scores
radius = (X_scores.max() - X_scores) / (X_scores.max() - X_scores.min())
[Link](X[:, 0], X[:, 1], s=1000 * radius, edgecolors='r',
facecolors='none', label='Outlier scores')
[Link]('tight')
[Link]((-5, 5))
[Link]((-5, 5))
[Link]("prediction errors: %d" % (n_errors))
legend = [Link](loc='upper left')
[Link][0]._sizes = [10]
[Link][1]._sizes = [20]
[Link]()

In Summary , we have discussed various quick methods through we can identify outliers.
There are other advanced machine learning models which can also be used to identify
outliers, however we will discuss them in a separate post.

10/11
Image and code source: Scikit Learn

In summary, we have discussed various ways eleven different ways for detecting outliers
using Python.

seven different ways to detect outliers by visualization, statistics


four different ways to detect outliers by machine learning model

11/11

Common questions

Powered by AI

Machine learning models like Isolation Forest, One-Class SVM, and LOF provide more flexible and adaptive frameworks for detecting outliers, as they do not rely on assumptions about the data's underlying distribution, unlike traditional methods such as Z-scores or Grubb’s test. These models can efficiently handle large, high-dimensional datasets with varying distributions, adapt to complex non-linear relationships and account for varying densities within the data, thus offering greater accuracy in outlier detection .

Grubb’s test is designed to detect a single univariate outlier by identifying the data point with the largest standard deviation from the mean. In contrast, the Tietjen-Moore test is used to detect multiple outliers at once but requires the number of outliers to be specified beforehand, which is not ideal when the number of outliers is unknown .

Visualization techniques like box plots are useful for identifying outliers in one-dimensional data by showcasing deviations beyond the interquartile range. However, in multi-dimensional datasets, these plots become less effective due to their inability to represent complex interdependencies and multivariate distributions. Despite this limitation, visualization can still play a role in initial data exploration and highlighting potential outliers for further verification with more sophisticated methods .

One-Class SVM is a machine learning algorithm used for unsupervised outlier detection, where it constructs a boundary around the data points considered as normal. It separates these from potential outliers by maximizing the boundary using a kernel function. In outlier detection, it predicts -1 for outliers and 1 for non-outliers based on how each data point falls in relation to the constructed boundary .

The Robust Covariance method, also known as the Elliptic Envelope technique, detects outliers by assuming that most data points follow a Gaussian distribution. The method increases covariance due to the presence of outliers, which results in a larger range of data and an increased determinant of the covariance matrix. The approach reduces these effects by iteratively computing a new estimate of the mean and covariance using the smallest Mahalanobis distances until convergence is reached, indicating stable mean and covariance estimations .

Challenges with using Z-scores in real-world datasets include the assumption of normal distribution, which may not hold true, especially in datasets with skewed distributions or outliers that pull the mean. Z-scores also become less effective with small sample sizes, where the standard deviation can be easily influenced by extreme values, thus affecting the threshold for determining outliers .

The Interquartile Range (IQR) is calculated as the difference between the third quartile (Q3) and the first quartile (Q1) of the dataset, representing the middle 50% of data. Data points lying outside of the range defined by Q1 - 1.5*IQR and Q3 + 1.5*IQR are considered outliers. IQR is preferred in skewed datasets as it does not assume any specific data distribution and is less affected by extreme values .

Local Outlier Factor (LOF) advantages in outlier detection include its ability to calculate the local density deviation of data points compared to their neighbors, which makes it effective in identifying outliers in datasets with varying densities. LOF can discern points that have significantly lower density compared to neighbors, thus identifying them as outliers. This local consideration allows it to perform well on complex datasets where global methods might fail .

Isolation Forest detects outliers by isolating observations through recursive partitioning. It randomly selects a feature and a split value to partition the data, forming a tree structure where outliers are expected to be isolated closer to the root due to fewer partitioning steps. This approach contrasts with traditional ensemble methods that often rely on aggregating predictions from different models, as it purely depends on the structural qualities of the data for outlier identification .

Sorting methods can be effectively used for initial outlier detection in scenarios with smaller datasets where visual inspection of data ordering can reveal apparent anomalies. By arranging data in ascending or descending order, extreme values will stand out, allowing for quick identification of outliers in the dataset. This method is simple and effective as a preliminary step before applying more complex analyses .

You might also like