Data Preprocessing using R

Contents
• Data Preprocessing
• Forms of Data Preprocessing
• Data Cleaning
• Data Integration
• Data Transformation
• Data Reduction
 Data Preprocessing
• Data preprocessing is a crucial step in the data analysis and
machine learning pipeline.
• It involves cleaning and organizing raw data into a format that
is suitable for further analysis or for training machine learning
models.
• Enhance the quality of the data and make it more conducive
to effective analysis or modeling
Forms of Data Preprocessing
 Data Cleaning
• Data collected from various sources may be
incomplete, inconsistent, noisy or incorrect. In
order to remove all such anomalies from data
set we apply data cleaning techniques.
• Basically deals with:
1. Missing data.
2. Noisy data.
3. Inconsistent data.
 Missing Data
• In R missing values are represented by symbol NA.
• To identify missing values use [Link]() which returns a logical
vector with TRUE in the element locations that contain
missing values represented by NA.
• [Link]() will work on vectors, lists, matrices, and data frames.
• Basically we use two strategies to fill missing data:
1) By mean value 2) and By Ignoring the tuple
• We will learn how to:
1. Test for missing values
2. Recode missing values
3. Exclude missing values
Test for Missing values
Location and count of missing data
 Recode Missing values
• To recode missing values; or recode specific
indicators that represent missing values, we can use
normal subsetting and assignment operations.
• For example, we can recode missing values in
vector x with the mean values in x by first subsetting the
vector to identify NAs and then assign these elements a
value.
• if missing values are represented by another value
(i.e. 99) we can simply subset the data for the
elements that contain that value and then assign a
desired value to those elements.
Recode missing values with mean value
Recode missing values in a dataframe
[Link]
 Exclude Missing values
• We can exclude missing values in a couple different ways.
First, if we want to exclude missing values from mathematical
operations use the [Link] = TRUE argument. If you do not
exclude these values most functions will return an NA.
• A vector with missing values
• x <- c(1:4, NA, 6:7, NA)
• # including NA values will produce an NA output mean(x)
• [1] NA
• # excluding NA values will calculate the mathematical
operation for all non-missing values
• mean(x, [Link] = TRUE)
• [1] 3.833333
 [Link]()
• The [Link] function in R is used to identify complete cases in a data
structure such as a data frame, matrix, or vector.

• The function returns a logical vector with the same length as the input data
structure, where each element is TRUE if the corresponding row (for a data frame
or matrix) or element (for a vector) is complete (i.e., has no missing values), and
FALSE otherwise.
df <- [Link](x = c(1, 2, NA, 4), y = c(NA, 6, 7, 8))
[Link](df)

[1] FALSE TRUE FALSE TRUE

We can use the output of [Link] to select only the complete cases from a data set, and store them in a
new data set.
df <- [Link](x = c(1, 2, NA, 4), y = c(NA, 6, 7, 8))
df_complete <- df[[Link](df), ]

xy
226
448
 [Link]()

[Link] is a function in R that can be used to remove all incomplete cases (i.e., cases with missing
values) from a data frame
 Exercise
• Load any csv file that contains missing data.
Identify the tuples with missing values and
perform the following tasks:
1. Replace all missing data with mean values.
2. Create new dataset ignoring all missing
tuples.
Solution (mean values to replace all NA)
Ignore the tuple with NA values
 Noisy Data
• Noise is a random error or variance in the given dataset.
• Given below is the techniques used for removing noise in
the dataset.
– Binning
• Sort the attribute values and partition them into bins
• smooth by bin means, bin median, or bin boundaries.
– Clustering: group values in clusters and then detect and
remove outliers (automatic or manual)
– Regression: smooth by fitting the data into regression
functions.
• Correct inconsistent data: use domain knowledge or
expert decision.
 Binning
• Binning method smooth a sorted data by
consulting its neighborhood that is the values
around it. The Sorted value are distributed
into number of bucket/bins.
1. Smoothing by bin means
2. Smoothing by bin median
3. Smoothing by bin boundary
 Binning Function in R
• Binning in R refers to the process of grouping a set of
continuous or numerical data points into discrete intervals, or
"bins."

# Create a vector of numerical data

data <- c(10, 15, 25, 30, 40, 50, 60, 70, 80, 90)

# Define the breakpoints for the bins

breakpoints <- c(0, 20, 40, 60, 80, 100)

# Use the cut function to bin the data

binned_data <- cut(data, breaks = breakpoints, labels = c("Bin1", "Bin2", "Bin3", "Bin4",
"Bin5"))

# Print the binned data

print(binned_data)
Binning Method
 Smoothing by Bin Means
• Let’s say bin size=3 (provided in the question)
• Each value of a bin is replaced by the mean
value of each bin
• For Eg:
• Mean of values 4,8, and 15 is 9
• Therefore, original value of bin is replaced by
9
 Smoothing by Bin Median/Boundary
• Instead of mean value, each bin value is
replaced by median value in Smoothing by bin
median
• In smoothing by bin boundary, minimum &
maximum values in a given bin are identified
as bin boundaries
• Each bin is then replaced by closest boundary
value
 Question?
• Consider the sorted data for sales in price:
{4,8,15,21,21,24,25,28,34}
Apply following methods of binning to remove
noisy data
1) Bin Means
2) Bin Median
3) Bin Boundary
 Solution
• Partition into equi size bins (bin size=3)
• Bin 1: 4,8,15
• Bin 2: 21,21,24
• Bin 3: 25,28,34
 Smoothing by Bin Means
• Bin 1: 9,9,9
• Bin 2: 22,22,22
• Bin 3: 29,29,29
 Smoothing by Bin Boundary
• Bin 1: 4,4,15
• Bin 2: 21,21,24
• Bin 3: 25,25,34
Methods of dividing data into bins
• There are 2 methods of dividing data into bins
• Equal Frequency Binning : bins have equal
frequency.
• Equal Width Binning : bins have equal width
where range is(min+w,min+2w,…) width =
(max – min) / (no of bins).
Equal Frequency
 Equal Width
• In Equal width, we divide the data in equal widths.
In order to calculate width we have the formula.
• width=(max−min)/No. of bins
• So from the available data in the problem we have
(width = 70) (215-5)/3
• We divide the data with three categories:
• 5 to 75
• 75 to 145
• 145 to 215.
Contd..
 Question
• Data={0,4,12,16,16,18,24,26,28}
• Partition the data into 3 bins using Equal width
and Equal Partitioning method?
 Solution
• Equal Frequency Partitioning:
• Bin I: 0,4,12
• Bin II:16,16,18
• Bin III: 24,26,28
 Contd..
• Equal Width Partitioning:
• Width=28-0/3= 9(Approximately)
• 0+9=9 goes to bin 1, 9 to 18 goes to bin 2, 18
to 27 goes to bin 3
• {0,4,12,16,16,18,24,26,28}
• Bin I: 0,4
• Bin II:12,16,16,18
• Bin III: 24,26,28
Example
 Data Integration
• Data Integration involves combining data from
two or more sources. Theses sources may
include multiple databases, data cubes or flat
files. Problem associated with integration of
data includes:
1. Entity Identification
2. Redundancy
 Redundancy
• Redundant attributes is detected by
correlation and covariance analysis.
• For nominal data we use chi-square test.
• For numeric attribute we can use correlation
and covariance.
 Chi-square Test
• The chi-square test is a statistical test used to determine whether two categorical variables are
independent or not.
• It compares the observed frequencies of each category with the expected frequencies under the
assumption of independence
• If there is a significant difference between the observed and expected frequencies, then the two
variables are not independent.

For example, let's say we want to investigate whether there is a relationship between gender and smoking
habits. We can collect data from a sample of 200 people and ask them whether they smoke or not, and what
their gender is. The data might look like this:

We can use a chi-square test to determine whether there is a relationship between gender and smoking habits. Our null
hypothesis is that the two variables are independent, and our alternative hypothesis is that they are not independent.
 Chi-square Test
1. To calculate the expected frequencies, we need to first calculate the row and column totals:

2. The expected frequency for each cell is calculated using the formula:
expected frequency = (row total * column total) / grand total

For the first cell (Male/Smoker), the expected frequency is:

(120 * 80) / 200 = 48
We can calculate the expected frequencies for all the cells:
 Chi-square Test
3. Now we can use the formula for the chi-square statistic
chi-square = Σ ( (observed frequency - expected frequency)² / expected frequency )

We can calculate the chi-square statistic as follows:

((50-48)²/48) + ((30-32)²/32) + ((70-72)²/72) + ((50-48)²/48) = 2.88

The degrees of freedom for the chi-square test is (number of rows - 1) * (number of columns - 1)
= (2-1) * (2-1) = 1

4. We can use the chi-square distribution table or a statistical software to calculate the p-value associated with
the calculated chi-square statistic and the degrees of freedom.

In this case, the p-value is 0.08938, which is greater than the typical threshold for statistical significance
(0.05). Therefore, we fail to reject the null hypothesis and conclude that there is insufficient evidence to
suggest that there is a relationship between gender and smoking habits.
 Chi-square Test: Applications

1. Medical research: To investigate the relationship between a disease and a certain risk
factor such as smoking, alcohol consumption, or diet.
2. Market research: To determine whether there is a significant association between
product sales and demographic factors such as age, gender, or income.
3. Political polling: To analyze the relationship between voting preferences and
demographic factors such as age, gender, or political affiliation.
4. Quality control: To evaluate whether there is a significant difference between the
observed and expected frequencies of defects in a production process.
5. Education: To determine whether there is a significant relationship between the level
of education and job opportunities, income, or other factors.
 Chi-square Test: Using R
• First, we can create a table of the observed frequencies:

observed <- matrix(c(50, 30, 70, 50), nrow=2, byrow=TRUE)

colnames(observed) <- c("Male", "Female")
rownames(observed) <- c("Smoker", "Non")

• Then, we can calculate the expected frequencies using the [Link]() function:

expected <- [Link](table(row(observed), col(observed))) * sum(observed)

• we can calculate the chi-square statistic using the [Link]() function:

result <- [Link](observed, p=expected, rescale.p=TRUE)

 Example
Let's consider the following example to illustrate how to perform a chi-square goodness-of-fit test:

Suppose we have a survey of 500 customers at a coffee shop, and we want to test whether the distribution of
their coffee preferences (black, with milk, with sugar, or with both milk and sugar) is the same as the
distribution of coffee sales at the shop over the past year.

Step 1: State the null and alternative hypotheses.

● Null hypothesis: The distribution of coffee preferences among customers is the same as the
distribution of coffee sales at the shop over the past year.
● Alternative hypothesis: The distribution of coffee preferences among customers is different from the
distribution of coffee sales at the shop over the past year.

Step 2: Set the level of significance.

● Let's set the level of significance to 0.05.

Step 3: Collect and summarize the data.

● We have the following data on coffee sales over the past year: 40% black, 30% with milk, 20% with sugar,
and 10% with both milk and sugar.
● We also surveyed 500 customers at the coffee shop, and the results are as follows:
Step 4: Create a table of expected frequencies.
● We can calculate the expected frequency for each category by multiplying the proportion of coffee sales
by the sample size.
● The expected frequencies are as follows:
Step 5: Calculate the chi-square test statistic.
● We can calculate the chi-square test statistic using the formula:
χ² = ∑ (Observed Frequency - Expected Frequency)² / Expected Frequency
● The calculations are as follows:
χ² = [(200 - 200)² / 200] + [(150 - 150)² / 150] + [(100 - 100)² / 100] + [(50 - 50)² / 50]
=0+0+0+0
=0

Step 6: Find the critical value and p-value.

● The degrees of freedom for this test are (number of categories - 1) = 3.
● Using a significance level of 0.05 and 3 degrees of freedom, the critical value from the chi-
square distribution table is 7.815.
● The p-value for the test is the area to the right of the calculated chi-square test statistic
under the chi-square distribution with 3 degrees of freedom.
● Since our calculated chi-square value of 0 is less than the critical value of 7.815, we can
accept the null hypothesis and conclude that the distribution of coffee preferences among
customers is not significantly different from the distribution of coffee sales at the shop over
the past year.
Step 1: State the null and alternative hypotheses.
● Null hypothesis: The roulette wheel is unbiased and equally likely to land on any of the 38 numbers.
● Alternative hypothesis: The roulette wheel is not unbiased and does not land equally on all 38 numbers.
Step 2: Set the significance level and the degrees of freedom.
● Significance level: Let's use a significance level of 0.05.
● Degrees of freedom: The degrees of freedom for a chi-square goodness-of-fit test with k categories is k -
1, where k is the number of categories.
In this case, we have k = 38 categories, so the degrees of freedom are 38 - 1 = 37.

Step 3: Calculate the expected frequencies under the null hypothesis.

● Under the null hypothesis, the expected frequency for each category is equal to the total number
of observations (100) divided by the number of categories (38).
Expected frequency = Total number of observations / Number of categories
Expected frequency = 100 / 38
Expected frequency = 2.63158 (rounded to 2 decimal places)
Implementation in R
 Correlation Coefficient
• Pearson correlation (r), which measures a
linear dependence between two variables (x
and y). It’s also known as a parametric
correlation test because it depends to the
distribution of the data. It can be used only
when x and y are from normal distribution.
The plot of y = f(x) is named the linear
regression curve.
 Correlation in R
• Correlation coefficient can be computed using the
functions cor() or [Link]():
• cor() computes the correlation coefficient
• [Link]() test for association/correlation between paired
samples. It returns both the correlation coefficient and
the significance level(or p-value) of the correlation .
• The simplified formats are:
• cor(x, y, method = c("pearson", "kendall", "spearman"))
• [Link](x, y, method=c("pearson", "kendall",
"spearman"))
 Steps
• Import your data into R
• Prepare your data as specified here:
• Save your data in an external .txt tab or .csv
files
• Import your data into R as follow:
• # If .txt tab file, use this:
• my_data <- [Link]([Link]()) #
• Or, if .csv file, use this:
• my_data <- [Link]([Link]())
Example
 Access to the values returned by [Link]() function

• The function [Link]() returns a list containing

the following components:
• [Link]: the p-value of the test
• estimate: the correlation coefficient
• # Extract the [Link] res$[Link]
• [1] 1.293959e-10
• # Extract the correlation coefficient
res$estimate
• cor -0.8676594
Covariance coefficient
Example
 Data Transformation
• Package Caret() is available in R for data transformation and given below is a
quick summary of all of the transform methods supported in
the method argument of the preProcess() function in caret.
• “BoxCox“: apply a Box–Cox transform, values must be non-zero and positive.
• “YeoJohnson“: apply a Yeo-Johnson transform, like a BoxCox, but values can
be negative.
• “expoTrans“: apply a power transform like BoxCox and YeoJohnson.
• “zv“: remove attributes with a zero variance (all the same value).
• “nzv“: remove attributes with a near zero variance (close to the same value).
• “center“: subtract mean from values.
• “scale“: divide values by standard deviation.
• “range“: normalize values.
• “pca“: transform data to the principal components.
• “ica“: transform data to the independent components.
• “spatialSign“: project data onto a unit circle.
 Transformation
• Box-Cox transformation is a statistical
technique that transforms your target
variable so that your data closely resembles a
normal distribution.
• The Yeo-Johnson Transformation inflates low
variance data and deflates high variance data
to create a more uniform dataset.
Scale():The scale transform calculates the standard deviation for an
attribute and divides each value by that standard deviation.
Center():The center transform calculates the mean for an attribute and subtracts it from each value .
 Standardize: Combining the scale and center transforms will standardize
your data. Attributes will have a mean value of 0 and a standard deviation
of 1.
Normalize: Data values can be scaled into the range of [0, 1] which is called normalization.
Box-Cox Transform: When an attribute has a Gaussian-like distribution but is shifted, this is called a skew. The distribution of an attribute
can be shifted to reduce the skew and make it more Gaussian. The BoxCox transform can perform this operation (assumes all values are
positive).
Yeo-Johnson Transform: Another power-transform like the Box-Cox transform, but it supports raw
values that are equal to zero and negative.
PCA: Transform the data to the principal components. The transform keeps
components above the variance threshold (default=0.95) or the number of
components can be specified (pcaComp). The result is attributes that are
uncorrelated, useful for algorithms like linear and generalized linear
regression.
Independent Component Analysis: Transform the data to the independent
components. Unlike PCA, ICA retains those components that are
independent. You must specify the number of desired independent
components with [Link] argument. Useful for algorithms such as naive
bayes.
Output
 Data Reduction

• Data Cube Aggregation

• Principal Component Analysis
• Regression
• Histogram
 Principal Component Analysis (PCA)
• PCA is a way of identifying patterns in data, and
expressing the data in such a way as to highlight
their similarities and differences, since patterns
in data can be hard to find in data of high
dimension, where the luxury of graphical
representation is not available, PCA is a powerful
tool for analyzing data.
• Other advantage of PCA is that once you have
found these patterns in data, you can compress
the data, by reducing the number of dimensions.
 PCA (Method)
• Step I: Get Data
• Step II: Subtract the mean: For PCA to work
properly you have to subtract the mean from
each of the data dimensions. The mean
subtracted is the average across each
dimension. So, all x values have x’(mean of x
values) subtracted, and all y values have
y’(mean of y values). This produces a data set
whose mean is zero.
 EX1:
Data Data Adjust
x y X=x-x’ Y=y-y’
2.5 2.4 .69 .49
0.5 0.7 -1.31 -1.21
2.2 2.9 .39 .99
1.9 2.2 .09 .29
3.1 3.0 1.29 1.09
2.3 2.7 .49 .79
2 1.6 .19 -.31
1 1.1 -.81 -.81
1.5 1.6 -.31 -.31
1.1 0.9 -.71 -1.01
 PCA contd..

• Step III: Calculate the covariance matrix

Covariance is always measured between 2
dimensions if you have 3-dimensional data
set(x,y,z) we could measure the covariance
between (x,y),(x,z) and (y,z). Covariance
formula is given below:
Ex2.
 Ex:
• Covariance matrix obtained for two dimensional data
for ex 1 is:
Cov=.616555556 .615444444
.615444444 .716555556

• Covariance matrix for 3 dimensions is

(cov(a,b)=cov(b,a)):
 Contd..
• Step IV: Calculate Eigen vectors and Eigen
values of covariance Matrix:
• If A is a square matrix, a non-zero vector v is

(eigenvalue) such that 𝐴𝑣 = λ𝑣

an eigenvector of A if there is a scalar λ

• Example:
Eigenvectors
Eigenvalues and Eigenvectors of covariance
matrix
 Contd.
• Step V: Choosing components and forming a feature
vector:
• Here is where the notion of data compression and reduced
dimensionality comes into it.
• Eigenvector with the highest eigen value is the principle
component of the data set.
• In general, once eigenvectors are found from the
covariance matrix, the next step is to order them by
eigenvalue, highest to lowest. This gives you the
components in order of significance. Now, if you like, you
can decide to ignore the components of lesser significance.
Feature Vector
 Contd.
• Step VI: Deriving the new dataset:
• This the final step in PCA, and is also the easiest. Once we
have chosen the components (eigenvectors) that we wish
to keep in our data and formed a feature vector, we simply
take the transpose of the vector and multiply it on the left
of the original data set, transposed.
• Final Data= Row_Feature_Vector X Row_Data_Adjust
where is the matrix with the eigenvectors in the columns
transposed so that the eigenvectors are now in the rows,
with the most significant eigenvector at the top, and is the
mean-adjusted data transposed, ie. the data items are in
each column, with each row holding a separate dimension.
 Complete PCA Steps
• Steps for principal component analysis
• The procedure includes 5 simple steps :
• Prepare the data :
• Center the data : subtract the mean from each variables. This produces a data set whose mean is zero.
• Scale the data : If the variances of the variables in your data are significantly different, it’s a good idea to
scale the data to unit variance. This is achieved by dividing each variables by its standard deviation.
• Calculate the covariance/correlation matrix
• Calculate the eigenvectors and the eigenvalues of the covariance matrix
• Choose principal components : eigenvectors are ordered by eigenvalues from the highest to the lowest.
The number of chosen eigenvectors will be the number of dimensions of the new data set. eigenvectors
= (eig_1, eig_2,…, eig_n)
• compute the new dataset :
• transpose eigeinvectors : rows are eigenvectors
• transpose the adjusted data (rows are variables and columns are individuals)
• [Link] = [Link] X [Link].
There are several functions from different packages for performing PCA :
• The functions prcomp() and princomp() from the built-in R stats package.
• PCA() from FactoMineR package
• [Link]() from ade4 package
Implementation in R
Contd.
Contd.
Contd.