DATA
ANALYTICS
Module No: 2
Data Pre-processing and Cleaning
DATA QUALITY ISSUES
Student Database
NAME AGE MARKS
Rahul 20 87
Ram NA 75
Anita 19 NA
Student Database
NAME AGE MARKS
Rahul 20 87
Ram 19 75
Anita 20 87
Student Database
NAME AGE MARKS
Rahul -15 87
Ram 19 120
Student Database
NAME DOB MARKS
Rahul 12/18/2002 87
Ram 23/1/2002 75
Temperature
TIME TEMP
10 AM 30 degree
11 AM 120 degree
DATA QUALITY ISSUES
DATA QUALITY ISSUES
IDENTIFY ISSUES
At first, we need to examine the data to find the issues
Missing data
Duplicate records
Incorrect or invalid values
Inconsistent formats
Outliers
DATA QUALITY ISSUES
HANDLE MISSING DATA
Missing values are handled by:
Removing incomplete records
Replacing with mean/median/mode
Using previous or predicted values
DATA QUALITY ISSUES
REMOVE DUPLICATES
Duplicate records are deleted to avoid repetition.
FIX INCORRECT OR INVALID DATA
Invalid values are corrected or removed.
HANDLE INCONSISTENT DATA
Convert data into a standard format.
DATA QUALITY ISSUES
DETECT AND TREAT OUTLIERS
Outliers are identified and:
Removed
Replaced
Capped
VALIDATE CLEANED DATA
Re-check data after cleaning to ensure correctness.
INTRODUCTION TO
R
PROGRAMMING
WHAT IS R?
R is a programming language and software environment used mainly for:
Data analysis
Statistical computing
Data visualization
It is widely used in data analytics
WHY R IS USED IN DATA ANALYTICS?
Excellent data visualization capabilities
Large collection of packages
Free and open-source
FEATURES OF R
R is an interpreted computer programming language.
R is a comprehensive programming language that provides support for procedural
programming involving functions as well as object-oriented programming with
generic functions.
R programming can integrate with other programming languages like C, C++, JAVA,
Python, Java.
R is an open-source programming language. It is completely free for anybody to use.
FEATURES OF R
R’s graphical capabilities are powerful. It can produce publication-quality graphs and
plots of any kind with its base package. With added packages like ggplot2 and plotly
the possiblities are endless.
No need for a compiler. The R language is interpreted. It does not need a compiler to
convert the code into a program.
R is cross-platform supportive that is it can run on any OS and in any software
environment without any hassle.
FEATURES OF R
With R, we can perform statistical analysis, data analysis as well as machine learning.
The big data world is also accessible to R. We can connect R with different databases
like Spark and Hadoop.
For machine learning, the best algorithms can be implemented with R. Keras and
TensorFlow allow us to create high end machine learning techniques. R can use
TensorFlow through special R packages
R PROGRAMMING
BASIC ARITHMETIC OPERATIONS
R PROGRAMMING
BASIC ARITHMETIC OPERATIONS
5+3
10 - 4
6*2
8/4
2^3
R PROGRAMMING
VARIABLE ASSIGNMENT
R PROGRAMMING
VARIABLE ASSIGNMENT
a <- 10
b <- 20
sum <- a + b
sum
R PROGRAMMING
DATA TYPES CHECKING
R PROGRAMMING
DATA TYPES CHECKING
x <- 5 class(x)
y <- 3.5 class(y)
z <- "R Programming" class(z)
flag <- TRUE class(flag)
R PROGRAMMING
VECTORS
R PROGRAMMING
VECTORS
v <- c(10, 20, 30, 40) length(v)
v sum(v)
mean(v)
median(v)
R PROGRAMMING
HANDLING NA VALUES
R PROGRAMMING
HANDLING NA VALUES
v <- c(10, 20, NA, 40) mean(v, [Link] = TRUE)
median(v, [Link] = TRUE)
R PROGRAMMING
SEQUENCE AND REPETITION
R PROGRAMMING
SEQUENCE AND REPETITION
seq(1, 10) rep(5, 4)
seq(1, 10, by = 2) rep(c(1, 2), times = 3)
R PROGRAMMING
LOGICAL OPERATIONS
R PROGRAMMING
LOGICAL OPERATIONS
x <- 10
--------------------
x>5
x<5
x == 10
x != 10
R PROGRAMMING
CONDITIONAL STATEMENTS
R PROGRAMMING
CONDITIONAL STATEMENTS
x <- 15
if (x > 10) {
print("Greater than 10")
} else {
print("Less than or equal to 10")
}
R PROGRAMMING
Loops: for Loop
R PROGRAMMING
Loops: for Loop
for (i in 1:5) {
print(i)
}
R PROGRAMMING
Loops: while Loop
R PROGRAMMING
Loops: while Loop
i <- 1
while (i <= 5) {
print(i)
i <- i + 1
}
R PROGRAMMING
Functions
FUNCTION DECLARATION
DEFAULT ARGUMENTS
RETURN VALUE
R PROGRAMMING
Functions
add <- function(a, b) {
return(a + b)
}
add(5, 3)
R PROGRAMMING
Data Frames
DATA FRAME DECLARATION
FINDING SUM/MEAN/MEDIAN
ACCESSING ROW DATAS/ COLUMN DATAS
SORTING INDIVIDUAL DATA
SORTING THE COMPLETE TABLE
R PROGRAMMING
Data Frames
FILTERING THE ROW VALUES
UPDATING PARTICULAR VALUE
REMOVING COLUMN
REMOVING ROW
R PROGRAMMING
Data Frames
students <- [Link](
name = c("Amit", "Rita", "Suman"),
marks = c(70, 85, 90)
)
students
mean(students$marks)
R PROGRAMMING
Sorting Data
R PROGRAMMING
Sorting Data
x <- c(4, 2, 8, 1)
sort(x)
sort(x, decreasing = TRUE)
R PROGRAMMING
Reading User Input
R PROGRAMMING
Reading User Input
name <- readline("Enter your name: ")
print(paste("Hello", name))
R PROGRAMMING
Basic Plot
R PROGRAMMING
Basic Plot
x <- c(1, 2, 3, 4)
y <- c(10, 20, 25, 30)
R PROGRAMMING
ACCESSING EXCEL FILE
[Link]("readxl")
library(readxl)