Common steps involved in cleaning a
dataset in R
• Common Steps in Cleaning a Dataset in R
• Cleaning a dataset is an essential process to prepare data for analysis. Here’s a
summary of the typical steps involved, with relevant R functions:
• 1. Import the Data
Load the dataset into R using appropriate functions like [Link](), [Link](),
or read_excel() (requires the readxl package).
Example:
• data <- [Link]("[Link]")
• 2. Examine the Data
Check structure and summary:
o str(data) – Displays structure.
o summary(data) – Provides a statistical summary.
• str(data)
• summary(data)
• 3. Handle Missing Values
Identify missing values using [Link]().
Options:
o Remove rows/columns with [Link]() or drop_na() (from the tidyverse).
o Replace missing values with mean/median using mutate() or replace().
• # Remove rows with NA
• clean_data <- [Link](data)
• # Replace NA with column mean
• data$column <- ifelse([Link](data$column), mean(data$column, [Link] = TRUE),
data$column)
• 4. Correct Data Types
Convert columns to the appropriate data type using [Link](), [Link](),
or [Link]().
• data$category <- [Link](data$category)
• 5. Remove Duplicates
Identify and remove duplicate rows with duplicated().
• clean_data <- data[!duplicated(data), ]
• 6. Fix Inconsistent or Erroneous Data
Detect and fix typos, outliers, or inconsistencies.
• data$category <- tolower(data$category) # Standardize text
• 7. Rename Columns
Rename columns for clarity using colnames() or the rename() function from the
dplyr package.
• colnames(data)[1] <- "NewName"
• 8. Filter or Subset Data
Select relevant rows/columns using filter() (from dplyr) or indexing.
• filtered_data <- data[data$column > 10, ] # Base R
• filtered_data <- filter(data, column > 10) # dplyr
• 9. Transform or Add Columns
Create new variables or transform existing ones using mutate().
• data <- mutate(data, new_column = old_column * 2)
• 10. Normalize/Scale Data
Standardize data using scaling or normalization (scale()).
• data$scaled_column <- scale(data$column)
• 11. Save the Cleaned Dataset
Export the cleaned dataset using [Link]() or other appropriate functions.
• [Link](clean_data, "cleaned_data.csv", [Link] = FALSE)
• # Create a sample dataset data <- [Link]( ID = 1:8, Name =
c("John Smith", "Jane Doe", "Mary Johnson", "James Brown", NA,
"Michael Davis", "Sarah Wilson", "Chris White"), Age = c(29, NA,
35, 42, 28, 33, 29, 37), Department = c("Sales", "HR", "IT", "Sales",
"Marketing", NA, "HR", "Sales"), Salary = c(50000, 60000, 75000,
55000, 47000, 63000, 58000, 51000), Join_Date = c("2015-06-15",
"2018-08-20", "2010-03-10", "2013-11-01", "2019-02-25", NA,
"2016-07-12", "2014-05-05"), Experience_Years = c(6, 3, 11, NA, 2,
5, 5, 7) )
• print(data)
• 1. Identify Missing Values
• Check for missing values in the dataset.
• # Check for missing values
• colSums([Link](data))
• Observation:
Name has 1 missing value.
Age has 1 missing value.
Department has 1 missing value.
Join_Date has 1 missing value.
• Experience_Years has 1 missing value.
• 2. Handle Missing Values
• (a) Replace Missing Name Replace missing Name with a placeholder like
"Unknown":
• data$Name[[Link](data$Name)] <- "Unknown"
• (b) Handle Missing Age Replace missing Age with the mean (or median) of the
column:
• data$Age[[Link](data$Age)] <- mean(data$Age, [Link] = TRUE)
• (c) Handle Missing Department Replace missing Department with "Unknown"
or the most frequent value:
• data$Department[[Link](data$Department)] <- "Unknown"
• (d) Handle Missing Join_Date If Join_Date is missing, replace it with a
placeholder like "0000-00-00":
• data$Join_Date[[Link](data$Join_Date)] <- "0000-00-00"
• (e) Handle Missing Experience_Years Replace missing Experience_Years with
the median of the column:
• data$Experience_Years[[Link](data$Experience_Years)] <-
median(data$Experience_Years, [Link] = TRUE)
• 3. Convert Data Types
• Ensure columns have the correct data types:
Join_Date should be a date type.
Department should be a factor.
• # Convert Join_Date to Date type
• data$Join_Date <- [Link](data$Join_Date, format = "%Y-%m-%d")
• # Convert Department to a factor
• data$Department <- [Link](data$Department)
• 4. Remove Duplicates
• Check for duplicate rows and remove them if any.
• # Remove duplicate rows
• data <- data[!duplicated(data), ]
• 5. Standardize Data
• For consistency, ensure text values (like Department) are standardized. For
example, if "HR" appears as "hr", convert to uppercase.
• # Standardize Department values
• data$Department <- toupper(data$Department)
• 6. Filter or Validate Rows
• Check for logical errors in the data. For example:
If Experience_Years exceeds Age, it’s invalid.
If Join_Date is after the current date, it’s invalid.
• # Filter out rows with invalid Experience_Years
• data <- data[data$Experience_Years <= data$Age, ]
• # Filter out invalid Join_Date
• data <- data[!data$Join_Date > [Link](), ]
• 7. Export Cleaned Data
• Save the cleaned dataset to a new file.
• # Save cleaned data to a CSV file
• [Link](data, "cleaned_employee_data.csv", [Link] = FALSE)
• getwd()…..to find out where is it saved
• setwd("C:/Users/YourFolder/Path")
• [Link](data, "employee_data.csv", [Link] = FALSE)
• Summary of Cleaning Steps
1. Identify Missing Values and handle them using mean, median, mode, or
placeholders.
2. Ensure Proper Data Types (e.g., dates and factors).
3. Remove Duplicates to avoid redundancy.
4. Standardize Data for uniformity.
5. Validate Data to eliminate logical errors.
6. Export Cleaned Data for further analysis.