Module 2 - Data Wrangling and summary statis-

tics
Instructional Hours: 4

Module Overview
In this module, you will learn about how to view data, clean the data by remov-
ing duplicates, not a number and white spaces. Further, rename columns, use
the pipe, the dplyr verbs for data manipulation, tidyr verbs, combining rows and
columns, and the summary statistics. The module provides videos, discussion
forum, exercises and self-assessment in the learning process.

Learning Outcomes
1. Show how to clean data by removing duplicates and handling missing
values.
2. Construct different datasets from a given dataset using dplyr and tidyr
verbs.
3. Evaluate the summary statistics from dataset variable.

4. Discuss the summary statistics findings from a dataset.

Learning Activities
1. Read the lecture notes (PDF)
2. Read the assigned reference materials
3. Watch lecture videos
4. Complete the module assessment/Quiz

Data cleaning
Data cleaning is an essential step in the data analysis pipeline, ensuring that the
dataset is accurate, consistent, and ready for analysis. In R, several packages
provide tools and functions for effective data cleaning.

1
Handling missing values
Handling missing values is a crucial step in data cleaning and preparation. Here
are some common methods and strategies to handle missing values in a dataset:
1. Identify Missing Values:

• Use functions like [Link](), sum([Link]()), anyNA(), and sapply() in R

to detect missing values.
• Get an overview of missing values using summary() to see how many
missing values are present in each column.

2. Remove Missing Values:

• Rows: Use [Link]() or drop na() from the tidyr package to remove
rows with missing values. This is useful when the amount of missing
data is small and randomly distributed.
• If a column has too many missing values, consider removing the entire
column using select() to avoid potential biases.
3. Impute Missing Values:
• Replace missing values with the mean, median, or mode of the re-
spective column using functions like mean(), median(), and custom
mode functions.
• Replace missing values based on conditional means, for example, the
mean of a subset of data defined by another variable.
• Predictive Imputation: Use regression models, k-nearest neighbors
(KNN), or machine learning algorithms to predict and fill in missing
values.
• Use fill() from tidyr to carry forward or backward the last observed
value.
4. Multiple Imputation:
• Use the mice package for multiple imputation, which generates sev-
eral imputed datasets and combines results to account for uncertainty
in the imputations.

Steps in data cleaning

1. Set up the environment by ensuring you have the necessary packages in-
stalled and loaded. The tidyverse package is a collection of R packages
designed for data science, which includes dplyr, tidyr, readr, and others
useful for data cleaning.

2
 [Link]("tidyverse")
[Link]("lubridate") # for date-time manipulation
library(tidyverse)
library(lubridate)

2. Import the data or read the data from the datasets packages in R base.
Type datasets:: in the console. The colon after the word datasets implies
that the is a package in R.

df <- [Link](age = c(25, 30, " ", 45, 37,40, 20),

weight =cc(85,75, 62, 70, 59, 67,80),
gender=c("Male", "Female", "Female",
"Male", "Female", "Male", "Female"))

# We can view the data

View(df)

#To view the first five or last five entries in the dataset, use
head(df)
tail(df)

# We confirm if the dataset is a dataframe

class(df)

3. To enable us manage data cleaning in a seamless manner, you introduce

the pipe % > %, where in R Studio the shortcut is Control + shift + m.
The pipe operator % > % in R is a powerful tool provided by the magrittr
package, which is part of the tidyverse. The primary purpose of the pipe
is to improve the readability and efficiency of code by enabling a clear,
concise, and logical flow of data transformations.
4. Check the data type for each column, confirm if duplicates exists, and any
missing values in the columns. Remove the duplicates and the missing
values

# Confirm if gender is factor (data type) and if age and

# weight are numeric (data types)
class(df$gender)
class(df$age)
class(df$weight)

# gender should be factor and age be numeric

df$gender <- [Link](df$gender)
df$age <- [Link](df$gender)

3
 # Check for duplicates in the dataset
sum(duplicated(df))

# Remove duplicate rows - one row is removed from the data

df_clean1 <- df %>% distinct()

# Check for missing values

sum([Link](df_clean1))

#Remove the missing values (not a number)

df_clean2 <- [Link](df_clean1)

Renaming columns
You can use the names() function to rename columns in a data frame. You can
change the column names from gender to Gender, age to Age years and weight
to Weight.
# Rename columns using names()
names(df_clean2) <- c("Gender", "Age_years", "Weight")
View(df_clean2)

Combining rows and columns

Combining rows and columns in R can be done using various functions depend-
ing on the specific needs and data structures.
1. To combine rows, you can use the rbind() function.

# Create sample data frames

df1 <- [Link](id = 1:3, name = c("Alice", "Bob", "Charlie"))
df2 <- [Link](id = 4:6, name = c("David", "Eve", "Frank"))

# Combine rows
combined_rows <- rbind(df1, df2)
print(combined_rows)

2. To combine columns (also known as merging or joining data frames), you

can use the cbind() function for a simple column bind or merge() for more
complex joins.

# Create sample data frames

df1 <- [Link](id = 1:3, name = c("Alice", "Bob", "Charlie"))
df2 <- [Link](age = c(25, 30, 35), city = c("New York", "Los Angeles", "Chicago"))

# Combine columns
combined_columns <- cbind(df1, df2)
print(combined_columns)

4
 3. Combining Rows with bind rows()

library(dplyr)

# Create sample data frames

df1 <- [Link](id = 1:3, name = c("Alice", "Bob", "Charlie"))
df2 <- [Link](id = 4:6, name = c("David", "Eve", "Frank"))

# Combine rows
combined_rows <- bind_rows(df1, df2)
print(combined_rows)

4. Combining Columns with left join(), right join(), inner join(), full join()

library(dplyr)

# Create sample data frames

df1 <- [Link](id = 1:3, name = c("Alice", "Bob", "Charlie"))
df2 <- [Link](id = 1:3, age = c(25, 30, 35), city = c("New York", "Los Angeles", "C

# Combine columns using a left join

combined_left_join <- left_join(df1, df2, by = "id")
print(combined_left_join)

# Combine columns using a full join

combined_full_join <- full_join(df1, df2, by = "id")
print(combined_full_join)

Practical data cleaning session

Create a code run for the students to practice the data cleaning process. The
student should be able to create a data frame with three columns (gender, age
and weight), remove the duplicates, remove na, view the data, use the pipe and
rename the columns as illustrated above.

Quiz
1. Which function is used to remove rows with missing values from a data
frame in R? (2 Marks)
(a) [Link]()
(b) [Link]() (Answer)
(c) [Link]()
(d) [Link]()
2. Which of the following functions is used to rename columns in a data
frame in R using the dplyr package? (2 Marks)

5
 (a) rename columns()
(b) rename vars()
(c) rename() (Answer)
(d) change names()
3. Which method is not commonly used for handling missing values in a
dataset? (2 Marks)
(a) Mean imputation
(b) Median imputation
(c) Deleting the entire dataset (Answer)
(d) Forward fill

Tidyr and dplyr verbs

The tidyverse is a collection of R packages designed for data science, including
ggplot2, dplyr, tidyr, readr, purrr, and others. Within tidyverse, verbs refer to
functions that are used for data manipulation and analysis. These verbs help
make data wrangling more intuitive and readable. There is a tendency to use
the dplyr verbs or tidyverse verbs interchangeably. Dplyr package is a subset of
the tidyverse package.

Dplyr verbs reference material

• Visit the website for the different dplyr verbs.

Dplyr Verbs website

Projects and datasets repository

Create a link in LMS where the students can access the datasets.
• UCL machine learning repository
Projects and Datasets Repository

Dplyr verbs
The dplyr package in R is a powerful and user-friendly tool for data manipula-
tion and transformation. It is part of the tidyverse, a collection of R packages
designed for data science. dplyr provides a set of functions, known as ”verbs,”
that help you to work with data in a clear and efficient manner.

• Data Manipulation: dplyr simplifies the process of manipulating data by

providing a consistent set of functions for filtering, selecting, arranging,
summarizing, and mutating data.

6
• Pipes (% > %): dplyr works seamlessly with the pipe operator % > %
from the magrittr package.

The iris dataset (part of R base datasets) is to be used for this exercise

1. select()is used to choose specific columns from a data frame.

library(dplyr)

# Create a new dataset with two columns, [Link] and [Link]

iris_select <- iris %>% select(c([Link], [Link]))

2. filter() - for rows based on specific conditions

# To select a section* of the data that has setosa as the species

iris_filter <- iris %>% filter(Species=="setosa")

3. mutate() - to create or transform columns

# Create an extra column which is the product of [Link] and [Link],

# and name it [Link]
iris_mutate <- iris %>% mutate([Link] = [Link] * [Link])

4. arrange() the rows in a specific order.

# Sort the column ([Link]) in descending order

iris_arrange <- iris %>% arrange(desc([Link]))

5. rename() - to rename columns

# Renamed the column by changing from [Link] to Petal_Length

#(replaced dot with underscore in the column name)
iris_rename <- iris %>% rename(Petal_Length = "[Link]")

6. summarize() or summarise() - create summary statistics

# to calculate the mean of numerical data for the [Link] and [Link]
iris_sum <- iris %>% summarise(mean_sepal.len = mean([Link]),
mean_sepal.wid = mean([Link]))

7. group by() - to group the data by one or more variables for summary
operations. This verb works well with other dplyr verbs

# Find the mean of the sepal and petal based on the species
iris_grp <- iris %>% group_by(Species)%>%
summarise(mean([Link]), mean([Link]),
mean([Link]), mean([Link]))

7
Tidyr verbs
The tidyr package is part of the tidyverse in R and is used to create tidy data,
a standard way of mapping the meaning of a dataset to its structure. Tidy data
principles make it easier to manipulate, model, and visualize datasets. Here’s
an overview of tidyr and some common functions it provides:
1. gather() (superseded by pivot longer()) - to convert data from wide to
long format.

# Combine [Link], [Link], [Link] and [Link] into two columns

iris_long <- iris %>%
pivot_longer(cols = c("[Link]", "[Link]",
"[Link]", "[Link]"),
names_to = "[Link]", values_to = "value")

2. spread() (superseded by pivot wider()) - to convert data from long to wide

format.

# From the long format to the wide format of the dataset

iris_wide <- iris_long %>%
pivot_wider(names_from = [Link], values_from = value)

3. unite() - unite multiple columns into one.

# combine two or more columns into one

iris_unite <- iris %>%
unite("Petal", [Link], [Link], sep = "_")

4. separate() - separate one column into multiple columns.

# Remove the unite function above to revert to the normal data format
iris_sep <- iris_unite %>%
separate(Petal, into = c("[Link]", "[Link]"), sep = "_")

Video on dplyr and tidyr verbs

Watch the following video and attempt the Quiz: Video Visit the URL below

to view a video:

[Link]

Video on dplyr and tidyr verbs

8
After Watching the Video, attempt the quiz
1. (Insert the question after minute 8:15). Given the iris dataset, select the
column ”[Link]” and ”Species” and slice the row number 40 to
number 100.

Provide a discussion forum for the students to discuss how they

were able to construct a new dataset from the iris dataset.

2. Which dplyr verbs will you use to select the columns where Species is the
versicolor and the [Link] that is 5.0 and above? (4 Marks)

a) iris %>% select(c(Species, [Link])) %>% filter([Link]>4.9) %>%

filter(Species=="versicolor") (Answer)
b) iris %>% select(c(Species, [Link], [Link])) %>%
filter([Link]>5.0) %>% filter(Species=="versicolor")
c) iris %>% select(c(Species, [Link])) %>% filter([Link]>5.0) %>%
filter(Species=="versicolor")

Summary Statistics
Summary statistics are numerical values that summarize and provide informa-
tion about the distribution, central tendency, and variability of a dataset. They
are essential for understanding the general characteristics of data, making com-
parisons between datasets, and informing further statistical analyses.
• Summary statistics are essential in data analysis as they provide a concise
overview of the data, highlighting key characteristics and patterns.
• Understanding and effectively using summary statistics are crucial for any
data-driven analysis, enabling analysts and researchers to draw meaningful
conclusions and make data-informed decisions.
Summary statistics can be broadly categorized into measures of central ten-
dency, measures of variability (dispersion), and measures of shape. The defini-
tion of these measures are:
• mean: The average value.
• sd: Standard deviation, a measure of the amount of variation or dispersion
of a set of values.
• median: The middle value when the data is ordered.
• trimmed: Mean after trimming a fraction of extreme values from both
ends.
• mad: Median absolute deviation, a robust measure of the variability of a
univariate sample.

9
 • min: Minimum value.
• max: Maximum value.
• range: The difference between the maximum and minimum values.
• skew: Skewness, a measure of the asymmetry of the probability distribu-
tion.
• kurtosis: Kurtosis, a measure of the ”tailedness” of the probability distri-
bution.
• se: Standard error, the standard deviation of the sample mean estimate.

The above measures can be estimated from numerical data using the sum-
mary() and describe() function
#Using the summary function
summary(iris$[Link])

# Using the describe function from the psych package

[Link]()
library(psych)

describe(iris$[Link])

# The describe() function output for the iris dataset:

describe(iris)

10
Practical summary statistics session

1. Create a code run for the students to practice how to construct

different datasets as subset of the main dataset using the dplyr
and tidyr verbs. Install the package ggplot2 which contains the
dataset diamonds.
2. Create a code run for the students to practice how to apply the
describe() function from the psych package using the iris dataset.

Attempt the Quiz

Using the diamonds dataset, (1)select the columns ”carat”, ”cut” and
”price”. (2) group by ”cut”, and (3) find the mean value (using the
summarise verb) of the ”carat” and ”price” as grouped by the ”cut”.
Select the correct answer for the mean of the price and carat respec-
tively (This is a multiple choice question and the answer is A) (6 Marks)
Choice Variable Fair Good Very good Premium Ideal
A Price 0.436 0.393 0.398 0.458 0.346
A Carat 1.05 0.849 0.806 0.892 0.703
B Price 0.436 0.393 0.898 0.458 0.346
B Carat 1.00 0.849 0.806 0.892 0.703
C Price 0.436 0.393 0.398 0.458 0.346
C Carat 1.25 0.849 0.806 0.892 0.703
D Price 0.436 1.393 0.398 0.458 0.346
D Carat 1.55 0.849 0.806 0.892 0.573

(A) CREATE A DISCUSSION FORUM

1. The students should discuss how the mean price differs based on
the diamond cut.
2. What is the meaning of cut and carat in diamonds? How do they
affect prices?

Reading Materials
1. Douglas, A., Roos, D., Mancino, F., Couto, A., and Lusseau, D. (2024).
An introduction to R. eBook (Pages 80 - 106) Read the selected pages

Summary
These concepts are fundamental for performing effective data analysis and en-
suring data quality using R.

11
1. Summary Statistics using R:
• Descriptive Statistics: Calculation of mean, median, mode, standard
deviation, variance, quartiles, and range to summarize data.
• Exploratory Data Analysis (EDA): Using functions like summary(),
describe(), and str() to get an overview of the dataset’s structure and
basic statistics.
2. Data Wrangling using R:
• Data Manipulation with dplyr: Using functions like filter(), select(),
mutate(), arrange(), and summarize() to manipulate and transform
data.
• Data Transformation with tidyr: Functions like gather(), spread(),
unite(), and separate() to reshape data for analysis. Combining
Datasets: Merging and joining datasets using functions like left join(),
right join(), inner join(), and full join().

3. Data Cleaning:
• Handling Missing Values: Identifying and managing missing data
using functions like [Link](), [Link](), and fill() from tidyr.
• Data Type Conversion: Ensuring correct data types using functions
like [Link](), [Link](), [Link](), and lubridate for date-
time manipulation.

12