BIG DATA ANALYTICS

COURSE CODE:
CREDITS: 3

UNIT-1

[Link]
Assistant Professor
Department of CSE

Shiv Nadar University Chennai

 Data Analytics using R Programming

• The R programming language is named after the first letters of the names of its
inventors, Ross Ihaka and Robert Gentleman.

Data Analytics
What is R Programming? & Why R Programming?
R Installation
After R Installation
Variables in R
Variables in R
 DATA ANALYSIS USING R

• Data Analysis is a subset of data analytics, it is a process where the objective has to be made clear, collect the relevant
data, preprocess the data, perform analysis(understand the data, explore insights), and then visualize it.

The process of data analysis would include all these steps for the given problem statement.
Example- Analyze the products that are being rapidly sold out and details of frequent customers of a retail shop.

• Defining the problem statement – Understand the goal, and what is needed to be done. In this case, our problem statement is
– “The product is mostly sold out and list of customers who often visit the store.”
• Collection of data – Not all the company’s data is necessary, understand the relevant data according to the problem. Here
the required columns are product ID, customer ID, and date visited.
• Preprocessing – Cleaning the data is mandatory to put it in a structured format before performing analysis.
 DATA ANALYSIS USING R

• Removing outliers( noisy data).

• Removing null or irrelevant values in the columns. (Change null values to mean value of that column.)
• If there is any missing data, either ignore the tuple or fill it with a mean value of the column.

Data Analysis using the Titanic dataset

• You can download the titanic dataset (it
contains data from real passengers of the
titanic)from [Link]
15db6BpWU3NBi8LK0paPA7USQicF9UR
B2/view
• Save the dataset in the current working
directory, now we will start analysis
(getting to know our data).

titanic=[Link]("[Link]")
head(titanic)

Our dataset contains all the columns like name, age, gender of the passenger and class they have traveled in, whether they
have survived or not, etc. To understand the class(data type) of each column sapply() method can be used.
 DATA ANALYSIS USING R

sapply(train, class)

We can categorize the value “survived” into “dead” to 0 and “alive” to 1 using factor() function.

train$Survived=[Link](train$Survived)
train$Sex=[Link](train$Sex)
sapply(train, class)

We analyze data using a summary of all the columns, their values, and data types. summary() can be used for this purpose.
 DATA ANALYSIS USING R

summary(train)

From the above summary we can extract below observations:

•Total passengers: 891
•The number of total people who survived: 342
•Number of total people dead: 549
•Number of males in the titanic: 577
•Number of females in the titanic: 314
•Maximum age among all people in titanic: 80
•Median age: 28
 DATA ANALYSIS USING R

Preprocessing of the data is important before analysis, so null values have to be checked and removed.
sum([Link](train))

dropnull_train=train[rowSums([Link](train))<=0,]

• dropnull_train contains only 631 rows because (total rows in dataset (808) – null value rows (177) =
remaining rows (631) )
• Now we will divide survived and dead people into a separate list from 631 rows.
survivedlist=dropnull_train[dropnull_train$Survived == 1,]
notsurvivedlist=dropnull_train[dropnull_train$Survived == 0,]
 DATA ANALYSIS USING R

Now we can visualize the number of males and females dead and survived using
• bar plots ([Link]
• histograms ([Link]
• piecharts ([Link]

mytable <- table(titanic$Survived)

lbls <- paste(names(mytable), "\n", mytable, sep="")
pie(mytable,
labels = lbls,
main="Pie Chart of Survived column data\n (with sample sizes)")
DATA ANALYSIS USING R
 DATA ANALYSIS USING R

hist(survivedlist$Age,
xlab="gender",
ylab="frequency")

Now let’s draw a bar plot to visualize the number of males and females who were there on the titanic ship.
 DATA ANALYSIS USING R

barplot(table(notsurvivedlist$Sex),
xlab="gender",
ylab="frequency")

From the barplot above we can analyze that there are nearly 350 males, and 50 females those are not survived in titanic.
 DATA ANALYSIS USING R

temp<-density(table(titanic$Fare))
plot(temp, type="n",
main="Fare charged from Passengers")
polygon(temp, col="lightgray",
border="gray")

• Here we can observe that there are some passengers who are charged extremely high.
• So, these values can affect our analysis as they are outliers.
• Let’s confirm their presence using a boxplot.

boxplot(titanic$Fare,
main="Fare charged from passengers")
 DATA ANALYSIS USING R

Performing Clustering
• Import required R libraries for data manipulation, clustering, and visualization.
• Read the Titanic dataset from the specified file path.
• Keep only the relevant columns for analysis.
• Transform categorical variables into numeric format using factor() function.
• Impute missing values for Age with the mean value and ensure there are no remaining missing values.
• Remove any rows with remaining missing values if necessary.
• Scale the data to ensure that all features contribute equally to clustering.
• Verify that the standardized data does not contain NaN or Inf values, which could affect clustering.
• Use the Elbow Method to find the optimal number of clusters; if this method fails, manually try different values.
• Run the K-means algorithm with the chosen number of clusters (e.g., k = 3) and a set seed for reproducibility.
• Ensure that the K-means clustering results have been successfully created.
• Append the cluster assignments to the original dataset for further analysis.
• Create a cluster plot to visualize the results of the K-means clustering.
 DATA ANALYSIS USING R
# Load necessary libraries
library(tidyverse)
library(cluster)
library(factoextra)

# Load the dataset

titanic_data <- [Link]("C:/Users/Tonmoy/Downloads/[Link]")

# Data preprocessing
# Remove unnecessary columns
titanic_data <- titanic_data %>%
select(PassengerId, Survived, Pclass, Sex, Age, SibSp, Parch, Fare)

# Convert categorical variables to numeric

titanic_data$Sex <- [Link](factor(titanic_data$Sex))

# Handle missing values

# Impute missing values for Age
titanic_data$Age[[Link](titanic_data$Age)] <- mean(titanic_data$Age, [Link] = TRUE)

# Ensure there are no remaining missing values

missing_values <- sum([Link](titanic_data))
print(paste("Remaining missing values after imputation:", missing_values))
 DATA ANALYSIS USING R

# If missing values still exist, handle them (e.g., impute with mean or remove rows/columns)
if (missing_values > 0) {
# Remove rows with remaining missing values (if any)
titanic_data <- [Link](titanic_data)
}

# Standardize the data

titanic_scaled <- scale(titanic_data)

# Check for NaNs or Infs in the scaled data

if (any([Link](titanic_scaled)) || any([Link](titanic_scaled))) {
stop("Scaled data contains NaN or Inf values")
}

# Double-check the data for any anomalies

print(summary(titanic_scaled))

# Determine the optimal number of clusters

# If Elbow method still fails, manually try different k values
fviz_nbclust(titanic_scaled, kmeans, method = "wss") + labs(subtitle = "Elbow Method")
 DATA ANALYSIS USING R

# Perform K-means clustering with the optimal number of clusters (e.g., k = 3)

[Link](123)
kmeans_result <- kmeans(titanic_scaled, centers = 3, nstart = 25)

# Check if kmeans_result was created successfully

if (!exists("kmeans_result")) {
stop("K-means clustering failed to create kmeans_result")
}

# Add cluster assignments to the original dataset

titanic_data$Cluster <- kmeans_result$cluster

# Visualize the clustering

fviz_cluster(kmeans_result, data = titanic_scaled, geom = "point", stand = FALSE)
 DATA ANALYSIS USING R
Predictive Model
•Load a collection of R packages for data manipulation and visualization. It includes dplyr and ggplot2, among others.
•The caret package for training and evaluating machine learning models. It provides functions for data splitting
(createDataPartition), model training (train), and performance evaluation (confusionMatrix).
•Loads the dataset from a specified file path into a data frame.
•Chooses relevant columns from the dataset to use for modeling. Here, it selects columns related to survival status and
passenger features.
•Converts the Sex variable into a factor, which is then converted to numeric values. This is necessary because logistic
regression models require numerical input.
•Computes the mean age (excluding missing values) to impute the missing values in the Age column.
•Counts remaining missing values.
•Removes rows with any remaining missing values.
•Converts the Survived variable to a factor. This is essential for classification tasks in logistic regression.
•Ensures reproducibility of the data split.
•Creates an 80-20 split of the data into training and testing sets.
•Trains a logistic regression model (method = “glm”) with a binomial family for binary classification.
•Generates predictions on the test set using the trained model.
•Ensures that the factor levels of titanic_test$Survived match those of predictions.
•Computes and prints the confusion matrix to evaluate model performance.
 DATA ANALYSIS USING R
# Load necessary libraries
library(tidyverse)
library(caret) # For createDataPartition, train, and confusionMatrix

# Load the dataset

titanic_data <- [Link]("C:/Users/Tonmoy/Downloads/[Link]")

# Data preprocessing
titanic_data <- titanic_data %>%
select(Survived, Pclass, Sex, Age, SibSp, Parch, Fare)

# Convert categorical variables to numeric

titanic_data$Sex <- [Link](factor(titanic_data$Sex))

# Handle missing values by imputing with mean for Age

titanic_data$Age[[Link](titanic_data$Age)] <- mean(titanic_data$Age, [Link] = TRUE)

# Check for remaining missing values

missing_values <- sum([Link](titanic_data))
print(paste("Remaining missing values after imputation:", missing_values))
 DATA ANALYSIS USING R

# Remove rows with remaining missing values (if any)

if (missing_values > 0) {
titanic_data <- [Link](titanic_data)
}

# Convert Survived to factor for classification

titanic_data$Survived <- [Link](titanic_data$Survived)

# Split the data into training and testing sets

[Link](123)
trainIndex <- createDataPartition(titanic_data$Survived, p = .8, list = FALSE)
titanic_train <- titanic_data[trainIndex, ]
titanic_test <- titanic_data[-trainIndex, ]

# Train a logistic regression model

model <- train(Survived ~ ., data = titanic_train, method = "glm", family = binomial)
 DATA ANALYSIS USING R

# Make predictions on the test set

predictions <- predict(model, titanic_test)

# Ensure both factors have the same levels

levels(titanic_test$Survived) <- levels(predictions)

# Evaluate the model

conf_matrix <- confusionMatrix(predictions, titanic_test$Survived)
print(conf_matrix)
 DATA ANALYSIS USING R

Explanation of the output –

Accuracy: 100% – The model predicts all test cases correctly.
Sensitivity: 100% – The model identifies all positive cases correctly.
Specificity: 100% – The model identifies all negative cases correctly.
Kappa: 1 – A measure of agreement between the predicted and observed classifications.
The code trains a logistic regression model to predict survival based on various features of the Titanic dataset.
The model shows perfect accuracy, but this might be due to issues in the data or its split. Further validation is recommended.