0% found this document useful (0 votes)
19 views4 pages

Data Analysis with R: Boxplots & Imputation

Week2 R Program

Uploaded by

pz253
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
19 views4 pages

Data Analysis with R: Boxplots & Imputation

Week2 R Program

Uploaded by

pz253
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

R program:

library(ggplot2)

# Step 1: Read and Examine Data


# Reading data from CSV
data <- [Link]("HMEQ_Loss.csv", [Link] = "")

# Display structure of the data


str(data)

# Display summary statistics


summary(data)

# Display first six records


head(data)

# Step 2: Box-Whisker Plots for numeric variables


create_boxplots <- function(data) {
# Get numeric columns
numeric_cols <- names(data)[sapply(data, [Link])]
numeric_cols <- numeric_cols[numeric_cols != "TARGET_BAD_FLAG"]

# Set up plotting area for multiple plots


par(mfrow = c(3, 3)) # Adjust based on number of variables

# Create box plots for each numeric variable


for(col in numeric_cols) {
boxplot(data[[col]] ~ data$TARGET_BAD_FLAG,
main = paste("Distribution of", col),
xlab = "Loan Status (0 = Good, 1 = Bad)",
ylab = col,
col = c("lightblue", "lightgreen"))
}

# Reset plotting area


par(mfrow = c(1, 1))
}

# Step 3: Create Histogram with Density Line


create_histogram <- function(data, variable) {
# Create histogram
hist(data[[variable]],
freq = FALSE,
breaks = 30,
main = paste("Distribution of", variable),
xlab = variable,
col = "lightblue",
border = "white")

# Add density line


lines(density(data[[variable]], [Link] = TRUE),
col = "red",
lwd = 2)
}

# Step 4: Handle Missing Values


impute_data <- function(data) {
# Create copy of original data
imputed_data <- data

# Handle TARGET variables


imputed_data$TARGET_BAD_FLAG[[Link](imputed_data$TARGET_BAD_FLAG)] <- 0
imputed_data$TARGET_LOSS_AMT[[Link](imputed_data$TARGET_LOSS_AMT)] <- 0

# Get numeric columns for imputation (excluding TARGET variables)


numeric_cols <- names(data)[sapply(data, [Link])]
numeric_cols <- numeric_cols[!numeric_cols %in% c("TARGET_BAD_FLAG",
"TARGET_LOSS_AMT")]

# Complex imputation for numeric variables


for(col in numeric_cols) {
# Create missing indicator
imputed_data[paste0("M_", col)] <- ifelse([Link](data[[col]]), 1, 0)

# Perform imputation using median by TARGET_BAD_FLAG group


imputed_values <- tapply(data[[col]], data$TARGET_BAD_FLAG, median, [Link] = TRUE)

# Create new imputed column


imputed_data[paste0("IMP_", col)] <- data[[col]]

# Impute missing values by group


for(flag in c(0, 1)) {
mask <- [Link](imputed_data[paste0("IMP_", col)]) & imputed_data$TARGET_BAD_FLAG ==
flag
imputed_data[mask, paste0("IMP_", col)] <- imputed_values[[Link](flag)]
}
# Remove original column
imputed_data[[col]] <- NULL
}

return(imputed_data)
}

# Step 5: One Hot Encoding


one_hot_encode <- function(data) {
# Identify character columns
char_cols <- names(data)[sapply(data, [Link])]

# Create dummy variables for each character column


for(col in char_cols) {
# Get unique values
unique_values <- unique(data[[col]][![Link](data[[col]])])

# Create dummy variables


for(value in unique_values) {
new_col_name <- paste0(col, "_", [Link](value))
data[[new_col_name]] <- ifelse(data[[col]] == value, 1, 0)
}

# Remove original column


data[[col]] <- NULL
}

return(data)
}

# Main execution
main <- function() {
# Read data
cat("Reading data...\n")
data <- [Link]("HMEQ_Loss.csv", [Link] = "")

# Step 1: Examine Data


cat("\nData Structure:\n")
str(data)

cat("\nData Summary:\n")
print(summary(data))

cat("\nFirst Six Records:\n")


print(head(data))

# Step 2: Create Box Plots


cat("\nCreating box plots...\n")
create_boxplots(data)

# Step 3: Create Histogram for LOAN amount


cat("\nCreating histogram for LOAN amount...\n")
create_histogram(data, "LOAN")

# Step 4: Handle Missing Values


cat("\nHandling missing values...\n")
imputed_data <- impute_data(data)

cat("\nSummary after imputation:\n")


print(summary(imputed_data))

# Print sum of missing value indicators


m_cols <- names(imputed_data)[startsWith(names(imputed_data), "M_")]
cat("\nNumber of imputed values per variable:\n")
print(colSums(imputed_data[m_cols]))

# Step 5: One Hot Encoding


cat("\nPerforming one-hot encoding...\n")
final_data <- one_hot_encode(imputed_data)

cat("\nFinal Data Structure:\n")


print(str(final_data))

return(final_data)
}

# Run the analysis

Common questions

Powered by AI

Challenges during one-hot encoding include the creation of many new columns if categories are numerous, leading to high-dimensional datasets that can increase computational expense and complexity. The R program addresses this by systematically iterating through character columns, creating binary indicators for each unique category, and then removing the original columns, thus transforming the dataset efficiently. This minimizes dimensionality issues by focusing only on columns with existing categories and excluding missing or irrelevant category values .

The structure of the data is initially examined by using 'str(data)' to view the types of variables and their sample contents, and 'summary(data)' to review summary statistics of each variable. Further, the first six records are displayed using 'head(data)'. Examining data structure initially is essential for understanding the data types, detecting any anomalies or unexpected data types, and planning subsequent analysis steps. It provides an overview, allowing the analyst to ensure the data is in the expected format and to identify areas needing cleaning or transformation .

The 'par(mfrow = c(3, 3))' command in R configures the plotting area to a grid layout, allowing up to nine plots to be displayed simultaneously in a 3-by-3 matrix format. This is crucial for multi-plot analysis as it facilitates the comparison of various plots side-by-side, enabling analysts to quickly observe patterns and anomalies across multiple variables without switching between separate windows or screens. Such a layout is particularly useful in exploratory data analysis for visualizing relationships and distributions efficiently .

Box-whisker plots are used to visualize the distribution of numeric variables and detect outliers within groups. In the provided R program, they are implemented by first identifying numeric columns in the dataset, excluding the 'TARGET_BAD_FLAG'. Multiple plots are arranged in a grid layout using 'par(mfrow = c(3, 3))'. For each numeric variable, a box-whisker plot is created, displaying different distributions based on the 'TARGET_BAD_FLAG' variable, which distinguishes between good and bad loans .

The R program distinguishes between different loan statuses using the 'TARGET_BAD_FLAG' variable during imputation. Missing numeric values are replaced by the median of the corresponding 'TARGET_BAD_FLAG' group ('0' for good, '1' for bad). This stratified imputation approach maintains the variability associated with different loan statuses, ensuring that imputed values reflect the specific characteristics of each status group. It prevents distortion of the dataset's structure by preserving distinct distributions, which might otherwise be homogenized if a single median were used for all data .

In the R program, density lines are added to histograms to provide a continuous visual representation of the distribution's shape. By overlaying density lines on histograms, the program highlights the estimated probability density function of the variable, offering insights into the skewness, modality, and overall distribution shape that histograms with fixed bins might obscure. This combination aids in identifying any non-uniform distribution features that could influence subsequent analyses, such as modeling assumptions about normality .

The 'main' function orchestrates the execution of all data analysis steps. It starts with reading the data, followed by structural overview through summary statistics and displaying initial records. Then it calls 'create_boxplots' to visualize data distributions, 'create_histogram' for the 'LOAN' variable's distribution, 'impute_data' to handle missing values, and 'one_hot_encode' to transform categorical variables. By integrating these functions, the 'main' function streamlines the workflow, ensuring each analysis step is carried out in sequence and producing a final processed dataset ready for further analysis or modeling .

Visualizing the 'LOAN' variable's distribution involves creating a histogram with a density line overlay. The function 'create_histogram' is used, which first constructs a histogram with specified breaks and color settings. Then, a density line is added to the same plot using the 'density' function to show the data's overall distribution shape more clearly. This visualization helps identify the distribution characteristics like skewness, spread, and potential modalities in loan amounts, which are crucial for understanding the data's underlying patterns and identifying any potential issues such as outliers .

The R program handles missing values by creating a copy of the original dataset for modification. For the 'TARGET_BAD_FLAG' and 'TARGET_LOSS_AMT' columns, missing values are set to zero. For other numeric columns, missing indicators are created, and missing values are imputed using the median of the respective 'TARGET_BAD_FLAG' group. This involves identifying columns with missing values, calculating medians within each 'TARGET_BAD_FLAG' group, and replacing missing values with these medians. Original columns are subsequently removed .

One-hot encoding transforms categorical variables into a format suitable for machine learning algorithms that cannot directly handle categorical data. In this R program, character columns are first identified, and for each, dummy variables are created for all unique values. Each unique value results in a new column indicating the presence (1) or absence (0) of that category. The original string columns are removed afterward. This process allows algorithms to process categorical inputs effectively by representing them as binary data, which preserves the information without assuming any ordinal relationship .

You might also like