Module-4

EXPLORE AND ANALYZE DATA

Dr. Sonali Mahure

What is Data Exploration?
Data exploration is the initial step in data analysis.
We dive into a dataset to know what it contains.
It's like detective work for our data.
Goal = understand data patterns, trends, anomalies (unusual/unexpected things)
or problems.
Example: Student marks dataset → check highest, lowest, average.
Analogy: Like looking at a map before traveling → exploration helps us plan.

Dr. Sonali Mahure

How Data Exploration Works?

i. Data Collection
ii. Data Cleaning
iii. Exploratory Data Analysis (EDA)
iv. Feature Engineering
v. Model Building and Validation

Dr. Sonali Mahure

Steps involved in Data Exploration
Data exploration is an iterative process, but there are generally some key steps involved:
1. Data Understanding
 Familiarization: Get an overview of the data format, size, and source.
 Variable Identification: Understand the meaning and purpose of each variable in the dataset.

2. Data Cleaning
 Identifying Missing Values: Locate and address missing data points .
 Error Correction: Find and rectify any inconsistencies or errors within the data.
 Outlier Treatment: Identify and decide how to handle outliers that might skew (mislead) the analysis.

Dr. Sonali Mahure

3. Exploratory Data Analysis (EDA)
 Univariate Analysis: Analyze individual variables to understand their distribution (e.g., histograms,
boxplots for numerical variables; frequency tables for categorical variables).

 Bivariate Analysis: Explore relationships between two variables using techniques like scatterplots to
identify potential correlations.

Dr. Sonali Mahure

Applications/ Advantages of Data Exploration

1. Business

Retail: Analyzing sales data to optimize inventory management and forecast demand.

Manufacturing: Identifying production inefficiencies or predicting equipment failures

through data analysis.

Marketing: Understanding customer behavior for targeted and personalized marketing

campaigns.

Dr. Sonali Mahure

2. Healthcare and Medicine

Disease Prediction: Analyzing patient data to predict and prevent diseases based on
risk factors.

Treatment Optimization: Identifying effective treatments or therapies by analyzing

patient response data.

Dr. Sonali Mahure

3. Financial Sector
Fraud Detection
Risk Assessment: Assessing investment risks by analyzing market data and economic
indicators.
Portfolio Management: Optimizing investment portfolios based on historical
performance and market trends.

Dr. Sonali Mahure

4. E-commerce and Customer Experience:

Customer Personalization: Analyzing browsing and purchasing patterns to personalize

recommendations.
Supply Chain Optimization: Optimizing inventory and logistics by analyzing demand
and supply data.

Dr. Sonali Mahure

4.1 Create quick table calculations
4.1.1 Create calculated fields
Introduction to Calculated Fields
A calculated field is a new column in a dataset that is derived from existing columns
by applying a formula or expression.
It does not exist in the raw data → we create it during analysis.
Purpose:
To derive new insights (e.g., % marks).
To avoid manual calculation every time.
To simplify analysis by adding meaningful variables.


Dr. Sonali Mahure

Arithmetic Operators in R
To create calculated fields, we use operators:
Addition: + → a + b
Subtraction: - → a - b
Multiplication: * → a * b
Division: / → a / b
Power: ^ → a ^ 2 (square of a)
Parentheses control order: (a + b) / 2
These are the building blocks of calculated fields.

Dr. Sonali Mahure

Example Dataset (Students Marks)

Task: Add a new column Percentage = (Marks / Total) × 100.

Dr. Sonali Mahure

R Example: Creating a Calculated Field
students <- [Link](
name = c("Ravi","Sneha","Anil"),
marks = c(85,92,78),
total = c(100,100,100))
students$percentage <- (students$marks / students$total) * 100
print(students)
Output:

Dr. Sonali Mahure

Creating Multiple Calculated Fields

We can add more derived columns:

students$grade <- ifelse(students$percentage >= 85, "A",

ifelse(students$percentage >= 70, "B", "C"))

students$status <- ifelse(students$percentage >= 40, "Pass", "Fail")

Now the dataset contains: Name , total, Marks, Percentage, Grade, Status.

Dr. Sonali Mahure

Hands-on Activity
Create a dataset: employee(name, salary, bonus)
Add a calculated field: total_income = salary + bonus
Add another: tax = total_income * 0.1
Add another: net_income = total_income - tax
Expected output: A table with 3 new fields.

Dr. Sonali Mahure

4.1.2 Moving average
What is a Moving Average?
It is the average of a fixed number of previous observations in a dataset.
Widely used in:
Stock market (trend analysis)
Weather forecasting (average temperature)
Sales prediction (average sales per week)

Dr. Sonali Mahure

Types of Moving Average
Simple Moving Average (SMA):
Just average of previous k observations.
Weighted Moving Average (WMA):
Gives more weight to recent values.
Exponential Moving Average (EMA):
Applies exponential weights (heavier weight to most recent).
 For beginners, we will start with Simple Moving Average.

Dr. Sonali Mahure

Formula for Simple Moving Average (SMA)
For a window size of k = 3:

Example (Sales):
Day 1: 100, Day 2: 120, Day 3: 150
3-day moving average on Day 3 = (100 + 120 + 150) / 3 = 123.3
The window keeps sliding forward → hence “moving” average.

Dr. Sonali Mahure

Example Dataset (Daily Sales)

We want to calculate the 3-day moving average of sales.

Dr. Sonali Mahure

R Example (Moving Average)

library(zoo) # for rollmean()

sales <- c(100,120,150,130,170,160,180)

# 3-day moving average

ma <- rollmean(sales, k=3, fill=NA, align="right")

[Link](Day=1:7, Sales=sales, Moving_Avg=ma)

Dr. Sonali Mahure

Output

Day Sales Moving_Avg How R calculated it

1 100 NA Not enough days yet

2 120 NA Not enough days yet

Average of (100,
3 150 123.33
120, 150)

Average of (120,
4 130 133.33
150, 130)

Average of (150,
5 170 150.00
130, 170)

Dr. Sonali Mahure

Visualization in R
plot(sales, type="o", col="blue", xlab="Day", ylab="Sales")
lines(ma, type="o", col="red")
legend("topleft", legend=c("Actual Sales","Moving Average"),
col=c("blue","red"), lty=1, pch=1)

Blue line = Actual Sales

Red line = Smoothed Moving Average

Dr. Sonali Mahure

Why Use Moving Average?
Highlights trends more clearly.
Helps businesses make better forecasts.
Example:
Raw sales fluctuate daily.
Moving average shows steady growth pattern.
 Like seeing the overall journey instead of every speed bump.

Dr. Sonali Mahure

Hands-on Activity
Create dataset: temperature = c(30,32,35,33,36,38,37,40,42)
Calculate 3-day moving average.
Plot original vs moving average in graph.
Discuss: Is moving average smoother than raw values?

Dr. Sonali Mahure

4.1.3 Percent of Total
What is Percent of Total?
The Percent of Total tells us what share or contribution each value has out of the
total.
Formula:

Examples:
City’s sales contribution to company’s total sales.
Age group’s percentage in a population survey.

Dr. Sonali Mahure

Example Dataset (Sales by City)

Question: What % of total sales does each city contribute?

Dr. Sonali Mahure

Manual Calculation Example

Total Sales = 500 + 300 + 200 + 100 = 1100

Percent of Total (Delhi) = (500 / 1100) × 100 = 45.45%

Percent of Total (Mumbai) = (300 / 1100) × 100 = 27.27%

Percent of Total (Chennai) = (200 / 1100) × 100 = 18.18%

Percent of Total (Kolkata) = (100 / 1100) × 100 = 9.09%

 Percentages always add up to 100%.

Dr. Sonali Mahure

R Example: Percent of Total

sales <- [Link](

city = c("Delhi", "Mumbai", "Chennai", "Kolkata"),
sales = c(500, 300, 200, 100)
)

total_sales <- sum(sales$sales)

sales$percent <- (sales$sales / total_sales) * 100

print(sales)

Dr. Sonali Mahure

Visualization (Pie Chart in R)

pie(sales$sales, labels = paste(sales$city,

round(sales$percent,1), "%"),
main="Sales Contribution by City")

Each city is shown as a slice of the pie.

Size of slice = Percent of Total.

Dr. Sonali Mahure

Hands-on Activity
Create dataset: expenses = (Rent=8000, Food=5000, Travel=2000, Misc=1000)
Calculate percent of total expenses.
Draw a bar chart or pie chart to visualize.
Discuss: Which category dominates your spending?

Dr. Sonali Mahure

What is a Running Total?
A Running Total is the cumulative sum of values up to a given point.
It shows how a quantity grows over time.
Formula:

Example:
Daily sales = 100, 200, 300
Running total = 100, 300, 600

Dr. Sonali Mahure

Example Dataset (Daily Sales)

Question: What is the running total of sales?

Dr. Sonali Mahure

Manual Calculation Example

Day 1: 100 → Running total = 100

Day 2: 200 → Running total = 100 + 200 = 300

Day 3: 300 → Running total = 300 + 300 = 600

Day 4: 150 → Running total = 600 + 150 = 750

Day 5: 250 → Running total = 750 + 250 = 1000

 Final Running Total = 1000

Dr. Sonali Mahure

R Example: Running Total

sales <- c(100, 200, 300, 150, 250)

# running total using cumsum()

running_total <- cumsum(sales)

[Link](Day = 1:5, Sales = sales, Running_Total = running_total)

Dr. Sonali Mahure

Output:

Dr. Sonali Mahure

Visualization in R

plot(running_total, type="o", col="blue",

xlab="Day", ylab="Running Total Sales",
main="Cumulative Sales Over Time")

Graph shows how total sales grow step by step.

Dr. Sonali Mahure

Hands-on Activity
Create dataset: marks = c(10, 15, 20, 25, 30)
Calculate running total of marks.
Plot both marks per test and running total.
Discuss: Which graph shows better progress tracking?

Dr. Sonali Mahure

Dr. Sonali Mahure
4.1.5 Percentile
What is a Percentile?
A percentile tells us the value below which a given percentage of data falls.
Example:
50th percentile (median) → half of data is below this value.
90th percentile → 90% of data lies below this value.
Analogy: In a class, if you are in the 90th percentile, you scored better than 90% of
students.

Dr. Sonali Mahure

Formula for Percentile

Pk​=Value at position 100k​×(n+1) PkP_kPk​ = k-th percentile

nnn = number of observations
Example: If 10 students’ marks are sorted, the 25th percentile is the
value at position (25/100)×(10+1)=2.75(25/100) \times (10+1) =
2.75(25/100)×(10+1)=2.75.
 If position is fractional, take an average between nearest ranks.
Dr. Sonali Mahure
Example Dataset (Marks of 10 Students)

Question: What are the 25th, 50th, and 75th percentiles?

Dr. Sonali Mahure

Manual Calculation Example
n = 10
25th percentile (P25): Position = 0.25 × (10+1) = 2.75 → between 2nd and 3rd values
(30 and 35).
Value = 30 + 0.75 × (35–30) = 33.75
50th percentile (P50): Position = 0.50 × (11) = 5.5 → between 5th (45) and 6th (50).
Value = 47.5
75th percentile (P75): Position = 0.75 × (11) = 8.25 → between 8th (60) and 9th (65).
Value = 61.25

Dr. Sonali Mahure

R Example: Percentiles
marks <- c(25,30,35,40,45,50,55,60,65,70)

# calculate percentiles
quantile(marks, probs = c(0.25, 0.5, 0.75))

Output:

25% → 33.75
50% → 47.50 (Median)
75% → 61.25
 R handles interpolation automatically.
Dr. Sonali Mahure
Visualization in R
boxplot(marks, main="Student Marks with Percentiles")

Boxplot shows:
Minimum
25th percentile (Q1)
50th percentile (Median, Q2)
75th percentile (Q3)
Maximum
 Percentiles are built-in in boxplots!

Dr. Sonali Mahure

Hands-on Activity
Create dataset: heights = c(150,155,160,162,165,168,170,172,175,180)
Calculate 10th, 50th, and 90th percentiles using quantile().
Draw a boxplot.
Discuss: Are students’ heights normally distributed or skewed?

Dr. Sonali Mahure

4.1.6 Custom Calculations
What Are Custom Table Calculations?
A custom calculation = a formula created by the analyst, beyond built-in ones like
moving average or percent of total.
It allows you to design your own metrics that fit the business or study needs.
Example:
Built-in → Moving Average.
Custom → Growth Rate = (Current – Previous) / Previous × 100.
 Analogy: Built-in = “ready-made dosa mix.”
 Custom = “you decide the ingredients and make it your own recipe.”

Dr. Sonali Mahure

Why Custom Calculations?
Not all analysis needs are covered by built-in functions.
Each dataset is unique.
Provides flexibility:
Education → Relative performance of students.
Business → Profit Margin % = (Profit / Revenue) × 100.
Sports → Win Rate = Wins / Matches Played.
 Custom calculations let you ask your own questions.

Dr. Sonali Mahure

Refresher: How to Create New Columns in R
In R, custom calculations are created by:
dataset$new_column <- formula
Uses existing columns.
Formula can include arithmetic, conditions, or built-in functions
Example:
sales$profit_margin <- (sales$profit / sales$revenue) * 100

Dr. Sonali Mahure

Example Dataset (Company Sales)

Task: Add two custom fields:

Profit = Revenue – Cost
Profit Margin = (Profit / Revenue) × 100

Dr. Sonali Mahure

R Example: Custom Calculations
sales <- [Link](
product = c("A","B","C"),
revenue = c(1000,2000,1500),
cost = c(700,1200,1000)
)

sales$profit <- sales$revenue - sales$cost

sales$profit_margin <- (sales$profit / sales$revenue) * 100

print(sales)

Dr. Sonali Mahure

Output

Dr. Sonali Mahure

Another Example: Growth Rate
Compare current year vs previous year:
yearly_sales <- [Link](
year = c(2022, 2023, 2024),
sales = c(1000, 1200, 1500)
)

yearly_sales$growth_rate <- c(NA, diff(yearly_sales$sales) /

yearly_sales$sales[-
length(yearly_sales$sales)] * 100)

print(yearly_sales)

Dr. Sonali Mahure

Output:

Growth Rate is a custom formula defined by the analyst.

Dr. Sonali Mahure

Visualization of Custom Calculations
barplot(sales$profit, [Link]=sales$product,
col="lightblue", main="Profit per Product")

plot(yearly_sales$year, yearly_sales$growth_rate, type="o", col="red",

xlab="Year", ylab="Growth Rate (%)", main="Sales Growth Rate")

First graph = profit comparison.

Second graph = growth rate trend.

Dr. Sonali Mahure

Hands-on Activity
Create dataset: students(name, marks, total).
Custom field 1: Percentage = (marks / total) × 100.
Custom field 2: Grade (A/B/C) using ifelse().
Custom field 3: Pass/Fail using ifelse().
Plot bar chart of percentages.

Dr. Sonali Mahure

4.1.7 Create and Use Filters
What is a Filter?
A filter selects only the rows of data that meet certain conditions.
Purpose: To focus on relevant data and ignore unnecessary parts.
Example:
Show only students who scored above 50.
Show only sales from Delhi.
 Analogy: A sieve filters out unwanted stones while keeping the rice you need.

Dr. Sonali Mahure

Types of Filters
Dimension Filter (categorical variables)
Select rows based on categories.
Example: Keep only “Delhi” and “Mumbai” from city.
Measure Filter (numeric variables)
Select rows based on numerical conditions.
Example: Keep only marks > 50.
 Dimensions = categories
 Measures = numbers

Dr. Sonali Mahure

R Example: Dimension Filter
students <- [Link](
name = c("Ravi","Sneha","Anil","Kiran"),
marks = c(85, 40, 72, 90),
city = c("Delhi","Mumbai","Delhi","Chennai")
)

# filter students from Delhi

subset(students, city == "Delhi")

Output keeps only students from Delhi.

Dr. Sonali Mahure

R Example: Measure Filter
# filter students who passed (marks >= 50)
passed <- subset(students, marks >= 50)
print(passed)

Output shows only students who scored 50 or more.

Dr. Sonali Mahure

Combining Filters (AND, OR)
# Students from Delhi AND marks > 70
subset(students, city == "Delhi" & marks > 70)

# Students from Delhi OR Mumbai

subset(students, city %in% c("Delhi","Mumbai"))

Logical operators:
& = AND
| = OR
%in% = match multiple values

Dr. Sonali Mahure

Context Filters (Step-by-Step)
Sometimes filters need to be applied in order.
Example:
First → Filter students from Delhi.
Then → From those, select marks > 70.
 In R, this can be done step by step:
delhi_students <- subset(students, city == "Delhi")
high_scorers <- subset(delhi_students, marks > 70)

Dr. Sonali Mahure

Using Parameters (Interactive Filters)
A parameter is a variable whose value can be changed by the user.
Example in R:
cutoff <- 60
subset(students, marks >= cutoff)

Here, changing cutoff changes the filter condition dynamically.

Dr. Sonali Mahure

Hands-on Activity
1) Create dataset: employees(name, dept, salary)
2) Apply filters:
Employees only in “IT” dept.

Employees with salary > 30,000.

Employees in “IT” dept AND salary > 30,000.

3) Make cutoff = 40,000 as a parameter and filter again.

Dr. Sonali Mahure

Geographic Data in R
R supports geographic data visualization using libraries:
maps → world, countries, states
ggplot2 + maps → advanced visualization
Data needed:
Location names (city, state, country)
Values (sales, population, cases, etc.)

Dr. Sonali Mahure

4.1.8 Mapping Data Geographically
Why Map Data Geographically?
Many datasets have a location component (city, state, country).
Mapping helps us see where things happen, not just numbers.
Examples:
Sales across cities.
Population density by state.
Covid-19 cases by country.
 Analogy: Instead of just saying “100 sales in Delhi,” put it on a map to see the regional
distribution.

Dr. Sonali Mahure

Example Dataset (City Sales)

Goal: Show sales by city on India map.

Dr. Sonali Mahure

Mapping in R (Basic World Map)

library(maps)
map("world") # draws world map

This draws the outline of the world.

Later we overlay our data points.

Dr. Sonali Mahure

Adding Points to the Map

# Example: mark Delhi and Mumbai

map("world", "India")
points(77.1, 28.7, col="red", pch=19) # Delhi (longitude, latitude)
points(72.8, 19.0, col="blue", pch=19) # Mumbai

Each point is placed using longitude & latitude.

Dr. Sonali Mahure

Mapping with ggplot2
library(ggplot2)
library(maps)

india_map <- map_data("world", region="India")

ggplot() +
geom_map(data=india_map, map=india_map,
aes(x=long, y=lat, map_id=region),
fill="lightyellow", color="black") +
coord_fixed(1.3)
Produces a clean India map.

Dr. Sonali Mahure

Overlay Data on Map
sales_data <- [Link](
city=c("Delhi","Mumbai","Chennai","Kolkata"),
lon=c(77.1, 72.8, 80.3, 88.4),
lat=c(28.7, 19.0, 13.1, 22.6),
sales=c(500,300,200,150)
)

ggplot() +
geom_map(data=india_map, map=india_map,
aes(x=long, y=lat, map_id=region),
fill="lightyellow", color="black") +
geom_point(data=sales_data,
aes(x=lon, y=lat, size=sales, color=city)) +
scale_size(range=c(3,10))

Each bubble size = sales value.

Dr. Sonali Mahure

Real-world Applications
Epidemiology: Map of Covid-19 cases by district.
Marketing: Sales performance by city.
Education: Student enrollment by state.
Government: Budget allocation by region.
Geographic maps show “where” patterns exist.

Dr. Sonali Mahure

Hands-on Activity
Create dataset of cities, lon, lat, population.
Use map("world","India") to draw map.
Add points using points().
Try coloring bubbles by population size.
Discuss: Which cities dominate population?

Dr. Sonali Mahure

4.1.9 Summarize, Model, and Customize Data using Analytics Features
What Does “Analytics Features” Mean?
Analytics = discovering patterns, insights, and trends from data.
Features in R for analytics:
Summarization – condensing large datasets.
Modeling – building simple predictive models.
Customization – adjusting analysis to answer specific questions.
Analogy: Imagine reading a 400-page novel.
Summarization = plot summary.
Modeling = predicting what happens next.
Customization = analyzing only your favorite character.

Dr. Sonali Mahure

Summarizing Data (Descriptive Stats)
Summarization reduces raw data into meaningful statistics:
Mean, Median, Mode
Minimum, Maximum, Range
Standard Deviation, Variance
R functions:
summary(dataset)
mean(x); median(x); sd(x); var(x); min(x); max(x)

Example: Summarize marks of 100 students → average, highest, lowest.

Dr. Sonali Mahure

Summarization Example in R

marks <- c(45, 67, 89, 76, 54, 92, 38)

summary(marks)
mean(marks)
sd(marks)

Output
Min: 38, Max: 92
Mean: 65.8
Median: 67
Standard Deviation: ~20
 With just a few functions, we understand the spread of marks.

Dr. Sonali Mahure

Modeling Data (Simple Prediction)
Modeling = using math/statistics to predict outcomes.
Example:
Data: Hours studied vs Marks scored.
Build a model → “more hours = higher marks.”
Common beginner models in R:
Linear Regression → predict continuous values.
Logistic Regression → predict categories (pass/fail).
 In this class we’ll start with Linear Regression (simplest).

Dr. Sonali Mahure

Linear Regression Example in R
hours <- c(2, 4, 6, 8, 10)
marks <- c(40, 50, 65, 80, 95)

model <- lm(marks ~ hours) # fit regression model

summary(model)

Formula: Marks = Intercept + (Slope × Hours)

If slope = 5, then each extra study hour adds ~5 marks.
This is a basic predictive model.

Dr. Sonali Mahure

Visualization of Model

plot(hours, marks, col="blue", pch=19, main="Hours vs Marks")

abline(model, col="red", lwd=2)

Scatterplot shows actual marks (blue dots).

Red line shows predicted trend.

Dr. Sonali Mahure

Customizing Analysis
Sometimes we need to customize analysis to answer questions:
Focus on only one subject.
Analyze male vs female students separately.
Compare urban vs rural sales.
 In R, customization = using filters + grouping + custom formulas.

subset(marks_data, gender=="Male")

aggregate(marks ~ subject, data=marks_data, mean)

Dr. Sonali Mahure

Hands-on Activity
Create dataset: hours studied vs marks.
Summarize dataset → mean, min, max.
Build linear regression model → predict marks.
Plot scatterplot + regression line.
Discuss: Does the line fit well?

Dr. Sonali Mahure

Hands-on:
1) Perform quick table calculations including creating calculated fields,
calculating moving averages, and computing percentages of the total.

Dr. Sonali Mahure

2) Write a program to apply filters to dimensions and measures.

Name <- c("Ram","Sita","Arun","Ashok","Vinay","Nani","Rahul","Priya","Kiran","Meena")

Department <- c("CSE","CSE","ISE","ECE","CSE","ISE","CSE","ECE","ISE","CSE")

Marks <- c(85,90,78,88,82,76,91,87,80,84)

students <- [Link](Name, Department, Marks)

print(students)

Dr. Sonali Mahure

# Now filter students whose marks are greater than 85.

high_marks <- students[students$Marks > 85, ]

print(high_marks)

Dr. Sonali Mahure

Apply Multiple Filters: We can also apply both dimension and measure filters together.
Example: Students from CSE department with marks greater than 85.

result <- students[students$Department == "CSE" & students$Marks > 85, ]

print(result)

Dr. Sonali Mahure

3) Design a program to Map a data geographically.
[Link]("ggplot2")
[Link]("maps")
library(ggplot2)
library(maps)

city <- c("Bangalore","Hyderabad","Chennai","Mumbai","Delhi","Kolkata","Pune","Ahmedabad","Jaipur","Mysore")

sales <- c(1200,1500,1300,2000,2200,1400,1100,900,700,800)

data <- [Link](city, sales)

print(data)

Dr. Sonali Mahure

# Load Geographic Map Data: Now load the world map dataset.

world_map <- map_data("world")

head(world_map)
ggplot() +
geom_map(data = world_map,
map = world_map,
aes(x = long, y = lat, map_id = region),
fill = “red",
color = “green") +
coord_fixed(1.3)

Dr. Sonali Mahure

Display Map of India
india_map <- subset(world_map, region == "India")
ggplot() + geom_polygon(data = india_map, aes(x = long, y = lat, group = group),
fill = “orange", color = "black") + ggtitle("Map of India")

Dr. Sonali Mahure