UNIT-II: DATA EXPLORATION

Beginner Level Study Notes for 5-Mark Answers

1. Types of Data
Definition
Data is information collected from various sources. It can be classified into two main
categories based on its nature.

Types
Qualitative (Categorical) Data:

Data that represents labels, names, or categories

Cannot be measured numerically
Examples: color (red, blue, green), gender (male, female), grade (A, B, C), blood type
(O, A, B, AB)
Quantitative (Numerical) Data:
Data that consists of numbers and can be measured
Arithmetic operations can be performed on it
Examples: height, weight, marks, salary, temperature, age

Sub-types of Quantitative Data

Discrete Data:

Countable values only

Cannot take any value between two points
Examples: number of students (5, 10, 15... not 5.5), number of books, number of cars
Sub-types of Qualitative Data
Nominal Data:

No natural order or ranking

Categories cannot be arranged in sequence
Examples: blood group, nationality, eye color, hobby
Ordinal Data:
Natural order or ranking exists
Categories can be arranged in sequence
Examples: grade (F < D < C < B < A), satisfaction level (poor < average < good <
excellent)
Simple Classification Diagram
DATA
│
├── QUALITATIVE (Categories/Labels)
│ ├── Nominal (No order): Blood group, Color, Gender
│ └── Ordinal (With order): Grade, Satisfaction level
│
└── QUANTITATIVE (Numbers)
├── Discrete (Countable): Students, Books, Cars
└── Continuous (Any value): Height, Weight, Temperature

Example
In a student database:
Roll number: Discrete quantitative
Name: Nominal qualitative
Height: Continuous quantitative
Class rank: Ordinal qualitative

2. Data Collection Methods

Definition
Data collection is the process of gathering information from various sources for analysis
and decision-making.

Primary Data Collection

Definition: Data collected directly for the current study/project
Methods:

Survey: Questionnaires given to a large number of people

Example: Online survey asking "How satisfied are you with our service?" (1-5
scale)
Interview: One-on-one or group conversations
Example: Company interviewing employees about workplace satisfaction
Observation: Direct watching and recording of events
Example: Observing how customers browse in a store
Experiment: Controlled environment testing
Example: Testing how temperature affects crop growth

Secondary Data Collection

Definition: Data collected previously by someone else and reused for new purposes
Sources:

Government reports and census data

Company databases and records
Books, journals, and research papers
 Websites and online databases
Historical records

Data Collection Design

Cross-sectional Data:
Data collected at a single point in time
Example: Survey on student preferences conducted in December 2025

Longitudinal Data:
Same units (people/items) tracked over time
Example: Recording monthly sales of a shop for 12 months

Data Collection Methods Flowchart

DATA COLLECTION
│
├── PRIMARY (Direct Collection)
│ ├── Survey (Questionnaires)
│ ├── Interview (One-on-one)
│ ├── Observation (Direct watching)
│ └── Experiment (Controlled testing)
│
└── SECONDARY (Existing Data)
├── Government Reports
├── Company Databases
├── Books & Journals
└── Websites

Example
For a project on "Student Eating Habits":

Primary: Conduct survey asking students what they eat daily

Secondary: Use existing nutrition database from health ministry

3. Data Characteristics (5 Vs of Big Data)

The Five V's
Volume:

Refers to the amount or size of data

Measured in: kilobytes (KB), megabytes (MB), gigabytes (GB), terabytes (TB)
Example: A company receives 1 TB of customer data daily
Velocity:
Speed at which new data is generated and arrives
Example: A weather sensor sending temperature readings every 1 second
Fast velocity requires real-time processing
Variety:
Different types and formats of data
Includes: text, numbers, images, audio, video, social media posts
Example: A hospital system handles patient reports (text), X-rays (images), and test
results (numbers)

Veracity:
Quality and trustworthiness of data
Affected by: errors, missing values, noise, inconsistencies
Example: A dataset with many spelling errors has low veracity
Value:

Usefulness of data for solving problems or making decisions

Not all data is valuable; some may be irrelevant
Example: Student marks data has high value for performance analysis

Attributes and Properties

Outliers:
Extreme values that differ significantly from other data points
Can affect analysis and need special handling
Example: In a class where most students score 70-90, one student scoring 30 is an
outlier
Range:

Difference between maximum and minimum values

Example: If marks range from 30 to 95, range = 95 - 30 = 65

Characteristics Summary Table

Characteristi
Meaning Example
c
Volume Amount of data 500 GB of sales records
Speed of data
Velocity 1000 tweets per second
arrival
Variety Different data types Images, text, numbers
Veracity Data quality 95% accurate data
Value Usefulness Data for decision-making
One very high salary in
Outliers Extreme values
dataset
4. Handling Missing Values
What are Missing Values?
Missing values are blank spaces, NA, or NULL entries in a dataset where information should
be present.
Reasons for Missing Values:
Non-response: Person skips a question in survey
Equipment failure: Sensor stops recording data
Manual error: Data entry mistake
Loss of data: File corruption or deletion
Privacy reasons: Sensitive information omitted

Impact of Missing Values

Reduces data quality and reliability
Can bias analysis results
Reduces effective sample size for analysis

Methods to Handle Missing Values

Method 1: Deletion
Remove rows or columns containing missing values
When to use: Few missing values (less than 5%), missing data is random
Advantage: Simple and fast
Disadvantage: Loses information
Example: Delete a survey response where respondent didn't fill age
Method 2: Imputation with Mean/Median/Mode

Replace missing values with statistical measures

For Numerical Data:
Mean: Average of all non-missing values
Example: Marks are 80, 75, NA, 90 → Replace NA with mean = (80+75+90)/3 =
81.67
Median: Middle value when sorted
Better than mean when outliers exist
Example: 20, 25, NA, 100 → Median = 22.5

For Categorical Data:

Mode: Most frequently occurring category
Example: Colors are Red, Red, Red, NA, Blue → Replace NA with Red (most
frequent)
Method 3: Advanced Methods

Prediction Model: Use other variables to predict missing value

Nearest Neighbor: Use similar records' values
 Forward Fill/Backward Fill: Use previous or next value (for time-series data)

Handling Missing Values Flowchart

MISSING VALUE DETECTED?
│
├─ YES
││
│ ├─ Is it few and random?
│ │ └─ YES → DELETE ROW
││
│ └─ Is it systematic?
│ ├─ Numerical data? → Replace with MEAN/MEDIAN
│ └─ Categorical data? → Replace with MODE
│
└─ NO → Continue with analysis

Example with Steps

Original data: Salary (in thousands)
30, 35, NA, 40, 45, 50

Step 1: Identify missing value (NA in position 3)

Step 2: Calculate mean = (30+35+40+45+50)/5 = 40
Step 3: Replace NA with 40
Step 4: New data = 30, 35, 40, 40, 45, 50

5. Introduction to Data Visualization

Definition
Data visualization is representing data using visual elements like charts, graphs, and
diagrams to make patterns and insights easy to understand.

Importance
Makes large amounts of data understandable at a glance
Reveals patterns, trends, and anomalies
Helps in presentation and communication
Supports decision-making

Common Visualization Types

Bar Chart:
Used for: Comparing categories
Data type: Categorical with numerical values
Example: Number of students in different branches (CSE, ECE, ME, Civil)
Number of Students
│
60│ ┌─────┐
│││
 50│ │ CSE │ ┌─────┐
│││││
40│ │ │ ECE │ │
│ │ │ ┌─────┐│ │
30│ │ │ │ ││ │ ┌─────┐
│ │ │ │ ││ │ │ │
┼─────┴─────┴─┴─────┴┴─────┴─┴─────┴──
CSE ECE ME Civil
Histogram:

Used for: Distribution of numerical data

Shows frequency of ranges
Example: Distribution of student marks (0-20, 20-40, 40-60, etc.)
Line Chart:
Used for: Trends over time
Example: Sales growth month by month

Pie Chart:
Used for: Showing parts of a whole (percentages)
Example: Market share of different mobile brands
Scatter Plot:

Used for: Relationship between two variables

Example: Relationship between study hours and marks

Rules for Good Visualization

1. Clear Title: Describe what the chart shows
2. Labeled Axes: Show what each axis represents with units
3. Readable Scales: Numbers should be easy to read
4. Appropriate Chart Type: Match chart to data type
5. No Misleading Distortion: Don't exaggerate or minimize values
6. Color Usage: Use colors to highlight, not confuse

Visualization Selection Guide

Data Type Best Visualization

Categorical comparison Bar chart
Numerical distribution Histogram
Trends over time Line chart
Parts of whole Pie chart
Relationship between variables Scatter plot
6. Data Exploration (Exploratory Data Analysis - EDA)
Definition
Data exploration or Exploratory Data Analysis (EDA) is the first detailed examination of
data after collection to understand its structure, patterns, and characteristics before
formal analysis.

Why EDA is Important

Identifies data quality issues early
Reveals patterns and relationships
Helps select appropriate analysis methods
Detects outliers and anomalies
Supports hypothesis formation

Steps in EDA
Step 1: Data Overview
Check data shape (number of rows and columns)
Examine first few rows
Check data types of each column
Example: 150 rows × 8 columns, 3 text columns, 5 numeric columns

Step 2: Summary Statistics

Calculate minimum, maximum, mean, median for numerical data
Example:
Student Marks Dataset:
Minimum: 25
Maximum: 98
Mean: 72.5
Median: 75
Step 3: Frequency Distribution

Count occurrences of each category

Example:
Grade Distribution:
A (90-100): 15 students
B (80-89): 35 students
C (70-79): 40 students
D (60-69): 20 students
Step 4: Visualization
Create histograms, bar charts, box plots
Identify visual patterns and distributions

Step 5: Outlier Detection

Find values far from the typical range
 Example: In a class with average salary 50,000, one person with 500,000 is an outlier
Step 6: Missing Value Check

Identify percentage and pattern of missing values

Decide handling strategy

EDA Workflow
RAW DATA
│
├─ Check types and structure
│
├─ Calculate summary statistics (Min, Max, Mean, Median)
│
├─ Create visualizations (Charts, graphs)
│
├─ Detect missing values
│
├─ Identify outliers
│
└─ Form hypotheses
│
└─ Ready for Analysis

Example: Student Performance Dataset

Original Data: 100 students, marks ranging from 20-98
EDA Process:

1. Overview: 100 rows, 1 numeric column (marks)

2. Summary: Min=20, Max=98, Mean=72, Median=75
3. Frequency: 20 students in 60-70 range, 35 in 70-80 range, etc.
4. Visualization: Histogram shows near-normal distribution
5. Outliers: One student with 20 (unusually low)
6. Missing: No missing values found
7. Hypothesis: Most students performed well (Mean > 70)

7. Data Analysis
Definition
Data analysis is the process of inspecting, transforming, and modeling data to discover
useful insights, draw conclusions, and support decision-making.
Types of Data Analysis
Descriptive Analysis:
What: Describes what happened
Methods: Averages, counts, percentages, charts
Example: Average monthly sales = 50,000 rupees
Diagnostic Analysis:

What: Explains why something happened

Methods: Comparison, correlation, regression
Example: Sales increased 20% because of new marketing campaign
Predictive Analysis:
What: Forecasts what may happen in future
Methods: Machine learning models, trend analysis
Example: Next month's sales will be approximately 60,000 rupees

Prescriptive Analysis:
What: Recommends actions to take
Methods: Optimization, simulation
Example: Increase advertising budget by 10% to boost sales further

Common Analysis Techniques

Regression:

Studies relationship between variables

Example: Relationship between advertising spend and sales revenue
Classification:
Categorizes data into classes/groups
Example: Email filtering (Spam vs. Not Spam)

Clustering:
Groups similar items together
Example: Customer segmentation by spending patterns
Time-Series Analysis:

Analyzes data over time to find trends

Example: Stock price movement over months

Data Analysis Process

CLEAN DATA
│
├─ Select analysis type (Descriptive/Diagnostic/etc.)
│
├─ Choose appropriate technique (Regression/Classification/etc.)
│
├─ Apply statistical or computational methods
│
├─ Calculate results and metrics
│
└─ Draw conclusions and make recommendations

Example: Sales Analysis

Business Question: Why did sales drop last month?
Analysis Steps:

1. Gather sales data for last 12 months

2. Compare current month with previous months
3. Investigate external factors (promotions, competition, holidays)
4. Identify correlation with marketing spend
5. Conclusion: Sales dropped due to lower advertising (Diagnostic)
6. Recommendation: Increase marketing to boost sales (Prescriptive)

8. Data Engineering
Definition
Data engineering is the practice of designing, building, and maintaining systems and
infrastructure that collect, store, process, and deliver data efficiently and reliably for
analysis and decision-making.

Key Responsibilities of Data Engineers

Build data collection systems
Design databases and data storage solutions
Create data pipelines (automated data flow)
Ensure data quality and security
Maintain data accessibility and performance

Components of Data Engineering

Data Sources:
Applications, websites, sensors, IoT devices
Databases, APIs, social media
Example: Mobile app sends user behavior data
Data Ingestion:

Collecting data from various sources

Methods: APIs, file uploads, log collection
Example: Collecting Twitter data through API
Data Storage:
Organizing and storing data
 Solutions: Databases, Data Warehouses, Data Lakes
Database: Structured, organized data (SQL)
Data Warehouse: Historical data organized for analysis
Data Lake: Large storage of raw data in any format
Data Processing:

Cleaning and transforming data

Removing duplicates, handling missing values
Combining data from multiple sources
Example: Converting temperatures from Celsius to Fahrenheit
Data Delivery:
Making processed data available to analysts
Creating reports, dashboards, APIs
Example: Dashboard showing daily sales

Complete Data Engineering Pipeline

DATA SOURCES (Apps, Sensors, Databases)
│
├─ INGESTION (Collect data)
│
├─ STORAGE (Database/Warehouse/Lake)
│
├─ PROCESSING (Clean, Transform)
│
├─ QUALITY CHECK (Validate data)
│
├─ DELIVERY (API, Dashboard, Reports)
│
└─ END USERS (Analysts, Data Scientists, Business)

Importance of Data Engineering

Enables reliable data flow for decision-making
Ensures data quality and security
Makes data accessible to all stakeholders
Supports scalability as data grows
Reduces manual work through automation

Example: E-Commerce Company

Sources: Customer clicks, orders, reviews from website
↓
Ingestion: API collects data every hour
↓
Storage: Stored in data warehouse
↓
Processing: Clean data, remove duplicates, combine with inventory
↓
Quality Check: Verify all fields have valid values
↓
Delivery: Dashboard shows daily sales, customer trends
↓
Use: Management uses insights for business decisions

Summary: Key Concepts Quick Reference

Topic Key Points
Types of Qualitative (nominal, ordinal) and Quantitative
Data (discrete, continuous)
Data Primary (direct) and Secondary (existing); Cross-
Collection sectional or Longitudinal
Data
Characteri Volume, Velocity, Variety, Veracity, Value + Outliers
stics
Missing Delete, Impute (Mean/Median/Mode), or use advanced
Values methods
Bar chart (categories), Histogram (distribution), Line
Visualizati
chart (trends), Pie chart (parts), Scatter plot
on
(relationships)
Examine structure, calculate statistics, visualize, detect
EDA
outliers, check missing values
Data Descriptive, Diagnostic, Predictive, or Prescriptive
Analysis approaches
Data
Collect, store, process, and deliver data through
Engineeri
automated pipelines
ng

Practice Questions (5-Mark Level)

1. Explain the difference between qualitative and quantitative data with examples.
2. What are primary and secondary data? Give examples of each.
3. Define missing values and explain three methods to handle them.
4. Why is data visualization important? Name four types of charts and their uses.
5. What is Exploratory Data Analysis (EDA)? What are its main steps?
6. Explain the types of data analysis (descriptive, diagnostic, predictive, prescriptive).
7. What are the 5 Vs of Big Data? Explain each with examples.
8. What is data engineering? Describe the complete data pipeline.
 9. What are outliers? How do they affect analysis?
10. Explain the concept of imputation using mean, median, and mode with examples.

End of Unit-II Data Exploration Notes

These notes are designed for 5-mark exam answers at beginner level with easy-to-understand
language, practical examples, and simple diagrams.