Unit I –

Data Analysis
Pipeline

By: Er. Aashish Nepal

 Learning Objectives
Students will be able to:

● Define data and data science in precise terms.

● Describe the evolution from raw data to knowledge (KDD).

● Identify different types of data.

● Explain each stage of data processing in detail.

● Recognize the role of preprocessing and transformation in reliable analytics.

What is Data?
Definition:

Data are raw facts, figures, or symbols that represent ideas, objects, or events. They lack meaning
until they are processed or interpreted.

Explanation:

● Data can be quantitative (numbers, measurements) or qualitative (labels, text, descriptions).

● Data are the building blocks of information. When organized, processed, and contextualized, data become
information; further interpretation produces knowledge.

● Example:

○ Raw data → “20°C”

○ Information → “Room temperature is 20°C.”

○ Knowledge → “A 20°C room is comfortable for humans.”

 Mini Quiz 1
1. True or False: Data and information are the same thing.

2. Which of the following is not data?

a) Temperature readings b) Customer reviews c) Insight
about sales pattern d) Sensor outputs

3. Give one example of qualitative and one of quantitative data.

 What is Data Science?
Definition :

Data Science is the multidisciplinary field that uses scientific methods, algorithms, and
systems to extract knowledge and insights from structured and unstructured data.

Expanded Explanation:

● Combines elements of statistics, computer science, mathematics, and domain

knowledge.
● Focus: making data-driven decisions using computational models.
● Core tasks: data collection, cleaning, analysis, visualization, and interpretation.
● Data Science = Data + Algorithms + Insights + Decisions.
● Real-world examples:
○ Netflix recommending shows
○ Weather forecasting
○ Predicting disease outbreaks

“In your daily life, where have you seen data science in action?”
History of Data & Knowledge Discovery

● Humans have gathered data for millennia (agricultural calendars, census, trade).

● 1960s–70s: Statistical data analysis became automated (SPSS, SAS).

● 1989: Gregory Piatetsky-Shapiro introduced the term KDD – Knowledge Discovery

in Databases.

● 1990s: Data mining gained prominence for uncovering hidden patterns.

● Today: Big Data, IoT, and AI have made data science central to decision-making.
 What Is a Data Pipeline?
A data pipeline is a series of automated processes that move data from one or more sources
to a destination system for storage, analysis, or visualization.

In simpler terms:

A pipeline collects raw data → cleans it → transforms it → loads it where analysis happens.

Core Idea:
It connects every stage of the KDD process in an operational, repeatable way.

Analogy: Like a water pipeline — raw data flows through pipes (processes) until it becomes
clean, useful information.
 Stage Description Common Tools / Examples

1. Data Sources Where data originates (sensors, APIs, IoT sensors, CRM systems
databases, logs).

2. Ingestion Layer Collects and transfers raw data into the pipeline. Kafka, Airbyte, custom
scripts

3. Storage Layer Holds raw or processed data. MySQL, Hadoop, Cloud

Storage

4. Processing Layer Cleans, transforms, aggregates data. Spark, Pandas, Airflow

5. Analytics / ML Layer Performs data mining and modelling. Python, R, scikit-learn

6. Visualization / Reporting Presents knowledge to users. Power BI, Tableau,

Matplotlib
 What is the KDD?
KDD is the process of discovering valid, novel, and potentially useful patterns from data, transforming raw
data into knowledge.

KDD Steps (detailed):

1. Selection – identify relevant data sources.

2. Preprocessing/Data Cleaning – remove noise, handle missing data.

3. Transformation – normalize, aggregate, or derive features.

4. Data Mining – apply algorithms (classification, clustering, etc.).

5. Interpretation/Evaluation – verify patterns, visualize insights.

Key Idea:
KDD = Data → Information → Knowledge.
Where “data cleaning” and “feature selection”happens?
 Types Of Data
1. Structured Data

● Organized in rows/columns (databases, spreadsheets).

● Easy to store, query (SQL).

2. Semi-Structured Data

● Not fully tabular; includes tags or metadata (JSON, XML, logs).

3. Unstructured Data

● No predefined format (text, images, audio, video).

4. Mixed / Multi-Modal Data

● Combination of multiple data types (e.g., image + text + sensor readings).

 Example Table:

Type Example Storage Analysis Method

Structured Sales records Database SQL, Pandas

Semi-structured JSON logs NoSQL Parsing, mapping

Unstructured Tweets, images Data lake NLP, CNNs

 Match the following:

a) JSON → ___
b) Customer table → ___
c) Audio recordings → ___
d) Chat transcripts → ___

(Answers: semi-structured, structured, unstructured, unstructured)

 Overview of Data Processing
Data processing converts raw inputs into meaningful outputs.

Phases:

1. Data Collection: sensors, surveys, APIs.

2. Data Cleaning: handle errors, missing values.

3. Data Integration: merge sources into a single dataset.

4. Transformation: convert formats, normalize, discretize.

5. Data Reduction: remove redundancy, apply dimensionality reduction.

6. Data Analysis / Mining: apply algorithms to find patterns.

7. Interpretation & Presentation: visualizations, reports, dashboards.

Example:
Bank transaction data → cleaned → grouped by customer → analyzed for fraud → visualized for managers.
 Mini Quiz 4
1. What is the goal of data cleaning?

2. Give two examples of data transformation methods.

3. Which step ensures the model’s outputs are

understandable to non-experts?
 Real-World Connection

How this pipeline applies — e.g., milk-collection analytics:

Collect daily data from sensors & agents.

Clean and transform fat percentage & volume data.

Integrate region and weather data.

Analyze seasonal patterns & forecast demand.

Present results via dashboards to cooperative managers.

 Data Cleaning — “Making data trustworthy”
Purpose:
To detect and correct errors, inconsistencies, and missing values in datasets so that analysis reflects reality.

Common Issues Found:

● Missing values (NaN, blanks)

● Outliers or extreme values
● Duplicate records
● Inconsistent formats (e.g., “Nepal”, “NPL”)
● Measurement or entry errors

Typical Techniques:

● Imputation: Fill missing values (mean, median, mode, regression, interpolation).

● Outlier handling: Remove, cap, or transform outlier values.
● Deduplication: Identify and remove repeated entries.
● Standardization: Convert units, names, date formats to a consistent scheme.

Key Idea:

Clean data = accurate conclusions. No cleaning → garbage in, garbage out.

Example:
Replacing missing humidity readings in an IoT dataset with median regional humidity.
Data Integration — “Bringing data together”
Purpose:
To merge data from multiple sources into a single coherent dataset for analysis.

Challenges:

● Different data formats or naming conventions.

● Conflicting values (e.g., two systems show different customer ages).
● Duplicate or overlapping records.
● Keys that don’t match exactly (e.g., “Cust_ID” vs. “CustomerNumber”).

Techniques:

● Schema matching: Align table structures and field names.

● Entity resolution: Identify and merge duplicate entities across datasets.
● Data fusion: Combine values from multiple sources into one trusted record.
● Data warehousing: Centralize storage for integrated data (OLAP systems).

Example:
Combining milk-collection records from multiple cooperatives into one national database.

Key Idea:

Integration creates a “single version of truth” for all subsequent analysis.

Data Transformation — “Shaping data for analysis”
Purpose:
To convert raw, cleaned data into formats or structures suitable for models and algorithms.

Types of Transformations:

● Normalization / Standardization: Scale features so they share a common range (e.g., 0–1 or z-scores).

● Aggregation: Summarize detailed data (daily → monthly totals).

● Encoding: Convert categorical variables into numeric form (one-hot, label encoding).

● Feature construction: Derive new variables from existing ones (e.g., BMI = weight/height²).

● Log/Box-Cox transformations: Reduce skewness in data distributions.

Example:
Transforming temperature data from Fahrenheit to Celsius, scaling it, and creating a “heat index” feature.

Key Idea:

Transformation tailors data to the needs of algorithms—bridging raw information and mathematical models
Discretization — “Turning numbers into categories”
Purpose:
To convert continuous numeric variables into discrete categories or intervals, simplifying analysis and improving
interpretability.

When Useful:

● For decision-tree or rule-based models that handle categorical features better.

● When patterns appear clearer in grouped form.

Techniques:

● Equal-width binning: Divide data range into equal intervals.

● Equal-frequency binning: Ensure each bin has roughly the same number of observations.
● Clustering-based discretization: Use K-means or similar to form natural groupings.
● Domain-driven binning: Apply expert-defined thresholds (e.g., “Low Fat < 3%”, “Medium 3–4.5%”, “High > 4.5%”).

Example:
Converting age (continuous) into “Young (18–30)”, “Adult (31–50)”, “Senior (51+)”.

Key Idea:

Discretization simplifies relationships, enhances interpretability, and can reduce model noise—but at the cost of
some detail.
 Step Goal Typical Tools Result

Cleaning Fix errors & Pandas, Excel, Reliable data

inconsistencies OpenRefine

Integration Combine sources SQL joins, ETL tools Unified dataset

coherently

Transformation Format & engineer Python/R, Spark Model-ready

features data

Discretization Simplify continuous Pandas cut(), binning Readable

variables categories
 Summary & Takeaways

Data are raw facts; information and knowledge emerge through

processing.

Data Science uses computational and statistical methods to extract

insights.

KDD provides the structure for discovery.

Understanding data types and processing steps is the foundation for all
later units.
 Homework!
1) Define “data” and “data science” in your own words.

2) List the five steps of the KDD process.

3) Distinguish between structured and unstructured data.

4) Why is preprocessing said to take 60–70% of a data science project’s

time?

5) Describe a scenario where poor data cleaning led to misleading

conclusions.
Enjoy!