11/19/2025

Data Science Course

Data Mining Methodology

&
Data Processing

Objectives
1. Data mining process
2. Data source
3. Exploratory Data Analysis - EDA
4. Data Cleaning Techniques
5. Practise

1
 11/19/2025

Reminding

The DIKW model

Mô hình DIKW (Data-Information-Knowledge-Wisdom) Source: [Link] 3

Data Mining Method

Reference: Altaf, Saud & Asad, Rimsha & Ahmad, Shafiq & Ahmed [Link]. PhD, Iftikhar &
Abdollahian, Mali & Zaindin, Mazen. (2023). A Hybrid Framework of Deep Learning
Techniques to Predict Online Performance of Learners during COVID-19 Pandemic.
Sustainability. 15. 11731. 10.3390/su151511731.

The CRISP-DM (Cross-Industry Standard Process for Data Mining) process

* Shearer C., The CRISP-DM model: the new blueprint for data mining, J Data Warehousing (2000); 5:13—22. 4

2
 11/19/2025

1. Data Source

Types of data
• Structured data: organized in rows and columns in a table, easy to
store and query using SQL, Excel, etc.
• Semi-structured data: has a flexible structure, not a rigid schema.
For example, email, XML, JSON.
• Unstructured data: no clear structure, difficult to analyze directly.
Examples: free text, images, videos, social media posts

3
 11/19/2025

Common data file formats

• CSV, TSV: Comma or tab-separated text; easy to handle using Excel,
Python, R.
• Excel (.xlsx): Formatted and computable tabular data; popular in office
analytics.
• XML: Hierarchical data with user-defined tags; suitable for transmitting
data on the web.
• PDF: Fixed presentation format; used in reports, forms and legal
documents.
• JSON: Key-valued, lightweight and easy to analyze; popular in
programming and data exchange
7

Popular data sources

• Relational databases (SQL, Oracle, MySQL): Store structured data in
transactional, customer, and forecast systems.
• Text files, spreadsheets, XML: often used in surveys, statements, and
administration.
• API & Web Services: Retrieve data from external systems such as
finance, weather, stocks.
• Web Scraping: Extracting data from the web without an API; used to
collect content, prices, and train models.
• Streaming data: Real-time data from IoT, GPS, social networks;
processed with Kafka, Spark, Storm.
8

4
 11/19/2025

Internal data
• Data files from the company:
• Most of them come from internal operations.
• Usually in the form of spreadsheets, texts.
• Coming from the database when operating the software
• Relational databases: Oracle, MS SQL Server, MySQL,…
• NoSQL: document, key-value, wide-column, and graph (MongoDB,
Cassandra, …)
• NewSQL: CockroachDB, NuoDB, VoltDB
Comparison sample: [Link] 9

Public data
• Open Data Portals: [Link]
• UCI Machine Learning Repository: [Link]
• Google Dataset Search: [Link]
• Other Resources:
• GitHub

• Kaggle

• …

10

10

5
 11/19/2025

Collect data through surveys

• Collect feedback from users.
• Popular Tools:
• Google Forms
• SurveyMonkey:
• Jotform,
• Typeform
• ProProfs Survey Make
• …
11

11

Big Data and Challenges

Big Data is a term used to refer to huge, complex, and diverse data sets that
exceed the processing power of traditional tools.

Big data ?V 12

12

6
 11/19/2025

Big Data and Challenges (cont.)

• Tools for storing and processing large amounts of data and big
data analysis tools are popular today

13

13

2. Exploratory Data
Analysis - EDA

14

14

7
 11/19/2025

Introduction
• EDA – Exploratory Data Analysis is the process of describing data
using statistical and visualization techniques that focus on
important aspects of the data for further analysis.
• This includes examining the dataset from multiple angles,
describing and summarizing it without making any other
assumptions about its contents.
• EDA is an important step that needs to be taken before diving into
statistical modeling or machine learning.

15

15

EDA aims
• This EDA step gives us a first look at the data.
• It is necessary to have a certain sense of what you have in your hands before having a strategy to build a
model.
• EDA helps you visualize the complexity of the problem and map out the first steps to take.

• Data discovery doesn't stop at the first time before building a feature, but it also needs to be
done throughout the system development process.
• Once you've built the features, you'll also need to redo the EDA again to see if the processed data is really
clean.
• In addition, after building and analyzing the model, we also often need to return to EDA to continue to
discover what is hidden in the problem data.
• The deeper you understand the data; the sooner you'll be able to interpret the behavior of
the model and make appropriate changes.
16

16

8
 11/19/2025

Basic descriptive statistics

Example EDA – dataset titanic : [Link]

• Data size
• What each data field means
• Data types for each field
•"Pclass": seat class. 1 = Upper class, 2 = Middle class, 3 = Lower class. Thus, the
• Probability distribution of each field "Pclass" information field can be considered both a category characteristic and a
numerical feature because it is orderly. This feature is likely to affect the survivability
• Correlation between data fields of passengers because more luxury classes may have better safety measures (or
vice versa, more subjective).
•"Sex": Passenger Gender.
•"Age": the age of the passenger. If the age is less than 1, it is in the form of an odd
number (0.42), if the age is an estimate, it is in the form xx.5. Obviously, this will also
be a potential feature to predict the outcome of the problem because children and the
elderly are in the higher risk group.
•"Sibsp": the number of siblings or spouses on board the ship.
•"Parch": Number of parents/children on board.
•"Ticket": Ticket number.
•"Fare": Fares.
Examples for descriptive statistics •"Cabin": Cabin code.
•"Embarked": Where to board, C = Cherbourg, Q = Queenstown, S = Southampton.
Library Referenceydata-profiling
17

17

3. Data Cleaning
Techniques

18

18

9
 11/19/2025

Introduction
• Data cleaning is the process of correcting or removing inaccurate,
incorrect format, duplicate, irrelevant, incomplete, or problematic
data in a database.
• Effective data cleaning will ensure that the data is consistent and
free of any errors that can cause problems during later use or
analysis.

19

19

Data quality issues

• Data can be problematic, especially when businesses combine data from many different
sources. Some problems in the quality of the data can be examples such as:
• Duplicate data: There are two or more records with identical information. This can cause bias in
numbers.
• Conflict data: When the same record produces two data streams with two different attributes, the data is
now in a conflicted state. For example, a company has only one headquarters but has two different
addresses in the database, which can be difficult for shippers.
• Incomplete data: Data that is missing some attributes. For example, an employee's payroll does not have
data on individual social security numbers.
• Invalid data: A data attribute that doesn't conform to established data standards. For example, phone
numbers are saved with 9 digits instead of 10 digits.

20

20

10
 11/19/2025

Data quality issues (cont.)

• Cause:
• Synchronization issues: This problem occurs when data is transmitted between
two systems but not shared in an appropriate manner. For example, if the bank's
sales software accepts a new mortgage but doesn't update it to the bank's
marketing system, if the marketing department is still sending notifications to
customers, it can make them feel confused.
• Errors in data processing software: Data processing software may have errors
when storing faulty data in the system, overwriting correct data, duplicate data,...
• Obfuscated information by users: Users may intentionally provide incomplete or
inaccurate data to protect their privacy.
21

21

Handling missing data (1/4)

1. Delete data
• Delete rows (called listwise deletion): Remove all rows that contain missing
values.
• Pros: Simple and easy to implement.

• Cons: Can lose a lot of important information and reduce statistical power if too
many rows are deleted.

• Delete columns: Remove all columns with a high percentage of missing

values.
• Pros: Helps simplify the model.

• Cons: Important features can be mistakenly removed.

22

22

11
 11/19/2025

Handling missing data (2/4)

2. Imputation
• Use of statistical values:

• Mean: Replace the missing value with the average of the remaining values in the column (consistent with continuous data and
standard distribution).

• Median: Substitutes with the median of the remaining values in the column, which is especially effective when the data has outlier
values (consistent with continuous data).

• Most Common Value (Mode): Replace with the most occurring value, usually used for categorical variables.

• Assign a fixed value: Replace the missing value with a predefined value (for example, or another value).

• Use advanced methods:

• Regression: Use a regression model to predict and fill in missing values based on other columns.

• KNN Imputation: TInsert the nearest data points (K-Neighbors) to the missing points and use their values to fill in the blanks.

• Multiple Imputation: Create multiple datasets with values, analyze them on each dataset, and then combine the results to reduce
bias.

23

23

Handling missing data (3/4)

The method of substituting missing values usually yields relatively good results.
However, the values assigned may be higher or lower than the actual value. Or
sometimes, these missing values contain some important signal.

3. Create new variables

• Create a new column to highlight which lines have missing values. This
retains information that the value was once missing, rather than just
filling in with no signal.

24

24

12
 11/19/2025

Handling missing data (4/4)

• Method selection criteria
• Cause Analysis:

• Understanding why data is missing (random, systematic) will help choose a more appropriate method.

• Missing data rate:

• If the data missing rate is low, it makes sense to delete the row or replace it with a statistical value. If the
deficiency rate is high, more complex methods should be considered.

• Objectives and Resources:

• Choose the method based on the requirements for accuracy, project objectives and available knowledge and
resources.

• Trial:

• No method is perfect, it is necessary to test different methods to find the best solution for your dataset.

Read more at: [Link]

25

25

Handling Duplicate Data (1/3)

• Concept:
• Duplicate data is any record or row in a dataset that is
identical or has the same value for some or all variables.
• Cause:
• Human error, data entry error, data consolidation problem, or
data collection error,….
• Classify:
• Exact Duplicate: are records that have the same value for all
variables,
• Near Duplicate: are records that have the same value for some
but not all variables.
26

26

13
 11/19/2025

Handling Duplicate Data (2/3)

• Identify duplicate data:
• There are many different methods and tools to do this, depending on the type and format
of your data.

• Example:
• If you're working with spreadsheets or tables, you can use the software's built-in features,
such as Excel's Remove Duplicates or Google Sheet's Unique, to highlight or filter
duplicate rows.

• If you're working with a programming language, such as Python or R, you can use libraries
or packages, such as pandas or dplyr, to detect and remove duplicate rows in your data
frame.

27

27

Handling Duplicate Data (3/3)

• Handling:
• Depending on the type and origin of the duplicate data, the purpose and scope of the analysis, and the
potential impact of duplicate data on the results, there are various options and strategies.

• Deleting duplicate rows can reduce the size and complexity of the dataset, but it can also cause bias. Retaining only one
instance of duplicate rows requires you to choose which instances to retain and remove.

• Aggregating or averaging duplicate rows can reduce noise or outliers in the data but may not properly reflect the true
distribution or variation.

• Modifying or merging duplicate rows can improve or enrich data but can also cause errors or inconsistencies.

• Therefore, when making decisions about how to handle duplicate data, you should carefully consider all
of these options and their potential implications.

28

28

14
 11/19/2025

Data Transformation

29

29

Data Transformation
• It is the process of processing raw data, including cleaning, restructuring, and adding information so
that it becomes accurate, consistent, and suitable for analysis, reporting, or other purposes.
• In data analytics, data transformation plays an extremely important role for the following reasons:
• Improved data quality: Data after conversion eliminates errors, outliers, and inconsistencies, making the data cleaner
and more accurate for further analysis.
• Data Normalization: Data transformation creates uniformity in format, units of measurement, and data structure,
making it easy to compare and combine data from a variety of sources.
• Increased analysis efficiency: New variables created through transformation techniques provide additional
perspectives, helping to uncover hidden relationships, trends, or data patterns that were not previously exploited.
• Optimize system performance: Standardized and properly grouped data will minimize the processing time and storage
space required, especially when deployed on platforms such as cloud servers or hosting services that P.A Vietnam
provides.

30

30

15
 11/19/2025

Common data transformation techniques (1/4)

1. Data Normalization: is one of the essential data transformation techniques in data preprocessing.
• The main goal of normalization is to transform raw data with different ranges and units of
measurement into values in a uniform scale, usually between 0 and 1 or -1 to 1.
• This increases the efficiency of analytical algorithms and machine learning models, and limits errors due to differences
in scales or units between variables.
• Common normalization methods include:
• Min-Max Scaling: Normalize the data to interval [0.1] based on the smallest and largest values of the data set.
• Z-score Standardization: Transform data based on mean and standard deviation, so that the data has a standard
distribution with a mean of 0 and a variance of 1.
• Decimal Scaling: Decimal shift of the data so that the value is scaled down to a certain range.
• Applying standardization correctly not only improves the convergence speed of algorithms, but also
improves the accuracy of the data analysis model.

31

31

Data Normalization Techniques

1.1. Min-Max Normalization: rescales a feature to a specific range, typically [0, 1]:
𝑋 − 𝑋𝑚𝑖𝑛 The minimum value maps to 0
𝑋𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑠𝑒𝑑 =
𝑋𝑚𝑎𝑥 − 𝑋𝑚𝑖𝑛 The maximum value maps to 1
Other values are scaled proportionally
𝑣
1.2. Decimal Scaling: normalizes data by shifting the decimal point of values: 𝑣 ′ = 10𝑗
j is the smallest integer such that the maximum absolute value of v′ is less than 1
1.3. Logarithmic Transformation: compresses large values and spreads out small values:
Reduces skewness in data
𝑋 ′ = log(𝑋 + 1) Stabilizes variance across features

𝑋
1.4. Unit Vector (Vector) Normalization: scales a data vector to have a magnitude of 1: 𝑋 ′ =
𝑋

𝑋 = 𝑋12 + 𝑋22 + ⋯ + 𝑋𝑛2

• Commonly used in text mining and machine learning algorithms like KNN

• Preserves direction but normalizes magnitude

32

32

16
 11/19/2025

Common Data Transformation Techniques (2/4)

2. Data Grouping/clustering is the process of classifying or breaking down data into homologous groups
based on certain characteristics or criteria.
• Purpose: This technique is often used to simplify data, detect trends, or support group analysis
methods such as clustering.
• The implications of data aggregation are important in exploring the intrinsic structure of data:
• Support statistical analysis by target group.
• Increased clarity and interpretability of analysis results.
• Improve processing performance with large data sets by working as a team.
• Common grouping techniques include:
• K-means clustering: Clustering data by optimizing the distance between data points and the center of the group.
• Hierarchical Clustering: Creates a hierarchical tree that groups data from the general to the detailed level.
• Group gathering based on Association Rules.

33

33

Common Data Transformation Techniques (3/4)

3. Feature Engineering (creating new features): is an important step in the data transformation process,
which enhances data representation so that analytical models can learn more accurately and
efficiently.
• Reasons to create a new variable include:
• Deepen your exploitation of the hidden features in the original dataset.
• Help the model better simulate complex relationships.
• Minimize data shortages by aggregating information from multiple sources.
• How to create new variables effectively:
• Combine variables: For example, create aggregate variables such as total revenue, average number of visits.
• Mathematical transformation: Such as logarithms, squares, or square roots to handle data with skewed distributions.
• Reclassify digital distortions into groups (binning) to reduce noise and increase comprehensibility.
• Extract chronological features, such as days, months, years, or special time periods.

34

34

17
 11/19/2025

Common Data Transformation Techniques (4/4)

4. Convert data formats: is an indispensable step in the data preparation process when
collecting diverse data sources in different formats such as CSV, JSON, XML, or image and
audio data.
• Common format conversion types include:
• Data type conversion: For example, from series to number, from date type to string, or vice versa.
• Convert data structures: Change flat data tables to hierarchical or pivot tables.
• Standardize the date and time format to ensure consistency in calculations or comparisons.
• Important notes when converting data formats:
• Make sure not to lose data or distort information.
• Double-check the data after conversion to detect potential errors.
• Make sure the final format is compatible with subsequent analysis or storage systems.

35

35

Outliers

36

36

18
 11/19/2025

Introduction
• In numerical form, the outlier data can be an unrealistic value
such as a negative age number, or a value that is far from the rest
of the values in that field.
• In the form of a categorical, the outlier data can be an unrealistic
value as a category that is outside the possibilities of a landmark
that is not on the map.
• Values with an extremely low frequency of occurrence in a data
column are also likely to be an outlier.

37

37

Identify outlier data

• Groups of outlier values:
• The values are not in the defined domain of the data. For example, age, income, or distance
cannot be negative.
• The values are likely to occur, but the probability is very low. For example, 120 years old, earn
$1 million/month. These values are likely to occur but are really rare.
• Handling:
• We can always delete columns or rows with exception data that waste data.
• For data in the first group, we can replace it with NaN and treat it as a missing value as the
missing data.
• For data in the second group, it is common to use the upper or lower blocking method
(clipping or capping).
38

38

19
 11/19/2025

Handling outlier values

• The most common ways:
• Using Box plot: help determine if the data has an outlier and also help find the largest and smallest
thresholds to use as cutoff points.
• Z-score: If you believe that the values in a data field follow the standard distribution, you can apply the
3𝜎 rule to the standard distribution.
• Assuming μ is the expectation and σ is the standard deviation, the 3σ rule for the standard distribution assumes that:

• 68% of data points are within 𝜇 ± 𝜎

• 95% of data points are in the range of 𝜇 ± 2𝜎
• 99.7% of data points are in the range of 𝜇 ± 3𝜎

𝑥−𝜇
• For a data point x, its z score is calculated by:
𝜎

• Points with z scores outside of the [−3,3] paragraph can be considered as outliers. A slight mathematical variation, this is
equivalent to the fact that points outside the [𝜇 − 3𝜎, 𝜇 + 3𝜎] segment are considered outliers..

39

39

Handling Outliers - Boxplox

• A box plot, or box and whisker chart, is a statistical graphical tool used to show the distribution of data through five
main values: the smallest value (min), the first quartile (Q1), the median, the third quartile (Q3), and the maximum
value (max). This chart helps visualize volatility, bias of data, and highlights outliers.

• Structure of Box plot

• Box: The length of the box represents the range of 50% of the central data, including Q1 to Q3.
• The line inside the box: Represents the median value.
• Outline of the box ("Whiskers"): Represents the range of data values outside of Q1 and Q3.
• Outliers: Data points outside the "whiskers" are considered outliers and are represented by separate dots.

40

40

20
 11/19/2025

Handling Outliers - Z-score

• Although Z-Score is a highly effective
method for detecting and eliminating
outlier values, we cannot use it with
every type of data.
• Khi nói như vậy, chúng tôi muốn nói
rằng nó chỉ hoạt động với dữ liệu hoàn
toàn hoặc gần phân phối chuẩn, điều
này cho thấy phương pháp này không
phù hợp với dữ liệu lệch , dù lệch
trái hay lệch phải.
41

41

Handling Outliers - Z-score

• Although Z-Score is a highly effective
method for detecting and eliminating
outlier values, we cannot use it with
every type of data.
• In saying that, we mean that it only
works with data that is completely or
near the standard distribution, which
suggests that this method is not
suitable for skewed data, whether
skewed left or right.
42

42

21
 11/19/2025

Summary
• DIKW Model
• Data mining methodology
• Data types, data sources, and data approaches.
• Exploratory Data Analysis (EDA)
• Some data cleaning techniques

43

43

PRACTISE
Using Orange Data Mining

[Link]

44

44

22
 11/19/2025

Requirements
• Purpose: Practice for data preprocessing using Orange tools
• Estimated Time: 90 minutes
• Form: team-work
• Workflow:
1. Understanding information (meta-data) about datasets horse-colic.
2. Practice with the Orange widgets File, FileInfo, Table, Features Statistics, Pivot,
Scatter Plot, …
3. Find out at least 3 assertions (information) in this dataset.
4. Conducting explanations.
45

45

Descriptive Statistics
• What information is
found from this step?
• Making an arguments.
• Form:
• Widget: ?
• Gained information: ?
• Explanation: ?

46

46

23
 11/19/2025

Feature Engineering
• Observe and make
comments.
• Do some formulas to
create new variables.
• Form:
• New variable: ?
• Formula: ?
• Explanation: ?
47

47

Handling missing and duplicate data

• Observe and make
comments.
• Discuss / argue about
these comments.

48

48

24
 11/19/2025

Normalize and transform data

• Observe the results, comment

and argue.

49

49

Data grouping

• Observe the results, comment

and argue.
• Make group by selections,
operations on other features
then comment and discuss.

50

50

25
 11/19/2025

Outlier Values Handling

• Observe the results, comment, discuss.

• Choose different strategies (algorithms) as well as the percentage of
outliers that are eliminated and then observe and discuss.

Đọc thêm tại: [Link]

51

51

Data transformation
• Perform complete preprocessing of the dataset.

• Observe and Discuss this outcome.

A sample of normalization process

52

52

26
 11/19/2025

Advanced Exercises
• Purpose: Practice for data
preprocessing using Orange tools
• Estimated Time: 180 minutes
• Form: Teamwork
• Workflow:
1. Find a dataset that is relevant to the student's major. Learn meta-data.
2. Consider dataset scenarios→Choose ways to clean your data. Arguments for
this.
3. Present the comments as a word file and submit them to the LMS system.
53

53

References
1. Andrew Park, Data Science for Beginners,ISBN 978-1914167508,
Independently Published, 11 November 2020
2. [Link]
3. [Link]
4. [Link]
5. [Link]
6. [Link]
7. …
54

54

27
 11/19/2025

Vietnam
[Link]

Thank You /

55

55

28