DATA

PROGRAMMING
데이터 프로그래밍
 I. Data Science & Programming 6
C I-1. Data Science 8

O
I-2. Data Programming 23

II. Basic Python Programming 37

N II-1. Introduction to Python

II-2. Control Structures
39
50

T II-3. Data Structures

II-4. Programming Paradigm
61
77

E III. Python Libraries - Numpy and Pandas 89

N
III-1. NumPy 91
III-2. Pandas 124

T IV. Data Collection

IV-1. Crawling
154
157

S IV-2. Using API 169

 V. Data Preprocessing 201
C V-1. Data Cleaning 208

O
V-2. Data Transformation 251

VI. Data Visualization 283

N VI-1. Matplotlib
VI-2. Seaborn
285
336

T VII. Regression Analysis 350

E VII-1. K-Nearest Neighbors (KNN)

VII-2. Linear Regression
356
369

N VII-3. Polynomial Regression

VII-4. Project
382
398

T VIII. Classification (1) 403

S VIII-1. Classification Model Evaluation

VIII-2. K-Nearest Neighbor Classification
405
419
VIII-3. Logistic Regression 433
VIII-4. Naive Bayes Classification 443
 IX. Classification (2) 464
C IX-1. Decision Tree 466

O
IX-2. Ensemble Learning 490
IX-3. Random Forest 499
IX-4. Project 518

N X. Clustering 525

T X-1. Introduction
X-2. k-Means
527
529

E X-3. Hierarchical Clustering

X-4. DBSCAN
541
552

N
X-5. Project 564

XI. Text Mining (1) 569

T XI-1. Introduction
XI-2. Text Preprocessing
571
572

S XI-3. Bag of words

XI-4. Text classification
585
592
 XII. Text Mining (2) 600
C XII-1. Sentiment Analysis 602

O
XII-2. Topic Modeling 615
XII-3. Documents Similarity 635
XII-4. Project 640

N XIII. Recommendation System 645

T XIII-1. Introduction
XIII-2. Content-based Filtering
647
648

E XIII-3. Collaborative Filtering

XIII-4. Project
656
682

N
T
S
 Ⅴ
CHAPTER

Data Preprocessing
V. Data Preprocessing
Objectives
 Grasp what Data Preprocessing is and its importance for analysis.

 Learn to filter data and remove unwanted information efficiently.

 Study how to handle outliers to reduce their analysis impact.

 Understand Missing Value treatment to improve dataset integrity.

 Learn to address Imbalanced Data for more precise models.

 Acquire Scaling methods for consistent feature comparison.

 Explore Categorical Variable Encoding to convert non-numeric into numeric data

for analysis.

DATA-PROGRAMMING 7
V. Data Preprocessing
Data Preprocessing
 Data Preprocessing
• Data preprocessing is an essential step performed before starting data analysis.
• Raw or unstructured data (text, images, audio, video, documents, etc.) cannot be directly input
into a machine learning model ⟹ The data must be preprocessed to make it usable.
 The goal is to prepare the data for analysis in a suitable state.

Data Data Data Model Model

Collection Preprocessing Exploration Building Evaluation

Process of Data Analysis

DATA-PROGRAMMING 8
V. Data Preprocessing
Data Preprocessing
 Why Data Preprocessing?
• Real-world data can generally be considered “dirty data,” which indicates data quality issues.

 Characteristics of real world data

1) Incomplete (불완전성)
• Incomplete refers to a condition in which the overall data set lacks completeness because
important parts of the data are missing.
2) Noisy (잡음이 섞임)
• Noisy refers to a state in which the accuracy or usefulness of data is reduced because the data
contains errors or outliers (이상치).
3) Inconsistent (비일관성)
• Inconsistent refers to a state in which information is contradictory (모순) or inconsistent
(일관되지 않은) within a data set or across multiple data sources.

DATA-PROGRAMMING 9
V. Data Preprocessing
Data Preprocessing
1) Incomplete (불완전성)
• Missing Values (누락된 값)
 ex) when some respondents in a survey
do not answer certain questions.

• Gaps in Time Series Data

(시계열 데이터의 간격)
 ex) Some time intervals are missing from
the observed data sequence.

• Inconsistencies in Merged Datasets

(결합된 데이터셋의 불일치)
 ex) In datasets joined from different
sources, some fields are present in one id age height
dataset but not in the other.
Kim 20 166 id age height weight

Kim 20 166 ?

id age weight Lee 23 ? 52

Lee 23 52

DATA-PROGRAMMING 10
V. Data Preprocessing
Data Preprocessing
2) Noisy (잡음이 섞임)
• Sensor Errors (센서 오류)
 ex) When the monitoring sensor records
inaccurate data due to failure or external
factors.

• Typos and Input Errors (오타 및 입력 오류)

 ex) Mistakes that occur when users
manually enter information. Personal Info

• Background Noise (배경 잡음) Age: ‐24

Submit
 ex) Including background noise or
unnecessary sounds in audio recordings.

[Link]
[Link]
DATA-PROGRAMMING 11
V. Data Preprocessing
Data Preprocessing
3) Inconsistent (비일관성)
• Inconsistent unit of measure from different data sources (측정 단위 불일치)
 Different unit of measure for the same item.
id Sales($)

5166 100 id Sales

5166 100

id Sales(\) 5248 10,000

5248 10,000

• Inconsistencies in data formats and rules (데이터 포맷 및 규칙의 불일치)

 When the same type of data is recorded in different formats.
id birthday

1234 10/05/2001 id birthday

1234 10/05/2001

id birthday 2345 2002-03-20

2345 2002-03-20

DATA-PROGRAMMING 12
V-1. Data Cleaning
Data Cleaning
 Data Cleaning
• Data Cleaning is a very important process in data analysis and data science.
• The purpose of this process is to modify or remove inaccurate or inappropriate information
(부적절한 정보) from the data set to prepare the data for analysis.

Powerful Machine
Learning Models

Garbage Garbage
Data Result

• Incomplete
• Noisy
• Inconsistent

DATA-PROGRAMMING 13
V-1. Data Cleaning
Exercise Data
 Employees datasets
• Various employees datasets exist on the internet. A dataset has been selected to practice
different data cleaning tasks.

 Preparation of data from Github

• [Link]

 First Name: The given name of the employee.

 Gender: The gender of the employee (Male/Female).
 Start Date: The date on which the employee started working at the company.
 Last Login Time: The time at which the employee last logged into the company's system.
 Salary: The annual salary of the employee in US dollars.
 Bonus %: The percentage of the salary that is given as a bonus.
 Senior Management: A boolean value (true/false) indicating whether the employee is part of the senior
management team.
 Team: The department or team within the company where the employee works.

DATA-PROGRAMMING 14
V-1. Data Cleaning
Exercise Data
 Look at the data that will be used in the data cleaning exercise.
In [1]: import pandas as pd
df = pd.read_csv('[Link]')
[Link]()
Out[1]:

• df = pd.read_csv('[Link]'): This line reads data from a CSV file named '[Link]'
and stores it in a 'df'. The pd.read_csv() function is used to read data from a CSV file and create
a DataFrame.
• [Link](): This line displays the first few rows of the 'df' using the .head() method.

DATA-PROGRAMMING 15
V-1. Data Cleaning
Exercise Data
 Obtain information about a DataFrame.
In [2]: [Link]()

Out[2]:

• [Link](): This code is used to obtain information about a DataFrame 'df.‘

• It presents a summary of the DataFrame's structure and offers details regarding the columns
and their data types, along with memory usage.
• The total number of rows in the DataFrame.
• The name of each column.
• The number of non-null values in each column.
• The data type of each column.
• Memory usage information.

DATA-PROGRAMMING 16
V-1. Data Cleaning
Dtype of Dataframe
 Dtype (Data Type) refers to the type of data stored in each column of a DataFrame.

Parameter Description
Typically represents string data.
object
Data that contains text or strings is classified under this type.
int64, int32, int16 Data of integer type.
float64, float32 Data of floating-point type.
Data of Boolean type.
bool
Used for data columns with values of True or False.
datetime64 Data type used to represent dates and times.
timedelta[ns] Data type used to represent time intervals.

DATA-PROGRAMMING 17
V-1. Data Cleaning
Exercise Data
 Look at summary statistics for the numeric columns
In [3]: [Link]()

Out[3]:

• .describe(): This method shows summary statistics for the numeric columns.
 Count: The number of non-null (non-missing) values in each column.
 Mean: The arithmetic mean value of the column.
 Std: The standard deviation, which measures the dispersion or spread of the data.
 Min: The minimum value in the column.
 25%: The 25th percentile value (first quartile).
 50%: The 50th percentile value (median or second quartile).
 75%: The 75th percentile value (third quartile).
 Max: The maximum value in the column.

DATA-PROGRAMMING 18
V-1. Data Cleaning
Exercise Data
 Look at summary statistics for the string columns.
In [4]: [Link](include='object')

Out[4]:

• .describe(): This method describes a statistical summary of columns of object data types
(categorical or textual information ) when include='object'.
 count: Indicates the number of non-null values in each column, representing the count of
valid data points in that column.
 unique: Represents the number of unique values in each column, displaying the count of
different unique values in that column.
 top: Represents the most frequently occurring value (mode) in each column, showing the
most commonly appearing value in that column.
 freq: Indicates the frequency with which the value shown in 'top' appears in the dataset,
revealing how often the most common value occurs.

DATA-PROGRAMMING 19
V-1. Data Cleaning
Data Filtering
 Data Filtering
• Data filtering is the process of selecting data from a dataset that is highly relevant or meets
specific criteria.
• It involves removing unnecessary or noisy information to improve the quality of data and
increase the accuracy of analyses.
• Data filtering is a crucial step in data preprocessing and is essential for various data science
tasks.
Data Filtering
Name Gender Start Date Last Time Salary Name Gender Salary

Douglas Male 8/6/1993 12:42 PM 97308 Douglas Male 97308

Row Filtering
(Gender=‘Male’)
Thomas Male 3/31/1996 6:53 AM 61933 Thomas Male 61933

Maria Female 4/23/1993 11:17 AM 130590 Jerry Male 130590

Jerry Male 3/4/2005 1:00 PM 130590

Larry Female 1/24/1998 4:47 PM NaN

Column Filtering
(‘Name’, ‘Gender’, ‘Salary’)

DATA-PROGRAMMING 20
V-1. Data Cleaning
Column Filtering
 Select only the 'First Name', 'Gender', and 'Salary' columns through column filtering.
In [5]: #Filter columns
[Link](['First Name', 'Gender', 'Salary'])
Out[5]:

• `[Link](['First Name', 'Gender', 'Salary'])`: A filter() method is used with a pandas DataFrame
object (`df`). It filters the columns of the DataFrame, selecting only the columns named 'First
Name', 'Gender', and 'Salary'.
• After this operation, the resulting DataFrame will include only these specified columns,
excluding all others.
• This method is useful for narrowing down a dataset to only the information of interest.

DATA-PROGRAMMING 21
V-1. Data Cleaning
Column Filtering – a single column
 Even if one column is specified, it is better to specify the column as a list.
In [6]: df[ ['Team'] ]
Out[6]:

Data type is ‘Dataframe’

In [7]: df['Team']
Out[7]:

Data type is ‘Series’

• `df[['Team']]`: This code returns a DataFrame object containing only the 'Team' column. This is
because the double brackets indicate a list of column names, and pandas returns a DataFrame
when you use a list inside the brackets.
• `df['Team']`: On the other hand, this code returns a Series object, which is a one-dimensional
array that contains the data of the 'Team' column.

DATA-PROGRAMMING 22
V-1. Data Cleaning
Row Filtering - indexing
 Let's filter only the selected rows using the index number.
In [8]: [Link]([0,1,2], axis=0)

Out[8]:

• [Link]([0, 1, 2], axis=0): This part of the code calls the filter method on the DataFrame 'df.'
• Inside the method: [0, 1, 2] specifies a list of index labels you want to filter and select. In this
case, it's a list containing the labels 0, 1, and 2.
• ‘axis=0’ indicates filtering rows. The axis parameter allows you to specify whether you want to
filter rows (0) or columns (1).

DATA-PROGRAMMING 23
V-1. Data Cleaning
Row filtering - slicing
 Let’s filter the rows in the selected section using slicing.
In [9]: df[2:5]

Out[9]:

• df[2:5]: This code is used to slice a pandas DataFrame `df`, selecting rows by their integer index.
 This specific slice `2:5` will return the rows at index positions 2, 3, and 4, because the slicing is
inclusive of the start index (2) and exclusive of the stop index (5).

DATA-PROGRAMMING 24
V-1. Data Cleaning
Row Filtering - Condition
 Let's filter the rows that satisfy the conditions.
In [10]: df[[Link]=='Finance']

Out[10]:

• df[[Link]=='Finance']: This code is used to filter and select rows in a DataFrame 'df' where the
'Team' column has values that are exactly equal to 'Finance.‘
• The result of [Link] == 'Finance' is a Boolean mask that evaluates to True for rows where the
'Team' column values match 'Finance.'
• Finally, df[[Link] == 'Finance'] uses this Boolean mask to filter the DataFrame and select only
the rows where the 'Team' column values are exactly equal to 'Finance.'

DATA-PROGRAMMING 25
V-1. Data Cleaning
Row Filtering - Condition
 Let's filter the rows that satisfy the conditions.
In [11]: df[ [Link](['Marketing','Finance']) ]

Out[11]:

• df[[Link](['Marketing', 'Finance'])]: This code filters the rows in the pandas DataFrame `df`
based on whether the values in the 'Team' column belong to either 'Marketing' or 'Finance'.
• ‘[Link](['Marketing', 'Finance'])`: This part creates a boolean mask. `[Link]` accesses the
'Team' column of the DataFrame. The `isin` method checks each element in the 'Team' column
to see if it matches any of the values in the list `['Marketing', 'Finance']`. It returns a Series of
boolean values (`True` or `False`) for each row in the DataFrame.
• `df[...]`: The DataFrame `df` is then indexed using this boolean mask. It selects and returns only
those rows where the condition inside the brackets is `True`, i.e., rows where the 'Team' column
has either 'Marketing' or 'Finance'.

DATA-PROGRAMMING 26
V-1. Data Cleaning
Row Filtering - Condition
 Let's filter the rows that satisfy the conditions.
In [12]: df[ ([Link] > 100000) & ([Link] < 110000) ]

Out[12]:

• df[([Link] > 100000) & ([Link] < 110000)]: This code filters rows in the pandas DataFrame
`df` based on the 'Salary' column values. Specifically, it selects rows where the 'Salary' is greater
than 100,000 and less than 110,000.
• ([Link] > 100000): This condition checks each row in the 'Salary' column to see if the salary
is greater than 100,000.
• ([Link] < 110000): This condition checks if the salary is less than 110,000 in each row.
• &: The ampersand operator is used to combine the two conditions. A row is selected only if
both conditions are true for that row.
• `df[...]`: The DataFrame `df` is indexed using this combined boolean condition. Only rows that
satisfy both conditions are returned in the resulting DataFrame.

DATA-PROGRAMMING 27
V-1. Data Cleaning
Row Filtering - Condition
 Let's filter the rows that satisfy the conditions.
In [13]: [Link]('Salary < 36000')

Out[13]:

• [Link]('Salary < 36000'): This code is used to filter rows in a pandas DataFrame `df`. It selects
and returns rows where the values in the 'Salary' column are less than 36,000.
• A `query` method is a more concise and readable way to perform query-like operations on a
DataFrame. This method is particularly useful when you want to filter rows based on a
condition or a set of conditions.
• Here, the condition inside the query string is `'Salary < 36000'`, which pandas evaluates to
select the appropriate rows.

DATA-PROGRAMMING 28
V-1. Data Cleaning
Handling Outliers
 Outliers
• Outliers are data points that significantly deviate from the majority of similar points.
• Outliers pose challenges when constructing predictive models.
 long model training times
 poor accuracy
 an increase in error variance
 a decrease in normality
 a reduction in the power of statistical tests.

10,000
Outlier
8,000

6,000

4,000

2,000

20 40 60 80 100
[Link]
DATA-PROGRAMMING 29
V-1. Data Cleaning
Detecting and Handling Outlier
 Box plot
• Box plot, also known as a box-and-whisker plot (상자 수염 플롯), is a method for visually
representing the distribution of data.
 It shows the median (중앙값), quartiles (사분위수), and outliers of the data.

• The median is the value that lies exactly in the middle of a dataset when it is arranged in order.
 If the number of data points is odd, the median is the single data point in the middle.
 If the number of data points is even, the median is calculated as the average of the two middle points.

1346678
35
𝑚𝑒𝑎𝑛 5
(산술평균) 7

𝑚𝑒𝑑𝑖𝑎𝑛 6

DATA-PROGRAMMING 30
V-1. Data Cleaning
Detecting and Handling Outlier
 Interquartile Range (사분위간 범위) (IQR)
• Interquartile Range (IQR) is a measure in statistics that represents the middle spread of a data
distribution.
 IQR is defined as the difference between the third quartile (Q3) and the first quartile (Q1).
 Q1 corresponds to the value below which 25% of the data falls when arranged in ascending order,
 Q3 corresponds to the value below which 75% of the data falls.

• The IQR represents the range of the middle 50% of the data, helping to understand the central
tendency and spread of the distribution.

Data that exists in this area

is judged to be outliers.

[Link]
DATA-PROGRAMMING 31
V-1. Data Cleaning
Detecting and Handling Outlier
 Check outliers with IQR
In [14]: attribute = 'Salary'
Q1 = df[attribute].quantile(.25)
Q3 = df[attribute].quantile(.75)
IQR = Q3 - Q1
outlier_step = 1.5 * IQR
df[ (df[attribute]< Q1 - outlier_step) | (df[attribute]> Q3 + outlier_step) ]
Out[14]:

• Q1 = df[attribute].quantile(.25): This line calculates the first quartile (Q1) of the 'Salary' attribute.
• Q3 = df[attribute].quantile(.75): This line calculates the third quartile (Q3) of the 'Salary'
attribute.
• IQR = Q3 - Q1: This line calculates the Interquartile Range (IQR), which is the range between
the first and third quartiles and provides a measure of data spread.
• outlier_step = 1.5 * IQR: This line calculates a step size for identifying potential outliers.
• df[(df[attribute] < Q1 - outlier_step) | (df[attribute] > Q3 + outlier_step)]: This line filters the
DataFrame to select rows.

DATA-PROGRAMMING 32
V-1. Data Cleaning
Detecting and Handling Outlier
 Z-score (표준점수)
• Z-score (표준점수) is a statistical measure of how far a data point is from the mean.
 If the Z-score is high, it is located above the mean, and if it is low, it is located below the mean.
 Outliers are situated in the tail of the normal distribution curve and are distant from the mean.
 Typically, if the Z-score is greater than ±2.5 or ±3, the data point is considered an outlier.

Original Data Mean

𝑥 𝜇
𝑍
𝜎

Standard Deviation

[Link]
DATA-PROGRAMMING 33
V-1. Data Cleaning
Detecting and Handling Outlier
 Check outliers with IQR
In [15]: upper_limit = df[attribute].mean() + 3 * df[attribute].std()
lower_limit = df[attribute].mean() - 3 * df[attribute].std()

df[(df[attribute] > upper_limit) | (df[attribute] < lower_limit)]

Out[15]:

• upper_limit = df[attribute].mean() + 3 * df[attribute].std(): This line calculates the upper limit for
identifying outliers for the specified 'attribute.'
• lower_limit = df[attribute].mean() - 3 * df[attribute].std(): This line calculates the lower limit for
identifying outliers for the same 'attribute.'
• df[(df[attribute] > upper_limit) | (df[attribute] < lower_limit)]: This line filters the DataFrame 'df'
to select rows where the values in the 'attribute' column are either greater than the
'upper_limit' or less than the 'lower_limit.'

DATA-PROGRAMMING 34
V-1. Data Cleaning
Detecting and Handling Outlier
 Using the Scatter Plot
• Outliers can be detected and handled using the Scatter Plot.
• Data points that do not align with the typical data distribution trends are referred to as outliers.

10,000
Outlier
8,000

6,000

4,000

2,000

20 40 60 80 100

[Link]
DATA-PROGRAMMING 35
V-1. Data Cleaning
Missing Value
 Missing Value
• In data analysis, Missing Value refers to instances in which one or more fields are blank among
the observations in a data set.
 This may be due to information being missing or not collected, or errors in transmitting or recording the data.

• Missing values can negatively impact the performance of data analysis, statistical modeling,
and machine learning algorithms.
 Analysts and data scientists must handle them carefully.

DATA-PROGRAMMING 36
V-1. Data Cleaning
Drop Missing Values
 Drop Missing Values
• The easiest way to handle missing values is to remove rows or columns containing missing
information.
• While this approach may be the quickest, it's not always the most desirable option as it
involves data loss. If possible, other methods are preferable.

[Link]() [Link](axis=1)

A1 A2 A1 A2 A1 A2 A1
1 A 1 A 1 A 1
2 B 2 B 2 B 2
3 C 3 C 3 C 3
4 NaN 4 NaN 4

drop drop

DATA-PROGRAMMING 37
V-1. Data Cleaning
Handling Missing Values with Dropping
 Remove the row containing missing data.
In [16]: df
Out[16]:

In [17]: drop_df=[Link]()
drop_df
Out[17]:

• drop_df=[Link](): This method is used to remove rows with missing values. `dropna()`
method removes rows in the DataFrame that contain missing values (NaNs). By default, it
removes any row where at least one element is missing. It returns a new DataFrame with the
missing values dropped. The result of [Link]() is stored in the new DataFrame drop_df.

DATA-PROGRAMMING 38
V-1. Data Cleaning
Handling Missing Values with Dropping
 Handling Missing Values with Imputation
• Instead of drop, there is also a way, known as imputation (결측값 대체), to fill in missing values.
• There are more sophisticated methods for performing imputation.
 Simple imputation
 k nearest neighbors
 Model-based Imputations (Random Forest, Deep Learning)

df. fillna(0) df. fillna(df['A1'].mean())

A1 A2 A1 A2 A1 A2 A1 A2
1 A 1 A 1 A 1 A
2 B 2 B 2 B 2 B
NaN C 0 C NaN C 3 C
4 D 4 D 6 D 6 D

DATA-PROGRAMMING 39
V-1. Data Cleaning
Simple Imputation
 The simple imputation is calculating the mean / median / mode of the non-missing
values and then replacing the missing values.
In [1]: df['Salary'] = df['Salary'].fillna(df['Salary'].mean())
df
Out[1]:

• .fillna(): This method is provided by Pandas and is used to fill missing values (nulls) in the
dataframe with a specific value.
• df['Salary'].mean(): This part calculates the average value of the 'Salary' column. It represents
the average of all valid (non-missing) values in the 'Salary' column.
• The entire line of code replaces all missing values (nulls) in the 'Salary' column with the
average value of that column.
• In other words, if the salary information of an employee is missing, it is replaced with the
average salary from the 'Salary' column.

DATA-PROGRAMMING 40
V-1. Data Cleaning
Simple imputation
 SimpleImputer
• Scikit Learn offers SimpleImputer methods.
• SimpleImputer provides a simple way to fill in missing values and is widely used in data
preprocessing.

from [Link] import SimpleImputer

• Key features of SimpleImputer

 SimpleImputer provides the ability to replace missing values with statistics mean, median,
mode.
 provides a variety of missing value imputation strategies, allowing users to choose the
strategy that fits their problem.

DATA-PROGRAMMING 41
V-1. Data Cleaning
Simple Imputation
 Let's use SimpleImputer
In [1]: from [Link] import SimpleImputer
import numpy as np
In [2]: data = [[1, 2, [Link]], [3, [Link], 1], [7, 6, 5]]
imputer = SimpleImputer(strategy='mean')
imputed_data = imputer.fit_transform(data)
imputed_data
Out[2]:

• from [Link] import SimpleImputer : Import the SimpleImputer class from the Scikit-learn library.
• data = [[1, 2, [Link]], [3, [Link], 1], [7, 6, 5]]: Here, a list named data is created, and this list contains three
lists. Each sublist contains a number and a [Link] value. [Link] is a value representing 'Not a Number' in
NumPy, meaning missing data.
• imputer = SimpleImputer(strategy='mean'): Creates a SimpleImputer object. The strategy='mean' parameter
means that the missing data will be replaced with the average value of the corresponding column.
• imputed_data = imputer.fit_transform(data): Uses the fit_transform method to perform an imputation operation
on data. This method first analyzes (fit) the data, calculates the average of each column, and then replaces
(transform) missing values with that average.
• Strategy parameter: mean/ median/ most_frequent/ constant

DATA-PROGRAMMING 42
V-1. Data Cleaning
Simple Imputation
 Imputation Using (Mean/Median) Values
• Pros
• Simple (간단함)
 The calculations are simple and easy to understand.
• Preserve data central tendency (데이터 중심 경향 유지)
 By maintaining the average value of the entire data set, the central tendency of the data can
be preserved.

• Cons
• Impact of outliers (이상치에 영향을 받음)
 Means are very sensitive to outliers.
• Changes in data distribution (데이터의 분포가 바뀜)
 If there are many missing values, replacing them with the mean can reduce variance and make
the data appear narrower than it actually is.
• Information loss (정보 손실)
 Simply replacing missing values with an average value ignores any patterns or structures the
missing values may have, which can result in loss of information.

DATA-PROGRAMMING 43
V-1. Data Cleaning
Simple Imputation
 Imputation Using (Most Frequent) or (Zero/Constant) Values
• Pros
• Best for categorical data (범주형 데이터에 적합)
 This method is especially useful for categorical data, as it allows you to retain the most
common categories.
• Insensitive to outliers (이상치에 둔감함)
 It is not affected by outliers because it uses the most frequently occurring values.

• Cons
 Data distortion (데이터 왜곡)
 Especially if there are many missing values, replacing the most frequent values can
overestimate the weight of that category.
• Not suitable for numeric data (수치형 데이터에 적합하지 않음)
 The most frequent values in numeric data may not provide important information and may
skew the distribution of the data.
• Distortion of the true distribution of the data (데이터 실제 분포 왜곡)
 Substituting constant values can cause the distribution of the data to be simplified or more
distorted than it actually is.
DATA-PROGRAMMING 44
V-1. Data Cleaning
k-NN Imputation
 k-NN Imputation
• k-NN (k-Nearest Neighbors) Imputation is a method that uses k nearest neighbors to replace
data points with missing values.
 The principle behind k-NN Imputation is to estimate the missing values based on the 'k'
nearest neighbors, where 'k' is a user-defined parameter.
 k-NN algorithm is covered in detail in ch.8 classification.
1) Check the remaining attributes excluding those with
A1 A2 A3 missing values.
61.0
3 20 4 ➡ A2, A3 = (80, 15) (From now on, let's call this the
50.8 'remaining tuple’.)
5 30 6
2) Using the Euclidean distance, calculate the distance
NaN 80 15 between the 'remaining tuple' and the tuples of other data.
12 100 22 𝑒𝑥 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 3𝑡ℎ, 4𝑡ℎ 80 100 15 22 21.2
21.2
10 70 14 3) When k=2, the two closest tuples are selected.
10.0 ➡ 4th, 5th tuples
4th, 5th tuples are 2-Nearest Neighbors of 3th tuple.
11 remaining 4) Replace missing values with the average of the A1 attribute
tuple values of the 4th and 5th tuples. (12+10)/2 = 11

DATA-PROGRAMMING 45
V-1. Data Cleaning
k-NN Imputation
 Scikit-Learn offers KNNImputer
In [1]: from [Link] import KNNImputer
import numpy as np
In [2]: data = [[1, 2, 3], [3, [Link], 1], [4, 6, 5], [7, 3, 8]]
imputer = KNNImputer(n_neighbors=2)
imputed_data = imputer.fit_transform(data)
imputed_data
Out[2]:

• from [Link] import KNNImputer: Import the KNNImputer class from the Scikit-learn library.
• data = [[1, 2, 3], [3, [Link], 1], [4, 6, 5], [7, 3, 8]]: Assign a list of lists to the variable 'data.' Here, [Link]
represents missing values.
• imputer = KNNImputer(n_neighbors=2): Create a KNNImputer object. The parameter n_neighbors=2 sets the
number of 'nearest neighbors' to be used in the KNN algorithm to 2. This means that when determining
replacement values for missing data, it considers the two closest neighboring data points.
• imputed_data = imputer.fit_transform(data): Perform imputation (data replacement) on the 'data' using the
fit_transform method. This method first analyzes (fits) the data to prepare for applying the KNN algorithm and
then replaces missing values based on the closest neighboring data points.

DATA-PROGRAMMING 46
V-1. Data Cleaning
Imbalanced Data
 Imbalanced Data
• Imbalanced Data (불균형 데이터) refers to a data set that contains very few examples of a class.
 It occurs when the number of observations in one class is much more or less than that of another class.

• Examples of imbalanced data

 Binary classification problem: In a fraud transaction detection system, fraudulent transactions (positive class)
occur very rarely compared to normal transactions (negative class).
 Multi-class problem : When diagnosing specific diseases, some may occur much more rarely than others.

4 800

Number of samples
2 600
0
400
-2
200
-4 Class 0
Class 1
0
-4 -2 0 2 4 0.0 1.0
class

DATA-PROGRAMMING 47
V-1. Data Cleaning
Problem of Imbalanced Data
 Problem of Imbalanced Data
• Poor performance (낮은 성능)
 Most machine learning algorithms are designed assuming balanced data.
 When training on imbalanced data, the model becomes biased toward the majority class,
increasing the probability of misclassifying the minority class.

• Skewed evaluation metrics (치우쳐진 평가 지표)

 Traditional performance metrics such as Accuracy can overestimate a model’s performance on
imbalanced data.
 For example, in a data set where only 1% of fraudulent transactions account, a model that
always predicts that something is 'normal' may be 99% accurate, but in reality it may not
detect any fraudulent transactions at all.

DATA-PROGRAMMING 48
V-1. Data Cleaning
Random Undersampling and Oversampling
 Random Undersampling and Oversampling
• Random undersampling and oversampling are two basic resampling techniques to deal with
imbalanced data.
 Random undersampling is a method to balance the ratio of the minority class and the
majority class by randomly removing observations from the majority class.
 Random oversampling is a method of randomly replicating observations in the minority class
to match the number of observations in the majority class.

Undersampling Oversampling
Copies of the
minority class

Samples of
majority class

Original dataset Original dataset

DATA-PROGRAMMING 49
V-1. Data Cleaning
Random Undersampling and Oversampling
 Disadvantage of Undersampling
• Information loss (정보 손실)
 Important information can be lost in multiple classes, which can lead to poor model performance.

• Risk of biased model

 Removing instances from the majority class may lead to a biased model.

Undersampling Imbalanced Dataset Oversampling

Loss information Redundant

information

[Link]
DATA-PROGRAMMING 50
V-1. Data Cleaning
Random Undersampling and Oversampling
 Disadvantage of Oversampling
• Risk of overfitting (과적합 위험)
 Duplicating the same data can cause your model to overfit to certain patterns.

• Increased computational cost (계산비용증가)

 As the size of the dataset increases, training time and memory usage increase.

Undersampling Imbalanced Dataset Oversampling

Loss information Redundant

information

[Link]
DATA-PROGRAMMING 51
V-1. Data Cleaning
Synthetic Minority Oversampling Technique
 Synthetic Minority Oversampling Technique
• Synthetic Minority Oversampling Technique (SMOTE) is an advanced oversampling technique
widely used to handle imbalanced data sets.
 This method enriches the minority class by generating new synthetic samples, rather than
simply replicating samples from the minority class.

Generating new synthetic samples

[Link]
DATA-PROGRAMMING 52
V-1. Data Cleaning
Synthetic Minority Oversampling Technique
 Create an artificial classification dataset using the make_classification function.
In [1]: !pip install imbalanced-learn
In [2]: from imblearn.over_sampling import SMOTE
from [Link] import make_classification
from sklearn.model_selection import train_test_split
from collections import Counter
In [3]: # Create a synthetic imbalanced classification dataset
X, y = make_classification(n_classes=2, class_sep=2, flip_y=0,
weights=[0.1, 0.9], n_features=20, n_clusters_per_class=1, n_samples=1000)
print(f"Before SMOTE: {Counter(y)}")
Out[3]:

• make_classification: This function is used to generate a synthetic dataset for classification.

 n_classes: Number of classes to generate.
 class_sep: Argument used to separate classes, the larger the value, the clearer the separation between classes.
 flip_y: The probability that the label will flip randomly.
 weights: Determines the proportion of samples for each class. [0.1, 0.9] causes the first class to occupy 10% of the
dataset and the second class to occupy 90%.
 n_features: Total number of features to create. Here it is set to 20, so there are a total of 20 features in the dataset.
 n_clusters_per_class: Number of clusters to create per class. Here it is set to 1, so each class consists of one cluster.
 n_samples: Total number of samples to generate.

• X, y: The generated synthetic dataset is assigned to X (features) and y (labels).

• Counter(y): This counts the occurrences of each class label in y using the Counter class from the
collections module.
DATA-PROGRAMMING 53
V-1. Data Cleaning
Synthetic Minority Oversampling Technique
 Data oversampling with SMOTE
In [4]: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

In [5]: sm = SMOTE(random_state=42)
X_res, y_res = sm.fit_resample(X_train, y_train)
print(f"After SMOTE: {Counter(y_res)}")
Out[5]:

• X_train, X_test, y_train, y_test: These variables will store the resulting training and testing sets
for features (X) and labels (y).
• train_test_split(X, y, test_size=0.25, random_state=42): This function splits the dataset into
training and testing sets. Here's what each parameter does:
 X: The features or input data.
 y: The corresponding labels or output data.
 test_size=0.25: Specifies the proportion of the dataset to include in the test split. In this case, 25% of the data
will be used for testing, and 75% will be used for training.

• SMOTE(random_state=42): Creates an instance of the SMOTE oversampling technique. The

random_state parameter is set to ensure reproducibility of results.
• fit_resample: This method applies SMOTE to the training data, oversampling it by synthesizing
samples from the minority class. As a result, the number of samples in the minority class will
be equal to the number of samples in the majority class.
• X_res, y_res: Features and labels of the oversampled dataset. This dataset is now balanced
across classes and can be used to train the model.

DATA-PROGRAMMING 54
V-1. Data Cleaning
Synthetic Minority Oversampling Technique
 Check the generated results in a graph
In [6]: import [Link] as plt
[Link](figsize=(10, 5))
[Link](1, 2, 1)
[Link](X_train[y_train == 0][:, 0], X_train[y_train == 0][:, 1], label="Class #0", alpha=0.5, linewidth=0.15)
[Link](X_train[y_train == 1][:, 0], X_train[y_train == 1][:, 1], label="Class #1", alpha=0.5, color='r',
linewidth=0.15)
[Link]('Original class distribution')
[Link]()

In [7]: [Link](1, 2, 2)
[Link](X_res[y_res == 0][:, 0], X_res[y_res == 0][:, 1], label="Class #0", alpha=0.5, linewidth=0.15)
[Link](X_res[y_res == 1][:, 0], X_res[y_res == 1][:, 1], label="Class #1", alpha=0.5, color='r', linewidth=0.15)
[Link]('After SMOTE class distribution')
[Link]()
plt.tight_layout()
[Link]()

Out[7]:

DATA-PROGRAMMING 55
V-2. Data Transformation
Scaling
 Scaling
• Scaling means to arbitrarily adjust the range of data.
 The shape of the data distribution remains unchanged,
 The range is adjusted while maintaining the same proportion as the original data.

• When analyzing using multiple variables, if each variable has a different unit, the importance of
the variables reflected in the model can vary.
 Ex) When using variables such as height and age, the height variable, due to its larger unit,
can have a greater influence on the learning process.

• Scaling improves the stability of analysis algorithms by ensuring that all variables have a similar
impact on the model.

Scaling

[Link]
DATA-PROGRAMMING 56
V-2. Data Transformation
Multiple Scaling Techniques
 Multiple Scaling Techniques
• The Maximum Absolute Scaling
• The Min-max Scaling
• The Z-score Method
• The Robust Scaling

DATA-PROGRAMMING 57
V-2. Data Transformation
Maximum Absolute Scaling
 Maximum Absolute Scaling
• Maximum Absolute Scaling scales data for each characteristic to a range between -1 and 1.
• This scaling method works by scaling the data by dividing it by the maximum absolute value of
each feature.

Maximum Absolute
Original data Scaled data
A B A B
1 -10 𝑥 0.2 -0.5
𝑥
𝑚𝑎𝑥 𝑥
2 5 0.4 0.25
3 -15 0.6 -0.75
𝑀𝑎𝑥 𝐴𝑏𝑠 5, 20
4 20 0.8 1
5 -5 1 -0.25

𝑀𝑎𝑥 𝐴𝑏𝑠 5 𝑀𝑎𝑥 𝐴𝑏𝑠 20

DATA-PROGRAMMING 58
V-2. Data Transformation
Maximum Absolute Scaling
 Scaling using MaxAbsScaler
In [1]: from [Link] import MaxAbsScaler
import numpy as np
In [2]: X = [Link]([[1, -10], [2, 5], [3, -15], [4, 20], [5, -5]])
In [3]: scaler = MaxAbsScaler()
X_scaled = scaler.fit_transform(X)
In [4]: print("Original data:\n", X)
print("Scaled data:\n", X_scaled)
Out[4]:

• from [Link] import MaxAbsScaler: Import the MaxAbsScaler class from the scikit-learn library.
• X = [Link]([[1, -10], [2, 5], [3, -15], [4, 20], [5, -5]]): Create the original data to be scaled as a NumPy array.
• scaler = MaxAbsScaler(): Instantiate an object of the MaxAbsScaler class.
• X_scaled = scaler.fit_transform(X): Use the fit_transform method to scale the original data X, and store the
result in X_scaled.

DATA-PROGRAMMING 59
V-2. Data Transformation
Min-max Scaling
 Min-max Scaling
• Min-max scaling (or normalization) is one of the simplest ways to scale features in your data.
• This method converts the values of all features to a range between 0 and 1.

Min-max
Original data Scaled data
A B A B
1 -10 𝑥 𝑥 0 0.143
𝑥
𝑥 𝑥
2 5 0.25 0.571
3 -15 0.5 0
𝑀𝑎𝑥 5, 20
4 20 𝑀𝑖𝑛 1, 15 0.75 1
5 -5 1 0.286

𝑀𝑎𝑥 5 𝑀𝑎𝑥 20
𝑀𝑖𝑛 1 𝑀𝑖𝑛 15

DATA-PROGRAMMING 60
V-2. Data Transformation
Min-max Scaling
 Scaling using MinMaxScaler
In [5]: from [Link] import MinMaxScaler
In [6]: scaler = MaxAbsScaler()
X_scaled = scaler.fit_transform(X)
In [7]: print("Original data:\n", X)
print("Scaled data:\n", X_scaled)
Out[7]:

• from [Link] import MaxAbsScaler: Import the MaxAbsScaler class from scikit-
learn, which scales each feature by its maximum absolute value.
• scaler = MaxAbsScaler(): Instantiate an object of the MaxAbsScaler class.
• X_scaled = scaler.fit_transform(X): Use the fit_transform method to scale the original data X,
and store the result in X_scaled.

DATA-PROGRAMMING 61
V-2. Data Transformation
Z-score Standardization
 Z-score Standardization
• Z-score standardization, also known as standard scaling or just standardization, is a scaling
technique.
• Z-score standardization used to transform the features of your data so that they have the
properties of a standard normal distribution with a mean of 0 and a standard deviation of 1.
• The formula for calculating the Z-score of a given value is as follows.
Z-score
Original data Scaled data
A B A B
1 -10 𝑥 𝜇 -1.414 -0.725
𝑥
𝜎
2 5 -0.707 0.484
3 -15 0 -1.128
𝑀𝑒𝑎𝑛 3, 1
4 20 𝑆𝑡𝑑. 1.414, 12.409 0.707 1.692
5 -5 1.414 -0.322
x is the original value
𝑀𝑒𝑎𝑛 3 𝑀𝑒𝑎𝑛 1 μ is the mean of the feature
𝑆𝑡𝑑. 1.414 𝑆𝑡𝑑. 12.409 σ is the standard deviation

DATA-PROGRAMMING 62
V-2. Data Transformation
Z-score Standardization
 Scaling using StandardScaler
In [8]: from [Link] import StandardScaler
In [9]: scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
In [10]: print("Original data:\n", X)
print("Scaled data:\n", X_scaled)
Out[10]:

• from [Link] import StandardScaler: Import the StandardScaler class from scikit-
learn, which standardizes features by removing the mean and scaling to unit variance.
• scaler = StandardScaler(): Instantiate an object of the StandardScaler class.
• X_scaled = scaler.fit_transform(X): Use the fit_transform method to standardize the original data
X, and store the result in X_scaled.

DATA-PROGRAMMING 63
V-2. Data Transformation
Robust Scaling
 Robust Scaling
• Robust scaling is a scaling method that is robust to outliers.
• Robust scaling performs scaling using the median and interquartile range (IQR) of the data.
• IQR refers to the range between the first quartile (Q1, 25th percentile) and the third quartile
(Q3, 75th percentile).

Robust
Original data Scaled data
A B A B
1 -10 𝑥 𝑄 𝑥 -1 -0.33
𝑥
2 5
𝑄 𝑥 𝑄 𝑥 -0.5 0.667
3 -15 0 -0.667
𝑄 2, 10
4 20 𝑄 3, 5 0.5 1.667
5 -5 𝑄 4, 5 1 0

𝑄 2 𝑄 10 𝑄 𝑄 𝑄
𝑄 3 𝑄 5 0 25% 50% 75% 100%
𝑄 4 𝑄 5 A 1 2 3 4 5
B -15 -10 -5 5 20

DATA-PROGRAMMING 64
V-2. Data Transformation
Robust Scaling
 Scaling using RobustScaler
In [8]: from [Link] import RobustScaler
In [9]: scaler = RobustScaler()
X_scaled = scaler.fit_transform(X)
In [10]: print("Original data:\n", X)
print("Scaled data:\n", X_scaled)
Out[10]:

• from [Link] import RobustScaler: Import the RobustScaler class from scikit-learn. RobustScaler
scales features using statistics that are robust to outliers.
• scaler = RobustScaler(): Creates an instance of RobustScaler. This scaler scales the data by subtracting the
median of each feature and dividing by the interquartile range (IQR). This method is useful when there are
outliers in your data.
• X_scaled = scaler.fit_transform(X): Apply scaling to the X data using the fit_transform method. The fit part
analyzes the data and calculates the parameters needed for scaling (median, IQR, etc.), and the transform part
uses these parameters to scale the data.

DATA-PROGRAMMING 65
V-2. Data Transformation
Categorical Variable Encoding
 Categorical Variable Encoding
• Categorical variable encoding refers to the process of converting categorical data into a format
that machine learning algorithms can understand and process.
 Categorical variables are typically represented as text labels.
 Machine learning algorithms find it easier to work with numerical data.
 Therefore, it is necessary to convert categorical variables into numerical form.

• Categorical variables can be broadly categorized into two types.

• Nominal Variables (명목 변수)

 Nominal variables represent cases where there is no inherent order or hierarchy among categories.
 For example, a 'color' nominal variable may have categories such as 'red,' 'blue,' and 'yellow,' with no specific
ranking between these categories.

• Ordinal Variables (순서 변수)

 Ordinal variables indicate cases where there is a ranking or order among categories.
 For instance, an 'education' ordinal variable may include categories like 'high school,' 'college graduate,'
'master's degree,' and 'doctorate,' with a clear hierarchy or ranking among these categories.

DATA-PROGRAMMING 66
V-2. Data Transformation
Various Encoding Techniques
 Various Encoding Techniques
• One-Hot Encoding
 One-Hot Encoding involves creating new binary features for each category, with a value of 1 indicating the
presence of the category and 0 for all others.

• Label Encoding
 Label Encoding is used for encoding ordinal variables.
 Assigns each category a unique integer value, maintaining the order information.

• Integer Encoding
 Integer Encoding can be used for encoding nominal variables by mapping each category to a unique integer
value.
 This method does not consider the order among categories for nominal variables.

• Category Embedding
 Category Embedding is a method of mapping categorical variables to real-valued vectors.
 It is primarily used in deep learning models and allows for learning similarities between categories.

DATA-PROGRAMMING 67
V-2. Data Transformation
One Hot Encoding
 One Hot Encoding
• One-hot Encoding is one of the methods for transforming categorical values into vectors
• each category is represented as independent binary columns (i.e., 0 and 1)

 Operation Method
1) Generate a new binary feature (column) for each unique category present in the dataset.
2) For each data point, assign a 1 to the corresponding column of its category and assign 0 to all
other newly created category columns.

Pet Cat Dog Turtle Fish

Cat 1 0 0 0
Dog 0 1 0 0
Turtle 0 0 1 0
Fish 0 0 0 1
Cat 1 0 0 0
Categorical values One-hot Encoding

DATA-PROGRAMMING 68
V-2. Data Transformation
One Hot Encoding
 Prepare categorical data.
In [1]: import pandas as pd

In [2]: df = [Link]({
'Color': ['Red', 'Green', 'Blue', 'Green', 'Red'],
'Size': ['S', 'M', 'L', 'S', 'M'],
'Price': [10, 15, 20, 15, 10]
})
df
Out[2]:

• [Link](...): Use the DataFrame constructor from Pandas to create a tabular data structure called a
DataFrame.
• {...}: Inside the DataFrame constructor, provide a dictionary where keys represent column names, and values
represent lists of data for each column.
• 'Color': ['Red', 'Green', 'Blue', 'Green', 'Red']: Create a 'Color' column with categorical data.
• 'Size': ['S', 'M', 'L', 'S', 'M']: Create a 'Size' column with categorical data.
• 'Price': [10, 15, 20, 15, 10]: Create a 'Price' column with numerical data.

DATA-PROGRAMMING 69
V-2. Data Transformation
One Hot Encoding
 Create a one-hot encoding using get_dummies.
In [3]: df_encoded = pd.get_dummies(df, columns=['Color', 'Size'])
df_encoded
Out[3]:

• pd.get_dummies(...): This function from Pandas is used to perform one-hot encoding on

categorical columns of a DataFrame.
 df: The DataFrame (df) that you want to encode.
 columns=['Color', 'Size']: Specify the columns to be one-hot encoded. In this case, 'Color' and 'Size' columns
are selected.

• The df_encoded DataFrame is created by applying one-hot encoding to the specified

categorical columns ('Color' and 'Size'). The original categorical columns are replaced with
binary columns representing each category using 0s and 1s (dummy variables). This
transformation is useful for machine learning algorithms that require numerical input.

DATA-PROGRAMMING 70
V-2. Data Transformation
One Hot Encoding
 Expressing one-hot encoding as 0,1
In [4]: df_encoded = pd.get_dummies(df, columns=['Color', 'Size'], dtype=int)
df_encoded
Out[4]:

• 'dtype': Specifies the data type of the encoded result columns as integers.

DATA-PROGRAMMING 71
V-2. Data Transformation
One Hot Encoding
 Scikit-learn library provides OneHotEncoder.
In [5]: from [Link] import OneHotEncoder

In [6]: encoder = OneHotEncoder(sparse_output=False)

encoded_data = encoder.fit_transform(df[['Color', 'Size']])
encoded_data
Out[6]:

• from [Link] import OneHotEncoder: Import the OneHotEncoder class from the
Scikit-learn library. This class is used to convert categorical data to one-hot encoding format.
• encoder = OneHotEncoder(sparse_output=False): Creates an instance of OneHotEncoder. Here,
the sparse_output=False parameter sets the encoded data to be returned in a dense array
(Dense Array). By default, OneHotEncoder returns data in the form of a sparse matrix. But here
we want to receive the results in a dense form.
• encoded_data = encoder.fit_transform(df[['Color', 'Size']]): One-hot encoding the 'Color' and
'Size' columns of dataframe df using the fit_transform method. Apply. The fit part analyzes the
data to identify each category, and the transform part transforms these categories into one-hot
encoding format.

DATA-PROGRAMMING 72
V-2. Data Transformation
One Hot Encoding
 Convert one-hot encoding results into a Dataframe.
In [7]: encoded_df = [Link](encoded_data, columns=encoder.get_feature_names_out(['Color', 'Size']))
encoded_df['Price'] = df['Price']
encoded_df
Out[7]:

• encoded_df = [Link](encoded_data, columns=encoder.get_feature_names_out(['Color',

'Size'])): This line uses one-hot encoded data(encoded_data) to generate new pandas data.
Create a frame (encoded_df). The columns parameter specifies the names of the columns
generated using encoder.get_feature_names_out(['Color', 'Size']). The get_feature_names_out
method generates column names for each one-hot encoded feature. For example, if the
column 'Color' has the values 'Red', 'Blue', and 'Green', this method returns column names like
'Color_Red', 'Color_Blue', and 'Color_Green'. returns.
• encoded_df['Price'] = df['Price']: Here we add column 'Price' from original dataframe df to new
dataframe encoded_df. By doing this, you can combine the one-hot encoded feature data with
other important information from the original dataframe.

DATA-PROGRAMMING 73
V-2. Data Transformation
Label Encoding
 Label Encoding
• Label Encoding is a method of converting each unique category of a categorical variable into
ordinal numerical values.
• For example, if a feature such as ‘Pet' has four values, such as [‘Cat‘, ‘Dog‘, ‘Turtle‘, ‘Fish’], Label
Encoding can be used to map them to numerical labels. like [0, 1, 2, 3].

Pet Pet
Cat 0
Dog 1
Turtle 2
Fish 3
Cat 0
Categorical values Label Encoding

DATA-PROGRAMMING 74
V-2. Data Transformation
Label Encoding
 Ordinality (순서성)
• Label Encoding assigns values with an order, which can be risky as it may impose an order on
nominal categorical variables where there is no inherent order.
• This can lead machine learning models to learn non-existent data structures.

 Simplicity (간단함)
• It is straightforward to implement and easy to understand.

 Model Dependency (모델 종속성)

• Some machine learning models perform well with data having ordinal labels.
• For example, decision trees and random forests work well with data encoded using Label
Encoding, while linear models may make incorrect assumptions.

 Utilizing Rank or Order Information (순위 및 순서 정보 활용)

• For categorical data where the order holds meaning (e.g., 'High', 'Medium‘, 'Low'),
• Label Encoding can be an effective way to convey ranking information to the model.

DATA-PROGRAMMING 75
V-2. Data Transformation
Label Encoding
 Creating label encoding using factorize()
In [8]: # Color와 Size 열에 대해 Label Encoding 수행
df['Color_encoded'] = [Link](df['Color'])[0]
df['Size_encoded'] = [Link](df['Size'])[0]
df
Out[8]:

• [Link](...): factorize() function takes categorical data as an argument and assigns unique
integer values to each category.
• factorize() function returns a tuple with two elements: the first element is an array containing
the encoded labels, and the second element is an array containing the unique categories used
for encoding.
• [0] is used to select only the first element of the tuple, which is the array of encoded labels.

DATA-PROGRAMMING 76
V-2. Data Transformation
Label Encoding
 Creating label encoding using Scikit-learn
In [9]: from [Link] import LabelEncoder
le_color = LabelEncoder()
le_size = LabelEncoder()
In [10]: df['Color_encoded'] = le_color.fit_transform(df['Color'])
df['Size_encoded'] = le_size.fit_transform(df['Size'])
df
Out[10]:

• from [Link] import LabelEncoder: This imports the LabelEncoder class from
scikit-learn, which is used to convert categorical labels into numerical form.
• le_color = LabelEncoder(): Creates a LabelEncoder object for the 'Color' column.
• le_size = LabelEncoder(): Creates a LabelEncoder object for the 'Size' column.
• df['Color_encoded'] = le_color.fit_transform(df['Color']): Applies label encoding to the 'Color'
column and creates a new column 'Color_encoded' with the encoded values.
• df['Size_encoded'] = le_size.fit_transform(df['Size']): Applies label encoding to the 'Size' column
and creates a new column 'Size_encoded' with the encoded values.

DATA-PROGRAMMING 77
V-2. Data Transformation
Ordinal Encoding
 Ordinal Encoding
• Ordinal Encoding differs in that it considers the order or ranking between categories.
• In other words, when categorical variable values have an inherent order (e.g., 'High,' 'Medium,' 'Low')
• This method assigns unique numbers to each category while preserving the order information.

Grade Grade
Poor 1
Good 2
Very Good 3
Excellent 4
Categorical values Ordinal Encoding

DATA-PROGRAMMING 78
V-2. Data Transformation
Ordinal Encoding
 Utilization of Order Information (순서 정보 활용)
• Ordinal Encoding encodes categories while taking into account their order.
• This is useful when the order of data is significant and needs to be learned by the model.

 Performance Improvement (성능 향상)

• For categorical variables where order matters, appropriately encoding them can lead to better
performance of machine learning models, resulting in overall improvement.

 Simplicity and Efficiency (단순성과 효율성)

• The encoding process is simple and computationally efficient.
• Additionally, the encoded data doesn't introduce additional dimensions, thus not increasing the
dimensionality of the data.

DATA-PROGRAMMING 79
V-2. Data Transformation
Ordinal Encoding
 Prepare data with embedded order information.
In [11]: data = {
'Education': ['High School', 'Bachelors', 'Masters', 'PhD', 'Associates', 'Bachelors', 'Masters'],
'Income': ['Low', 'Low', 'Medium', 'High', 'Low', 'Medium', 'High'] }
df = [Link](data)
df
Out[11]:

• data: This is a Python dictionary containing two lists, 'Education' and 'Income', with
corresponding values.
• 'Education': ['High School', 'Bachelors', 'Masters', 'PhD', 'Associates', 'Bachelors', 'Masters']:
Defines the 'Education' column with different education levels for each entry in the DataFrame.
• 'Income': ['Low', 'Low', 'Medium', 'High', 'Low', 'Medium', 'High']: Defines the 'Income' column
with income levels corresponding to each entry in the DataFrame.
• df = [Link](data): Creates a pandas DataFrame named 'df' using the provided data
dictionary.

DATA-PROGRAMMING 80
V-2. Data Transformation
Ordinal Encoding
 Specify the label to which each category data corresponds.
In [12]: education_order = {
'High School': 1,
'Associates': 2,
'Bachelors': 3,
'Masters': 4,
'PhD': 5
}
income_order = {
'Low': 1,
'Medium': 2,
'High': 3
}

• education_order: This is a Python dictionary defining an order or numerical representation for

different education levels. Each education level is associated with a numerical value, indicating
its position in the order.
 'High School': 1, 'Associates': 2, 'Bachelors': 3, 'Masters': 4, 'PhD': 5
• income_order: Similar to education_order, this dictionary defines an order or numerical
representation for income levels.
 'Low': 1, 'Medium': 2, 'High': 3

DATA-PROGRAMMING 81
V-2. Data Transformation
Ordinal Encoding
 Create Ordinal Encoding using the map() function.
In [13]: df['Education_encoded'] = df['Education'].map(education_order)
df['Income_encoded'] = df['Income'].map(income_order)
df
Out[13]:

• df['Education'].map(education_order): Apply the map method to the 'Education' column to

encode values according to the order defined in the 'education_order' dictionary.
• The map function applies the mapping from the dictionary to each item in the given column
and converts it to a new value.
• df['Income'].map(income_order): Perform the same process for the 'Income' column.

DATA-PROGRAMMING 82
V-2. Data Transformation
Ordinal Encoding
 Create Ordinal Encoding using OrdinalEncoder.
In [14]: from [Link] import OrdinalEncoder

In [15]: # 범주의 순서를 정의

education_categories = ['High School', 'Associates', 'Bachelors', 'Masters', 'PhD']
income_categories = ['Low', 'Medium', 'High']
In [16]: # OrdinalEncoder 인스턴스 생성
encoder = OrdinalEncoder(categories=[education_categories, income_categories])
# 인코딩할 열을 선택하고 fit_transform 메소드를 적용
df_encoded = encoder.fit_transform(df[['Education', 'Income']])

• from [Link] import OrdinalEncoder: Imports the OrdinalEncoder class from

scikit-learn, which is used to encode categorical features as ordinal integers.
• education_categories and income_categories: Lists defining the order of categories for the
'Education' and 'Income' columns, respectively. This is necessary for ordinal encoding, as it
specifies the order of categories.
• encoder = OrdinalEncoder(categories=[education_categories, income_categories]): Creates an
instance of the OrdinalEncoder class, specifying the custom order of categories for the
'Education' and 'Income' columns.
• df_encoded = encoder.fit_transform(df[['Education', 'Income']]): Applies the ordinal encoding to
the selected columns ('Education' and 'Income') of the DataFrame df. The result is stored in the
df_encoded DataFrame.

DATA-PROGRAMMING 83
V-2. Data Transformation
Ordinal Encoding
 Create Ordinal Encoding using OrdinalEncoder.
In [17]: # 인코딩된 데이터를 새로운 열에 할당
df['Education_encoded'] = df_encoded[:, 0]
df['Income_encoded'] = df_encoded[:, 1]
df
Out[17]:

• df['Education_encoded'] = df_encoded[:, 0]: Creates a new column in the DataFrame df called

'Education_encoded' and assigns the values from the first column of the df_encoded array. This
column represents the ordinal encoding of the 'Education' category.
• df['Income_encoded'] = df_encoded[:, 1]: Creates another new column in the DataFrame df
called 'Income_encoded' and assigns the values from the second column of the df_encoded
array. This column represents the ordinal encoding of the 'Income' category.

DATA-PROGRAMMING 84
V. Data Preprocessing
Summary
 Data Preprocessing: A crucial step in data analysis involving cleaning, transforming,
and organizing raw data to improve its quality and ensure it's in the right format
for analysis and modeling.

 Data Filtering: The process of screening and removing irrelevant, incorrect, or

redundant data from a dataset to focus on only the most relevant information for
analysis.

 Handling Outliers: This involves identifying and managing data points that
significantly differ from the rest of the data, as they can distort statistical analyses
and models.

 Missing Value: Addressing the absence of data in a dataset. Strategies include

imputation, deletion, and weighing the implications of missing data on analysis
outcomes.

DATA-PROGRAMMING 85
V. Data Preprocessing
Summary
 Imbalanced Data: Techniques to manage datasets where some classes are
significantly under- or over-represented, which can bias predictive models and
affect accuracy.

 Scaling: Adjusting the range of variables to normalize the level of variance within
the data, which is critical for algorithms that are sensitive to the scale of data.

 Categorical Variable Encoding: Transforming non-numeric categories into a

numerical format so that they can be properly utilized in machine learning
algorithms.

DATA-PROGRAMMING 86
V. Data Preprocessing
References
 [Link]

 [Link]

DATA-PROGRAMMING 87