Data Cleaning Cheat Sheet in
Data Cleaning
Python Using
- By Eugenia Python
Anello
for Data Science Project
Table of Contents:
1. Dealing with Missing Data
2. Dealing with Duplicates
3. Outlier Detection
4. Encode Categorical Features
5. Transformation
1. Dealing with Missing data
Check missing data in each column of the dataset
[Link]().sum()
Delete missing data
[Link](how='all')
Drop columns that have missing values
[Link](how='columns')
Drop specific columns that have missing values
[Link](subset=[‘municipal,'city'])
Replace missing values with Mean/Median/Mode
df[‘price’].fillna(df[‘price’].mean())
df[‘age’].fillna(df[‘age’].median())
df[‘type_building’].fillna(df[‘type_building’].mode())
Replace missing values with Mean/Median/Mode of the group
df['price'].fillna([Link]('type_building')['price'].transform(‘mean’),
inplace=True)
Forward Fill - Fill missing values with values before them
df['stock_price'].fillna(method='ffill')
Forward Fill within Groups
df['stock_price'] = [Link]('type_stock').ffill()
Backward Fill - FIll missing values with values after them
df['stock_price'].fillna(method='bfill')
Backward Fill within Groups
df['stock_price'] = [Link]('type_stock')['stock_price'].bfill()
Fill missing values using the interpolation method
df['stock_price'] =
df['stock_price'].interpolate(method='polynomial',order=2)
Fill missing values using the interpolation method within groups
df['stock_price'] = [Link]('type_stock')['stock_price'].apply(lambda
x: [Link](method='polynomial',order=2))
2. Dealing with Duplicates
Check if there are duplicates
[Link]().sum()
Extract duplicate rows from the dataframe
df[[Link]()]
Drop duplicates
df.drop_duplicates()
Aggregate data
[Link]('id').agg({'price':'mean'}).reset_index()
3. Outlier detection
Detect range of values for each column of the dataset
[Link]([x*0.1 for x in range(10)])
Display boxplot to display the distribution of a column
import seaborn as sns
[Link](x=df['age'])
Display histogram to display the distribution of a column
[Link](data=df[‘column1’])
Remove outliers
df = df[df['age']<df[‘age'].quantile(0.9)]
Outlier detection with machine learning models, like Isolation Forest
if = IsolationForest(random_state=42)
[Link](X)
y_pred = [Link](X)
4. Encode categorical features
Apply one-hot-encoding to a categorical column
from [Link] import OneHotEncoder
ohe = OneHotEncoder()
encoded_data = [Link](ohe.fit_transform(df[[‘type_build’]]).toarray())
new_df = [Link](encoded_data)
Apply label-encoding to a categorical column
from [Link] import LabelEncoder
le = LabelEncoder()
df[‘type_build’] = le.fit_transform(df[‘type_build’])
Apply ordinal-encoding to a categorical column to retain its ordinal nature
from [Link] import OrdinalEncoder
le = OrdinalEncoder()
df['price_level'] = le.fit_transform(df['price_level'])
5. Transformation
Standardize features by removing the mean and scaling to unit variance
from [Link] import StandardScaler
X_std = StandardScaler().transform(X)
Rescale features into the range [0,1]
from [Link] import MinMaxScaler
X_mms = MinMaxScaler().transform(X)
Scale features exploiting statistics that are robust to outliers
from [Link] import RobustScaler
X_rs = RobustScaler().transform(X)