Scribd
Upload
2 views3 pages

Data Cleaning With Python Examples

The document outlines essential data cleaning techniques for beginners using Python, including handling missing values, removing duplicates, fixing data types, and managing outliers. It provides practical examples with code snippets for each technique, such as standardizing text data and validating data integrity. The document emphasizes the importance of documenting each step in the data cleaning process.

Uploaded by

sumicode.exe
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views3 pages

Data Cleaning With Python Examples

The document outlines essential data cleaning techniques for beginners using Python, including handling missing values, removing duplicates, fixing data types, and managing outliers. It provides practical examples with code snippets for each technique, such as standardizing text data and validating data integrity. The document emphasizes the importance of documenting each step in the data cleaning process.

Uploaded by

sumicode.exe
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Data Cleaning Techniques for Beginners (with

Python Examples)

1. Remove or Handle Missing Values


import pandas as pd

data = {'age': [25, None, 30, 28, None],


'city': ['New York', 'Paris', None, 'Berlin', 'Paris']}
df = [Link](data)

# Remove rows with missing values


df_drop = [Link]()

# Fill missing values


df_fill = [Link]()
df_fill['age'].fillna(df_fill['age'].mean(), inplace=True)
df_fill['city'].fillna('Unknown', inplace=True)

print(df_fill)

2. Remove Duplicates
data = {'name': ['John', 'Anna', 'John', 'Mike'],
'age': [28, 22, 28, 32]}
df = [Link](data)

# Remove duplicate rows


df_unique = df.drop_duplicates()
print(df_unique)

3. Fix Data Types


data = {'age': ['25', '30', '35'],
'date': ['2024-01-01', '2024-02-01', '2024-03-01']}
df = [Link](data)

# Convert data types


df['age'] = df['age'].astype(int)
df['date'] = pd.to_datetime(df['date'])

print([Link])

4. Handle Outliers
import numpy as np

data = {'salary': [3000, 3200, 3500, 100000, 3600, 3400]}


df = [Link](data)

# Detect and remove outliers using IQR


Q1 = df['salary'].quantile(0.25)
Q3 = df['salary'].quantile(0.75)
IQR = Q3 - Q1
df_clean = df[(df['salary'] >= Q1 - 1.5 * IQR) & (df['salary'] <= Q3 + 1.5 * IQR)]

print(df_clean)

5. Standardize Text Data


data = {'city': [' New York ', 'paris', 'BERLIN', 'new york ']}
df = [Link](data)

# Standardize text
df['city'] = df['city'].[Link]().[Link]()
print(df)

6. Handle Categorical Variables


data = {'gender': ['Male', 'Female', 'Female', 'Male']}
df = [Link](data)

# One-Hot Encoding
df_encoded = pd.get_dummies(df, columns=['gender'])
print(df_encoded)

7. Normalize or Scale Data


from [Link] import StandardScaler

data = {'age': [20, 25, 30, 35],


'salary': [20000, 30000, 40000, 50000]}
df = [Link](data)

# Standardization (Z-score scaling)


scaler = StandardScaler()
df[['age', 'salary']] = scaler.fit_transform(df[['age', 'salary']])
print(df)

8. Fix Inconsistent Formats


data = {'date': ['01-01-2024', '2024/02/01', 'March 3, 2024']}
df = [Link](data)

# Convert all dates to a single format


df['date'] = pd.to_datetime(df['date'])
print(df)

9. Validate Data Integrity


data = {'id': [1, 2, 2, 4],
'age': [25, -5, 30, 40]}
df = [Link](data)
# Remove duplicate IDs
df = df.drop_duplicates(subset='id')

# Remove invalid ages


df = df[df['age'] > 0]
print(df)

10. Document Every Step


# Example workflow with comments and checkpoints

# Step 1: Load data


df = pd.read_csv("[Link]")

# Step 2: Handle missing values


df['age'].fillna(df['age'].median(), inplace=True)

# Step 3: Fix data types


df['date'] = pd.to_datetime(df['date'])

# Step 4: Remove duplicates


df.drop_duplicates(inplace=True)

# Step 5: Save cleaned data


df.to_csv("cleaned_dataset.csv", index=False)
print("Data cleaning completed and saved.")

You might also like