DATA MINING USING PYTHON LAB MANUAL
[Link] Using Numpy and Pandas for data manipulation.
import numpy as np
import pandas as pd
# Create a 1D array
arr = [Link]([1, 2, 3, 4, 5, 6])
print("Original array:", arr)
# Reshape to a 2x3 matrix
reshaped_arr = [Link](2, 3)
print("Reshaped array (2x3):", reshaped_arr)
Output:
Reshaped array (2x3):[[1 2 3]
[4 5 6]]
toy_prices = [Link]([5, 8, 3, 6])
# Decrease all prices by 2 without a loop
sale_prices = toy_prices - 2
print("Prices after $2 sale:", sale_prices)
Output:
Prices after $2 sale: [3 6 1 4]
data = [Link]([10, 20, 30, 40, 50])
print("Mean:", [Link](data))
print("Median:", [Link](data))
print("Sum:", [Link](data))
Output:
Mean: 30.0
Median: 30.0
Sum: 150
data = {
'City': ['Tokyo', 'Delhi', 'Shanghai', 'Sao Paulo', 'Mumbai'],
'Country': ['Japan', 'India', 'China', 'Brazil', 'India'],
'Population_Millions': [37, 32, 28, 22, 21]
}
df = [Link](data)
print("Original DataFrame:")
print(df)
Output:
Original DataFrame:
City Country Population_Millions
0 Tokyo Japan 37
1 Delhi India 32
2 Shanghai China 28
3 Sao Paulo Brazil 22
4 Mumbai India 21
# Select only cities in India
india_cities_df = df[df['Country'] == 'India']
print("\nCities in India:")
print(india_cities_df)
Output:
Cities in India:
City Country Population_Millions
1 Delhi India 32
4 Mumbai India 21
[Link] and handling missing values using Pandas, Normalization and scaling
techniques, one-hot encoding for categorical data.
Implementation for Data Cleaning
Database Cleaning using titanic dataset.
Step 1: Import Libraries and Load Dataset
We will import all the necessary libraries i.e pandas and numpy.
Output:
Step 2: Check for Duplicate Rows
[Link](): Returns a boolean Series indicating duplicate rows.
[Link]()
Output:
Step 3: Identify Column Data Types
List comprehension with .dtype attribute to separate categorical and numerical columns.
object dtype: Generally used for text or categorical data.
Output:
Step 4: Count Unique Values in the Categorical Columns
df[numeric_columns].nunique(): Returns count of unique values per column.
Output:
Step 5: Calculate Missing Values as Percentage
[Link](): Detects missing values, returning boolean DataFrame.
Sum missing across columns, normalize by total rows and multiply by 100.
Output:
Step 6: Drop Irrelevant or Data-Heavy Missing Columns
[Link](columns=[]): Drops specified columns from the DataFrame.
[Link](subset=[]): Removes rows where specified columns have missing values.
fillna(): Fills missing values with specified value (e.g., mean).
Step 7: Detect Outliers with Box Plot
[Link](): Displays distribution of data, highlighting median, quartiles and
outliers.
[Link](): Renders the plot.
Output:
Step 8: Data formatting (Normalization)
Output:
[Link] data using Matplotlib and Seaborn, Calculating summary statistics and distribution
analysis, Identifying and handling outliers.
Step 1: Importing necessary Libraries
We will be using Numpy, Pandas, Matplotlib, Seaborn and Scikit-learn libraries for its
implementation.
Step 2: Loading the Dataset
Output:
Step 3: Identifying and Removing invalid Blood Types
Output:
unique_blood_types_main = set(main_data['blood_type'])
valid_blood_types_set = set(blood_type_categories['blood_type'])
invalid_blood_types = unique_blood_types_main.difference(valid_blood_types_set)
invalid_blood_types
Output:
{'C+', 'D-'}
Once the invalid values are found the corresponding rows can be dropped from the dataset.
invalid_records_index = main_data['blood_type'].isin(invalid_blood_types)
without_invalid_records = main_data[~invalid_records_index].copy()
without_invalid_records['blood_type'].unique()
Output:
array(['A+', 'B+', 'A-', 'AB-', 'AB+', 'B-', 'O-', 'O+'], dtype=object)
Step 4: Handling Inconsistent Marriage Status Categories
Output:
Standardizing the categories by converting all text to lowercase.
Output:
Now we will standardize the categories by stripping extra spaces:
Output:
Step 5: Grouping Income into Meaningful Bins
:
Output:
Now, let us create the range and labels for the income feature. Pandas cut method is used here.
Output:
Step 6: Visualizing Income Group Distribution
Now lets visualize the distribution of income groups:
Output:
Step 7: Cleaning Phone Number Data
Simulating phone numbers with inconsistent formats and cleaning them:
Output:
Step 8: Visualizing Categorical Data
Now we can see the relationship between income and the marital status of a person using
a boxplot.
Output:
Step 9: Encoding Categorical Data
Certain learning algorithms like regression and neural networks require their input to be
numbers. Hence categorical data must be converted to numbers to use these algorithms. Let us
see some encoding methods.
1. Label Encoding
2. One-hot Encoding in Python
Output:
[Link] Apriori Algorithm to find frequent item sets, generating Association rules for
interpreting and evaluation.
import pandas as pd
from mlxtend.frequent_patterns import apriori, association_rules
from [Link] import TransactionEncoder
# 1. Prepare/Load Data: Apriori needs data in a one-hot encoded format.
# Assume 'transactions' is a list of lists (e.g., [['milk', 'bread'],
['milk', 'butter']]).
# The mlxtend TransactionEncoder can convert this format.
# For a full example of data loading and preparation, refer to the
[GeeksforGeeks
tutorial]([Link]
apriori-algorithm-in-python/).
# Example of one-hot encoding a simple dataset:
dataset = [
['milk', 'bread', 'butter'],
['milk', 'butter'],
['milk', 'diapers', 'beer'],
['milk', 'bread', 'diapers', 'beer'],
['bread', 'butter', 'diapers']
]
te = TransactionEncoder()
te_array = [Link](dataset).transform(dataset)
df_encoded = [Link](te_array, columns=te.columns_)
# 2. Find Frequent Item Sets: Set a minimum support threshold.
# 'min_support=0.6' means the itemset must appear in at least 60% of
transactions.
frequent_itemsets = apriori(df_encoded, min_support=0.6, use_colnames=True)
print("Frequent Itemsets:")
print(frequent_itemsets)
# 3. Generate Association Rules: Set a minimum confidence or lift threshold.
# 'min_threshold=0.7' for confidence or 'metric="lift", min_threshold=1' for
lift.
rules = association_rules(frequent_itemsets, metric="confidence",
min_threshold=0.7)
print("\nAssociation Rules:")
print(rules)
Output:
Frequent Itemsets:
support itemsets
0 0.8 {milk}
1 0.6 {bread}
2 0.6 {butter}
3 0.6 {diapers}
4 0.6 {milk, bread}
5 0.6 {milk, butter}
Association Rules:
Antecedents consequents support confidence lift
0 {milk} {bread} 0.6 0.75 1.25
1 {bread} {milk} 0.6 1.00 1.25
2 {milk} {butter} 0.6 0.75 1.25
3 {butter} {milk} 0.6 1.00 1.25
[Link] K-means Clustering Algorithm, hierarchical clustering using Scipy , visualizing
Clustering and evaluating clustering results.
K-means Clustering Algorithm
Output:
Agglomerative hierarchical clustering
Output :
[Link] principle Component Analysis (PCA) for feature reduction, Visualizing reduced dimensional
data, Understanding the trade-offs dimensionality reduction.
Step 1: Importing Required Libraries
We import the necessary library like pandas, numpy, scikit
learn, seaborn and matplotlib to visualize results.
Step 2: Creating Sample Dataset
We make a small dataset with three features Height, Weight, Age and Gender.
Output:
Step 3: Standardizing the Data
Step 4: Applying PCA algorithm
Step 5: Evaluating with Confusion Matrix
Output:
Step 6: Visualizing PCA Result
Output:
[Link] decision tree and random forest classifiers, training and evaluating a Naïve Bayes
classifier, Comparing classification performance metrics.
Decision Tree
Step 1: Install Libraries and Import Dependencies
Import pandas, numpy for data handling and matplotlib for plotting.
import scikit learn for Decision Tree implementation
Import metrics for evaluating model performance
Step 2: Import Dataset
Load the dataset from the UCI repository.
Display dataset length, shape and first few rows.
Returns the dataset for further processing.
Step 3: Split Dataset into Features and Labels
Separate input features (X) and target labels (Y).
Split data into training and testing sets.
Return both the complete dataset and split sets for modeling.
Step 4: Train Decision Tree Using Gini Index
Initialize DecisionTreeClassifier with gini criterion.
Set max_depth and min_samples_leaf to control tree complexity.
Fit the model on training data and return the trained classifier.
Step 5: Train Decision Tree Using Entropy
Initialize classifier with entropy criterion.
Same depth and leaf settings to compare with Gini.
Fit the model on training data and return the trained classifier.
Step 6: Make Predictions
Use the trained classifier to predict target labels for the test set.
Step 7: Evaluate Model Accuracy
Calculate and display the confusion matrix.
Compute accuracy score.
Show detailed classification report including precision, recall and F1-score.
Step 8: Visualize the Decision Tree
Plot the trained decision tree using matplotlib.
Include feature names and class names for better readability.
Output:
Random Forest
1. Import Required Libraries
We will be importing Pandas, matplotlib, seaborn and sklearn to build the model.
2. Import Dataset
For this we'll use the Iris Dataset which is available within scikit learn. This dataset contains
information about three types of Iris flowers and their respective features (sepal length, sepal
width, petal length and petal width).
Output:
3. Data Preparation
Here we will separate the features (X) and the target variable (y).
4. Splitting the Dataset
We'll split the dataset into training and testing sets so we can train the model on one part and
evaluate it on another.
X_train, y_train: 80% of the data used to train the model.
X_test, y_test: 20% of the data used to test the model.
test_size=0.2: means 20% of data goes to testing.
random_state=42: ensures you get the same split every time
5. Feature Scaling
6. Building Random Forest Classifier
7. Evaluation of the Model
We will evaluate the model using the accuracy score and confusion matrix.
Output:
Accuracy: 100.00%
Naïve Bayes classifier, Comparing classification performance metrics
1. Importing Libraries
Importing necessary libraries:
math: for mathematical operations
random: for random number generation
pandas: for data manipulation
numpy: for scientific computing
2. Encoding Class
The encode_class function converts class labels in the dataset into numeric values. It assigns a
unique numeric identifier to each class.
3. Splitting the Data
The splitting function is used to split the dataset into training and testing sets based on the given
ratio.
4. Grouping Data by Class
The groupUnderClass function takes the data and returns a dictionary where each key is a class
label and the value is a list of data points belonging to that class.
5. Calculating Mean and Standard Deviation for Class
The MeanAndStdDev function takes a list of numbers and calculates the mean and standard
deviation.
The MeanAndStdDevForClass function takes the data and returns a dictionary where each
key is a class label and the value is a list of lists, where each inner list contains the mean and
standard deviation for each attribute of the class.
6. Calculating Gaussian and Class Probabilities
7. Predicting for Test Set
8. Calculating Accuracy
9. Loading and Preprocessing Data
10. Splitting Data into Training and Testing Sets
Output:
Total number of examples: 768
Training examples: 537
Test examples: 231
11. Training and Testing the Model
Output:
Accuracy of the model: 100.0
12. Evaluating Model
We will plot different types of visualizations for evaluation:
1. Confusion Matrix
The confusion matrix summarizes prediction results by showing true positives, false positives,
true negatives and false negatives. It helps visualize how well the classifier distinguishes
between different classes.
Output:
2. Precision, Recall and F1 score
The F1 score is the harmonic mean of precision and recall, balancing both metrics into a single
value. It’s useful when the class distribution is imbalanced or when false positives and false
negatives are costly.
Output:
8. Implementing Linear regression for predicting numerical values, Analyzing and
visualizing time series data.
Step 1: Import Libraries and Load Data
# Importing the Required libraries
import pandas as pd
import numpy as np
import [Link] as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from [Link] import mean_squared_error
# Loading the dataset from the URL
url = '[Link]
data = pd.read_csv(url, parse_dates=['Month'], index_col='Month')
[Link]()
Step 2: Visualize the Time Series Data
# plotting the time series data to visualize the trends or patterns
[Link](figsize=(10, 6))
[Link](data, label='Monthly airline passengers')
[Link]('Airline Passengers Over Time')
[Link]('Date')
[Link]('Number of Passengers')
[Link]()
# saving the plot as an image
[Link]('timeseries_plot.png')
[Link]()
Step 3: Data Preparation for Linear Regression
We can create lagged features to do this task, and then you can split the final data into the
training and testing sets in proportion of 80% and 20% respectively.
Step 4: Fitting the Linear Regression Model
Now train the linear regression model using the training data available. Below is the code for
fitting the linear regression model.
# Initializing the linear regression model
model = LinearRegression()
# Training the model on the training data available
[Link](X_train, y_train)
Step 5: Make Predictions
Now predict the values for the testing set, using the trained model. Below is the code for making
predictions.
# Making predictions on the testing data
y_pred = [Link](X_test)
Step 6: Model Evaluation
In this step, you need to evaluate the model performance using evaluation metrics such as Mean
Squared Error (MSE). Below is the code for printing the value of mean squared error of the
model.
# Evaluating the models using MSE metric
mse = mean_squared_error(y_test, y_pred)
print(f'Mean Squared Error: {mse}')
Step 7: Visualize the Results
Finally plot the actual vs predicted values to visually understand the performance of the model.
# Plotting the actual vs predicted values inorder to visualize model performance
[Link](figsize=(10, 6))
[Link]([Link], data['Passengers'], label='Actual')
[Link](X_test.index, y_pred, label='Predicted', color='red')
[Link]('Actual vs Predicted Airline Passengers')
[Link]('Date')
[Link]('Number of Passengers')
[Link]()
# Saving the plot as an image
[Link]('actual_vs_predicted.png')
[Link]()
Output:
Mean Squared Error: 5450.723647259961
9. Forecasting using basic time series data.
import pandas as pd
import numpy as np
import [Link] as plt
from [Link] import ExponentialSmoothing
# 1. Generate Basic Time Series Data (10 years, monthly)
[Link](42)
periods = 120
time_index = pd.date_range("2015-01-01", periods=periods, freq="M")
# Trend + Seasonality + Noise
trend = [Link](10, 50, periods)
seasonality = 10 * [Link](2 * [Link] * [Link](periods) / 12)
noise = [Link](0, 2, periods)
ts = trend + seasonality + noise
# Put into DataFrame
df = [Link]({'value': ts}, index=time_index)
# 2. Split Data (Train: 9 years, Test: 1 year)
train = [Link][:-12]
test = [Link][-12:]
# 3. Build and Fit Model (Holt-Winters Exponential Smoothing)
model = ExponentialSmoothing(train['value'], trend='add', seasonal='add',
seasonal_periods=12)
fit = [Link]()
# 4. Forecast
forecast = [Link](12)
# 5. Plot Results
[Link](figsize=(10, 6))
[Link]([Link], train['value'], label='Train')
[Link]([Link], test['value'], label='Test')
[Link]([Link], forecast, label='Forecast', color='red')
[Link]()
[Link]("Time Series Forecasting Using Exponential Smoothing")
[Link]()
Output:
Time Series Forecasting Using Exponential Smoothing