Name:Jassika

Class:BCA with IBM(AI&DS)

Reg No.: 72311364

SUBMITTED TO: Rashi Prashar

Q1: Handling Missing Data in DataFrames and Data Aggregation and Grouping with
Pandas?

Handling missing data in Pandas involves techniques like detecting, removing, or

imputing missing values, while aggregation and grouping allow for summarizing data
based on specific criteria. Utilizing methods like groupby can help manage missing
values effectively during data analysis.

Handling Missing Data in Pandas

• Types of Missing Values:

• None: A Python object representing missing values in object-type arrays.

• NaN: A special floating-point value from NumPy recognized by systems

using IEEE floating-point standards.

• Detecting Missing Values:

• Use isnull() or isna() to identify missing values in a DataFrame.

• Example:

import pandas as pd

import numpy as np

data = {'First Score': [100, 90, [Link], 95],

'Second Score': [30, 45, 56, [Link]],

'Third Score': [[Link], 40, 80, 98]}

df = [Link](data)
 missing_values = [Link]()

print(missing_values)

Removing Missing Values:

• Use dropna() to remove rows or columns with missing values.

• Example to drop rows with any missing values:

df_cleaned = [Link]()

Filling Missing Values:

• Use fillna() to replace missing values with a specified value or method.

• Example to fill with zero:

df_filled = [Link](0)

Data Aggregation and Grouping with Pandas

• GroupBy Functionality:

• The groupby() method allows you to group data based on one or more
keys.

• Example:

grouped = [Link]('First Score')

Aggregation Methods:

• Common aggregation functions include sum(), mean(), median(), min(),

and max().

• Example to calculate the mean of a grouped DataFrame:

mean_scores = [Link]('First Score')['Second Score'].mean()

Applying Functions:

• Use apply() to apply custom functions to each group.

• Example:

def custom_function(x):

return [Link]() - [Link]()

 result = [Link]('First Score').apply(custom_function)

Transforming Data:

• Use transform() to return a DataFrame with the same shape as the input.

• Example to center data by subtracting the group mean:

centered_data = [Link]('First Score').transform(lambda x: x - [Link]())

Filtering Groups:

• Use filter() to keep groups that meet certain criteria.

• Example to filter groups with a standard deviation greater than a threshold:

filtered_groups = [Link]('First Score').filter(lambda x: x['Second Score'].std() > 10)

By utilizing these techniques, you can effectively manage missing data and perform
data aggregation and grouping in Pandas, enhancing your data analysis capabilities.

Q2: Numerical Operations and Array Manipulation using NumPy?

Numerical operations and array manipulation using NumPy are fundamental for
efficient data processing in Python. NumPy provides a powerful array object and a
collection of functions to perform mathematical operations on these arrays. Here’s a
concise overview of key concepts and operations.

1. Importing NumPy
To use NumPy, you first need to import it:
import numpy as np

2. Creating Arrays
You can create NumPy arrays using various methods:

From a list or tuple:

arr = [Link]([1, 2, 3, 4])

Using built-in functions:

• Zeros:
zeros_array = [Link]((2, 3)) # 2x3 array of zeros

Ones: ones_array = [Link]((3, 2)) # 3x2 array of ones

Empty: empty_array = [Link]((2, 2)) # 2x2 array with uninitialized values

Arange: range_array = [Link](0, 10, 2) # Array of values from 0 to 10 with a step of 2

Linspace: linspace_array = [Link](0, 1, 5) # 5 values evenly spaced between 0

and 1

3. Array Properties
You can access various properties of NumPy arrays:
• Shape:
shape = [Link] # Returns the dimensions of the array

Size: size = [Link] # Total number of elements in the array

Data Type: dtype = [Link] # Data type of the array elements

4. Basic Numerical Operations

NumPy supports element-wise operations:
• Addition:
result = arr + 5 # Adds 5 to each element
Subtraction: result = arr - 2 # Subtracts 2 from each element

Multiplication: result = arr * 3 # Multiplies each element by 3

Division: result = arr / 2 # Divides each element by 2

Exponentiation: result = arr ** 2 # Squares each element

5. Array Manipulation

You can manipulate arrays in various ways:

• Reshaping:

reshaped_array = [Link]((2, 2)) # Reshape to 2x2

Flattening: flat_array = [Link]() # Convert to a 1D array

Transposing: transposed_array = arr.T # Transpose the array

Concatenation: concatenated_array = [Link]((arr1, arr2), axis=0) #

Concatenate along the first axis

. Advanced Operations

• Broadcasting: NumPy automatically expands the dimensions of arrays to

perform operations on arrays of different shapes.

• Statistical Functions:

mean_value = [Link](arr) # Mean of the array

 sum_value = [Link](arr) # Sum of the array

std_dev = [Link](arr) # Standard deviation

Logical Operations: boolean_array = arr > 2 # Returns a boolean array where the
condition is met

Indexing and Slicing

You can access and modify elements using indexing and slicing:

• Indexing:

element = arr[0] # Access the first element

Slicing: sub_array = arr[1:3] # Access elements from index 1 to 2

Boolean Indexing: filtered_array = arr[arr > 2] # Get elements greater than 2

Creating visualizations such as line charts, bar charts, histograms, and pie charts using
Matplotlib is straightforward. Below is a guide on how to create each type of chart with
examples.

1. Importing Matplotlib

First, you need to import the necessary libraries:

import [Link] as plt

import numpy as np

. Creating a Basic Line Chart

A line chart is useful for showing trends over time.

# Sample data

x = [Link](0, 10, 100) # 100 points from 0 to 10

y = [Link](x) # Sine function

# Create a line chart

[Link](figsize=(10, 5))

[Link](x, y, label='Sine Wave', color='blue')

[Link]('Basic Line Chart')

[Link]('X-axis')

[Link]('Y-axis')

[Link]()

[Link]()

[Link]()

. Creating a Bar Chart

A bar chart is useful for comparing quantities corresponding to different groups.

# Sample data

categories = ['A', 'B', 'C', 'D']

values = [3, 7, 5, 2]

# Create a bar chart

[Link](figsize=(10, 5))

[Link](categories, values, color='orange')

[Link]('Basic Bar Chart')

[Link]('Categories')

[Link]('Values')

[Link]()

. Creating a Histogram

A histogram is useful for showing the distribution of a dataset.

# Sample data

data = [Link](1000) # 1000 random numbers from a normal distribution

# Create a histogram

[Link](figsize=(10, 5))

[Link](data, bins=30, color='green', alpha=0.7)

[Link]('Basic Histogram')

[Link]('Value')
[Link]('Frequency')

[Link]()

[Link]()

Creating a Pie Chart

A pie chart is useful for showing the proportions of a whole.

# Sample data

sizes = [15, 30, 45, 10] # Proportions

labels = ['Category A', 'Category B', 'Category C', 'Category D']

colors = ['gold', 'lightcoral', 'lightskyblue', 'lightgreen']

# Create a pie chart

[Link](figsize=(8, 8))

[Link](sizes, labels=labels, colors=colors, autopct='%1.1f%%', startangle=140)

[Link]('Basic Pie Chart')

[Link]('equal') # Equal aspect ratio ensures that pie is drawn as a circle.

[Link]()

Q3: Predicting House Prices Using Linear Regression?

To predict house prices using linear regression in Python, you typically start by preparing
your dataset, which includes features like size, location, and number of rooms. Then,
you can use libraries like Scikit-Learn to build and train your linear regression model,
followed by evaluating its performance on test data.

Steps to Predict House Prices Using Linear Regression

1. Import Required Libraries

Start by importing the necessary libraries for data manipulation, visualization, and
model building.

import pandas as pd

import numpy as np
import [Link] as plt

import seaborn as sns

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LinearRegression

from [Link] import mean_squared_error, mean_absolute_error

Load the Dataset

Load your dataset, which should contain features relevant to house pricing.

# Load the dataset

df = pd.read_csv('house_prices.csv') # Replace with your dataset path

Data Preprocessing

• Check for Missing Values: Identify and handle any missing values in your
dataset.

print([Link]().sum())

[Link]([Link](), inplace=True) # Example: Filling missing values with mean

• Feature Selection: Choose the features that will be used for prediction.

X = df[['Size', 'Bedrooms', 'Location']] # Example features

y = df['Price'] # Target variable

. Split the Dataset

Divide the dataset into training and testing sets to evaluate the model's performance.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Create and Train the Model

Instantiate the linear regression model and fit it to the training data.

model = LinearRegression()

[Link](X_train, y_train)

Make Predictions

Use the trained model to make predictions on the test set.

y_pred = [Link](X_test)

Evaluate the Model

Assess the model's performance using metrics such as Mean Absolute Error (MAE) and
Mean Squared Error (MSE).

mae = mean_absolute_error(y_test, y_pred)

mse = mean_squared_error(y_test, y_pred)

print(f'Mean Absolute Error: {mae}')

print(f'Mean Squared Error: {mse}')

Visualize the Results

Visualize the predicted vs actual prices to understand the model's performance better.

[Link](y_test, y_pred)

[Link]('Actual Prices')

[Link]('Predicted Prices')

[Link]('Actual vs Predicted Prices')

[Link]([min(y_test), max(y_test)], [min(y_test), max(y_test)], color='red') # Diagonal

line

[Link]()

Q5:Predicting Student Performance using Logistic Regression?

Logistic regression is a statistical method used to predict the probability of a binary

outcome based on one or more predictor variables. In the context of student
performance, it can effectively model factors influencing academic success, achieving
high accuracy and precision in predictions.

Steps to Predict Student Performance Using Logistic Regression

1. Import Required Libraries

Begin by importing the necessary libraries for data manipulation, visualization, and
model building.

2. Load the Dataset

Load your dataset, which should contain features relevant to student performance.

3. Data Preprocessing
 • Handle Categorical Variables: Convert categorical features into numerical
values using label encoding

• Create a Mean Score Column: Combine the individual scores into a mean
score for easier analysis.

. Define Features and Target Variable:Identify the features (independent variables)

and the target variable (dependent variable).
5. Split the Dataset

Divide the dataset into training and testing sets to evaluate the model's performance.

6. Create and Train the Model

Instantiate the logistic regression model and fit it to the training data.

7. Make Predictions

Use the trained model to make predictions on the test set.

import pandas as pd

import numpy as np

import [Link] as plt

import seaborn as sns

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression

from [Link] import accuracy_score, confusion_matrix

# Load the dataset

df = pd.read_csv('[Link]') # Replace with your dataset path

from [Link] import LabelEncoder

lc = LabelEncoder()

df['lunch'] = lc.fit_transform(df['lunch'])

df['test preparation course'] = lc.fit_transform(df['test preparation course'])

df['gender'] = lc.fit_transform(df['gender'])

df['race/ethnicity'] = lc.fit_transform(df['race/ethnicity'])

df['parental level of education'] = lc.fit_transform(df['parental level of education'])

df["mean score"] = ((df["math score"] + df["reading score"] + df["writing score"]) /
3).round()

df = [Link](['math score', 'reading score', 'writing score'], axis=1)

X = [Link](['mean score'], axis=1) # Features

y = (df['mean score'] >= 70).astype(int) # Target variable: 1 if mean score >= 70, else 0

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

y_pred = [Link](X_test)

Q6: Outlier Detection using Z-Score and IQR Methods?

Outlier detection is an important step in data analysis, as outliers can significantly

affect the results of statistical analyses and machine learning models. Two common
methods for detecting outliers are the Z-Score method and the Interquartile Range (IQR)
method. Below is a simple guide on how to use both methods for outlier detection.

1. Import Required Libraries

First, you need to import the necessary libraries for data manipulation and visualization.

import pandas as pd

import numpy as np

import [Link] as plt

import seaborn as sns

2. Create a Sample Dataset

For demonstration purposes, let's create a sample dataset with some outliers.

# Create a sample dataset

data = {'Values': [10, 12, 12, 13, 12, 14, 15, 16, 18, 19, 20, 100]} # 100 is an outlier

df = [Link](data)

3. Outlier Detection Using Z-Score Method

The Z-Score method identifies outliers based on how many standard deviations a data
point is from the mean. A common threshold is a Z-Score of 3 or -3.

# Calculate Z-Scores
df['Z-Score'] = (df['Values'] - df['Values'].mean()) / df['Values'].std()

# Identify outliers

z_threshold = 3

outliers_z = df[[Link](df['Z-Score']) > z_threshold]

print("Outliers using Z-Score method:")

print(outliers_z)

4. Outlier Detection Using IQR Method

The IQR method identifies outliers based on the interquartile range, which is the range
between the first quartile (Q1) and the third quartile (Q3). Outliers are typically defined
as points that fall below $ Q1 - 1.5 \times IQR $ or above $ Q3 + 1.5 \times IQR $.

# Calculate Q1 and Q3

Q1 = df['Values'].quantile(0.25)

Q3 = df['Values'].quantile(0.75)

IQR = Q3 - Q1

# Identify outliers

lower_bound = Q1 - 1.5 * IQR

upper_bound = Q3 + 1.5 * IQR

outliers_iqr = df[(df['Values'] < lower_bound) | (df['Values'] > upper_bound)]

print("Outliers using IQR method:")

print(outliers_iqr)

5. Visualizing Outliers

You can visualize the data and the detected outliers using box plots.

[Link](figsize=(10, 5))

# Box plot for IQR method

[Link](1, 2, 1)

[Link](x=df['Values'])

[Link]('Box Plot (IQR Method)')

# Scatter plot for Z-Score method

[Link](1, 2, 2)

[Link](df['Values'], np.zeros_like(df['Values']), alpha=0.5)

[Link](outliers_z['Values'], np.zeros_like(outliers_z['Values']), color='red',

label='Outliers (Z-Score)', zorder=5)

[Link]('Scatter Plot (Z-Score Method)')

[Link]()

plt.tight_layout()

[Link]()

Q7: Implementing Normalization and Standardization on a Dataset?

Normalization and standardization are two common techniques used to preprocess

data, especially when preparing it for machine learning algorithms. Below is a simple
guide on how to implement both techniques using Python with the help of libraries like
Pandas and Scikit-Learn.

1. Import Required Libraries

First, you need to import the necessary libraries for data manipulation and
preprocessing.

import pandas as pd

import numpy as np

from [Link] import MinMaxScaler, StandardScaler

2. Create a Sample Dataset

For demonstration purposes, let's create a sample dataset.

# Create a sample dataset

data = {

'Feature1': [10, 20, 30, 40, 50],

'Feature2': [100, 200, 300, 400, 500],

'Feature3': [1, 2, 3, 4, 5]

df = [Link](data)

print("Original Dataset:")

print(df)

3. Normalization

Normalization (Min-Max Scaling) rescales the features to a fixed range, usually [0, 1].
This is useful when you want to ensure that all features contribute equally to the
distance calculations in algorithms like K-Nearest Neighbors.

# Normalize the dataset

scaler = MinMaxScaler()

normalized_data = scaler.fit_transform(df)

# Convert the normalized data back to a DataFrame

normalized_df = [Link](normalized_data, columns=[Link])

print("\nNormalized Dataset:")

print(normalized_df)

4. Standardization

Standardization (Z-score normalization) rescales the features so that they have a mean
of 0 and a standard deviation of 1. This is useful when the data follows a Gaussian
distribution.

# Standardize the dataset

scaler = StandardScaler()

standardized_data = scaler.fit_transform(df)

# Convert the standardized data back to a DataFrame

standardized_df = [Link](standardized_data, columns=[Link])

print("\nStandardized Dataset:")

print(standardized_df)

5. Summary of Results

You can summarize the results to see how normalization and standardization have
transformed the data.

print("\nSummary of Original, Normalized, and Standardized Data:")

print("Original Data:\n", [Link]())

print("\nNormalized Data:\n", normalized_df.describe())

print("\nStandardized Data:\n", standardized_df.describe())

Q8: Implementing Encoding Techniques for Categorical Data?

Encoding categorical data is an essential step in data preprocessing, especially when

preparing data for machine learning algorithms that require numerical input. Below are
some common encoding techniques, including Label Encoding and One-Hot Encoding,
along with examples of how to implement them using Python with the Pandas library.

1. Import Required Libraries

First, you need to import the necessary libraries.

import pandas as pd

2. Create a Sample Dataset

For demonstration purposes, let's create a sample dataset with categorical features.

# Create a sample dataset

data = {

'Gender': ['Male', 'Female', 'Female', 'Male', 'Female'],

'City': ['New York', 'Los Angeles', 'New York', 'Chicago', 'Los Angeles'],

'Score': [85, 90, 78, 88, 92]

df = [Link](data)
print("Original Dataset:")

print(df)

3. Label Encoding

Label Encoding converts categorical values into numerical values. Each unique
category is assigned an integer value.

from [Link] import LabelEncoder

# Initialize the LabelEncoder

label_encoder = LabelEncoder()

# Apply Label Encoding to the 'Gender' column

df['Gender_Encoded'] = label_encoder.fit_transform(df['Gender'])

print("\nDataset after Label Encoding:")

print(df)

4. One-Hot Encoding

One-Hot Encoding creates binary columns for each category in a categorical feature.
This is useful for nominal categorical variables where there is no ordinal relationship.

# Apply One-Hot Encoding to the 'City' column

df_one_hot = pd.get_dummies(df, columns=['City'], prefix='City')

print("\nDataset after One-Hot Encoding:")

print(df_one_hot)

5. Combining Encoded Features

If you want to keep both the original and encoded features, you can combine them as
needed.

# Combine the original DataFrame with the one-hot encoded DataFrame

final_df = [Link](['City'], axis=1).join(df_one_hot[['City_New York', 'City_Los Angeles',

'City_Chicago']])
print("\nFinal Dataset with Encoded Features:")

print(final_df)

Q9: Customer Segmentation Using K-Means Clustering?

Customer segmentation using K-Means clustering involves grouping customers based

on their characteristics to identify distinct segments for targeted marketing. Various
resources, including tutorials and practical guides, can help you implement this
technique in Python effectively.

Overview of Customer Segmentation

• Purpose: To divide customers into distinct groups based on shared

characteristics such as demographics, spending behavior, and preferences.

• Benefits: Enables targeted marketing strategies, improves customer

engagement, and enhances product offerings.

Steps for Implementing K-Means Clustering

1. Data Preparation

• Import necessary libraries and load the dataset.

• Clean the data by handling missing values and encoding categorical

variables.

import pandas as pd

from [Link] import StandardScaler

# Load dataset

data = pd.read_csv('Mall_Customers.csv')

Feature Selection

• Select relevant features for clustering, typically Annual Income and Spending
Score.

X = data[['Annual Income (k$)', 'Spending Score (1-100)']].values

Data Standardization

• Standardize the features to ensure they contribute equally to the distance

calculations.
 scaler = StandardScaler()

X_scaled = scaler.fit_transform(X)

Determine Optimal Number of Clusters (K)

• Use the Elbow Method to find the optimal number of clusters by plotting the
Within-Cluster Sum of Squares (WCSS).

from [Link] import KMeans

import [Link] as plt

wcss = []

for i in range(1, 11):

kmeans = KMeans(n_clusters=i, init='k-means++', random_state=42)

[Link](X_scaled)

[Link](kmeans.inertia_)

[Link](range(1, 11), wcss)

[Link]('Elbow Method')

[Link]('Number of clusters')

[Link]('WCSS')

[Link]()

Apply K-Means Clustering

• Fit the K-Means algorithm to the data using the optimal number of clusters
determined from the Elbow Method.

optimal_k = 4 # Example optimal value from the Elbow Method

kmeans = KMeans(n_clusters=optimal_k, init='k-means++', random_state=42)

y_kmeans = kmeans.fit_predict(X_scaled)

Visualize the Clusters

• Create a scatter plot to visualize the clusters and their centroids.

[Link](X_scaled[y_kmeans == 0, 0], X_scaled[y_kmeans == 0, 1], s=100, c='red',

label='Cluster 1')
 [Link](X_scaled[y_kmeans == 1, 0], X_scaled[y_kmeans == 1, 1], s=100, c='blue',
label='Cluster 2')

[Link](X_scaled[y_kmeans == 2, 0], X_scaled[y_kmeans == 2, 1], s=100, c='green',

label='Cluster 3')

[Link](X_scaled[y_kmeans == 3, 0], X_scaled[y_kmeans == 3, 1], s=100, c='cyan',

label='Cluster 4')

[Link](kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], s=300,

c='yellow', label='Centroids')

[Link]('Customer Segments')

[Link]('Annual Income (scaled)')

[Link]('Spending Score (scaled)')

[Link]()

[Link]()

Q10: Market Basket Analysis Using Apriori Algorithm?

Market Basket Analysis (MBA) is a data mining technique used to identify patterns in
transaction data, particularly to find associations between items purchased together.
The Apriori algorithm is a popular method for performing market basket analysis. Below
is a simple guide on how to implement the Apriori algorithm using Python.

Steps for Market Basket Analysis Using the Apriori Algorithm

1. Import Required Libraries

First, you need to import the necessary libraries.

import pandas as pd

from mlxtend.frequent_patterns import apriori, association_rules

2. Load the Dataset

Load your transaction dataset. For this example, we will create a sample dataset.

# Sample transaction data

data = {

'TransactionID': [1, 1, 1, 2, 2, 3, 3, 3, 4, 4],

 'Item': ['Bread', 'Milk', 'Eggs', 'Bread', 'Diaper', 'Milk', 'Diaper', 'Beer', 'Bread', 'Diaper']

df = [Link](data)

print("Original Dataset:")

print(df)

3. repare the Data for Apriori Algorithm

Transform the dataset into a format suitable for the Apriori algorithm, typically a one-hot
encoded DataFrame.

# Create a basket matrix

basket = [Link](['TransactionID',
'Item'])['Item'].count().unstack().reset_index().fillna(0).set_index('TransactionID')

# Convert counts to 1s and 0s

basket = [Link](lambda x: 1 if x > 0 else 0)

print("\nBasket Matrix:")

print(basket)

4. Apply the Apriori Algorithm

Use the Apriori algorithm to find frequent itemsets.

# Apply the Apriori algorithm

frequent_itemsets = apriori(basket, min_support=0.2, use_colnames=True)

print("\nFrequent Itemsets:")

print(frequent_itemsets)

5. Generate Association Rules

Generate association rules from the frequent itemsets.

# Generate association rules

rules = association_rules(frequent_itemsets, metric="confidence", min_threshold=0.5)

print("\nAssociation Rules:")

print(rules)
 6. Interpret the Results

The resulting DataFrame contains the association rules, including metrics like support,
confidence, and lift, which help evaluate the strength of the rules.