Python for Data Science MCS104A

Python for Data Science

Module 1
Data Science is an interdisciplinary field that utilizes scientific methods, processes, algorithms,
and systems to extract insights and knowledge from structured and unstructured data. It combines
various techniques from statistics, mathematics, computer science, and domain- specific
knowledge to analyze data and inform decision-making. Data Science encompasses several areas,
including data mining, data analysis, machine learning, and data visualization.
Skill Sets Required for Data Scientists
Data scientists typically possess a diverse skill set that includes technical, analytical, and soft skills.
Essential skills:
1. Technical Skills
● Programming Languages: Proficiency in languages such as Python and R, which are widely used
for data analysis and machine learning. ● Data Manipulation and Analysis: Familiarity with
libraries like Pandas, NumPy, andtools like SQL for data manipulation and analysis. ● Machine
Learning: Understanding machine learning algorithms and frameworks (e.g., Scikit-learn,
TensorFlow, Keras) for building predictive models. ● Data Visualization: Skills in visualization
libraries (e.g., Matplotlib, Seaborn, Plotly) and tools (e.g., Tableau, Power BI) for creating
insightful visualizations.
● Statistical Analysis: Strong foundation in statistics, including probability, hypothesis
testing, regression analysis, and statistical modeling.
2. Analytical Skills
● Problem Solving: Ability to identify problems, formulate questions, and develop analytical
solutions. ● Critical Thinking: Evaluating information and data critically to make informed
decisions and recommendations. ● Domain Knowledge: Understanding the specific industry (e.g.,
finance, healthcare, marketing) to interpret data within context and derive actionable insights. 3.
Soft Skills
● Communication: Ability to convey complex technical concepts to non-technical stakeholders
clearly and effectively, often through storytelling and visualization. ● Collaboration: Working
effectively in cross-functional teams with data engineers, business analysts, and domain experts.
● Adaptability: Willingness to learn and adapt to new tools, technologies, and methodologies as
the field evolves.
4. Data Engineering Skills (optional but beneficial)

1
ISE, JSSATEB
Python for Data Science MCS104A

● Database Management: Understanding of relational and non-relational databases (e.g., MySQL,

MongoDB) for efficient data storage and retrieval. ● Big Data Technologies: Familiarity with big
data frameworks (e.g., Hadoop, Spark) for
processing large datasets.

Disciplines of Data Science

Data Science is a broad field that incorporates several key disciplines, including:
[Link]:
•Foundation: Statistics is the backbone of Data Science, providing the mathematical framework
for collecting, analyzing, interpreting, and presenting data.
•Applications: It involves hypothesis testing, probability theory, inferential statistics,
and the development of statistical models to make predictions and understand underlying data
patterns.
[Link] Learning:
•Foundation: Machine Learning (ML) is a subset of artificial intelligence that involves building
algorithms that allow computers to learn from and make decisions based on data.
•Applications: ML techniques such as supervised learning, unsupervised learning, and
reinforcement learning are widely used in Data Science to predict outcomes, recognize patterns,
and automate processes.
[Link] Science and Software Engineering:
•Foundation: This discipline involves the design, development, and implementation of software
systems that can handle large datasets efÏciently.
•Applications: Key areas include data structures, algorithms, databases, and high-
performance computing, which are crucial for processing and analyzing large volumes of data.
[Link]:
•Foundation: Mathematics, particularly linear algebra, calculus, and discrete mathematics,
underpins many Data Science algorithms and models.
•Applications: It is essential for understanding the workings of machine learning algorithms,
optimization techniques, and statistical models.
[Link] Expertise:
•Foundation: Domain expertise refers to specialized knowledge in a particular field, such as
healthcare, finance, or marketing.

2
ISE, JSSATEB
Python for Data Science MCS104A

•Applications: Understanding the domain helps data scientists to frame the right
questions, interpret results accurately, and create solutions that are relevant and impactful.

[Link] Engineering:
•Foundation: Data Engineering focuses on the practical aspects of data management,
including the development, construction, and maintenance of scalable data pipelines and
infrastructure.
•Applications: It involves tasks like data warehousing, ETL (Extract, Transform, Load)
processes, and database management to ensure that data is clean, accessible, and ready for analysis.
[Link] Visualization:
•Foundation: Data Visualization is the art and science of representing data in visual formats like
graphs, charts, and dashboards to make the data understandable and actionable.
•Applications: Effective visualization helps in communicating complex data insights to
stakeholders clearly and concisely, facilitating better decision-making.
[Link] and Data Governance:
•Foundation: This discipline addresses the ethical implications of data use, ensuring that data
practices comply with legal and ethical standards.
•Applications: It involves data privacy, security, and governance policies that protect sensitive
information and ensure ethical use of [Link]
Datafication:
Datafication refers to the process of transforming various aspects of life and activities
intoquantifiable data. It involves the collection, storage, and analysis of data generated
fromeveryday activities, behaviors, and interactions, often through digital means. This
transformationallows for the extraction of insights and patterns from these data points that can
informdecision-making, enhance services, and drive innovation.
Key Features:
● Quantification: Almost any activity, whether physical or digital, can be quantified. For
example, social interactions on social media, purchasing behaviors, health metrics (likeheart rate),
and even environmental conditions can be turned into data.
● Collection: Datafication relies heavily on technology to collect data. This can involve sensors,
mobile devices, social media platforms, and other digital tools. ● Analysis: Once data is collected,
it can be analyzed to identify trends, correlations, and actionable insights. This analysis can be
used in various fields, including marketing, healthcare, urban planning, and more. ● Value

3
ISE, JSSATEB
Python for Data Science MCS104A

Creation: Datafication can lead to enhanced decision-making, improved customer experiences, and
the creation of new business models by leveraging data insights.
Examples:
● Social Media: Platforms like Facebook and Twitter collect data on user interactions, preferences,
and behaviors, allowing companies to target advertising more effectively.
● Smart Devices: IoT (Internet of Things) devices, such as fitness trackers, collect data on user
health and activity levels, enabling personalized health recommendations. ● Transportation:
Companies like Uber collect data on ride patterns, traffic conditions, and user preferences to
optimize services and pricing. ii) Statistical Modelling

Statistical modelling is a mathematical framework used to represent and analyze relationships

between variables in a dataset. It involves using statistical techniques to create models that can
predict outcomes, explain relationships, and identify patterns based on empirical data.
Key Features:
● Model Structure: Statistical models typically have a defined structure, which includes
dependent and independent variables. The dependent variable is the outcome being predicted or
explained, while independent variables are the predictors or factors believed to influence the
dependent variable.
● Estimation: Parameters of the model are estimated using statistical methods, such as least
squares, maximum likelihood estimation, or Bayesian inference.
● Assumptions: Statistical models often rely on assumptions about the underlying data, such as
normality, independence, and homoscedasticity (constant variance).
Validating these assumptions is crucial for accurate model interpretation.
● Interpretation: The results from statistical models can provide insights into the strength hand
nature of relationships between variables, as well as the overall model fit.

Fitting a model refers to the process of training a statistical or machine learning model on a dataset
to learn the underlying patterns or relationships present in the data. This is a crucial step in
predictive modeling and data analysis, as it enables the model to make accurate predictions or
provide insights based on new, unseen data.
Key Components of Fitting a Model
1. Training Data:

4
ISE, JSSATEB
Python for Data Science MCS104A

○ The data used to fit the model is called the training dataset. It contains input features
(independent variables) and the corresponding target variable (dependent variable) that the model
aims to predict.
2. Model Selection:
○ Selecting an appropriate model is essential, as different algorithms may be suitedfor different
types of problems (e.g., linear regression for continuous outcomes, logistic regression for binary
outcomes, decision trees for classification, etc.).
3. Loss Function:
○ A loss function (or cost function) quantifies how well the model's predictions
match the actual data. The goal of fitting the model is to minimize this loss.
Common loss functions include:
■ Mean Squared Error (MSE) for regression tasks.
■ Cross-Entropy Loss for classification tasks.
4. Optimization Algorithm:
○ An optimization algorithm (like gradient descent) is used to minimize the loss function by
adjusting the model parameters (coefficients). This involves iteratively updating the model's
parameters based on the gradients of the loss
function.
5. Hyperparameters:
○ These are the settings that are not learned during training (e.g., learning rate, number of trees in
a random forest). Hyperparameter tuning is often performed to find the best set of hyperparameters
for optimal model performance.
6. Validation:
○ The fitted model is usually evaluated on a separate validation dataset to assess its performance
and generalization capability. Metrics like accuracy, precision, recall, F1-score, or R-squared may
be used, depending on the task.
Steps in Fitting a Model
1. Data Preprocessing:
○ Clean the data, handle missing values, and preprocess features (e.g., normalization, encoding
categorical variables).
2. Split the Data:

5
ISE, JSSATEB
Python for Data Science MCS104A

○ Divide the dataset into training, validation, and test sets (commonly in a 70/15/15or 80/10/10
split) to prevent overfitting and evaluate model performance.
3. Choose a Model:
○ Select an appropriate algorithm based on the problem type (regression, classification, clustering,
etc.).
4. Fit the Model:
○ Use the training data to train the model by fitting it using the selected
optimization algorithm to minimize the loss function.
5. Evaluate the Model:
○ Use the validation dataset to assess the model's performance and make
adjustments if necessary. This might involve tuning hyperparameters or trying different
algorithms. 6. Testing:
○ Finally, assess the model's performance on the test set to evaluate how well it generalizes to new
data.

Population and samples

i) Population
A population refers to the entire set of individuals or items that are of interest in a particular
study. It includes all possible observations that meet a certain criterion and can be finite (e.g., all
students in a university) or infinite (e.g., all possible rolls of a die).
Key Features:
● Comprehensive: The population encompasses all possible members of the group being studied.
● Parameters: Characteristics of a population are described by parameters, such as the population
mean (μ\muμ), population variance (σ2\sigma^2σ2), etc.
● Cost and Time: Studying the entire population may be impractical due to cost, time, or
accessibility constraints. Example: Suppose you are conducting research on the average height of
adult men in a city. Inthis case:
● Population: All adult men living in that city. This includes every man aged 18 and older
in the city.

6
ISE, JSSATEB
Python for Data Science MCS104A

ii) Sample
A sample is a subset of the population selected for analysis. Samples are used to make
inferencesabout the population without needing to collect data from every individual within the
population.
Key Features:
● Representative: Ideally, the sample should be representative of the population to ensure that
the findings can be generalized.
● Statistics: Characteristics of a sample are described by statistics, such as the sample mean,
sample variance
● Sampling Methods: There are various methods to select samples, including random sampling,
stratified sampling, systematic sampling, and convenience sampling. Example: Continuing with
the previous example:
● Sample: If you randomly select 100 adult men from different neighborhoods in the city to
measure their heights, that group of 100 men represents your sample. Comparison and Importance
● Scope: The population is the whole group, while a sample is a part of that group.
● Data Collection: Collecting data from a population can be time-consuming and
expensive, while sampling allows for quicker and less costly data collection.
● Inference: By analyzing the sample, researchers can make inferences about the
population.
Statistical techniques to estimate population parameters based on samples statistics:
Let’s consider a scenario to illustrate the concepts further. Scenario: A university wants to
determine the average GPA of all its students (population) but
finds it impractical to collect GPAs from every student due to time constraints. 1. Population: All
enrolled students at the university. ○ If there are 10,000 students, the population includes each one
of them. 2. Sample: The university decides to randomly select 500 students fromits list of
enrolledstudents to participate in the study. ○ The average GPA of these 500 students will be
calculated to estimate the averageGPA of all 10,000 students.

Histogram Program
import pandas as pd
import numpy as np
import [Link] as plt
from [Link] import fetch_california_housing

7
ISE, JSSATEB
Python for Data Science MCS104A

# Load the California Housing dataset

housing_data = fetch_california_housing(as_frame=True)
data = housing_data['data']
print(data)
data['MedHouseVal'] = housing_data['target'] # Adding target variable for
completeness
print(data)

# Histograms for all numerical features

print("Creating histograms for all numerical features...")
for column in [Link]:
[Link](figsize=(8, 5))
[Link](data[column], bins=50, edgecolor='k', alpha=0.7)
[Link](f'Distribution of {column}')
[Link](column)
[Link]('Frequency')
[Link](axis='y', linestyle='--', alpha=0.7)
[Link]()

# Box plots for all numerical features

print("Creating box plots for all numerical features to identify
outliers...")
for column in [Link]:
[Link](figsize=(8, 5))
[Link](data[column], vert=False,
patch_artist=True,boxprops=dict(facecolor='skyblue', color='blue'))
[Link](f'Box Plot of {column}')
[Link](column)
[Link](axis='x', linestyle='--', alpha=0.7)
[Link]()

# Identify outliers using IQR

print("Identifying potential outliers using the IQR method...")
outliers = {}
for column in [Link]:
Q1 = data[column].quantile(0.25)
Q3 = data[column].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
outliers[column] = data[(data[column] < lower_bound) | (data[column] >
upper_bound)]
print(type(data[column]))
print(data[column])

8
ISE, JSSATEB
Python for Data Science MCS104A

print(outliers[column])
print(f"{column}:")
print(f"Lower Bound: {lower_bound}, Upper Bound: {upper_bound}")
print(f"Number of outliers: {len(outliers[column])}")
print("---")

Module 2

Exploratory Data Analysis (EDA) is a critical step in the data analysis process, aimed at
summarizing the main characteristics of a dataset, often using visual methods. EDA helps
inunderstanding the underlying patterns, detecting anomalies, testing hypotheses, and
checkingassumptions, which can guide subsequent analysis and modeling.

EDA Process
The EDA process typically involves the following steps:

1. Data Collection:
○ Gather the data from various sources, which could be databases, CSVfiles, APIs, etc.
2. Data Cleaning:
○ Identify and handle missing values, outliers, and inconsistencies in the data. ○ Remove
duplicates and ensure that data types are correct.
3. Descriptive Statistics:
○ Calculate summary statistics (mean, median, mode, standard deviation, quartiles)
to understand the data distribution. ○ Use measures like skewness and kurtosis to analyze the shape
of the data
distribution.
4. Data Visualization:
○ Create visualizations to explore data patterns, trends, and relationships. ○ Common
visualizations include:
■ Histograms for distribution
■ Box plots for identifying outliers

9
ISE, JSSATEB
Python for Data Science MCS104A

■ Scatter plots for relationships between variables

■ Heatmaps for correlation analysis
5. Feature Exploration:
○ Analyze individual features and their distributions. ○ Examine relationships between features to
identify potential predictors for
modeling.
6. Hypothesis Testing (optional):
○ Formulate and test hypotheses based on initial findings fromthe data.
7. Documentation:
○ Document the insights gained during the EDA process to informfurther analysisor modeling.

Example of EDA Process

Let’s consider a dataset containing information about a company’s employees,
includingfeaturessuch as age, salary, department, and years of experience. Here’s a step-by-step
illustrationof theEDA process:

Step 1: Data Collection

● Load the dataset using pandas in Python. python

import pandas as pd
# Load dataset
data = pd.read_csv('employee_data.csv')

Step 2: Data Cleaning

● Check for missing values and data types. python

# Check for missing values

missing_values = [Link]().sum()
# Data types

10
ISE, JSSATEB
Python for Data Science MCS104A

data_types = [Link]
● Fill or drop missing values as necessary.

Step 3: Descriptive Statistics

● Generate summary statistics. python
# Descriptive statistics
summary_statistics = [Link]()

Step 4: Data Visualization

● Visualize the data using various plots. python
import [Link] as plt
import seaborn as sns
# Histogram of employee ages
[Link](figsize=(10, 5))
[Link](data['age'], bins=10, kde=True)
[Link]('Distribution of Employee Ages')
[Link]('Age')
[Link]('Frequency')
[Link]()
# Box plot of salaries by department
[Link](figsize=(10, 5))
[Link](x='department', y='salary', data=data)
[Link]('Salaries by Department')
[Link]('Department')
[Link]('Salary')
[Link]()

11
ISE, JSSATEB
Python for Data Science MCS104A

Step 5: Feature Exploration

● Analyze the correlation between features.
# Correlation heatmap
[Link](figsize=(10, 5))
[Link]([Link](), annot=True, cmap='coolwarm')
[Link]('Correlation Matrix')
[Link]()
Step 6: Hypothesis Testing (Optional)
● For example, test whether the average salary differs by department using ANOVA.
Step 7: Documentation
● Summarize insights such as:
○ The average age of employees is around 35 years. ○ There are outliers in the salary distribution.
○ There is a positive correlation between years of experience and salary.
The EDA process is fundamental in data analysis as it provides insights into the dataset that
inform further statistical analysis and model development. By systematically exploring
andvisualizing the data, analysts can better understand the data’s characteristics, which aids
inmaking informed decisions based on the analysis.

The K-Nearest Neighbors (K-NN) algorithm is a simple, yet effective, supervised

machinelearning algorithm used for classification and regression tasks. It works on the principle
of
finding the 'k' nearest data points in the feature space and making predictions based on the
majority class (for classification) or the average value (for regression) of these neighbors.
1. Data Representation:
○ Each data point is represented as a point in a multidimensional feature space
based on its attributes. For example, if you have a dataset with features like height
and weight, each data point can be represented as a point in a 2Dspace.
2. Choosing the Value of K:
○ The user must select the number of neighbors kkk. A smaller value of kkkcanlead to noise
affecting the prediction, while a larger value may include points

12
ISE, JSSATEB
Python for Data Science MCS104A

from other classes, which can dilute the classification accuracy.

3. Distance Metric:
○ K-NN uses a distance metric to determine the 'closeness' of data points. Commondistance metrics
include:
■ Euclidean Distance: Most commonly used, calculates the straight-linedistance between two
points. ■ Manhattan Distance: The sum of the absolute differences of the
Cartesian coordinates. ■ Minkowski Distance: A generalization of both Euclidean and
Manhattandistances.

4. Finding Neighbors:
○ For a given test point, the algorithm calculates the distance between this point andall other points
in the training dataset. It then identifies the kkk points that areclosest to the test point.

5. Making Predictions:
○ Classification: The algorithm assigns the class that is most frequent amongthekkk nearest
neighbors to the test point. ○ Regression: The algorithm takes the average (or weighted average)
of the target
values of the kkk nearest neighbors to make a prediction. Example of K-NN Algorithm
K-NN algorithm with a simple example.
Imagine you have a dataset of fruits, characterized by their weight and color (in terms of
RGBvalues). You want to classify a new fruit based on its features.

13
ISE, JSSATEB
Python for Data Science MCS104A

14
ISE, JSSATEB
Python for Data Science MCS104A

15
ISE, JSSATEB
Python for Data Science MCS104A

Data Science Process Life Cycle

Some steps are necessary for any of the tasks that are being done in the field of data sciencetoderive
any fruitful results from the data at hand.
● Data Collection – After formulating any problem statement the main task is tocalculate data
that can help us in our analysis and manipulation. Sometimes datais
collected by performing some kind of survey and there are times when it is donebyperforming
scrapping.
● Data Cleaning – Most of the real-world data is not structured and requires cleaningand
conversion into structured data before it can be used for any analysis or modeling.
● Exploratory Data Analysis – This is the step in which we try to find the hiddenpatterns in the
data at hand. Also, we try to analyze different factors which affect thetarget variable and the extent
to which it does so. How the independent features arerelated to each other and what can be done
to achieve the desired results all theseanswers can be extracted from this process as well. This also
gives us a directioninwhich we should work to get started with the modeling process. ● Model
Building – Different types of machine learning algorithms as well as
techniques have been developed which can easily identify complex patterns inthedata which will
be a very tedious task to be done by a human.
● Model Deployment – After a model is developed and gives better results on the
holdout or the real-world dataset then we deploy it and monitor its performance. Thisis the main
part where we use our learning from the data to be applied in real-worldapplications and use cases

16
ISE, JSSATEB
Python for Data Science MCS104A

Module 3
Bayes' Theorem
Bayes' Theorem is a statistical method used for classification tasks - spam classification.

It is a probabilistic classifier that assumes independence between the features.

It is widely used for text classification, spam filtering, and other tasks involving high-
dimensional data. Though simple, performs well in practice and is computationally efficient.

Bayes' Theorem describes the probability of an event, based on prior knowledge of conditions
that might be related to the event. It is mathematically expressed as:

P(A∣B)=P(B∣A)⋅P(A) / P(B)

Where:

 P(A∣B) is the posterior probability (the probability of A given B has occurred).

 P(B∣A) is the likelihood (the probability of B given A).
 P(A) is the prior probability (the probability of A).
 P(B) is the evidence (the probability of B).

Spam Classification Using Naive Bayes

Class 1: Spam

Class 2: Not Spam (Ham)

1. Prior Probabilities: These are the probabilities of an email being spam or ham before
seeing any specific words in the email.

Assume 30% of emails are spam and 70% are not, the prior probabilities would be:
P(Spam)=0.30, P(Ham)=0.70

2. Likelihoods (Conditional Probabilities): Probabilities of observing certain words in the

email, given that the email is spam or ham.

For instance, if the word "buy" is very common in spam emails but less common in ham
emails:
P(buy∣Spam)=0.8, P(buy∣Ham)=0.1

3. Evidence (Normalizing Constant): This is the probability of seeing the word "buy" in
any email, regardless of whether it's spam or ham. It will cancel out when comparing the
posterior probabilities for spam and ham, so no explicit calculation required.

17
ISE, JSSATEB
Python for Data Science MCS104A

4. Calculate Posterior Probabilities: According to Bayes' Theorem, the probability of an

email being spam given the word "buy" is:
P(Spam∣buy)=P(buy∣Spam)⋅P(Spam) / P(buy)

5. Similarly, the probability of it being ham is:

P(Ham∣buy)=P(buy∣Ham)⋅P(Ham) / P(buy)

If P(Spam∣buy)>P(Ham∣buy), classify the email as spam. Otherwise, classify it as ham.

—-------------------------------------------------------------------------------------------------------

Example Setup:

Consider a training dataset and consider classifying a test email based on certain words.

Assume Out of 100 emails, 40 are spam and 60 are not spam (ham).

 Given some common words and their occurrences in spam and ham emails:

| Word | P(word | Spam) | P(word | Ham) | |-------|------------------|----------------| | "free" | 0.6 | 0.1 |

| "offer" | 0.7 | 0.2 | | "buy" | 0.8 | 0.1 | | "hello" | 0.2 | 0.9 | | "discount" | 0.7 | 0.2 |

Test Email:
Classify the following test email:
Test Email: "Buy now and get a free discount offer!"
The words in this email are: "buy", "free", "discount", "offer".

1. Prior Probabilities:
P(Spam)=0.40, P(Ham)=0.60

2. Likelihoods:
 P(buy∣Spam)=0.8 P(buy∣Ham)=0.1
 P(free∣Spam)=0.6 P(free∣Ham)=0.1
 P(discount∣Spam)=0.7 P(discount∣Ham)=0.2
 P(offer∣Spam)=0.7 P(offer∣Ham)=0.2

3. Posterior Probabilities:
P(Spam∣email) ∝ P(buy∣Spam) ⋅ P(free∣Spam) ⋅ P(discount∣Spam) ⋅ P(offer∣Spam) ⋅
P(Spam)

P(Spam∣email) ∝ (0.8)⋅(0.6)⋅(0.7)⋅(0.7)⋅(0.40)
P(Spam∣email) ∝ 0.09408
For Ham:
P(Ham∣email) ∝ 0.00024

Result: Posterior probability for spam is higher, hence classify this email as Spam.

18
ISE, JSSATEB
Python for Data Science MCS104A

The K-Means algorithm is a popular unsupervised machine learning technique used for
clustering. It partitions a dataset into kkk distinct, non-overlapping clusters based on
featuresimilarity. The primary goal of K-Means is to minimize the variance within each cluster
whilemaximizing the variance between clusters.
1. Initialization:
○ Select kkk initial centroids (cluster centers). This can be done randomly or byusing techniques
like K-Means++ for better initialization.
2. Assignment Step:
○ Assign each data point to the nearest centroid based on a distance metric
(commonly Euclidean distance). This forms kkk clusters.
3. Update Step:
○ Calculate the new centroids for each cluster by taking the mean of all points
assigned to that cluster.
4. Repeat:
○ Repeat the Assignment and Update steps until the centroids do not change
significantly, indicating that the algorithm has converged, or until a specifiednumber of iterations
is reached. 5. Output:
○ The final clusters and their centroids.
Example of K-Means Algorithm
Let’s consider a simple example to illustrate how the K-Means algorithm works.
Suppose you have a dataset representing the scores of students in two subjects: Math andScience.
The goal is to cluster the students into groups based on their performance. Step 1: Prepare the Data
Let's say the dataset is as follows:

19
ISE, JSSATEB
Python for Data Science MCS104A

20
ISE, JSSATEB
Python for Data Science MCS104A

21
ISE, JSSATEB
Python for Data Science MCS104A

Module 4
Dimensionality Problem:
The dimensionality problem refers to a datasets that have a large number of features, making it
harder to process, analyze, and visualize. High-dimensional data can lead to overfitting, where the
model becomes too complex and performs poorly on new data. Dimensionality reduction
techniques like Singular Value Decomposition (SVD) and Principal Component Analysis (PCA)
reduce the number of features while retaining important information.
Singular Value Decomposition (SVD) Algorithm
Singular Value Decomposition (SVD) is a technique for reducing the dimensionality of data. It
decomposes a matrix X into three components:
X = U ⋅ S ⋅ VT
Where:

22
ISE, JSSATEB
Python for Data Science MCS104A

• X is the original matrix (e.g., users' ratings of items).

• U contains the user features (latent factors for users).
• S is a diagonal matrix with singular values, representing the importance of latent factors.
• VT contains item features (latent factors for items).
How SVD works:
1) Decompose X into U, S, and VT
2) Truncate the matrices to a smaller rank d for dimensionality reduction.
3) The reduced approximation Xd = Ud ⋅ Sd ⋅ 𝑉𝑑𝑇
captures the important data patterns.
In recommendation systems, SVD identifies latent features, allowing for better user-item
representation and more efficient recommendations.

Principal-component analysis, or PCA, is a technique for taking a dataset consisting of a set of

tuples representing points in a high-dimensional space and finding the directions along which the
tuples line up best. It achieves this by finding new axes (principal components) that best describe
the distribution of the data. These axes correspond to the eigenvectors of the covariance matrix,
and their importance is determined by their eigenvalues.
Example: Consider a dataset with four points:

23
ISE, JSSATEB
Python for Data Science MCS104A

The transformed data shows that most of the variance lies along the first principal component,
meaning the second component can be discarded to reduce dimensionality.

Feature Extraction and Feature Selection.

Feature extraction refers to taking the raw dump of data you have and curating it more carefully,
to avoid the “garbage in, garbage out” scenario you get if you just feed raw data into an algorithm
without enough forethought.

24
ISE, JSSATEB
Python for Data Science MCS104A

Feature extraction is a crucial process in data preprocessing and machine learning, particularly in
the context of high-dimensional data. It involves transforming rawdata intoaset
of usable features that can effectively represent the underlying patterns in the data while
reducingits dimensionality. This process aims to simplify the data without losing important
information, making it easier for machine learning algorithms to learn from it.
Key Concepts of Feature Extraction
1. Dimensionality Reduction:
○ Feature extraction helps in reducing the number of input variables (features) inadataset. This can
improve model performance, reduce computation time, andmitigate overfitting.
2. Feature Selection vs. Feature Extraction:
○ Feature Selection involves selecting a subset of relevant features fromthe original
dataset without transforming the data. ○ Feature Extraction, on the other hand, involves creating
new features by
transforming or combining the original features.
3. Representation:
○ The goal is to represent the data in a way that highlights the most important
aspects or patterns that are relevant to the problem being solved.

Techniques for Feature Extraction

Several techniques can be used for feature extraction, depending on the type of data and the
problem at hand:
1. Statistical Methods:
○ Techniques like Principal Component Analysis (PCA) reduce dimensionalitybyprojecting data
onto the directions of maximum variance.
2. Text Data:
○ For text data, methods like Term Frequency-Inverse Document Frequency(TFIDF) and word
embeddings (like Word2Vec or GloVe) are commonly usedtoextract meaningful features from
raw text.
3. Image Data:
○ In image processing, features can be extracted using techniques like edge
detection, histograms of oriented gradients (HOG), or deep learning methods

25
ISE, JSSATEB
Python for Data Science MCS104A

using convolutional neural networks (CNNs) which automatically learn featuresfrom images.
4. Signal Processing:
○ For time-series or audio data, features like frequency components, wavelet
transforms, or statistical measures (mean, variance) can be extracted.
Example of Feature Extraction: Image Classification
Consider a scenario where you want to classify images of cats and dogs. The rawpixel dataof
the images can be high-dimensional (especially for large images), making it difficult for a model
to learn effectively.

Significance of Feature Selection

• Reduces Overfitting: Too many features can lead to overfitting, where the model learns noise
instead of patterns.
• Improves Model Accuracy: Removing irrelevant features improves model generalization.
• Enhances Computational Efficiency: Fewer features mean faster training and inference times.
Example: In an e-commerce dataset, "Website color theme" may not impact customer purchases,
so we remove it to improve the model.
Feature selection is the process of constructing a subset of the data or functions of the data to be
the predictors or variables for your models and algorithms.
Feature Extraction Significance:
• Handles High-Dimensional Data: Useful when raw data has too many variables.
• Creates Better Representations: Helps when existing features don't fully capture patterns.
• Improves Model Interpretability: Transforms raw data into more meaningful representations.
Example: In image classification, instead of using raw pixels, we extract edges, color histograms,
or use PCA to reduce dimensions while keeping key information.
Both methods enhance machine learning models by improving data quality and reducing
complexity.

Decision Trees and Random Forest.

Decision Trees
A decision tree is a popular supervised machine learning algorithm used for both

26
ISE, JSSATEB
Python for Data Science MCS104A

classification and regression tasks. It models decisions and their possible consequences as a tree
like structure, where each internal node represents a feature (attribute), each branch represents
decision rule, and each leaf node represents an outcome (label or value).

Structure of a Decision Tree

1. Root Node: The top node in the tree represents the entire dataset, which gets split intotwo or
more subsets based on the feature that results in the most significant informationgain or reduction
in impurity.
2. Internal Nodes: These nodes represent the features used to split the data. Each internal
node corresponds to a decision based on a specific feature.
3. Branches: These are the outcomes of the decision made at the internal node and connect
the nodes.
4. Leaf Nodes: The end points of the tree that represent the final output or decision (classlabels in
classification tasks or continuous values in regression tasks).

Role of Decision Trees in Feature Selection

Decision trees inherently perform feature selection as they determine the best features to split the
data based on certain criteria. Here’s how decision trees contribute to feature selection:
1. Information Gain: Decision trees evaluate the effectiveness of a feature in classifyingthe
training data by calculating the information gain or reduction in entropy. Features that
provide the most information gain are preferred for splitting.
2. Impurity Measures: Metrics such as Gini impurity or entropy are used to assess thequality of
splits. Features that result in lower impurity are selected.
3. Feature Importance: Once a decision tree is trained, it provides a measure of feature
importance, which indicates how valuable each feature was in making predictions. This
information can be used to select the most relevant features for a model. Implementation of
Decision Tree in Python

27
ISE, JSSATEB
Python for Data Science MCS104A

Step 3: Split the Dataset

28
ISE, JSSATEB
Python for Data Science MCS104A

Check the feature importance provided by the decision tree:

# Get feature importances

importances = clf.feature_importances_
features = iris.feature_names

29
ISE, JSSATEB
Python for Data Science MCS104A

importance_df = [Link]({'Feature': features, 'Importance': importances})

print(importance_df.sort_values(by='Importance', ascending=False))

• Decision Trees are classification algorithms that break down complex decisions into a series of
simple yes/no (binary) questions. They mimic human decision-making, making them highly
interpretable.
• In the Chasing Dragons example, a decision tree can predict whether a user will return to the
game next month based on various factors like: Number of dragons slain, Age of the player and
Total hours spent playing.

• At each step, the model selects the most informative feature—the one that provides the most
separation between classes. This selection process is similar to playing "20 Questions," where you
try to ask the most effective question first.
• The tree is constructed using a mathematical approach, often based on entropy and information
gain. The goal is to reduce uncertainty at each step, ensuring that each decision node helps separate
the data efficiently.
• However, a major downside of decision trees is overfitting—they tend to memorize data rather
than generalize well to unseen examples.

30
ISE, JSSATEB
Python for Data Science MCS104A

Random Forests
Random Forests solve the overfitting problem by using bagging (bootstrap aggregating) to create
multiple decision trees and averaging their predictions. This increases accuracy but reduces
interpretability.
The process works as follows:
• Bootstrapping: Each tree is trained on a random subset of the training data, often 80% of the
total dataset, with some points appearing multiple times.
• Random Feature Selection: At each split, only a subset of the total features is considered (e.g.,
selecting 5 out of 100 features).
• Decision Tree Construction: Each tree grows independently, following the same entropy-
information gain method.
• Final Prediction: The predictions from all trees are combined, usually by majority voting for
classification tasks.
Unlike decision trees, random forests do not require pruning. They can capture complex patterns
and reduce the risk of overfitting by averaging out noise from individual trees. However, their
complexity makes them harder to interpret.

Random Forest is an ensemble learning method primarily used for classification andregression
tasks. It builds multiple decision trees during training and merges their outputs toimprove the
accuracy and robustness of the predictions. The underlying idea is to combine thepredictions of
several models (in this case, decision trees) to achieve better performance thananyindividual
model.

Key Characteristics of Random Forest

1. Ensemble Learning: Random Forest is based on the concept of ensemble learning, where
multiple models (decision trees) are trained to make predictions.
2. Bootstrap Aggregation (Bagging): It uses a technique called bagging, which involvestraining
each decision tree on a random subset of the training data. This subset is createdby sampling with
replacement (bootstrap sampling).
3. Random Feature Selection: At each node of a decision tree, a randomsubset of featuresis
selected for splitting, rather than considering all features. This adds diversity amongthe trees and
helps to reduce overfitting.

31
ISE, JSSATEB
Python for Data Science MCS104A

4. Voting Mechanism: For classification tasks, the final prediction is made by majorityvoting
among all the trees. For regression tasks, the average of all tree predictions is
taken.
Working of the Random Forest Algorithm
1. Data Preparation:
○ Start with a training dataset.
2. Bootstrap Sampling:
○ Create multiple bootstrap samples (subsets) from the original dataset. Eachsubset
will be used to train a different decision tree.
3. Tree Building:
○ For each bootstrap sample, build a decision tree:
■ At each internal node, randomly select a subset of features. ■ Choose the best feature among the
selected subset to split the data at that
node. ■ Repeat this process recursively until a stopping criterion is met (e.g., maximum tree depth
or minimum samples per leaf).
4. Aggregation:
○ For classification tasks, predict the class label based on majority voting fromall
the trees. For regression tasks, average the predictions from all trees.
5. Final Prediction:
○ The aggregated output from all the trees is the final prediction of the randomforest model.
Example of Random Forest Algorithm

Let’s implement a Random Forest classifier using the scikit-learn library with the Iris dataset.

Step 1: Import Libraries

import pandas as pd
import numpy as np
from [Link] import load_iris

32
ISE, JSSATEB
Python for Data Science MCS104A

from sklearn.model_selection import train_test_split

from [Link] import RandomForestClassifier
from [Link] import accuracy_score, classification_report

Step 2: Load the Dataset

# Load the Iris dataset

iris = load_iris()
X = [Link] # Features (sepal length, sepal width, petal length, petal width)
y = [Link] # Target variable (species)

Step 3: Split the Dataset

# Split the dataset into training and testing sets (70% train, 30% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

Step 4: Initialize and Train the Random Forest Classifier

# Initialize the Random Forest Classifier

rf_classifier = RandomForestClassifier(n_estimators=100, random_state=42)
# Train the model
rf_classifier.fit(X_train, y_train)

Step 5: Make Predictions

# Make predictions on the test set

y_pred = rf_classifier.predict(X_test)

33
ISE, JSSATEB
Python for Data Science MCS104A

Step 6: Evaluate the Model

# Calculate accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy: {accuracy:.2f}')
# Display classification report
print(classification_report(y_test, y_pred, target_names=iris.target_names))

Feature importance scores from the Random Forest model:

# Get feature importances

importances = rf_classifier.feature_importances_
features = iris.feature_names
# Create a DataFrame for better visualization
importance_df = [Link]({'Feature': features, 'Importance': importances})
print(importance_df.sort_values(by='Importance', ascending=False))

Nearest Neighbor algorithm and the problems with Nearest Neighbor in Recommendation
System.
The Nearest Neighbor (k-NN) algorithm is a simple and intuitive instance-based learning method
used for classification and regression tasks. The idea is to predict the label of a data point based
on the majority label or average of its closest neighbors.
k-NN Working:
1) Choose the number of neighbors (k): Select the number of nearest neighbors, k, that will be
used to make the prediction. A common value for k is 3, but it can vary based on the data.
2) Calculate distance: Measure the distance between the input point and all other points in the
dataset.
Common distance metrics include:
• Euclidean distance for continuous data
• Manhattan distance for grid-like data

34
ISE, JSSATEB
Python for Data Science MCS104A

• Cosine similarity for text or high-dimensional data

3) Identify nearest neighbors: Select the k closest data points to the input based on the calculated
distance.
4) Prediction:
• For classification, the label most common among the k nearest neighbors is assigned to the input
point.
• For regression, the average (or weighted average) of the k nearest neighbors' values is predicted.
Example: Imagine you are predicting whether a user will return to an online game. You could use
k-NN,
where:
• Input data: User features like age, number of games played, time spent.
• Prediction: The majority label of the closest users (based on the similarity of features) would
determine if this user returns.
Problems with k-NN in Recommendation Systems:
• Scalability: k-NN is computationally expensive, especially with large, high-dimensional
datasets, as it requires calculating the distance between all users/items.
• Cold Start Problem: For new users or items, there’s no prior data, making it difficult to find
similar neighbors, leading to poor recommendations.
• Sparsity: In sparse user-item matrices, finding meaningful neighbors is hard due to limited
interactions, reducing the effectiveness of similarity measures.
• Neighborhood Selection: The choice of k is critical; too small or large can lead to overfitting or
overly general recommendations.
• Lack of Temporal Adaptation: k-NN doesn’t account for changing user preferences over time,
leading to outdated recommendations.
• Bias Toward Popularity: k-NN often favors popular items, reducing diversity and
personalization in
recommendations.

User Retention and feature generation

User retention refers to the ability to keep users engaged with an app or service over time. In the
context of the Chasing Dragons app, the goal is to predict whether a user will return after their

35
ISE, JSSATEB
Python for Data Science MCS104A

first month based on their activities during that period. By identifying patterns and key behaviors
associated with retention, the app can take proactive steps to encourage users to stay longer.
Feature generation plays a critical role in this process. In the feature generation phase, the team
brainstorms a wide range of potential features that might influence retention. For example, features
like the number of days the user visited, the time between visits, total points earned, or whether
the user completed their profile could all provide valuable insights. Domain expertise from game
designers, software engineers, statisticians, and marketers help to ensure that a comprehensive set
of features is generated, making the model more effective. Redundancies and correlations between
features are acceptable at this stage as they can later be addressed during feature selection.
Brainstorming and the Role of Domain Expertise in Feature Generation
• Domain expertise plays a critical role in feature generation by guiding which features are most
likely to impact the outcome.
• Understanding Data Context: Domain experts can provide insight into what aspects of the data
are most relevant, which helps in selecting the right features that align with business goals.
• Identifying Hidden Patterns: Experts can recognize patterns or relationships in the data that
may not be obvious to non-experts, such as correlations between user behavior and retention.
• Creating Derivative Features: Experts help brainstorm new features, like transforming raw data
into more useful forms (e.g., aggregating activity over time, segmenting users by behavior, etc.).
• Prioritizing Features: They can prioritize features based on their importance, guiding the
machine learning process towards the most impactful data for predictions.
By collaborating with domain experts during feature brainstorming, data scientists can better tailor
their models to the problem domain, improving both the relevance and effectiveness of their
machine learning solutions.

Filters and Wrappers

a) Filters: Filters are a feature selection method that ranks features based on their individual
predictive power, usually using a statistical metric such as correlation with the outcome variable.
In this approach, features are ranked and selected based on their strength in predicting the target
variable. However, filters do not account for feature interactions or redundancies. This means that
correlated features may be selected together even though one may not provide any additional value.
Filters are often useful for a quick first pass over the features to get rid of irrelevant ones, but they
might miss important interactions between features that could enhance predictive power.
b) Wrappers: Wrappers approach feature selection by evaluating subsets of features for their
performance using a predictive model. It involves selecting subsets of features, training the model
on those features, and evaluating the model’s performance using a predefined criterion (e.g.,
accuracy). Wrappers are more computationally expensive because they involve testing many

36
ISE, JSSATEB
Python for Data Science MCS104A

combinations of features, and this can lead to overfitting, especially when dealing with large
datasets.
The key challenge with wrappers is the exponential growth in the number of possible feature
subsets as the size of the feature set increases. Despite the computational expense, wrappers can
often find better feature subsets since they take into account the interactions between features.

Criteria for selection of features

As a data scientist, selecting the right criterion for feature selection is an important decision.
Different criteria might produce different models, and choosing the best one depends on your goals
and the nature of the data. Here are some selection criteria to consider:
1) R-squared (R²):

• Measures the proportion of variance explained by the model.

• A higher R² indicates a better fit, with more of the variance in the target variable explained by
the features. But it doesn’t account for overfitting and might be misleading if used alone.
2) p-values:
The p-value is derived from the null hypothesis test, where the null hypothesis assumes that the
coefficient of a feature is zero. For a given feature β, p-value = P(Observed data | Null hypothesis).
• A low p-value (typically < 0.05) suggests that a feature significantly impacts the target variable.
• p-values can be unreliable if the model is mis specified or if they’re interpreted in isolation.
3) AIC (Akaike Information Criterion):
AIC = 2k – 2 ln(L)
where k is the number of parameters in the model and L is the maximized value of the likelihood
function.
• Aims to minimize the trade-off between model fit and complexity.
• A lower AIC indicates a better fit, considering both the model's complexity and the data's
likelihood.
• Limitation: AIC may favor overly complex models if not carefully balanced with other criteria.
4) BIC (Bayesian Information Criterion):

37
ISE, JSSATEB
Python for Data Science MCS104A

BIC = k ⋅ ln(n) – 2 ln(L)

where k is the number of parameters, n is the number of observations, and L is the maximized
value of the likelihood function.
• Like AIC but imposes a stronger penalty for models with more parameters, particularly in large
datasets.
• The goal is to minimize BIC for a better balance between model fit and complexity.
• BIC can favor simpler models, especially when the dataset is large.
5) Entropy:

where H(X) is the entropy of a random variable X, and P(xi) is the probability of each outcome xi.
• Measures the uncertainty or disorder in a target variable, with lower entropy indicating more
certainty.
• Commonly used in decision trees to identify the most informative features by maximizing
information gain.
• Limitation: May not capture feature interactions effectively on its own.

Data wrangling and importance of data wrangling

Data wrangling is the process of transforming raw data into a structured, usable format. It ensures
that
the data is consistent, complete, and suitable for analysis and visualization.
[Data Collection] --> [Data Cleaning] --> [Data Transformation] --> [Data Integration] --> [Data
Validation] --> [Data Structuring]
Steps Involved in Data Wrangling:
• Data Collection: Collect data from various sources (databases, CSV files, APIs, etc.).
• Data Cleaning: Remove duplicates: Identify and remove duplicate records. Handle missing
values: Impute or remove rows/columns with missing data. Standardize formats: Ensure
consistency in units, date formats, and categorical variables.

38
ISE, JSSATEB
Python for Data Science MCS104A

• Data Transformation: Normalization/Scaling: Scale numerical data to a uniform range.

Encoding categorical variables: Convert categorical variables into numerical representations (e.g.,
one-hot encoding).
Feature engineering: Create new features from existing ones based on domain knowledge.
• Data Integration: Combine data from different sources, ensuring that they align well (merging
DataFrames, joining tables).
• Data Validation: Check for data integrity (ensuring values make sense, no out-of-range values).
Remove outliers if necessary or treat them properly.
• Data Structuring: Organize data into a format suitable for analysis, such as tabular data in a
DataFrame (rows and columns). Importance of Data Wrangling Before Creating Visualizations
• Accurate Insights: Clean, structured data is necessary to avoid misleading visualizations.
Inconsistent or missing data can lead to incorrect interpretations.
• Efficient Analysis: Wrangling ensures data is in the right format, allowing visualizations to
display the data clearly and efficiently.
• Improved Decision-Making: Well-prepared data enables stakeholders to make data-driven
decisions based on reliable visualizations.
• Better User Experience: Visualizations with clean data are easier to interpret and help users
grasp the underlying trends and patterns.

Module 5
Types of Social networks as graphs and methods of discovery of communities in grapshs.
a) Betweenness to find communities with its variant.
b) Betweenness with Algorithm.
Types of Social Networks as Graphs
• Telephone Networks: Nodes represent phone numbers; edges indicate communication within a
fixed period. Communities form from frequent communicators.
• Email Networks: Nodes are email addresses; edges represent exchanged emails. Strong or weak
edges show the communication direction. Communities form based on frequent interactions.
• Collaboration Networks: Nodes are authors of research papers. Edges connect co-authors.
Communities form based on shared topics.
• Wikipedia Editing Networks: Nodes are editors or articles. Edges indicate common editing.

39
ISE, JSSATEB
Python for Data Science MCS104A

Communities form around similar subjects.

• Collaborative Filtering Networks: Nodes are customers/products. Edges represent purchased
items.
Communities form based on shared purchase patterns.
Methods of Discovery of Communities in Graphs:
• Communities in a graph are groups of nodes that are more densely connected to each other than
to the rest of the network.
There are several methods to discover communities in social networks:
• Modularity Optimization: Maximizes the modularity of a network to find densely connected
clusters.
• Spectral Clustering: Uses eigenvalues and eigenvectors of the graph's adjacency matrix to find
communities.
• Label Propagation: Iteratively propagates labels through the network to find community
structures.
• Betweenness Centrality: Identifies central nodes that act as bridges between communities.
• Girvan-Newman Algorithm: Removes edges with the highest betweenness centrality to identify
communities.
a) Betweenness to Find Communities
The betweenness of an edge (a, b) is the number of shortest paths between any two nodes (x, y)
that passthrough edge (a, b). High betweenness indicates that the edge is a bridge between different
communities. An edge with high betweenness should be removed to separate the network into
smaller communities.
Example: In a graph, an edge (B, D) might have the highest betweenness, indicating it connects
two distinct groups. Removing this edge can help reveal the underlying communities.

40
ISE, JSSATEB
Python for Data Science MCS104A

b) Betweenness with Algorithm

Procedure:
• Start with the full graph and calculate the betweenness scores for all edges.
• Remove the edge with the highest betweenness, which is likely a bridge between two
communities.
• Repeat this process iteratively, removing edges with high betweenness, until the graph is broken
into smaller connected components (communities).
Example: After removing edges with high betweenness, the graph splits into two communities:
{A, B, C}
and {D, E, F, G}. Continuing the removal process reveals more granular communities.
The edges with high betweenness are typically the ones connecting separate groups within the
network, and their removal helps identify community boundaries.

Partitioning of graphs and neighborhood properties in graphs.

Partitioning a graph involves dividing its nodes into two or more subsets, aiming to minimize the
number of edges that connect nodes in different subsets. A cut in the graph refers to the set of
edges that connect nodes from different subsets. The goal of partitioning is to minimize the cut
while ensuring that the resulting subsets are approximately equal in size. This is important in
network analysis and community detection.

41
ISE, JSSATEB
Python for Data Science MCS104A

Neighborhood properties of a graph describe the relationships between a node and its immediate
neighbors. These properties help us understand how nodes interact with one another, and they are
fundamental for many graph-based algorithms.
1) Degree of a Node:
The degree of a node is the number of edges incident to it, representing the number of direct
connections the node has. Nodes with high degrees tend to be more central in the graph and may
play important roles in the structure of communities.
2) Local Clustering Coefficient:
The clustering coefficient measures the tendency of nodes to form clusters, i.e., whether their
neighbors are also connected to each other. A high clustering coefficient indicates a dense
subgraph of interconnected nodes, which often suggests a tightly-knit community.
3) Neighborhood Overlap:
This property quantifies the number of common neighbors shared by two nodes. Nodes with high
neighborhood overlap are likely to be part of the same community, as they share many connections.
4) Eccentricity:
The eccentricity of a node is the greatest distance from that node to any other node in the graph. A
low eccentricity means the node is close to many other nodes, making it a potential hub or central
node in the network.

42
ISE, JSSATEB