#1 Explain Supervised Learning and Unsupervised Learning with the help of an

example.

Supervised Learning is the machine learning approach defined by its use of labeled datasets to train
algorithms to classify data and predict outcomes.

The labeled dataset has output tagged corresponding to input data for the machine to understand
what to search for in the unseen data.

Here's how it looks in practice.

Supervised Machine Learning Methods

There are two main areas where supervised machine learning comes in handy: classification
problems and regression problems.

Classification
Classification refers to taking an input value and mapping it to a discrete value. In classification
problems, our output typically consists of classes or categories. This could be things like trying to
predict what objects are present in an image (a cat/ a dog) or whether it is going to rain today or not.

Regression

Regression is related to continuous data (value functions). In Regression, the predicted output
values are real numbers. It deals with problems such as predicting the price of a house or the trend
in the stock price at a given time, etc.

Regression

Regression is related to continuous data (value functions). In Regression, the predicted output
values are real numbers. It deals with problems such as predicting the price of a house or the trend
in the stock price at a given time, etc.

Some of the most common algorithms in Supervised Learning include Support Vector Machines
(SVM), Logistic Regression, Naive Bayes, Neural Networks, K-nearest neighbor (KNN), and
Random Forest.

Supervised Machine Learning Applications

Now, let's have a look at some of the popular applications of Supervised Learning:

 Predictive analytics (house prices, stock exchange prices, etc.)

 Text recognition

 Spam detection

 Customer sentiment analysis

 Object detection (e.g. face detection)

 What is Unsupervised Learning?

 Unsupervised Learning is a type of machine learning in which the algorithms
are provided with data that does not contain any labels or explicit instructions
 on what to do with it. The goal is for the learning algorithm to find structure in
the input data on its own.
 To put it simply—Unsupervised Learning is a kind of self-learning where the
algorithm can find previously hidden patterns in the unlabeled datasets and give
the required output without any interference.
 Identifying these hidden patterns helps in clustering, association, and detection
of anomalies and errors in data.

 Unsupervised Machine Learning Methods

 Unsupervised Learning models can perform more complex tasks than
Supervised Learning models, but they are also more unpredictable. Here are the
main tasks that utilize this approach.
 Clustering
 Clustering is the type of Unsupervised Learning where we find hidden patterns
in the data based on their similarities or differences. These patterns can relate to
the shape, size, or color and are used to group data items or create clusters.
 There are several types of clustering algorithms, such as exclusive, overlapping,
hierarchical, and probabilistic.
 Association
 Association is the kind of Unsupervised Learning where we can find the
relationship of one data item to another data item. We can then use those
dependencies and map them in a way that benefits us—e.g., understanding
consumers' habits regarding our products can help us develop better cross-
selling strategies.
 The association rule is used to find the probability of co-occurrence of items in
a collection. These techniques are often utilized in customer behavior analysis
in e-commerce websites and OTT platforms.
 Dimensionality reduction
 As the name suggests, the algorithm works to reduce the dimensions of the
data. It is used for feature extraction.
 Extracting the important features from the dataset is an essential aspect of
machine learning algorithms. This helps reduce the number of random variables
in the dataset by filtering irrelevant features.
 Finally, here's a nice visual recap of everything we've covered so far (plus the
Reinforcement Learning).
#2 What is the curse of dimensionality? How does it impact a predictive model? How is this
problem solved?

Curse of Dimensionality describes the explosive nature of increasing data dimensions and its

resulting exponential increase in computational efforts required for its processing and/or

analysis. This term was first introduced by Richard E. Bellman, to explain the increase in

volume of Euclidean space associated with adding extra dimensions, in area of dynamic

programming. Today, this phenomenon is observed in fields like machine learning, data

analysis, data mining to name a few. An increase in the dimensions can in theory, add more

information to the data thereby improving the quality of data but practically increases the

noise and redundancy during its analysis.

Behavior of a Machine Learning Algorithms — Need for data points and Accuracy of

Model

In machine learning, a feature of an object can be an attribute or a characteristic that defines it.

Each feature represents a dimension and group of dimensions creates a data point. This

represents a feature vector that defines the data point to be used by a machine learning

algorithm(s). When we say increase in dimensionality it implies an increase in the number of

features used to describe the data. For example, in the field of breast cancer research, age,

number of cancerous nodes can be used as features to define the prognosis of the breast cancer

patient. These features constitute the dimensions of a feature vector. But other factors like past

surgeries, patient history, type of tumor and other such features help a doctor to better

determine the prognosis. In this case by adding features, we are theoretically increasing the

dimensions of our data.

As the dimensionality increases, the number of data points required for good performance of

any machine learning algorithm increases exponentially.

Hughes (1968) in his study concluded that with a fixed number of training samples, the

predictive power of any classifier first increases as the number of dimensions increase, but

after a certain value of number of dimensions, the performance deteriorates. Thus, the

phenomenon of curse of dimensionality is also known as Hughes phenomenon.

Solutions to Curse of Dimensionality:

One of the ways to reduce the impact of high dimensions is to use a different measure of

distance in a space vector. One could explore the use of cosine similarity to replace Euclidean

distance. Cosine similarity can have a lesser impact on data with higher dimensions. However,

use of such method could also be specific to the required solution of the problem.

Other methods:

Other methods could involve the use of reduction in dimensions. Some of the techniques that

can be used are:

1. Forward-feature selection: This method involves picking the most useful subset of features

from all given features..

2. PCA/t-SNE: Though these methods help in reduction of number of features, but it does not

necessarily preserve the class labels and thus can make the interpretation of results a tough

task.

#3 Is it possible that gradient descent method may get stuck in local minima when used for

finding an optimum vector θ for linear regression? Justify your answer.

No, gradient descent is not prone to getting stuck in local minima when used for finding the
optimum vector θ (parameters) in linear regression. The reason for this is that the cost
function for linear regression, which is the mean squared error (MSE), is a convex function.
Convex functions have only one global minimum, and there are no local minima or other
optima.

In a convex function, the gradient points in the direction of the steepest increase of the
function. Gradient descent utilizes this property by iteratively moving towards the minimum
of the cost function by taking steps proportional to the negative of the gradient.

Since the cost function for linear regression is convex, gradient descent is guaranteed to
converge to the global minimum, and there are no concerns about getting stuck in local
minima. The convex nature of the cost function is a key reason why gradient descent is a
reliable optimization method for linear regression.

#4 Is it possible that gradient descent method may get stuck in local minima when used for

finding an optimum vector θ for linear regression? Justify your answer.

#5 What is the role of ridge regression in training a linear regression model?

Ridge Regression, a linear regression methodology, improves on the traditional least
squares approach by introducing a penalty element in the loss function. This penalty term
prevents the model from relying too much on a single feature, reducing the impacts of
multicollinearity. A regularization parameter (or alpha) is used to alter it.

#6 Why do we split data into trainig and test data?

Data splitting is when data is divided into two or more subsets. Typically, with a two-part
split, one part is used to evaluate or test the data and the other to train the model.
Data splitting is an important aspect of data science, particularly for creating models based on
data. This technique helps ensure the creation of data models and processes that use data
models -- such as machine learning -- are accurate.

How data splitting works

In a basic two-part data split, the training data set is used to train and develop models.
Training sets are commonly used to estimate different parameters or to compare different
model performance.

The testing data set is used after the training is done. The training and test data are compared
to check that the final model works correctly. With machine learning, data is commonly split
into three or more sets. With three sets, the additional set is the dev set, which is used to
change learning process parameters.

There is no set guideline or metric for how the data should be split; it may depend on the size
of the original data pool or the number of predictors in a predictive model. Organizations and
data modelers may choose to separate split data based on data sampling methods, such as the
following three methods:

1. Random sampling. This data sampling method protects the data modeling process
from bias toward different possible data characteristics. However, random splitting
may have issues regarding the uneven distribution of data.

2. Stratified random sampling. This method selects data samples at random within
specific parameters. It ensures the data is correctly distributed in training and test sets.

3. Non-random sampling. This approach is typically used when data modelers want the
most recent data as the test set.

#7 What are hyperparameters for linear regression?

Some set of parameters that are used to control the behaviour of the model/algorithm and
adjustable in order to obtain an improvised model with optimal performance is so-
called Hyperparameters.
#8 Give one real life application each of classification and regression.

Classification:

Classification is a process of finding a function which helps in dividing the dataset into
classes based on different parameters. In Classification, a computer program is trained on the
training dataset and based on that training, it categorizes the data into different classes. The
task of the classification algorithm is to find the mapping function to map the input(x) to the
discrete output(y). Example: The best example to understand the Classification problem is
Email Spam Detection. The model is trained on the basis of millions of emails on different
parameters, and whenever it receives a new email, it identifies whether the email is spam or
not. If the email is spam, then it is moved to the Spam folder.

Types of ML Classification Algorithms:

Classification Algorithms can be further divided into the following types:

Logistic Regression

K-Nearest Neighbours

Support Vector Machines

Kernel SVM

Naïve Bayes

Decision Tree Classification

Random Forest Classification

 Regression:

Regression is a process of finding the correlations between dependent and independent

variables. It helps in predicting the continuous variables such as prediction of Market
Trends, prediction of House prices, etc.

The task of the Regression algorithm is to find the mapping function to map the input
variable(x) to the continuous output variable(y).

Example: Suppose we want to do weather forecasting, so for this, we will use the Regression
algorithm. In weather prediction, the model is trained on the past data, and once the training

Regression Algorithm Classification Algorithm

In Regression, the output variable must be In Classification, the output variable must be a discrete
of continuous nature or real value. value.

The task of the regression algorithm is to The task of the classification algorithm is to map the input
map the input value (x) with the continuous value(x) with the discrete output variable(y).
output variable(y).

Regression Algorithms are used with Classification Algorithms are used with discrete data.
continuous data.

In Regression, we try to find the best fit line, In Classification, we try to find the decision boundary,
which can predict the output more which can divide the dataset into different classes.
accurately.

Regression algorithms can be used to solve Classification Algorithms can be used to solve
the regression problems such as Weather classification problems such as Identification of spam
Prediction, House price prediction, etc. emails, Speech Recognition, Identification of cancer cells,
etc.

The regression Algorithm can be further The Classification algorithms can be divided into Binary
divided into Linear and Non-linear Classifier and Multi-class Classifier.
Regression.

is completed, it can easily predict the weather for future days.

Types of Regression Algorithm:

1. Simple Linear Regression

2. Multiple Linear Regression
 3. Polynomial Regression
4. Support Vector Regression
5. Decision Tree Regression
6. Random Forest Regression

Difference between Regression and Classification

#9 Illustrate bias-variance trade off for regression

What is Bias?
The bias is known as the difference between the prediction of the values by the Machine
Learning model and the correct value. Being high in biasing gives a large error in training
as well as testing data. It recommended that an algorithm should always be low-biased to
avoid the problem of underfitting. By high bias, the data predicted is in a straight line
format, thus not fitting accurately in the data in the data set. Such fitting is known as
the Underfitting of Data. This happens when the hypothesis is too simple or linear in
nature. Refer to the graph given below for an example of such a situation.

In such a problem, a hypothesis looks like follows

What is Variance?
The variability of model prediction for a given data point which tells us the spread of our
data is called the variance of the model. The model with high variance has a very complex
fit to the training data and thus is not able to fit accurately on the data which it hasn’t seen
before. As a result, such models perform very well on training data but have high error rates
on test data. When a model is high on variance, it is then said to as Overfitting of Data.
Overfitting is fitting the training set accurately via complex curve and high order
hypothesis but is not the solution as the error with unseen data is high. While training a data
model variance should be kept low. The high variance data looks as follows.

In such a problem, a hypothesis looks like follows

Bias Variance Tradeoff

If the algorithm is too simple (hypothesis with linear equation) then it may be on high bias
and low variance condition and thus is error prone. If algorithm fit too complex (hypothesis
with high degree equation) then it mab be on high variance and low bias. In the latter
condition, the new entries will not perform well. Well, there is something between both of
these conditions, known as Trade-off or Bias Variance Trade-off. This tradeoff in complexity
is why there is a tradeoff between bias and variance. An algorithm can’t be more complex
and less complex at the same time. For the graph, the perfect tradeoff will be like this:
We try to optimize the value of the total error of the model by using the Variance Tradeoff:

The best fit will be given by the hypothesis on the tradeoff point. The error to complexity
graph to show trade-off is given as-

#10 Training error is 100% for a model trained by you. Will you be proud of your model?
Justify your answer.

When you are training a machine learning model, your main expectation from that model is
that: it should generalize well. The meaning of generalization is that once you train a model
on the given set of data points then that model should learn the pattern from those training
points and should be able to generalize that pattern for some unseen data points. So once the
model is trained, we need to assess the model on some basis to know that how well the model
is performing. And one of the very common( I would not say very popular) approaches is to
check the accuracy of the model.

In the case of machine learning, the meaning of accuracy is the number of correct predictions
out of the total predictions. By the definition you can see that there is no effect of the false
predictions on the measure of accuracy, that is the reason we should not prefer accuracy more
often. Two more versatile and concrete measures of the fitness of the model are precision and
recall. For this article, we are not going into this discussion that which fitness function is
better, so let’s rest this discussion here only.
Once a machine learning model is trained and the training accuracy is calculated, so there
might be a huge chance that the accuracy would result in a high range probably in the nineties
or even 100%. So, what does that mean? Does it mean that our model is 100% accurate and
no one could do better than us? The answer is “NO”. A high accuracy measured on the
training set is the result of Overfitting. So, what does this overfitting means?

Overfitting

Overfitting occurs when our machine learning model tries to cover all the data points or more
than the required data points present in the given dataset. Because of this, the model starts
caching noise and inaccurate values present in the dataset, and all these factors reduce the
efficiency and accuracy of the model. The overfitted model has low bias and high variance.

It means that the more you train your data on the same points, it will start to treat the noise
also as the data and will just imitate the entire pattern. The overfitting model is just like a
student cramming the entire syllabus for the exam and will perform well if presented in a way
he remembered but the same student will fail miserably when you ask the same question
reframed as another problem.

Low Bias:- No careful decision on what to study and what to not.

High Variance:- Trying to learn everything, even remember the page number of the book.

Underfitting

Underfitting occurs when our machine learning model is not able to capture the underlying
trend of the data. To avoid the overfitting in the model, the fed of training data can be stopped
at an early stage, due to which the model may not learn enough from the training data. As a
result, it may fail to find the best fit of the dominant trend in the data. The under-fitted model
has high bias and low variance.

It means that even if you are having enough data points to capture the pattern of the data but
you restrict your model with only a few data points. Taking back our example of a student,
then this will be the case of that over a smart student who tries to find the important topics
out of the entire syllabus and just finishes them. When it comes to testing the knowledge, this
student can’t perform well because he/she won't have complete knowledge.
The problem of overfitting is more serious than the problem of underfitting because in
overfitting one might not get an idea that whether the high accuracy is the result of
overfitting, or the model is actually performing well. While in the case of underfitting one
will directly get a hint that something is wrong with the model.

How to avoid the problem of overfitting

Both overfitting and underfitting cause the degraded performance of the machine learning
model. But the main cause is overfitting, so there are some ways by which we can reduce the
occurrence of overfitting in our model.

 Cross-Validation
 Training with more data
 Removing features
 Early stopping the training

#11 Describe K-fold cross validation procedure for training and evaluation of a learning
machine.

What is Cross-Validation?

Cross validation is a technique used in machine learning to evaluate the performance of a

model on unseen data. It involves dividing the available data into multiple folds or subsets,
using one of these folds as a validation set, and training the model on the remaining folds.
This process is repeated multiple times, each time using a different fold as the validation
set. Finally, the results from each validation step are averaged to produce a more robust
estimate of the model’s performance. Cross validation is an important step in the machine
learning process and helps to ensure that the model selected for deployment is robust and
generalizes well to new data.

What is cross-validation used for?

The main purpose of cross validation is to prevent overfitting, which occurs when a model
is trained too well on the training data and performs poorly on new, unseen data. By
evaluating the model on multiple validation sets, cross validation provides a more realistic
estimate of the model’s generalization performance, i.e., its ability to perform well on new,
unseen data.

Types of Cross-Validation

There are several types of cross validation techniques, including k-fold cross validation,
leave-one-out cross validation, and Holdout validation, Stratified Cross-
Validation. The choice of technique depends on the size and nature of the data, as well as
the specific requirements of the modeling problem.
1. Holdout Validation

In Holdout Validation, we perform training on the 50% of the given dataset and rest 50% is
used for the testing purpose. It’s a simple and quick way to evaluate a model. The major
drawback of this method is that we perform training on the 50% of the dataset, it may
possible that the remaining 50% of the data contains some important information which we
are leaving while training our model i.e. higher bias.

2. LOOCV (Leave One Out Cross Validation)

In this method, we perform training on the whole dataset but leaves only one data-point of
the available dataset and then iterates for each data-point. In LOOCV, the model is trained
on samples and tested on the one omitted sample, repeating this process for each data point
in the dataset. It has some advantages as well as disadvantages also.

An advantage of using this method is that we make use of all data points and hence it is
low bias.

The major drawback of this method is that it leads to higher variation in the testing model
as we are testing against one data point. If the data point is an outlier it can lead to higher
variation. Another drawback is it takes a lot of execution time as it iterates over ‘the
number of data points’ times.

3. Stratified Cross-Validation

It is a technique used in machine learning to ensure that each fold of the cross-validation
process maintains the same class distribution as the entire dataset. This is particularly
important when dealing with imbalanced datasets, where certain classes may be
underrepresented. In this method,

A) The dataset is divided into k folds while maintaining the proportion of classes in
each fold.
B) During each iteration, one-fold is used for testing, and the remaining folds are used
for training.
C) The process is repeated k times, with each fold serving as the test set exactly once.

Stratified Cross-Validation is essential when dealing with classification problems where

maintaining the balance of class distribution is crucial for the model to generalize well to
unseen data.

4. K-Fold Cross Validation

In K-Fold Cross Validation, we split the dataset into k number of subsets (known as folds)
then we perform training on the all the subsets but leave one(k-1) subset for the evaluation
of the trained model. In this method, we iterate k times with a different subset reserved for
testing purpose each time.

Note: It is always suggested that the value of k should be 10 as the lower value of k is takes
towards validation and higher value of k leads to LOOCV method.
Example of K Fold Cross Validation

The diagram below shows an example of the training subsets and evaluation subsets
generated in k-fold cross-validation. Here, we have total 25 instances. In first iteration we
use the first 20 percent of data for evaluation, and the remaining 80 percent for training ([1-
5] testing and [5-25] training) while in the second iteration we use the second subset of 20
percent for evaluation, and the remaining three subsets of the data for training ([5-10]
testing and [1-5 and 10-25] training), and so on.

Comparison between cross-validation and hold out method

Advantages of train/test split:

1. This runs K times faster than Leave One Out cross-validation because K-fold cross-
validation repeats the train/test split K-times.
2. Simpler to examine the detailed results of the testing process.

Advantages of cross-validation:

A. More accurate estimate of out-of-sample accuracy.

B. More “efficient” use of data as every observation is used for both training and
testing.

Advantages and Disadvantages of Cross Validation

Advantages:

1. Overcoming Overfitting: Cross validation helps to prevent overfitting by providing

a more robust estimate of the model’s performance on unseen data.
2. Model Selection: Cross validation can be used to compare different models and
select the one that performs the best on average.
 3. Hyperparameter tuning: Cross validation can be used to optimize the
hyperparameters of a model, such as the regularization parameter, by selecting the
values that result in the best performance on the validation set.
4. Data Efficient: Cross validation allows the use of all the available data for both
training and validation, making it a more data-efficient method compared to
traditional validation techniques.

Disadvantages:
1. Computationally Expensive: Cross validation can be computationally expensive,
especially when the number of folds is large or when the model is complex and requires
a long time to train.
2. Time-Consuming: Cross validation can be time-consuming, especially when there are
many hyperparameters to tune or when multiple models need to be compared.
3. Bias-Variance Tradeoff: The choice of the number of folds in cross validation can
impact the bias-variance tradeoff, i.e., too few folds may result in high variance, while
too many folds may result in high bias.

#12. Why do we need confusion matrix for evaluation of a machine trained for a
classification task?
In machine learning, classification is the process of categorizing a given set of data into
different categories. In machine learning, to measure the performance of the classification
model, we use the confusion matrix. Through this tutorial, understand the significance of
the confusion matrix.

What is a Confusion Matrix?

A confusion matrix is a matrix that summarizes the performance of a machine learning
model on a set of test data. It is a means of displaying the number of accurate and
inaccurate instances based on the model’s predictions. It is often used to measure the
performance of classification models, which aim to predict a categorical label for each
input instance.

The matrix displays the number of instances produced by the model on the test data.

True positives (TP): occur when the model accurately predicts a positive data point.
True negatives (TN): occur when the model accurately predicts a negative data point.
True positives (FP): occur when the model predicts a positive data point incorrectly.
False negatives (FN): occur when the model mispredicts a negative data point.

Why do we need a Confusion Matrix?

When assessing a classification model’s performance, a confusion matrix is essential. It

offers a thorough analysis of true positive, true negative, false positive, and false negative
predictions, facilitating a more profound comprehension of a model’s recall, accuracy,
precision, and overall effectiveness in class distinction. When there is an uneven class
distribution in a dataset, this matrix is especially helpful in evaluating a model’s
performance beyond basic accuracy metrics.
Let’s understand the confusion matrix with the examples:

Confusion Matrix For binary classification

A 2X2 Confusion matrix is shown below for the image recognition having a Dog image or
Not Dog image.

Actual

Dog Not Dog

True Positive False Positive

Dog (TP) (FP)

False Negative True Negative

Predicted Not Dog (FN) (TN)

True Positive (TP): It is the total counts having both predicted and actual values are Dog.
True Negative (TN): It is the total counts having both predicted and actual values are Not
Dog.
False Positive (FP): It is the total counts having prediction is Dog while actually Not Dog.
False Negative (FN): It is the total counts having prediction is Not Dog while actually, it is
Dog.

Metrics based on Confusion Matrix Data

1. Accuracy
Accuracy is used to measure the performance of the model. It is the ratio of Total correct
instances to the total instances.

For the above case:

Accuracy = (5+3)/(5+3+1+1) = 8/10 = 0.8

2. Precision
Precision is a measure of how accurate a model’s positive predictions are. It is defined as
the ratio of true positive predictions to the total number of positive predictions made by the
model.
For the above case:

Precision = 5/(5+1) =5/6 = 0.8333

3. Recall
Recall measures the effectiveness of a classification model in identifying all relevant
instances from a dataset. It is the ratio of the number of true positive (TP) instances to the
sum of true positive and false negative (FN) instances.

For the above case:

Recall = 5/(5+1) =5/6 = 0.8333

Note: We use precision when we want to minimize false positives, crucial in scenarios like
spam email detection where misclassifying a non-spam message as spam is costly. And we
use recall when minimizing false negatives is essential, as in medical diagnoses, where
identifying all actual positive cases is critical, even if it results in some false positives.

4. F1-Score

F1-score is used to evaluate the overall performance of a classification model. It is the

harmonic mean of precision and recall,

For the above case:

F1-Score: = (2* 0.8333* 0.8333)/( 0.8333+ 0.8333) = 0.8333

We balance precision and recall with the F1-score when a trade-off between minimizing
false positives and false negatives is necessary, such as in information retrieval systems.

Specificity:
Specificity is another important metric in the evaluation of classification models,
particularly in binary classification. It measures the ability of a model to correctly identify
negative instances. Specificity is also known as the True Negative Rate.

Specificity=3/(1+3)=3/4=0.75
6. Type 1 and Type 2 error

Type 1 error
Type 1 error occurs when the model predicts a positive instance, but it is actually negative.
Precision is affected by false positives, as it is the ratio of true positives to the sum of true
positives and false positives.

For example, in a courtroom scenario, a Type 1 Error, often referred to as a false positive,
occurs when the court mistakenly convicts an individual as guilty when, in truth, they are
innocent of the alleged crime. This grave error can have profound consequences, leading to
the wrongful punishment of an innocent person who did not commit the offense in question.
Preventing Type 1 Errors in legal proceedings is paramount to ensuring that justice is
accurately served and innocent individuals are protected from unwarranted harm and
punishment.

Type 2 error
Type 2 error occurs when the model fails to predict a positive instance. Recall is directly
affected by false negatives, as it is the ratio of true positives to the sum of true positives and
false negatives.

In the context of medical testing, a Type 2 Error, often known as a false negative, occurs
when a diagnostic test fails to detect the presence of a disease in a patient who genuinely
has it. The consequences of such an error are significant, as it may result in a delayed
diagnosis and subsequent treatment.

#14 Why should one perform exploratory data analysis? What are different conclusions can
you draw about data using exploratory data analysis? List some problems that you can
identify and their solution

Exploratory Data Analysis (EDA) is a critical data analysis process involving the initial
examination and exploration of a dataset. Its primary purpose is to summarize the main
characteristics of the data, often through visualization and summary statistics, to gain insights
into its structure, patterns, and potential issues.

Exploratory Data Analysis (EDA) is a fundamental and widely utilized approach within data
science. It involves thoroughly investigating and analyzing a dataset to summarize its main
characteristics through various visual and statistical methods. EDA serves multiple purposes,
including the discovery of data patterns, the identification of anomalies, hypothesis testing,
and the validation of assumptions. It closely aligns with Data analytics and data analysis,
which are interrelated processes to extract valuable insights from data to facilitate informed
decision-making.

In essence, EDA can be briefly defined as a method that empowers scientists to determine the
most effective means of manipulating a given dataset to achieve their desired goals and
extract meaningful information.

Exploratory Data Analysis Types

Processes to extract valuable insights from data to facilitate informed decision-making.

EDA typically involves various types of analyses and techniques. Here are some common
types of exploratory data analysis:

Univariate Non-graphical

Univariate Non Graphical is the most simplest form of data analysis. here it consists of just
one variable. Being a single variable, it doesn’t deal with causes or relationships. Instead, the
primary purpose of the univariate thematic analysis is to describe the data and find patterns
within it.

Univariate Graphical

Non-graphical methods cannot provide a complete picture of the data. Graphical methods are
therefore required here. The Common types of univariate graphics are:

Stem-and-leaf plots: These show all data values and the shape of the distribution.

Histograms a bar plot: in which each bar represents the frequency (count) or proportion
(count/total count) of cases for a range of values.

Box plots: graphically depict the five-number summary of minimum, first quartile, median,
third quartile, and maximum.

Multivariate Non-graphical

Multivariate data arises from more than one variable. Generally, Multivariate non-graphical
EDA techniques show the relationship between two or more data variables through cross-
tabulation or statistics.

Multivariate Graphical

Multivariate data uses graphics while displaying relationships between two or more Dataset.
The Most used graphic is a grouped bar plot or bar chart with each group representing one
level of one of the variables and each bar within a group representing the levels of the other
variable.
The Other common types of multivariate graphics include:

Scatter plot: Is used to plot data points on a horizontal and a vertical axis to show how much
one variable is affected by another.

Multivariate chart: Is a graphical representation of the relationships between factors and a

response.

Run chart: Is a line graph of data plotted over time.

Bubble chart: Is a data visualization that displays multiple circles (bubbles) in a two-
dimensional plot.

Heat map: Is a graphical representation of data where values are depicted by color.

EDA is an iterative process, and the choice of techniques and visualizations depends on the
data’s nature and the analysis’s specific goals. It helps analysts gain insights, make decisions,
and prepare the data for further modeling or analysis.

Steps of Exploratory Data Analysis Process

Exploratory Data Analysis (EDA) typically involves several key steps to gain insights into a
dataset. While the specific sequence of steps can vary, here is a general outline of the EDA
process:

Data Collection: Gather the dataset.

Data Cleaning: Handle missing values and outliers.

Data Exploration and Visualization: Analyze and visualize the data.

Feature Engineering: Enhance dataset for modeling and analysis.

Hypothesis Testing: Validate assumptions.

Communication and Documentation: Share findings and document the process.

EDA aims to gain a deep understanding of the data, identify patterns and relationships, and
make informed decisions about subsequent steps in the data analysis process, such as feature
selection, model building, or further data processing.

How Important Exploratory Data Analysis in Data Science

Exploratory Data Analysis (EDA) is a fundamental and crucial step in the field of data
science for several reasons:
Data Understanding:

EDA helps data scientists understand the dataset they are working with, including data
collection methods and the data source. It provides insights into the data set’s structure,
quality, and characteristics. Data scientists examine the data types, summary statistics, and
distribution of data points to grasp their fundamental properties.

Data Cleaning:

EDA often needs to uncover missing values, outliers, and inconsistencies in the data.
Cleaning and preprocessing the data is essential for building accurate and reliable models.
Handling missing matters is crucial during this stage, as they can significantly impact the
analysis and model performance.

Pattern Discovery:

EDA techniques reveal data patterns, relationships, and trends. A data scientist can identify
patterns and correlations by conducting univariate analysis and examining the relationships
between variables through bivariate or multivariate analysis. It can be valuable for making
informed decisions and formulating hypotheses to answer exciting questions about the data.

Data Visualization:

Visualizations created during EDA, such as histograms, scatter plots, and box plots, make it
easier to communicate findings and insights to technical and non-technical stakeholders.
They can be powerful tools for storytelling, helping to convey the data’s story effectively.

Model Selection:

Based on the insights gained from EDA, scientists can make informed decisions about model
selection. For instance, if EDA reveals non-linear relationships or interactions between
variables, it may lead to the choice of non-linear models like decision trees or neural
networks.

Quality Control:

EDA can uncover data quality issues early in the analysis process. Addressing these issues,
such as data outliers and inconsistencies, is crucial before modeling. Detecting and handling
data quality problems at this stage can save time and resources.

EDA is not just important but essential in data science. EDA helps a data scientist makes
informed decisions, understand the data’s limitations, and extract valuable information from
the data, ultimately leading to more accurate and meaningful results. It plays a critical role in
addressing missing values, summarizing data points, and ensuring the data set is ready for in-
depth analysis.
Exploratory Data Analysis Tools

Exploratory Data Analysis (EDA) is a necessary process in data analysis, and several tools
are available to help data scientists and analysts in this endeavor. Among the most popular
tools for EDA are R, Python, and SAS. Each tool has its strengths and weaknesses, making it
essential to choose the right one for your specific needs.

R:

The data science community widely acknowledges R as an exceptional tool for visualizing
data during EDA. R furnishes an extensive repertoire of plots and charts, encompassing
histograms, box plots, and scatter plots, which analysts can employ to investigate data from
various perspectives. R provides a rich set of visualization options and an extensive collection
of statistical functions, making it valuable for performing more advanced analyses. R is
favored by statisticians and data scientists for its versatility and comprehensive statistical
capabilities.

Python:

Python is another robust tool for EDA. It shares many features with R, making it a
compelling choice for data analysis. Python’s advantage lies in its user-friendliness and
versatility, making it a good option for beginners just getting started with data analysis. The
Python ecosystem includes libraries like Pandas, Matplotlib, Seaborn, and Plotly, which
facilitate data manipulation and visualization. Python’s readability and extensive community
support further enhance its appeal for data exploration.

SAS:

SAS is a powerful software package for statistical analysis and data exploration. It is
renowned for its capabilities in handling complex calculations and conducting advanced
statistical analyses. While SAS is a formidable tool, it’s important to note that it tends to be
more expensive than open-source alternatives like R and Python. However, the investment
can be worthwhile for organizations and professionals who require the robust statistical
features SAS offers.

In addition to these three primary tools, other software and platforms can be valuable for
EDA. For instance, tools like Tableau, Excel, Power BI, and KNIME offer user-friendly
interfaces for interactive data exploration and visualization, making them suitable for a
broader audience, including non-technical stakeholders.

Choosing an EDA tool depends on your specific goals, expertise, and budget. While R,
Python, and SAS are among the top choices, selecting the right tool for your project is
essential for conducting practical exploratory data analysis. Data scientists often encounter
challenges related to missing values, data sets, dependent variables, median values,
categorical variables, data points, and analysis techniques. Proper handling of these aspects is
crucial to deriving meaningful insights from raw data and preparing new data collections for
further analysis. Additionally, measures such as standard deviation and other statistical
metrics are essential for understanding the distribution and variability of data points.
QuestionPro and Exploratory Data Analysis

When you obtain your data from various sources, QuestionPro offers excellent support in
gathering survey data through multiple channels. However, what do you do when you wish to
delve deeper into the data beyond what’s already been collected? It is where exploratory data
analysis (EDA) comes into play.

QuestionPro’s integrated suite of analytical tools simplifies the initiation of EDA. It enables
you to swiftly access your data’s summary statistics, such as median values and cumulative
distribution functions. You can also craft interactive visualizations to uncover patterns and
relationships within the dataset. Furthermore, QuestionPro’s seamless integration with R
allows you to harness R’s formidable statistical capabilities, including regression analysis for
understanding dependent variables.

Suppose you’re ready to take your data analysis to a more advanced level, whether for data
analytics, machine learning, or deep learning tasks. QuestionPro stands out as the perfect tool
for the job. It equips you to handle missing values, work with diverse data sets, and extract
valuable insights from data points.

Conclusion

Exploratory data analysis (EDA) remains a well-established methodology, serving as a

valuable compass for a data scientist navigating intricate datasets. By leveraging the power of
visualizations, correlation coefficients, and other analytical techniques, EDA unveils intricate
patterns and relationships that might otherwise remain concealed. It plays a pivotal role in
detecting and addressing missing values, understanding the behavior of dependent variables,
and employing various analysis techniques, including those associated with data analytics and
machine learning.

Hence, EDA stands as a fundamental cornerstone of any data analysis endeavor. This article
has provided you with a solid introduction to this indispensable practice. In the ever-evolving
landscape of data science, EDA remains essential, enabling the transformation of raw data
into actionable insights.

#15. We deploy a K-NN classifier for a two class classification problem. The data has 10
dimensions and is known not to have many outliers. (a) What distance matrix will you use for
identifying K nearest neighbours. Write an expression for calculating the distance. (b) How
will you measure performance of your classifier?

The K-Nearest Neighbors (KNN) algorithm is a popular machine learning technique used
for classification and regression tasks. It relies on the idea that similar data points tend to
have similar labels or values.

During the training phase, the KNN algorithm stores the entire training dataset as a
reference. When making predictions, it calculates the distance between the input data
point and all the training examples, using a chosen distance metric such as Euclidean
distance.
Next, the algorithm identifies the K nearest neighbors to the input data point based on
their distances. In the case of classification, the algorithm assigns the most common
class label among the K neighbors as the predicted label for the input data point. For
regression, it calculates the average or weighted average of the target values of the K
neighbors to predict the value for the input data point.

The KNN algorithm is straightforward and easy to understand, making it a popular

choice in various domains. However, its performance can be affected by the choice of K
and the distance metric, so careful parameter tuning is necessary for optimal results.

When Do We Use the KNN Algorithm?

KNN Algorithm can be used for both classification and regression predictive problems.
However, it is more widely used in classification problems in the industry. To evaluate
any technique, we generally look at 3 important aspects:

1. Ease of interpreting output

2. Calculation time

3. Predictive Power

Conclusion

The KNN algorithm is one of the simplest classification algorithms. Even with such
simplicity, it can give highly competitive results. KNN algorithm can also be used for
regression problems. The only difference from the discussed methodology will be using
averages of nearest neighbors rather than voting from k-nearest neighbors. KNN can be
coded in a single line on R. I am yet to explore how we can use the KNN algorithm on
SAS.

Key Takeaways

KNN classifier operates by finding the k nearest neighbors to a given data point, and it
takes the majority vote to classify the data point.

The value of k is crucial, and one needs to choose it wisely to prevent overfitting or
underfitting the model.

One can use cross-validation to select the optimal value of k for the k-NN algorithm,
which helps improve its performance and prevent overfitting or underfitting. Cross-
validation is also used to identify the outliers before applying the KNN algorithm.

The above article provides implementations of KNN in Python and R, and it compares
the result with scikit-learn and the “Class” library in R.
Q1. What is K nearest neighbors algorithm?

A. KNN classifier is a machine learning algorithm used for classification and regression
problems. It works by finding the K nearest points in the training dataset and uses their
class to predict the class or value of a new data point. It can handle complex data and is
also easy to implement, which is why KNN has become a popular tool in the field of
artificial intelligence.

Q2. What is KNN algorithm used for?

A. KNN algorithm is most commonly used for:

1. Disease prediction – predicts the likelihood of a disease based on symptoms and the
data available.
2. Handwriting recognition – recognizes the handwritten characters.
3. Image classification – recognizes images in computer vision.

Q3. What is the difference between KNN and Artificial Neural Networks?

A. K-nearest neighbors (KNN) are mainly used for classification and regression
problems, while Artificial Neural Networks (ANN) are used for complex function
approximation and pattern recognition problems. Moreover, ANN has a higher
computational cost than KNN.

#17 Given the following two-dimensional data points. X = [x1, x2] = [1.5, 1.7], [2, 1.9],
[1.6, 1.8], [1.2, 1.5], [1.5, 1.0] (a) Given a new data point x = [1.4, 1.6] as a query, rank the
data points based on similarity with the query using Euclidean distance and Manhattan
distance. (b) If we center and scale the data and then use Euclidean distance on the
transformed data to the rank points, will the rank change? Comment on the result

a) To rank the data points based on similarity with the query using Euclidean distance and
Manhattan distance, we first calculate the distances between the query point and each data
point.

Euclidean distance between two points (x1, y1) and (x2, y2) is given by:
Euclidean Distance=(�2−�1)2+(�2−�1)2Euclidean Distance=(x2−x1)2+(y2−y1)2

Manhattan distance between two points (x1, y1) and (x2, y2) is given by:
Manhattan Distance=∣�2−�1∣+∣�2−�1∣Manhattan Distance=∣x2−x1∣+∣y2−y1∣

For the given data points and query point x = [1.4, 1.6]:

Euclidean Distance Rankings:Euclidean Distance Rankings:

1. [1.5, 1.7]
2. [1.6, 1.8]
3. [1.2, 1.5]
4. [2, 1.9]
5. [1.5, 1.0]
Manhattan Distance Rankings:Manhattan Distance Rankings:

1. [1.5, 1.7]
2. [1.6, 1.8]
3. [1.2, 1.5]
4. [2, 1.9]
5. [1.5, 1.0]

(b) If we center and scale the data and then use Euclidean distance on the transformed data,
the rankings may change. Centering and scaling involve subtracting the mean and dividing by
the standard deviation for each dimension.

Let's say we center and scale the data by subtracting the mean and dividing by the standard
deviation for each dimension. After transforming the data, if we apply Euclidean distance to
the query point [1.4, 1.6], the rankings might change due to the effect of the scaling.

In summary, centering and scaling can affect the rankings when using distance-based
measures, as the relative distances between points may change after the transformation.