SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Experiment No. 1

Name of the Experiment: Data preparation using NumPy and Pandas

Aim:
a. Derive an index field and add it to the data set.
b. Find out the missing values.
c. Obtain a listing of all records that are outliers according to the any field. Print
out a listing of the 10 largest values for that field.
d. Do the following for the any field.
i. Standardize the variable.
ii. Identify how many outliers there are and identify the most extreme outlier.

Requirements: Anaconda Distribution – Python Platform/ Jupyter Notebook/ PyCharm IDE/ Spyder
python IDE/ Google COLAB – Cloud based Jupyter Notebook Environment

Pre-requisite: Python Standard Libraries

Theory :-
Pandas is defined as an open-source library that provides high-performance data manipulation in
Python. It is built on top of the NumPy package, which means Numpy is required for operating the
Pandas. The name of Pandas is derived from the word Panel Data, which means an Econometrics from
Multidimensional data. It is used for data analysis in Python and developed byWes McKinney in 2008.
NumPy is mostly written in C language, and it is an extension module of Python. It is defined as a
Python package used for performing the various numerical computations and processing of the
multidimensional and single-dimensional array elements. The calculations using Numpy arrays are
faster than the normal Python array.
Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures
and data analysis tools for the Python programming language. Pandas makes importing, analyzing, and
visualizing data much easier. It builds on packages like NumPy and matplotlib to give you a single,
convenient, place to do most of your data analysis and visualization work.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

import numpy as np
import pandas as pd
# Creating the data frame
df=[Link]()
print(df)
o/p
Empty DataFrame
Columns: []
Index: []
#data frame can be created from lists, dictonaries etc
content =[0,1,2,3,4,5,6,7]
df=[Link](content)
print(df)
0
0 0
1 1
2 2
3 3
4 4
5 5
6 6
7 7

Basis for Pandas NumPy

Comparison

Works with Pandas moduleworks withthe tabular data. NumPy moduleworkswith

numerical data.

Powerful Tools Pandas has powerful tools like Series, NumPy has a powerful
DataFrame etc.
Tool like Arrays.

2
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Organizational Pandas is used in popular organizations NumPy is used in the popular

usage like Instacart,SendGrid, and organization like SweepSouth.
Sighten.

Performance Pandas has a better performance NumPy has a better performance

for 500K rows or more. for 50K rows or less.

Memory Utilization Pandas consume largememory ascompared NumPy consumes less

to NumPy. memory as compared to Pandas.

IndustrialCoverage Pandas is mentioned in 73 companystacks NumPy is mentioned

and 46 developer stacks. in 62 companystacksand 32
developer stacks.

Objects Pandas provides 2d table object NumPy provides a

called DataFrame. multi-dimensional array.

a. Derive an index field and add it to the data set.

Uploading file from Local Source

For Uploading on the colab drive

To Display the file Heading

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

#Create github account and you can upload the data set in new project folder to get global link for the
dataset. So that for every use you need not to upload the dataset on drive.
#df1=pd.read_csv("[Link]
m_8dd85q.csv")
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

b. Find out the missing values.

c. Obtain a listing of all records that are outliers according to the any field. Print out
alisting of the 10 largest values for that field.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

d. Obtain a listing of all records that are outliers according to the any field. Print
out alisting of the 10 largest values for that field.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

i. Standardize the variable.

ii. Identify how many outliers there are and identify the most extreme outlier.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Conclusion:

For the above AIDS experiment the dataset used is california_housing_test..csv. With help of
google colab platform it’s very easy to perform DataScience algorithm. Use of online storage
likegoogle drive in colab has increased availability of the dataset for all the time .Numpy and
pandaslibrary for variety of task on california_housing_test.csv has been studied successfully.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Experiment No. 2

Name of the Experiment: Data Visualization / Exploratory Data Analysis for the
selected data set using Matplotlib andSeaborn.

Aim:
a. Create a bar graph, contingency table using any 2 variables.
b. Create normalized histogram.
c. Describe what this graphs and tables indicates?

Requirements: Anaconda Distribution – Python Platform/ Jupyter Notebook/ PyCharm IDE/ Spyder
python IDE/ Google COLAB – Cloud based Jupyter Notebook Environment

Pre-requisite: Python Standard Libraries

Theory:

Matplotlib is an amazing visualization library in Python for 2D plots of arrays. Matplotlib is a

multi-platform data visualization library built on NumPy arrays and designed to work with the
broader SciPy stack. It was introduced by John Hunter in the year 2002.

One of the greatest benefits of visualization is that it allows us visual access to huge amounts of
data in easily digestible visuals. Matplotlib consists of several plots like line, bar, scatter,
histogrametc.

Seaborn is a library for making statistical graphics in Python. It builds on top of matplotlib and
integrates closely with pandas data structures.

Seaborn helps you explore and understand your data. Its plotting functions operate on
dataframesand arrays containing whole datasets and internally perform the necessary semantic
mapping and statistical aggregation to produce informative plots. Its dataset-oriented,
declarative API lets you focus on what the different elements of your plots mean, rather than on
the details of how to drawthem
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

a. Create a bar graph, contingency table using any 2 variables.

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

b. Create normalized histogram.

c. Describe what this graphs and tables indicates

While we can be productive using only seaborn functions, full customization of our graphics will require
some knowledge of matplotlib’s concepts and API. One aspect of the learning curve for new users of
seaborn will be knowing when dropping down to the matplotlib layer is necessary to achieve a particular
customization. On the other hand, users coming from matplotlib will find that much of their knowledge
transfers.

Matplotlib has a comprehensive and powerful API; just about any attribute of the figure can be changedto your
liking. A combination of seaborn’s high-level interface and matplotlib’s deep customizabilitywill allow you
both to quickly explore your data and to create graphics that can be tailored into a publication quality final
product.

Conclusion:

For the above AIDS experiment the dataset used is california_housing_test..csv. visualization library
like seaborn helps in API abstraction across visualizations, Statistical estimation and error bars,
Informative distributional summaries, Specialized plots for categorical data, Composite views onto
multivariate datasets, Classes and functions for making complex graphics. Seaborn’s integration with
matplotlib allows you to use it across the many environments that matplotlib supports, inlcuding
exploratory analysis in notebooks, real-time interaction in GUI applications, and archival output in a
number of raster and vector formats.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Experiment No. 3

Name of the Experiment: Data Modeling

Aim:
a. Partition the data set, for example 75% of the records are includedin the training data set and 25% are
included in the test data set. Use a bar graph to confirm yourproportions.
b. Identify the total number of records in the training data set.
c. Validate your partition by performing a two‐sample Z‐test.
*Apply this on your dataset only

Requirements: Anaconda Distribution – Python Platform/ Jupyter Notebook/ PyCharm IDE/ Spyder
python IDE/ Google COLAB – Cloud based Jupyter Notebook Environment

Pre-requisite: Python Standard Libraries

Theory:

Scikit-learn is an open source machine learning library that supports supervised and unsupervised
learning. It also provides various tools for model fitting, data preprocessing, model selection, model
evaluation, and many other utilities.

Scikit-learn provides dozens of built-in machine learning algorithms and models, called [Link]
estimator can be fitted to some data using its fit method.

Fitting a model to some data does not entail that it will predict well on unseen data. This needs to be
directly evaluated. We have just seen the train_test_split helper that splits a dataset into train and test
sets, but scikit-learn provides many other tools for model evaluation, in particular for cross-validation.

Scikit-learn provides tools to automatically find the best parameter combinations (via cross-
validation). In the following example, we randomly search over the parameter space of a random forest
with a RandomizedSearchCV object. When the search is over, the RandomizedSearchCV behaves as a
RandomForestRegressor that has been fitted with the best set of parameters.

Import sklearn
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Sample example:-
loading the Wine Data

a. Using train_test_split to Split Data into Training and Testing Data

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Returning a Non-Stratified Result

b. Seeing the split across training and testing datasets

Generate a Two Column DataFrame

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Split and Label the Data

Visualize the Data

c. total number of records in the training data set.

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

d. Validate your partition by performing a two‐sample Z‐test.

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Conclusion:

We have provided 2 normally distributed and independent populations, and we have drawn
samples at random from both populations. Here, we consider u1 and u2 be the population mean
X1 and X2 are the observed sample mean.

Type 1 error and Type II error:

Type I error: Type 1 error has occurred when we reject the null hypothesis, even when the
hypothesis is true. This error is denoted by alpha.

Type II error: Type II error has occurred when we didn’t reject the null hypothesis, even when
the hypothesis is false. This error is denoted by beta.

Null Hypothesis is
Null Hypothesis is FALSE
TRUE

Type I Error
Corre
Reject Null Hypothesis
(False ct
decisi
Positive)
on

Type II error
Fail to Reject the Null (False
Correct decision
Hypothesis Negati
ve)
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Experiment No. 4

Name of the Experiment: Implementation of Statistical Hypothesis Test using Scipy

and Sci-kit learn [Any one]

Aim:
1. Normality Tests
A. Shapiro-Wilk Test
B. D’Agostino’s K^2 Test
C. Anderson-Darling Test
2. Correlation Tests
A. Pearson’s Correlation Coefficient
B. Spearman’s Rank Correlation
C. Kendall’s Rank Correlation
D. Chi-Squared Test
3. Stationary Tests
A. Augmented Dickey-Fuller
B. Kwiatkowski-Phillips-Schmidt-Shin
4. Parametric Statistical Hypothesis Tests
A. Student’s t-test
B. Paired Student’s t-test
C. Analysis of Variance Test (ANOVA)
D. Repeated Measures ANOVA Test
5. Nonparametric Statistical Hypothesis Tests
A. Mann-Whitney U Test
B. Wilcoxon Signed-Rank Test
C. Kruskal-Wallis H Test
D. Friedman Test

Requirements: Anaconda Distribution – Python Platform/ Jupyter Notebook/ PyCharm IDE/ Spyder
python IDE/ Google COLAB – Cloud based Jupyter Notebook Environment

Pre-requisite: Python Standard Libraries

Theory:

Implementation of Statistical Hypothesis Test using Scipy and Sci-kit learn

1. Normality Tests
2. Shapiro-Wilk Test
3. D’Agostino’sK^2 Test
4. Anderson-Darling Test
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

A large fraction of the field of statistics is concerned with data that assumes that it was drawn from a
Gaussian distribution.

If methods are used that assume a Gaussian distribution, and your data was drawn from a different
distribution, the findings may be misleading or plain wrong.

There are a number of techniques that you can check if your data sample is Gaussian or sufficiently
Gaussian-like to use the standard techniques, or sufficiently non-Gaussian to instead use non-
parametric statistical methods.

This is a key decision point when it comes to choosing statistical methods for your data sample. We
can summarize this decision as follows:

If Data Is Gaussian:
Use Parametric Statistical Methods
Else:
Use Nonparametric Statistical Methods

There is also some middle ground where we can assume that the data is Gaussian-enough to use
parametric methods or that we can use data preparation techniques to transform the data to be
sufficiently Gaussian to use the parametric methods.

There are three main areas where you may need to make this evaluation of a data sample in a
machine learning project; they are:

Input data to the model in the case of fitting models.

Model evaluation results in the case of model selection.
Residual errors from model predictions in the case of regression.
For checking whether a sample of data is Gaussian:

Graphical Methods. These are methods for plotting the data and qualitatively evaluating whether the
data looks Gaussian.

Statistical Tests. These are methods that calculate statistics on the data and quantify how likely it is that
the data was drawn from a Gaussian distribution.

Methods of this type are often called normality tests.

Test Dataset

Before we start looking at normality tests, let’s first develop a test dataset that we can use throughout
this tutorial.

We will generate a small sample of random numbers drawn from a Gaussian distribution.

The choice of Gaussian random numbers for the test dataset means that we do expect each test to
correctly identify the distribution, nevertheless, the small-ish sample size may introduce some noise
into the results.

We will use the randn() NumPy function to generate random Gaussian numbers with a mean of 0 and a
standard deviation of 1, so-called standard, normal variables. We will then shift them to havea mean of
50 and a standard deviation of 5.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Statistical Normality Tests

There are many statistical tests that we can use to quantify whether a sample of data looks as though it
was drawn from a Gaussian distribution.

Each test makes different assumptions and considers different aspects of the data.
Each test will return at least two things:

Statistic: A quantity calculated by the test that can be interpreted in the context of the test via
comparing it to critical values from the distribution of the test statistic.

p-value: Used to interpret the test, in this case whether the sample was drawn from a Gaussian
distribution.

Each test calculates a test-specific statistic. This statistic can aid in the interpretation of the result,
although it may require a deeper proficiency with statistics and a deeper knowledge of the specific
statistical test. Instead, the p-value can be used to quickly and accurately interpret the statistic in
practical applications.

The tests assume that that the sample was drawn from a Gaussian distribution. Technically this iscalled
the null hypothesis, or H0. A threshold level is chosen called alpha, typically 5% (or 0.05), that is used
to interpret the p-value.

In the SciPy implementation of these tests, you can interpret the p value as

follows.p <= alpha: reject H0, not normal.

p > alpha: fail to reject H0, normal.

This means that, in general, we are seeking results with a larger p-value to confirm that our samplewas
likely drawn from a Gaussian distribution.

A result above 5% does not mean that the null hypothesis is true. It means that it is very likely true given
available evidence. The p-value is not the probability of the data fitting a Gaussian distribution; it can be
thought of as a value that helps us interpret the statistical test.

23
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Shapiro-Wilk Test

The Shapiro-Wilk test evaluates a data sample and quantifies how likely it is that the data was drawn
from a Gaussian distribution, named for Samuel Shapiro and Martin Wilk.
In practice, the Shapiro-Wilk test is believed to be a reliable test of normality, although there is some
suggestion that the test may be suitable for smaller samples of data, e.g. thousands of observations or
fewer.
The shapiro() SciPy function will calculate the Shapiro-Wilk on a given dataset. The function returns
both the W-statistic calculated by the test and the p-value.

D’Agostino’s K^2 Test

The D’Agostino’s K^2 test calculates summary statistics from the data, namely kurtosis and skewness,
to determine if the data distribution departs from the normal distribution, named for Ralph D’Agostino.

Skew is a quantification of how much a distribution is pushed left or right, a measure of asymmetryin the
[Link] quantifies how much of the distribution is in the tail.

It is a simple and commonly used statistical test for [Link] D’Agostino’s K^2 test is available
via the normaltest() SciPy function and returns the test statistic and the p-value.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Anderson-Darling Test

Anderson-Darling Test is a statistical test that can be used to evaluate whether a data sample comesfrom
one of among many known data samples, named for Theodore Anderson and Donald Darling.

It can be used to check whether a data sample is normal. The test is a modified version of a more
sophisticated nonparametric goodness-of-fit statistical test called the Kolmogorov-Smirnov test.

A feature of the Anderson-Darling test is that it returns a list of critical values rather than a single p-
value. This can provide the basis for a more thorough interpretation of the result.

The anderson() SciPy function implements the Anderson-Darling test. It takes as parameters the data
sample and the name of the distribution to test it against. By default, the test will check against the
Gaussian distribution (dist=’norm’).
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Conclusion:

From the above test sample methods for Normality Test, we can derive that which test to be used for
analysis depends on your data, where appropriate. The question then becomes, how do you interpret the
results? What if the tests disagree, which they often will? Thus the Normality test hasbeen observed for
the given dataset.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Experiment No. 5

Name of the Experiment: Regression Analysis

Aim:
a. Perform Logistic Regression to find out relation between variables.
b. Apply regression Model techniques to predict the data on above Dataset

Requirements: Anaconda Distribution – Python Platform/ Jupyter Notebook/ PyCharm IDE/ Spyder
python IDE/ Google COLAB – Cloud based Jupyter Notebook Environment

Pre-requisite: Python Standard Libraries

Theory :
Classification techniques are an essential part of machine learning and data mining applications.
Approximately 70% of problems in Data Science are classification problems. There are lots of
classification problems that are available, but the logistics regression is common and is a useful
regression method for solving the binary classification problem. Another category of classification is
Multinomial classification, which handles the issues where multiple classes are present in the target
variable. For example, IRIS dataset a very famous example of multi-class classification.

Logistic Regression can be used for various classification problems such as spam detection. Diabetes
prediction, if a given customer will purchase a particular product or will they churn another competitor,
whether the user will click on a given advertisement link or not.

Logistic Regression is one of the most simple and commonly used Machine Learning algorithms for
two-class classification. It is easy to implement and can be used as the baseline for any binary
classification problem. Its basic fundamental concepts are also constructive in deep learning. Logistic
regression describes and estimates the relationship between one dependent binary variable and
independent variables.

Logistic regression is a statistical method for predicting binary classes. The outcome or target variable
is dichotomous in nature. Dichotomous means there are only two possible classes. For example, it can
be used for cancer detection problems. It computes the probability of an event occurrence.

It is a special case of linear regression where the target variable is categorical in nature. It uses a log of
odds as the dependent variable. Logistic Regression predicts the probability of occurrence of a binary
event utilizing a logit function.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Step 1: Import Packages, Functions, and Classes

Step 2: Get Data

Step 3: Create a Model and Train It.

IT Department BVCOE, Navi Mumbai 28

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Step 4: Evaluate the Model

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Improve the Model

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Conclusion:

You now know what logistic regression is and how you can implement it for classification with
Python. You’ve used many open-source packages, including NumPy, to work with arrays and
Matplotlib to visualize the results. You also used both scikit-learn and StatsModels to create, fit,
evaluate, and apply models.

Generally, logistic regression in Python has a straightforward and user-friendly implementation. It

usually consists of these steps:

a) Import packages, functions, and classes

b) Get data to work with and, if appropriate, transform it
c) Create a classification model and train (or fit) it with existing data
d) Evaluate your model to see if its performance is satisfactory
e) Apply your model to make predictions
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Experiment No. 6

Name of the Experiment: Classification modeling.

Aim:
a. Choose classifier for classification problem.
b. Evaluate the performance of classifier.

Requirements: Anaconda Distribution – Python Platform/ Jupyter Notebook/ PyCharm IDE/ Spyder
python IDE/ Google COLAB – Cloud based Jupyter Notebook Environment

Pre-requisite: Python Standard Libraries

Theory : One of the classifier is explained here depending on the case study you can implement other
types of classifiers also.
Naive Bayes classifiers are a collection of classification algorithms based on Bayes’ Theorem. Itis not
a single algorithm but a family of algorithms where all of them share a common principle, i.e. every
pair of features being classified is independent of each [Link] start with, let us consider a dataset.
Consider a fictional dataset that describes the weather conditions for playing a game of golf. Given the
weather conditions, each tuple classifies the conditions as fit(“Yes”) or unfit(“No”) for playinggolf.

Here is a tabular representation of our dataset.

Outlook Temperature Humidity Windy Play Golf

0 Rainy Hot High False No

1 Rainy Hot High True No

2 Overcast Hot High False Yes

3 Sunny Mild High False Yes

4 Sunny Cool Normal False Yes

5 Sunny Cool Normal True No

6 Overcast Cool Norm True Yes

al

7 Rainy Mild High False No

8 Rainy Cool Normal False Yes

9 Sunny Mild Normal False Yes

10 Rainy Mild Normal True Yes

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

11 Overcast Mild High True Yes

12 Overcast Hot Normal False Yes

13 Sunny Mild High True No

The dataset is divided into two parts, namely, feature matrix and the response vector.

Feature matrix contains all the vectors(rows) of dataset in which each vector consists of the
value of dependent features. In above dataset, features are ‘Outlook’, ‘Temperature’,
‘Humidity’ and ‘Windy’.

Response vector contains the value of class variable(prediction or output) for each row of
featurematrix. In above dataset, the class variable name is ‘Play golf’.

Assumption:

The fundamental Naive Bayes assumption is that each feature makes

an:Independent Equal Contribution to the outcome.

With relation to our dataset, this concept can be understood as:

We assume that no pair of features are dependent. For example, the temperature being ‘Hot’ has
nothing to do with the humidity or the outlook being ‘Rainy’ has no effect on the winds. Hence, the
features are assumed to be independent.

Secondly, each feature is given the same weight(or importance). For example, knowing only
temperature and humidity alone can’t predict the outcome accurately. None of the attributes is
irrelevant and assumed to be contributing equally to the outcome.

Note: The assumptions made by Naive Bayes are not generally correct in real-world situations. In- fact,
the independence assumption is never correct but often works well in practice.

Now, before moving to the formula for Naive Bayes, it is important to know about Bayes’ theorem.
Bayes’ Theorem

Bayes’ Theorem finds the probability of an event occurring given the probability of another event that
has already occurred. Bayes’ theorem is stated mathematically as the following equation:

P(A|B) = \frac{P(B|A) P(A)}{P(B)}

where A and B are events and P(B) ≠ 0.

Basically, we are trying to find probability of event A, given the event B is true. Event B is also
termed as evidence.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

P(A) is the priori of A (the prior probability, i.e. Probability of event before evidence is seen). The
evidence is an attribute value of an unknown instance(here, it is event B).

P(A|B) is a posteriori probability of B, i.e. probability of event after evidence is [Link], with regards

to our dataset, we can apply Bayes’ theorem in following way: P(y|X) = \frac{P(X|y) P(y)}{P(X)}

where, y is class variable and X is a dependent feature vector (of size n) where:X = (x_1,x_2,x_3,,x_n)

Just to clear, an example of a feature vector and corresponding class variable can be: (refer 1st row of
dataset)

X = (Rainy, Hot, High, False)y = No

So basically, P(y|X) here means, the probability of “Not playing golf” given that the weather
conditions are “Rainy outlook”, “Temperature is hot”, “high humidity” and “no wind”.
Naive assumption

Now, its time to put a naive assumption to the Bayes’ theorem, which is, independence among the
features. So now, we split evidence into the independent parts.

Now, if any two events A and B are independent,

then,P(A,B) = P(A)P(B)

Hence, we reach to the result:

P(y|x_1,...,x_n) = \frac{ P(x_1|y)P(x_2|y)...P(x_n|y)P(y)}{P(x_1)P(x_2)...P(x_n)} Which can be

expressed as

P(y|x_1,...,x_n) = \frac{P(y)\prod_{i=1}^{n}P(x_i|y)}{P(x_1)P(x_2)...P(x_n)}

Now, as the denominator remains constant for a given input, we can remove that term:

P(y|x_1,...,x_n)\propto P(y)\prod_{i=1}^{n}P(x_i|y)

Now, we need to create a classifier model. For this, we find the probability of given set of inputs for all
possible values of the class variable y and pick up the output with maximum probability. This can be
expressed mathematically as:

y = argmax_{y} P(y)\prod_{i=1}^{n}P(x_i|y)

So, finally, we are left with the task of calculating P(y) and P(xi | y).

Please note that P(y) is also called class probability and P(xi | y) is called conditional probability.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

The different naive Bayes classifiers differ mainly by the assumptions they make regarding the
distribution of P(xi | y).

Let us try to apply the above formula manually on our weather dataset. For this, we need to do some
precomputations on our dataset.

We need to find P(xi | yj) for each xi in X and yj in y. All these calculations have been demonstratedin the
tables below:
So, in the figure above, we have calculated P(xi | yj) for each xi in X and yj in y manually in
the tables 1-4. For example, probability of playing golf given that the temperature is cool, i.e
P(temp.
= cool | play golf = Yes) = 3/9.

Gaussian Naive Bayes classifier

In Gaussian Naive Bayes, continuous values associated with each feature are assumed to be distributed
according to a Gaussian distribution. A Gaussian distribution is also called Normal distribution. When
plotted, it gives a bell shaped curve which is symmetric about the mean of thefeature values as shown
below:

The likelihood of the features is assumed to be Gaussian, hence, conditional probability is given by:

P(x_i | y) = \frac{1}{\sqrt{2\pi\sigma _{y}^{2} }} exp \left (-\frac{(x_i-\mu _{y})^2}{2\sigma

_{y}^{2}} \right )

Now, we look at an implementation of Gaussian Naive Bayes classifier using scikit-

learn.# load the iris dataset

# store the feature matrix (X) and response vector (y)

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

# splitting X and y into training and testing sets

# training the model on training set

# making predictions on the testing set

# comparing actual response values (y_test) with predicted response values (y_pred)

Conclusion:

For the above AIDS experiment the dataset used is [Link]. With the help of sklearn library the
functions of Gaussian Naive Bayes classifier observed the above model gave the accuracy of classifier
to 95%.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Experiment No. 7

Name of the Experiment: Clustering.

Aim:
a. Clustering algorithms for unsupervised classification.
b. Plot the cluster data.
Find the attribute and predicting attribute from dataset and Implement above listed points.

Requirements: Anaconda Distribution – Python Platform/ Jupyter Notebook/ PyCharm IDE/ Spyder
python IDE/ Google COLAB – Cloud based Jupyter Notebook Environment

Pre-requisite: Python Standard Libraries

Theory :
Clustering is a set of techniques used to partition data into groups, or clusters. Clusters are loosely
defined as groups of data objects that are more similar to other objects in their cluster than theyare to
data objects in other clusters. In practice, clustering helps identify two qualities of data:
Meaningfulness
Usefulness
Meaningful clusters expand domain knowledge. For example, in the medical field, researchers applied
clustering to gene expression experiments. The clustering results identified groups of patients who
respond differently to medical treatments.
Useful clusters, on the other hand, serve as an intermediate step in a data pipeline. For example,
businesses use clustering for customer segmentation. The clustering results segment customers into
groups with similar purchase histories, which businesses can then use to create targeted advertising
campaigns.
There are many other applications of clustering, such as document clustering and social network
analysis. These applications are relevant in nearly every industry, making clustering a valuable skill for
professionals working with data in any field.
Selecting an appropriate clustering algorithm for your dataset is often difficult due to the number of
choices available. Some important factors that affect this decision include the characteristics of the
clusters, the features of the dataset, the number of outliers, and the number of data objects.
Three popular categories of clustering algorithms:

Partitional clustering
Hierarchical clustering
Density-based clustering
The k-means clustering method is an unsupervised machine learning technique used to identify clusters
of data objects in a dataset. There are many different types of clustering methods, but k- means is one
of the oldest and most approachable. These traits make implementing k-means clustering in Python
reasonably straightforward, even for novice programmers and data scientists.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Conventional k-means requires only a few steps. The first step is to randomly select k centroids, where
k is equal to the number of clusters you choose. Centroids are data points representing the center of a
cluster.

The main element of the algorithm works by a two-step process called expectation-maximization. The
expectation step assigns each data point to its nearest centroid. Then, the maximization step computes
the mean of all the points for each cluster and sets the new centroid. Here’s what the conventional
version of the k-means algorithm looks like:

The quality of the cluster assignments is determined by computing the sum of the squared error (SSE)
after the centroids converge, or match the previous iteration’s assignment. The SSE is defined as the
sum of the squared Euclidean distances of each point to its closest centroid. Since this is a measure of
error, the objective of k-means is to try to minimize this value.

The figure below shows the centroids and SSE updating through the first five iterations from two

different runs of the k-means algorithm on the same dataset

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

The purpose of this figure is to show that the initialization of the centroids is an important step. It also
highlights the use of SSE as a measure of clustering performance. After choosing a number of clusters and
the initial centroids, the expectation-maximization step is repeated until the centroidpositions reach
convergence and are unchanged.

The random initialization step causes the k-means algorithm to be nondeterministic, meaning that
cluster assignments will vary if you run the same algorithm twice on the same dataset. Researchers
commonly run several initializations of the entire k-means algorithm and choose the cluster
assignments from the initialization with the lowest SSE.

# Implementation of K Means algorithm.

# importing required library.

41
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

# check scikit-learn version

# synthetic classification dataset

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

# K-menas clustering code with output

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Conclusion:

scikit-learn, an accessible and extensible tool for implementing k-means clustering in Python. it has
been learned that What the popular clustering techniques are and when to use them, What thek-means
algorithm is, How to implement k-means clustering in Python, How to evaluate the performance of
clustering algorithms, How to build and tune a robust k-means clustering pipeline in Python, How to
analyze and present clustering results from the k-means algorithm.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Experiment No. 8

Name of the Experiment: Using any machine learning techniques using available data
set to develop a recommendation system.

Aim:
Using any machine learning techniques using available data set to develop a recommendation system.

Requirements: Anaconda Distribution – Python Platform/ Jupyter Notebook/ PyCharm IDE/ Spyder
python IDE/ Google COLAB – Cloud based Jupyter Notebook Environment

Pre-requisite: Python Standard Libraries

Theory:

Recommender systems are computer programs that make recommendations to users based on a variety
of parameters. These systems forecast the most likely product that users will buy and that they will be
interested in. Netflix, Amazon, and other companies employ recommender systems toassist their users to
find the right product or movie for them.

The recommender system filters a large volume of data by focusing on the most significant information
based on the information provided by the user as well as other factors such as the user’s preferences and
interests. In order to offer recommendations, it determines the compatibilityof the user and the object, as
well as the similarities between users and products.

Recommender systems in use include playlist generators for video and music services, product
recommenders for online businesses, content recommenders for social media platforms, and open web
content recommenders. Within and across platforms, these systems can work with a single input, such
as music, or multiple inputs, such as news, books, and search queries.

It has the following benefits:

Users benefit from being able to find items that are of interest to [Link] item providers in getting

their products to the correct people.

Users will be able to identify products that are most relevant to [Link] that is tailored to the

individual.

Assist websites in increasing user engagement.

Approaches of Recommenders System

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Collaborative Filtering

Collaborative filtering is a popular method for the creation of recommender systems. Collaborative
filtering is predicated on the premise that people who agreed in the past will agree in the future, and
that they will prefer comparable types of goods in the past. The technology creates suggestions based
solely on rating profiles for various persons or things. They generate recommendations utilizing this
neighbourhood by seeking peer users/items with rating histories similar to the current user or item.

Content-Based Filtering

Another method that is widely utilized when building recommender systems is content-based filtering.
The description of an item and a profile of the user’s preferences are used in content-based filtering
systems. When there is known data about an item (name, location, description, etc.) but not on the user,
these strategies perform well. Content-based recommenders treat suggestions as a user-specific
classification issue, developing a classifier for a user’s likes and dislikes based on an item’s properties.

Session-Based Recommender System

The interactions of a user during a session are used to produce recommendations in these recommender
systems. Youtube and Amazon both utilize session-based recommender systems. When a user’s history
(such as previous clicks or transactions) is not available or relevant in the current user session, they are
particularly valuable. Video, e-commerce, travel, music, and other domains are all examples of when
session-based suggestions are useful. Most session-based recommender systems rely on the sequence
of recent interactions inside a session without requiring any further information about the user (history,
demographics)
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Multi-Criteria Recommender System

Multi-criteria recommender systems (MCRS) are recommender systems that take into account many
factors when making recommendations. Rather than developing recommendation techniques based ona
single criterion value, such as user u’s overall preference for an item I, these systems attempt to predict
a rating for unexplored items of u by leveraging preference information on multiple criteria thatinfluence
this overall preference value. Several researchers see MCRS as a multi-criteria decision- making
(MCDM) problem and construct MCRS systems using MCDM approaches and techniques.

Exploratory Data Analysis (EDA) on the data

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Build the Movie Recommonder System

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Get Recommendations for movie

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Results:-

Conclusion:

In this machine learning project, we build movie recommendation systems. We built a content-based
recommendation engine that makes recommendations given the title of the movie as input.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Experiment No. 9

Name of the Experiment: Exploratory data analysis using Apache Spark and Pandas.

Aim:
Exploratory data analysis using Apache Spark and Pandas

Requirements: Anaconda Distribution – Python Platform/ Jupyter Notebook/ PyCharm IDE/

Spyder python IDE/ Google COLAB – Cloud based Jupyter Notebook Environment

Pre-requisite: Python Standard Libraries

Theory :
EDA with spark means saying bye-bye to Pandas. Due to the large scale of data, every calculationmust
be parallelized, instead of Pandas, [Link] are the right tools you can use. It is, for sure,
struggling to change your old data-wrangling habit. I hope this post can give you a jump start to
perform EDA with Spark.

There are two kinds of variables, continuous and categorical. Each of them has different EDA
requirements:

Continuous variables EDA list:

missing values
statistic values: mean, min, max, stddev, quantilesbinning & distribution
correlation
Categorical variables EDA list:
missing values frequency table
Generation of charts on Databricks without any plot libraries like seaborn or matplotlib are also
shown.

Initialize pyspark framework and load data into pyspark's dataframe

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Overview of Dataset
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Detect missing values and abnormal zeroes

 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Conclusion:

PySpark is a Spark library written in Python to run Python application using ApacheSpark capabilities,
using PySpark we can run applications parallelly on the distributed cluster (multiple nodes).In other
words, PySpark is a Python API for Apache Spark. Apache Spark is an analytical processing engine for
large scale powerful distributed data processing and machine learning applications.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Experiment No. 10

Name of the Experiment: Batch and Streamed Data Analysis using Spark.

Aim:
Batch and Streamed Data Analysis using Spark.

Requirements: Anaconda Distribution – Python Platform/ Jupyter Notebook/ PyCharm IDE/ Spyder
python IDE/ Google COLAB – Cloud based Jupyter Notebook Environment

Pre-requisite: Python Standard Libraries

Theory :
Data streaming is a way of collecting data continuously in real-time from multiple data sources in the

form of data streams. Datastream can be thought of as a table that is continuously being appended.

Data streaming is essential for handling massive amounts of live data. Such data can be from a variety

of sources like online transactions, log files, sensors, in-game player activities, etc.

There are various real-time data streaming techniques like Apache Kafka, Spark Streaming,

Apache Flume etc.

Spark Streaming is an integral part of Spark core API to perform real-time data analytics. It allowsus
to build a scalable, high-throughput, and fault-tolerant streaming application of live data streams.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

Spark Streaming supports the processing of real-time data from various input sources and
storingthe processed data to various output sinks.

Spark Streaming has 3 major components as shown in the above image.

Input data sources: Streaming data sources (like Kafka, Flume, Kinesis, etc.), static data sources (like
MySQL, MongoDB, Cassandra, etc.), TCP sockets, Twitter, etc.

Spark Streaming engine: To process incoming data using various built-in functions, complex
algorithms. Also, we can query live streams, apply machine learning using Spark SQL and MLlib
respectively.

Output Sinks: processed data can be stored to file systems, databases(relational and NoSQL), live
dashboards etc.

Spark Streaming also allows receiving data streams using TCP sockets. So let’s write a simple streaming
program to receive text data streams on a particular port, perform basic text cleaning (like white space
removal, stop words removal, lemmatization, etc.), and print the cleaned text on the screen

1. Creating Streaming Context and Receiving data streams.

StreamingContext is the main entry point for any streaming application. It can be created by
instantiating StreamingContext class from pyspark. streaming module.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

While creating StreamingContext we can specify the batch duration, for e.g. here the batch duration is 3
seconds.

Once the StreamingContext is created, we can start receiving data in the form of DStream through
TCP protocol on a specific port. For e.g. here the hostname is specified as “localhost” and port
used is 8084.
 SARASWATI EDUCATION SOCIETY

YADAVRAO TASGAONKAR INSTITUTE OF ENGINEERING &TECHNOLOGY

DR.N.Y. TASGAONKAR TECHNICAL EDUCATIONAL COMPUS, CHANDHAI [Link]: -NASARPUR, TAL: -KARJAT, DIST: -RAIGAD

2. Performing operations on data streams

After creating a DStream object, we can perform operations on it as per the requirement. Here,
wewrote a custom text cleaning function.
This function first converts the input text into lower case, then removes extra spaces, non-
alphanumeric characters, links/URLs, stop words, and then further lemmatizes the text using the NLTK
library

3. Starting the Streaming service

The streaming service has not started yet. Use the start() function on top of the StreamingContext
object to start it and keep on receiving streaming data until the termination command (Ctrl + C or
Ctrl + Z) is not received by awaitTermination() function.

Now first we need to run the ‘nc’ command (Netcat Utility) to send the text data from the data server
to the spark streaming server. Netcat is a small utility available in Unix-like systems to read from and
write to network connections using TCP or UDP ports. Its two main options are –
-l: To allow nc to listen to an incoming connection rather than initiating a connection to a
remotehost.
-k: Forces nc to stay listening for another connection after its current connection is
[Link] run the following nc command in the terminal.

Similarly, run the pyspark script in a different terminal using the following
command in order to perform text cleaning on the received data

any text written in the terminal (running netcat server) will be cleaned and the
cleaned text is printed in another terminal after every 3 seconds (batch
duration).

Conclusion:

With help of python library Spark Streaming, its benefits in real-time data
streaming, and a sample application (using TCP sockets) to receive the live data
streams and process them as per the requirement has been studied successfully .

- 62 -