Unit-IV
Predictive Big Data Analytics with Python
Indira College of Engineering Management, Pune
Introduction-
• Predictive modelling is usually a pleasurable experience.
• Using past data, predictive analysis provides predictions about what might
happen in the future. The information is gathered in a base table that has three
key components: Population, Candidate Predictors and target.
• Population – The population is the set of persons or things for which you want to
make predictions.
• Candidate Predictors – Candidate Predictors are people in a population who,
given certain knowledge, could be used to forecast an event.
• Target - Target has information about the events to predict.
• Example -
2
Python Libraries -
Python is a multi-purpose programming language that is frequently used for data analysis
and data science.
What makes Python so handy for data manipulation?
• There are libraries that provides users with the capabilities they require when
processing data.
• There are key python libraries for working with data.
1. Numpy and Scipy
2. Pandas
3. Matplotlib
4. Scikit-learn
5. StatsModels
6. Seaborn
3
Python Libraries -
1. Pandas - Data Manipulation and Analysis
• Pandas is a programming language that may be used to do structured data
operations and manipulations. Its often used for data preprocessing and
munging.
• Pandas helped to increase Python’s popularity among data scientists.
4
Python Libraries -
2. NumPy and Scipy –
• NumPy – Numerical Python referred as NumPy. The n-diamentional array is
NumPy’s most powerful feature. Basic linear algebra operations, Fourier
transforms, additional random number capabilities, and tools for integration with
other low-level languages like Fortran, C, and C++ are also included in this
package.
• SciPy – Scientific Python is a Python programming language. It is based on the
NumPy programming language. SciPy is a handy library for wide range of high-
level science and engineering modules, including discrete Fourier transforms,
linear algebra, optimization, and sparse matrices.
5
Python Libraries -
3. Matplotlib – Plotting and Visualization
• Matplotlib is used to plot a wide range of graphs, from histogram to line plots to
heat plots.
4. Scikit-learn – Machine Learning and Data Mining
• For Machine Learning, use Scikit Learn.
• This library, which is based on NumPy, SciPy, and Matplotlib, includes a
number of useful tools for machine learning and statistical modelling, such as
Classification, Regression, Clustering and Dimensional reduction.
5. StatsModels – Statistical Modeling, Testing and Analysis.
6. Seaborn – For Statistical Data Visualization.
6
Data Processing -
Removing Duplicates: Dataset contains duplicates.
How to deal with duplicate values?
• There are two methods –
i. Remove the values from all duplicate rows – Using the drop duplicates function, you can
remove all rows with the same values()
ii. Remove the duplicates from the columns that have the most of them.
7
Data Preprocessing -
1. Removing Duplicates –
• Data points in a dataset that have the same values for all or part of the
characteristics are said to have duplicate values. Due to problems with data
input, data collecting, or other circumstances, duplicate values may appear.
• Pandas drop_duplicates() method helps in removing duplicates from the Pandas
Dataframe In Python.
8
Data Preprocessing -
1. Removing Duplicates –
• Example –
9
Data Preprocessing -
1. Removing Duplicates –
• Example –
10
Data Preprocessing -
2. Transformation of Data using Function or Mapping –
• Python’s Transform function returns a self-produced dataframe with
transformed values after applying the function specified in its
parameter. This dataframe has the same length as the passed
dataframe.
• Input->Split->Apply->Combine(Transform)
11
Data Preprocessing -
• Example –
• Multiply 10 to each element in a dataframe.
12
Data Preprocessing -
print([Link](func = lambda x : x*10))
//lambda – A lambda function can take any number of arguments,
but can only have one. (X= lambda a: a+10)
13
Data Preprocessing -
3. Replacing Values –
• [Link]() function is used to replace string, gender, list,
number etc.
• There are two ways –
1. Remove missing values
2. Filling missing values
14
Data Preprocessing -
3. Replacing Values –
i) Remove missing values –
• Example –
Employee ID Name Salary Employee ID Name Salary
101 Ram 50000 101 Ram 50000
102 Jay 20000 102 Jay 20000
103 Yash NaN 104 Riya 16000
104 Riya 16000
15
Data Preprocessing -
3. Replacing Values –
i) Remove missing values –
• Example –
Employee ID Name Salary Employee ID Name Salary
101 Ram 50000 101 Ram 50000
102 Jay 20000 102 Jay 20000
103 Yash NaN 103 Yash 28666.66
104 Riya 16000 104 Riya 16000
16
Analytics Types -
What is Data Analytics?
Data analytics is the process of manipulating data to extract useful
trends and hidden patterns which can help us derive valuable insights to
make business predictions.
• Types of Data Analytics -
1. Predictive (forecasting)
2. Descriptive (business intelligence and data mining)
3. Prescriptive (optimization and simulation)
4. Diagnostic analytics
17
Analytics Types -
18
Analytics Types -
1. Predictive Analytics -
• Predictive analytics turn the data into valuable, actionable information.
predictive analytics uses data to determine the probable outcome of an
event or a likelihood of a situation occurring. Predictive analytics holds a
variety of statistical techniques from modeling, machine learning, data
mining and game theory that analyze current and historical facts to make
predictions about a future event.
• Techniques that are used for predictive analytics are:
• Linear Regression
• Time Series Analysis and Forecasting
• Data Mining
19
Analytics Types -
2. Descriptive Analytics
• Descriptive analytics looks at data and analyze past event for insight as to how
to approach future events. It looks at past performance and understands the
performance by mining historical data to understand the cause of success or
failure in the past.
• Almost all management reporting such as sales, marketing, operations, and
finance uses this type of analysis.
• A predictive model that focuses on predicting the behavior of a single customer,
Descriptive analytics identifies many different relationships between customer
and product.
20
Analytics Types -
3. Prescriptive Analytics
• Prescriptive Analytics automatically synthesize big data, mathematical science,
business rule, and machine learning to make a prediction and then suggests a
decision option to take advantage of the prediction.
• Prescriptive Analytics not only anticipates what will happen and when to happen
but also why it will happen. Further, Prescriptive Analytics can suggest decision
options on how to take advantage of a future opportunity or mitigate a future
risk and illustrate the implication of each decision option.
21
Analytics Types -
4. Diagnostic Analytics
• In this analysis, we generally use historical data over other data to answer any
question or for the solution of any problem. We try to find any dependency and
pattern in the historical data of the particular problem.
• Common techniques used for Diagnostic Analytics are:
• Data discovery
• Data mining
• Correlations
22
Association Rules -
• Association rule learning is a type of unsupervised learning technique that
checks for the dependency of one data item on another data item and maps
accordingly so that it can be more profitable.
• It tries to find some interesting relations or associations among the variables of
dataset. It is based on different rules to discover the interesting relations between
variables in the database.
• The association rule learning is one of the very important concepts of machine
learning, and it is employed in Market Basket analysis, Web usage mining,
continuous production, etc.
23
Association Rules -
• For example, if a customer buys bread, he most likely can also buy butter, eggs,
or milk, so these products are stored within a shelf or mostly nearby. Consider
the below diagram:
24
Association Rules -
• Association rule learning can be divided into three types of algorithms:
1. Apriori
2. Eclat
3. F-P Growth Algorithm
25
Association Rules -
1. Apriori Algorithm –
• This algorithm uses frequent datasets to generate association rules. It is designed
to work on the databases that contain transactions. This algorithm uses a
breadth-first search and Hash Tree to calculate the itemset efficiently.
• It is mainly used for market basket analysis and helps to understand the products
that can be bought together. It can also be used in the healthcare field to find
drug reactions for patients.
2. Eclat Algorithm -
• Eclat algorithm stands for Equivalence Class Transformation. This algorithm
uses a depth-first search technique to find frequent itemsets in a transaction
database. It performs faster execution than Apriori Algorithm.
26
Association Rules -
3. F-P Growth Algorithm -
• The F-P growth algorithm stands for Frequent Pattern, and it is the improved
version of the Apriori Algorithm.
• It represents the database in the form of a tree structure that is known as a
frequent pattern or tree. The purpose of this frequent tree is to extract the most
frequent patterns.
27
Association Rules -
Applications of Association Rule Learning -
• It has various applications in machine learning and data mining. Below are some
popular applications of association rule learning:
1. Market Basket Analysis: It is one of the popular examples and applications of
association rule mining. This technique is commonly used by big retailers to
determine the association between items.
2. Medical Diagnosis: With the help of association rules, patients can be cured
easily, as it helps in identifying the probability of illness for a particular disease.
3. Protein Sequence: The association rules help in determining the synthesis of
artificial Proteins.
4. It is also used for the Catalog Design and Loss-leader Analysis and many more
other applications.
28
Regression -
• Regression analysis is a statistical method to model the relationship between a
dependent (target) and independent (predictor) variables with one or more
independent variables.
• Regression analysis helps us to understand how the value of the dependent
variable is changing corresponding to an independent variable when other
independent variables are held fixed.
• It predicts continuous/real values such as temperature, age, salary, price, etc.
• Regression is a supervised learning technique which helps in finding the
correlation between variables and enables us to predict the continuous output
variable based on the one or more predictor variables.
• It is mainly used for prediction, forecasting, time series modeling, and
determining the causal-effect relationship between variables.
29
Regression -
• In Regression, we plot a graph between the variables which best fits the given
datapoints, using this plot, the machine learning model can make predictions
about the data.
• "Regression shows a line or curve that passes through all the datapoints on
target-predictor graph in such a way that the vertical distance between the
datapoints and the regression line is minimum." The distance between
datapoints and line tells whether a model has captured a strong relationship or
not.
• Some examples of regression can be as:
1. Prediction of rain using temperature and other factors
2. Determining Market trends
3. Prediction of road accidents due to rash driving.
30
Regression -
• Types of Regression -
31
Regression -
1. Linear Regression –
• Linear regression is a statistical regression method which is used for predictive analysis.
• It is one of the very simple and easy algorithms which works on regression and shows
the relationship between the continuous variables.
• It is used for solving the regression problem in machine learning.
• Linear regression shows the linear relationship between the independent variable (X-
axis) and the dependent variable (Y-axis), hence called linear regression.
• If there is only one input variable (x), then such linear regression is called simple linear
regression. And if there is more than one input variable, then such linear regression is
called multiple linear regression.
• The relationship between variables in the linear regression model can be explained
using the below image. Here we are predicting the salary of an employee on the basis
of the year of experience.
32
Regression -
1. Linear Regression –
33
Regression -
1. Linear Regression –
Mathematical equation for Linear regression:
Y= aX+b
Y = dependent variables (target variables),
X= Independent variables (predictor variables),
a and b are the linear coefficients
Some popular applications of linear regression are:
• Analyzing trends and sales estimates
• Salary forecasting
• Real estate prediction
• Arriving at ETAs in traffic.
34
Regression -
2. Logistic Regression –
• Logistic regression is another supervised learning algorithm which is used to
solve the classification problems. In classification problems, we have
dependent variables in a binary or discrete format such as 0 or 1.
• Logistic regression algorithm works with the categorical variable such as 0 or 1,
Yes or No, True or False, Spam or not spam, etc.
• It is a predictive analysis algorithm which works on the concept of probability.
• Logistic regression is a type of regression, but it is different from the linear
regression algorithm in the term how they are used.
35
Regression -
2. Logistic Regression –
• Logistic regression uses sigmoid function or logistic function which is a complex cost
function. This sigmoid function is used to model the data in logistic regression. The
function can be represented as:
• f(x)= Output between the 0 and 1 value.
• x= input to the function
• e= base of natural logarithm.
36
Regression -
• 2. Logistic Regression –
• It uses the concept of threshold levels, values above the threshold level are
rounded up to 1, and values below the threshold level are rounded up to 0.
• There are three types of logistic regression:
• Binary(0/1, pass/fail)
• Multi(cats, dogs, lions)
• Ordinal(low, medium, high)
37
Classification -
• The Classification algorithm is a Supervised Learning technique that is used to identify
the category of new observations on the basis of training data.
• In Classification, a program learns from the given dataset or observations and then
classifies new observation into a number of classes or groups. Such as, Yes or No, 0 or
1, Spam or Not Spam, cat or dog, etc. Classes can be called as targets/labels or
categories.
• Unlike regression, the output variable of Classification is a category, not a value, such
as "Green or Blue", "fruit or animal", etc.
• Since the Classification algorithm is a Supervised learning technique, hence it takes
labeled input data, which means it contains input with the corresponding output.
• The main goal of the Classification algorithm is to identify the category of a given
dataset, and these algorithms are mainly used to predict the output for the categorical
data.
38
Classification -
• Classification algorithms can be better understood using the below diagram. In
the below diagram, there are two classes, class A and Class B. These classes
have features that are similar to each other and dissimilar to other classes.
39
Classification -
Types of ML Classification Algorithms:
• Classification Algorithms can be further divided into the Mainly two
category:
1. Linear Models
• Logistic Regression
• Support Vector Machines
2. Non-linear Models
• K-Nearest
• Kernel SVM
• Naïve Bayes
• Decision Tree Classification
• Random Forest Classification
40
Classification -
1. Naïve Bayes –
• Naïve Bayes algorithm is a supervised learning algorithm, which is based
on Bayes theorem and used for solving classification problems.
• It is mainly used in text classification that includes a high-dimensional training
dataset.
• Naïve Bayes Classifier is one of the simple and most effective Classification
algorithms which helps in building the fast machine learning models that can
make quick predictions.
• It is a probabilistic classifier, which means it predicts on the basis of the
probability of an object.
• Some popular examples of Naïve Bayes Algorithm are spam filtration,
Sentimental analysis, and classifying articles.
41
Classification -
1. Naïve Bayes –
The Naïve Bayes algorithm is comprised of two words Naïve and Bayes, Which can be
described as:
• Naïve: It is called Naïve because it assumes that the occurrence of a certain feature is
independent of the occurrence of other features. Such as if the fruit is identified on the
bases of color, shape, and taste, then red, spherical, and sweet fruit is recognized as an
apple. Hence each feature individually contributes to identify that it is an apple without
depending on each other.
• Bayes: It is called Bayes because it depends on the principle of Bayes' Theorem.
Bayes' Theorem:
• Bayes' theorem is also known as Bayes' Rule or Bayes' law, which is used to determine
the probability of a hypothesis with prior knowledge. It depends on the conditional
probability.
• The formula for Bayes' theorem is given as:
42
Classification -
1. Naïve Bayes –
Example – Fruit = {Yellow, Sweet, Long}
Fruit Yellow Sweet Long Total
Orange 350 450 0 650
Banana 400 300 350 400
Others 50 100 50 150
Total 800 850 400 1200
• Baye’s Theorem -
43
Classification -
2. Decision Tree –
• Decision Tree is a Supervised learning technique that can be used for both
classification and Regression problems, but mostly it is preferred for solving
Classification problems.
• It is a tree-structured classifier, where internal nodes represent the features of
a dataset, branches represent the decision rules and each leaf node
represents the outcome.
• In a Decision tree, there are two nodes, which are the Decision Node and Leaf
Node. Decision nodes are used to make any decision and have multiple
branches, whereas Leaf nodes are the output of those decisions and do not
contain any further branches.
• The decisions or the test are performed on the basis of features of the given
dataset.
44
Classification -
2. Decision Tree –
• It is a graphical representation for getting all the possible solutions to a
problem/decision based on given conditions.
• It is called a decision tree because, similar to a tree, it starts with the root node,
which expands on further branches and constructs a tree-like structure.
• In order to build a tree, we use the CART algorithm, which stands
for Classification and Regression Tree algorithm.
• A decision tree simply asks a question, and based on the answer (Yes/No), it
further split the tree into subtrees.
• A decision tree can contain categorical data (YES/NO) as well as numeric
data.
45
Classification -
2. Decision Tree –
46
Classification -
2. Decision Tree –
• Example: Suppose there is a candidate who has a job offer and wants to decide
whether he should accept the offer or Not. So, to solve this problem, the decision
tree starts with the root node (Salary attribute by ASM). The root node splits
further into the next decision node (distance from the office) and one leaf node
based on the corresponding labels. The next decision node further gets split into
one decision node (Cab facility) and one leaf node. Finally, the decision node
splits into two leaf nodes (Accepted offers and Declined offer).
47
Classification -
2. Decision Tree –
48
Introduction to Scikit-Learn-
• Installation
• Dataset
• Matplotlib
• Missing Values
• Regression & Classification using Scikit-learn
49