Big Data Analytics - Statistical

Methods
• When analyzing data, it is possible to have a
statistical approach. The basic tools that are
needed to perform basic analysis are −
– Correlation analysis
– Analysis of Variance
– Hypothesis Testing
 Advanced Methods
• Machine Learning for Data Analysis
• Naive Bayes Classifier
• K-Means Clustering
• Association Rules
• Big Data Analytics - Decision Trees
• Logistic Regression
• Big Data Analytics - Time Series
• Big Data Analytics - Text Analytics
• Machine learning can be divided in two types
of task −
– Supervised Learning
– Unsupervised Learning
 Naive Bayes Classifier
• Naive Bayes is a probabilistic technique for
constructing classifiers.
• The characteristic assumption of the naive
Bayes classifier is to consider that the value of
a particular feature is independent of the
value of any other feature, given the class
variable.
 K-Means Clustering
• k-means clustering aims to partition n observations into
k clusters in which each observation belongs to the
cluster with the nearest mean, serving as a prototype of
the cluster.
• Given a set of observations (x1, x2, …, xn), where each
observation is a d-dimensional real vector, k-means
clustering aims to partition the n observations into k
groups G = {G1, G2, …, Gk} so as to minimize the within-
cluster sum of squares (WCSS) defined as follows −
•
 Association Rules
• Let I = i1, i2, ..., in be a set of n binary attributes called
items. Let D = t1, t2, ..., tm be a set of transactions
called the database.
• Each transaction in D has a unique transaction ID and
contains a subset of the items in I.
• A rule is defined as an implication of the form X ⇒ Y
where X, Y ⊆ I and X ∩ Y = ∅.
• The sets of items (for short item-sets) X and Y are
called antecedent (left-hand-side or LHS) and
consequent (right-hand-side or RHS) of the rule.
Example
• An example rule for the supermarket could
be {milk, bread} ⇒ {butter} meaning that if
milk and bread is bought, customers also
buy butter. To select interesting rules from
the set of all possible rules, constraints on
various measures of significance and interest
can be used. The best-known constraints are
minimum thresholds on support and
confidence.
• The support supp(X) of an item-set X is
defined as the proportion of transactions in
the data set which contain the item-set. In the
example database in Table 1, the item-set
{milk, bread} has a support of 2/5 = 0.4 since it
occurs in 40% of all transactions (2 out of 5
transactions). Finding frequent item-sets can
be seen as a simplification of the
unsupervised learning problem.
• The confidence of a rule is defined conf(X ⇒ Y ) =
supp(X ∪ Y )/supp(X). For example, the rule {milk,
bread} ⇒ {butter} has a confidence of 0.2/0.4 =
0.5 in the database in Table 1, which means that
for 50% of the transactions containing milk and
bread the rule is correct. Confidence can be
interpreted as an estimate of the probability
P(Y|X), the probability of finding the RHS of the
rule in transactions under the condition that
these transactions also contain the LHS.
 Decision Trees
• A Decision Tree is an algorithm used for supervised learning
problems such as classification or regression.
• A decision tree or a classification tree is a tree in which each
internal (nonleaf) node is labeled with an input feature.
• The arcs coming from a node labeled with a feature are labeled
with each of the possible values of the feature.
• Each leaf of the tree is labeled with a class or a probability
distribution over the classes.
• A tree can be "learned" by splitting the source set into subsets
based on an attribute value test.
• This process is repeated on each derived subset in a recursive
manner called recursive partitioning.
• Decision trees used in data mining are of two
main types −
– Classification tree − when the response is a
nominal variable, for example if an email is spam
or not.
– Regression tree − when the predicted outcome
can be considered a real number (e.g. the salary
of a worker).
 Logistic Regression
• Logistic regression is a classification model in
which the response variable is categorical.
• It is an algorithm that comes from statistics
and is used for supervised classification
problems.
 Time Series Analysis
• Time series is a sequence of observations of
categorical or numeric variables indexed by a
date, or timestamp.
 Text Analytics
• Text analytics is the automated process of
translating large volumes of unstructured text
into quantitative data to uncover insights,
trends, and patterns.
• Combined with data visualization tools, this
technique enables companies to understand
the story behind the numbers and make
better decisions.
Data Science and Big Data Analytics
Chap 5: Adv Analytical Theory and
Methods: Association Rules
 Chapter Sections

• 5.1 Overview
• 5.2 Apriori Algorithm
• 5.3 Evaluation of Candidate Rules
• 5.4 Example: Transactions in a Grocery Store
• 5.5 Validation and Testing
• 5.6 Diagnostics
 5.1 Overview

• Association rules method

– Unsupervised learning method
– Descriptive (not predictive) method
– Used to find hidden relationships in data
– The relationships are represented as rules
• Questions association rules might answer
– Which products tend to be purchased together
– What products do similar customers tend to buy
 5.1 Overview

• Example – general logic of association rules

 5.1 Overview

• Rules have the form X -> Y

– When X is observed, Y is also observed
• Itemset
– Collection of items or entities
– k-itemset = {item 1, item 2,…,item k}
– Examples
• Items purchased in one transaction
• Set of hyperlinks clicked by a user in one session
 5.1 Overview – Apriori Algorithm

• Apriori is the most fundamental algorithm

• Given itemset L, support of L is the percent of
transactions that contain L
• Frequent itemset – items appear together “often
enough”
– Minimum support defines “often enough” (% transactions)
• If an itemset is frequent, then any subset is frequent
 5.1 Overview – Apriori Algorithm

• If {B,C,D} frequent, then all subsets frequent

 5.2 Apriori Algorithm
Frequent = minimum support
• Bottom-up iterative algorithm
• Identify the frequent (min support) 1-itemsets
• Frequent 1-itemsets are paired into 2-itemsets, and
the frequent 2-itemsets are identified, etc.
5.3 Evaluation of Candidate Rules
Confidence
• Frequent itemsets can form candidate rules
• Confidence measures the certainty of a rule

• Minimum confidence – predefined threshold

• Problem with confidence
– Given a rule X->Y, confidence considers only the
antecedent (X) and the co-occurrence of X and Y
– Cannot tell if a rule contains true implication
5.3 Evaluation of Candidate Rules
Lift
• Lift measures how much more often X and Y occur
together than expected if statistically independent

– Lift = 1 if X and Y are statistically independent

– Lift > 1 indicates the degree of usefulness of the rule
– Example – in 1000 transactions,
• If {milk, eggs} appears in 300, {milk} in 500, and {eggs} in 400, then
Lift(milk->eggs) = 0.3/(0.5*0.4) = 1.5
• If {milk, bread} appears in 400, {milk} in 500, and {bread} in 400,
then Lift(milk->bread) = 0.4/(0.5*0.4) = 2.0
5.3 Evaluation of Candidate Rules
Leverage
• Leverage measures the difference in the probability
of X and Y appearing together compared to statistical
independence

– Leverage = 0 if X and Y are statistically independent

– Leverage > 0 indicates degree of usefulness of rule
– Example – in 1000 transactions,
• If {milk, eggs} appears in 300, {milk} in 500, and {eggs} in 400, then
Leverage(milk->eggs) = 0.3 - 0.5*0.4 = 0.1
• If {milk, bread} appears in 400, {milk} in 500, and {bread} in 400,
then Leverage (milk->bread) = 0.4 - 0.5*0.4 = 0.2
 5.4 Applications of Association Rules
• The term market basket analysis refers to a specific implementation of
association rules mining that many companies use for a variety of
purposes, including these:
• • Broad-scale approaches to better merchandising- what products should
be included in or excluded from the inventory each month
• • Cross-merchandising between products and high-margin or high-ticket
items
• • Physical or logical placement of product within related categories of
products
• • Promotional programs-multiple product purchase incentives managed
through a loyalty card program
• Besides market basket analysis, association rules are commonly used for
recommender systems and clickstream analysis
• Many online service providers such as Amazon and Netflix use
recommender systems. Recommender systems can use
association rules to discover related products or identify
customers who have similar nterests.
• For example, association rules may suggest that those
customers who have bought product A have also bought product
B, or those customers who have bought products A, B, and Care
more similar to this customer. These findings provide
opportunities for re tailers to cross-sell their products.
• Clickstream analysis refers to the analytics on data related to web
browsing and user clicks, which is stored on the client or the
server side. Web usage log files generated on web servers contain
huge amounts of information, and association rules can
potentially give useful knowledge to web usage data analysts.
• For example, association rules may suggest that website visitors
who land on page X click on links A, B, and C much more often
than links 0, E, and F. This observation provides valuable insight
on how to better personalize and recommend the content to site
visitors.
 5.6 Validation and Testing

• The frequent and high confidence itemsets are found by pre-

specified minimum support and minimum confidence levels
• Measures like lift and/or leverage then ensure that
interesting rules are identified rather than coincidental ones
• However, some of the remaining rules may be considered
subjectively uninteresting because they don’t yield
unexpected profitable actions
– E.g., rules like {paper} -> {pencil} are not interesting/meaningful
• Incorporating subjective knowledge requires domain experts
• Good rules provide valuable insights for institutions to
improve their business operations
Data Science and Big Data Analytics
Chap 6: Adv Analytical Theory and
Methods: Regression
 Chapter Sections

• 6.1 Linear Regression

• 6.2 Logical Regression
• 6.3 Reasons to Choose and Cautions
• 6.4 Additional Regression Models
• Summary
 6 Regression

• Regression analysis attempts to explain the influence

that input (independent) variables have on the outcome
(dependent) variable
• Questions regression might answer
– What is a person’s expected income?
– What is probability an applicant will default on a loan?
• Regression can find the input variables having the
greatest statistical influence on the outcome
– Then, can try to produce better values of input variables
– E.g. – if 10-year-old reading level predicts students’ later
success, then try to improve early age reading levels
 6.1 Linear Regression

• Models the relationship between several input

variables and a continuous outcome variable
– Assumption is that the relationship is linear
– Various transformations can be used to achieve a
linear relationship
• Linear regression models are probabilistic
– Involves randomness and uncertainty
– Not deterministic like Ohm’s Law (V=IR)
 6.1.1 Use Cases

• Real estate example

– Predict residential home prices
• Possible inputs – living area, #bathrooms, #bedrooms, lot
size, property taxes
• Demand forecasting example
– Restaurant predicts quantity of food needed
• Possible inputs – weather, day of week, etc.
• Medical example
– Analyze effect of proposed radiation treatment
• Possible inputs – radiation treatment duration, freq
>

6.2 Logistic Regression

Introduction

• In linear regression modeling, the outcome variable

is continuous – e.g., income ~ age and education
• In logistic regression, the outcome variable is
categorical, and this chapter focuses on two-valued
outcomes like true/false, pass/fail, or yes/no
>

6.2.1 Logistic Regression

Use Cases

• Medical
– Probability of a patient’s successful response to a specific
medical treatment – input could include age, weight, etc.
• Finance
– Probability an applicant defaults on a loan
• Marketing
– Probability a wireless customer switches carriers (churns)
• Engineering
– Probability a mechanical part malfunctions or fails
>

6.2.2 Logistic Regression

Model Description
• Logical regression is based on the logistic function

– As y -> infinity, f(y)->1; and as y->-infinity, f(y)->0

>

6.3 Reasons to Choose and Cautions

• Linear regression – outcome variable continuous

• Logistic regression – outcome variable categorical
• Both models assume a linear additive function of
the inputs variables
– If this is not true, the models perform poorly
– In linear regression, the further assumption of normally
distributed error terms is important for many statistical
inferences
• Although a set of input variables may be a good
predictor of an output variable, “correlation does
not imply causation”
ADVANCED ANALYTICAL THEORY
AND METHODS: TEXT ANALYSIS
• A text analysis problem usually consists
of three important steps:
Parsing
search and retrieval
Text mining.
• Parsing is the process that takes unstructured text
and imposes a structure for further analysis.

• The unstructured text could be a plain text file, a

weblog, an Extensible Markup Language (XML) fi le, a
HyperText Markup Language (HTML) fi le, or a Word
document.

• Parsing deconstructs the provided text and renders it

in a more structured way for the subsequent steps.
• Search and retrieval is the identification of the
documents in a corpus that contain search items
such as specific words, phrases, topics, or entities like
people or organizations.

• These search items are generally called key terms.

• Search and retrieval originated from the field of

library science and is now used extensively by web
search engines.
• Text mining uses the terms and indexes produced by
the prior two steps to discover meaningful insights
pertaining to domains or problems of interest.

• With the proper representation of the text, many of

the Techniques such as clustering and classifica tion,
can be adapted to text mining.
 Part-of-Speech (POS) Tagging, Lemmatization and
Stemming
• The goal of POS tagging is to bu ild a model whose input is
a sentence, such as
he saw a fox
• whose output is a tag sequence. Each tag marks the POS for
the corresponding word, such as
PRPVBD DT NN
• four words are mapped to pronoun(personal), verb (past
tense), determiner and noun (singular) respectively
• lemmatization finds the correct dictionary base
form of a word.
• For example, given the sentence:
obesity causes many problems
• the output of lemmatization wou ld be:
obesity cause many problem
• stemming does not need a dictionary and it usually
refers to a crude process of stripping affixes based on a
set of heuristics with the hope of correctly achieving the
goal to reduce inflections or variant forms.
• words are stripped to become stems.
• A stem is not necessarily an actual word defined in the
natural language, but it is sufficient to differentiate itself
from the stems of other words
• A well-known rule-based stemming algorithm is Porter's
stemming algorithm. It defines a set of production rules
to iteratively transform words into their stems
• For the sentence shown previously:
obesity causes many problems
• the output of Porter's stemming algorithm is:
obes caus mani problem
A Text Analysis Example
ADVANCED ANALYTICAL THEORY
AND METHODS: TIME SERIES
ANALYSIS
• A time series can consist of the following
components:
• Trend
• Seasonality
• Cyclic
• Random
• The trend refers to the long-term movement in a
time series. It indicates whether the observation
values are increasing or decreasing over time.

Examples of trends are a

• Steady increase in sales month over month
• An annual decline of fatalities due to car
accidents.
• The seasonality component describes the fixed,
periodic fluctuation in the observations over
time.

• As the name suggests, the seasonality

component is often related to the calendar.

• For example, monthly retail sales can fluctuate

over the year due to the weather and holidays.
• A cyclic component also refers to a periodic
fluctuation, but one that is not as fixed as in
the case of a seasonality component.

• For example, retails sales are influenced by

the general state of the economy.

• Thus, a retail sales time series can often follow

the lengthy boom-bust cycles of the economy.
• random component - noise is certainly part of
this random component, there is often some
underlying structure to this random
component that needs to be modeled to
forecast future values of a given time series.
 Box-Jenkins methodology for time series analysis

– Involves the following three main steps:

 Condition data and select a model.
 Identify and account for any trends or seasonality in the
time series.
 Examine the remaining time series and determine a
suitable model.
 Estimate the model parameters.
 Assess the model and return to Step 1, if necessary.
Data Science and Big Data Analytics
Chap 7: Adv Analytical Theory and
Methods: Classification
 Chapter Sections

• 7.1 Decision Trees

• 7.2 Naïve Bayes
• 7.3 Diagnostics of Classifiers
 7 Classification

• Classification is widely used for prediction

• Most classification methods are supervised
• This chapter focuses on two fundamental
classification methods
– Decision trees
– Naïve Bayes
 7.1 Decision Trees

• Tree structure specifies sequence of decisions

– Given input X={x1, x2,…, xn}, predict output Y
• Input attributes/features can be categorical or continuous
– Node = tests a particular input variable
• Root node, internal nodes, leaf nodes return class labels
• Depth of node = minimum steps to reach node
– Branch (connects two nodes) = specifies decision
– Two varieties of decision trees
• Classification trees: categorical output, often binary
• Regression trees: numeric output
 7.1 Decision Trees
7.1.1 Overview of a Decision Tree

• Example of a decision tree

– Predicts whether customers will buy a product
• Decision trees are widely used in practice.
• For example, to classify animals, questions (like
cold-blooded or warm -blooded, mammal or not
mammal).
• Another example is a checklist of symptoms during
a doctor's evaluation of a patient.
• The artificial intelligence engine of a video game
commonly uses decision trees to control the
autonomous actions of a character in response to
various scenarios.
• Retailers can use decision trees to segment
customers or predict response rates to
marketing and promotions.
• Financial institutions can use decision trees
to help decide if a loan application should
be approved or denied.
• By limiting the number of splits, a short tree
can be created. Short trees are often used as
components (also called weak learners or base
learners) in ensemble methods.
• The simplest short tree is called a decision
stump, which is a decision tree with the root
immediately connected to the leaf nodes.
 7.1 Decision Trees
7.1.1 Overview of a Decision Tree
– Example: will bank client subscribe to term deposit?
 7.1 Decision Trees
7.1.2 The General Algorithm

• Construct a tree T from training set S

• Requires a measure of attribute information
– Simplistic method (data from previous Fig.)
• Purity = probability of corresponding class
• E.g., P(no)=1789/2000=89.45%, P(yes)=10.55%
– Entropy methods
• Entropy measures the impurity of an attribute
• Information gain measures purity of an attribute
• subsets of s recursively until one of the following
criteria is met:
All the leaf nodes in the tree satisfy the
minimum purity threshold.
The tree cannot be further split with the preset
minimum purity threshold.
Any other stopping criterion is satisfied (such as
the maximum depth of the tree).
 7.1 Decision Trees
7.1.4 Evaluating a Decision Tree

• Decision trees -> rectangular decision regions

 7.1 Decision Trees
7.1.4 Evaluating a Decision Tree

• Advantages of decision trees

– Computationally inexpensive
– Outputs are easy to interpret – sequence of tests
– Show importance of each input variable
– Decision trees handle
• Both numerical and categorical attributes
• Categorical attributes with many distinct values
• Variables with nonlinear effect on outcome
• Variable interactions
 7.1 Decision Trees
7.1.4 Evaluating a Decision Tree

• Disadvantages of decision trees

– Sensitive to small variations in the training data
– Overfitting can occur because each split reduces
training data for subsequent splits
– Poor if dataset contains many irrelevant variables
 7.2 Naïve Bayes

• The naïve Bayes classifier

– Based on Bayes’ theorem (or Bayes’ Law)
– Assumes the features contribute independently
– Features (variables) are generally categorical
• Discretization of continuous variables is the process of
converting continuous variables into categorical ones
– Output is usually class label plus probability score
• Log probability often used instead of probability
 7.2 Naïve Bayes
7.2.1 Bayes Theorem

• Bayes’ Theorem

where C = class, A = observed attributes

 7.2 Naïve Bayes
7.2.2 Naïve Bayes Classifier

• Example: client subscribes to term deposit?

• The following record is from a bank client. Is this
client likely to subscribe to the term deposit?
 7.2 Naïve Bayes
7.2.2 Naïve Bayes Classifier
– Compute probabilities for this record
 7.2 Naïve Bayes
7.2.2 Naïve Bayes Classifier
– Compute Naïve Bayes classifier outputs: yes/no

– The client is assigned the label subscribed = yes

– The scores are small, but the ratio is what counts
– Using logarithms helps avoid numerical underflow
 7.2 Naïve Bayes
7.2.4 Diagnostics
• Naïve Bayes advantages
– Handles missing values
– Robust to irrelevant variables
– Simple to implement
– Computationally efficient
– Handles high-dimensional data efficiently
– Often competitive with other learning algorithms
– Reasonably resistant to overfitting
• Naïve Bayes disadvantages
– Assumes variables are conditionally independent
• Therefore, sensitive to double counting correlated variables
– In its simplest form, used only for categorical variables
 7.3 Diagnostics of Classifiers

• The book covered three classifiers

– Logistic regression, decision trees, naïve Bayes
• Tools to evaluate classifier performance
– Confusion matrix
 7.3 Diagnostics of Classifiers

• Bank marketing example

– Training set of 2000 records
– Test set of 100 records, evaluated below
 7.3 Diagnostics of Classifiers

– Evaluation metrics
 7.3 Diagnostics of Classifiers

– Evaluation metrics on bank marketing 100 test set