Data Analytics

UNIT-4
Regression vs. Segmentation

 Regression analysis focuses on finding a relationship between a dependent variable and

one or more independent variables.
 Predicts the value of a dependent variable based on the value of at least one independent
variable.
 Explains the impact of changes in an independent variable on the dependent variable.

 We use linear or logistic regression technique for developing accurate models for
predicting an outcome of interest.
 Often, we create separate models for separate segments.
 Segmentation methods such as CHAID or CRT is used to judge their effectiveness
 Creating separate model for separate segments may be time consuming and not worth the
effort.
 But, creating separate model for separate segments may provide higher predictive power.

 Market Segmentation
 Dividing the target market or customers on the basis of some significant features which
could help a company sell more products in less marketing expenses.

 Companies have limited marketing budgets. Yet, the marketing team is expected to
makes large number of sales to ensure rising revenue & profits.
 A product is created in two ways:
 Create a product after analyzing (research) the needs and wants of target market –
For example: Computer. Companies like Dell, IBM, Microsoft entered this
market after analyzing the enormous market which this product upholds.
 Create a product which evokes the needs & wants in target market – For example:
iPhone.
 Once the product is created, the ball shifts to the marketing team’s court.
 As mentioned above, they make use of market segmentation techniques.
 This ensures the product is positioned to the right segment of customers with high
propensity to buy.

 How to create segments for model development?

 Commonly adopted methodology
 Let us consider an example.
 Here we’ll build a logistic regression model for predicting likelihood of a customer to
respond to an offer.

NMREC, Hyderabad
Data Analytics

 A very similar approach can also be used for developing a linear regression model.

NMREC, Hyderabad
Data Analytics

 Logistic regression uses 1 or 0 indicator in the historical campaign data, which indicates
whether the customer has responded to the offer or not.
 Usually, one uses the target (or ‘Y’ known as dependent variable) that has been identified
for model development to undertake an objective segmentation.
 Remember, a separate model will be built for each segment.
 A segmentation scheme which provides the maximum difference between the segments
with regards to the objective is usually selected.
 Below is a simple example of this approach.

 Fig: Sample segmentation for building a logistic regression – commonly adopted

methodology
 The above segmentation scheme is the best possible objective segmentation developed,
because the segments demonstrate the maximum separation with regards to the objectives
(i.e. response rate).

NMREC, Hyderabad
Data Analytics

Supervised and Unsupervised

Learning

There are two broad set of methodologies for segmentation:

 Objective (supervised) segmentation

 Non-Objective (unsupervised) segmentation

Objective Segmentation
 Segmentation to identify the type of customers who would respond to a particular offer.
 Segmentation to identify high spenders among customers who will use the e-commerce
channel for festive shopping.
 Segmentation to identify customers who will default on their credit obligation for a loan
or credit card.

Non-Objective Segmentation
 Segmentation of the customer base to understand the specific profiles which exist within
the customer base so that multiple marketing actions can be personalized for each
segment
 Segmentation of geographies on the basis of affluence and lifestyle of people living in
each geography so that sales and distribution strategies can be formulated accordingly.
 Segmentation of web site visitors on the basis of browsing behavior to understand the
level of engagement and affinity towards the brand.
 Hence, it is critical that the segments created on the basis of an objective segmentation
methodology must be different with respect to the stated objective (e.g. response to an
offer).
 However, in case of a non-objective methodology, the segments are different with respect
to the “generic profile” of observations belonging to each segment, but not with regards
to any specific outcome of interest.
 The most common techniques for building non-objective segmentation are cluster
analysis, K nearest neighbor techniques etc.
 Each of these techniques uses a distance measure (e.g. Euclidian distance, Manhattan
distance, Mahalanobis distance etc.)
 This is done to maximize the distance between the two segments.
 This implies maximum difference between the segments with regards to a combination of
all the variables (or factors).

NMREC, Hyderabad
Data Analytics

Tree Building

 Decision tree learning

o is a method commonly used in data mining.
o is the construction of a decision tree from class-labeled training tuples.

 goal
o to create a model that predicts the value of a target variable based on several input
variables.

 Decision trees used in data mining are of two main types.

o Classification tree analysis
o Regression tree analysis

 Classification tree analysis is when the predicted outcome is the class to which the
data belongs.
 Regression tree analysis is when the predicted outcome can be considered a real
number. (e.g. the price of a house, or a patient’s length of stay in a hospital).

 A decision tree
o is a flow-chart-like structure
o each internal (non-leaf) node denotes a test on an attribute
o each branch represents the outcome of a test,
o each leaf (or terminal) node holds a class label.
o The topmost node in a tree is the root node.

 Decision-tree algorithms:
o ID3 (Iterative Dichotomiser 3)
o C4.5 (successor of ID3)
o CART (Classification and Regression Tree)
o CHAID (CHI-squared Automatic Interaction Detector). Performs multi-level
splits when computing classification trees.
o MARS: extends decision trees to handle numerical data better. Conditional
Inference Trees.

 Statistics-based approach that uses non-parametric tests as splitting criteria, corrected for
multiple testing to avoid over fitting.
 This approach results in unbiased predictor selection and does not require pruning.
 ID3 and CART follow a similar approach for learning decision tree from training tuples.

NMREC, Hyderabad
Data Analytics

CHAID (CHI-squared Automatic Interaction Detector)

 A simple method for fitting trees to predict a quantitative variable proposed by Morgan
and Sonquist (1963).
 They called the method AID, for Automatic Interaction Detection.
 The algorithm performs stepwise splitting.
 It begins with a single cluster of cases and searches a candidate set of predictor variables
for a way to split this cluster into two clusters.
 Each predictor is tested for splitting as follows:
o Sort all the n cases on the predictor and examine all n-1 ways to split the cluster in
two.
o For each possible split, compute the within-cluster sum of squares about the mean
of the cluster on the dependent variable.
o Choose the best of the n-1 splits to represent the predictor’s contribution. Now do
this for every other predictor.
o For the actual split, choose the predictor and its cut point which yields the
smallest overall within-cluster sum of squares.
o Categorical predictors require a different approach. Since categories are
unordered, all possible splits between categories must be considered.
o For deciding on one split of k categories into two groups, this means that 2k-1
possible splits must be considered.
o Once a split is found, its suitability is measured on the same within-cluster sum of
squares as for a quantitative predictor.
 Morgan and Sonquist called their algorithm AID because it naturally incorporates
interaction among predictors. Interaction is not correlation.
 It has to do instead with conditional discrepancies.
 In the analysis of variance, interaction means that a trend within one level of a variable is
not parallel to a trend within another level of the same variable.

 In the ANOVA model, interaction is represented by cross-products between predictors.

 In the tree model, it is represented by branches from the same nodes which have different
splitting predictors further down the tree.

NMREC, Hyderabad
Data Analytics

 Regression trees parallel regression/ANOVA modeling in which the dependent variable

is quantitative.
 Classification trees parallel discriminant analysis and algebraic classification methods.
 Kass (1980) proposed a modification to AID called CHAID for categorized dependent
and independent variables.
 His algorithm incorporated a sequential merge and split procedure based on a chi-square
test statistic.
 Kass was concerned about computation time, so he decided to settle for a sub-optimal
split on each predictor instead of searching for all possible combinations of the
categories.
 Kass’s algorithm is like sequential cross-tabulation.
o For each predictor:
1. cross tabulate the m categories of the predictor with the k categories of the
dependent variable,
2. find the pair of categories of the predictor whose 2xk sub-table is least
significantly different on a chi-square test and merge these two categories;
3. if the chi-square test statistic is not “significant” according to a preset critical
value, repeat this merging process for the selected predictor until no non-
significant chi-square is found for a sub-table, and pick the predictor variable
whose chi-square is largest and split the sample into subsets, where l is the
number of categories resulting from the merging process on that predictor;
4. Continue splitting, as with AID, until no “significant” chi-squares result. The
CHAID algorithm saves some computer time, but it is not guaranteed to find
the splits which predict best at a given step. Only by searching all possible
category subsets can we do that. CHAID is also limited to categorical
predictors, so it cannot be used for quantitative or mixed categorical
quantitative models.

CART (Classification And Regression Tree)

 CART algorithm was introduced in Breiman et al. (1986).
 A CART tree is a binary decision tree that is constructed by splitting a node into two
child nodes repeatedly, beginning with the root node that contains the whole learning
sample.
 The CART growing method attempts to maximize within-node homogeneity.
 The extent to which a node does not represent a homogenous subset of cases is an
indication of impurity.
 For example, a terminal node in which all cases have the same value for the dependent
variable is a homogenous node that requires no further splitting because it is "pure."
 For categorical (nominal, ordinal) dependent variables the common measure of impurity
is Gini, which is based on squared probabilities of membership for each category.

NMREC, Hyderabad
Data Analytics

 Splits are found that maximize the homogeneity of child nodes with respect to the value
of the dependent variable.

 Impurity Measure:

 GINI Index Used by the CART (classification and regression tree) algorithm, Gini
impurity is a measure of how often a randomly chosen element from the set would be
incorrectly labeled if it were randomly labeled according to the distribution of labels in
the subset.
 Gini impurity can be computed by summing the probability fi of each item being chosen
times the probability 1-fi of a mistake in categorizing that item.
 It reaches its minimum (zero) when all cases in the node fall into a single target category.
 To compute Gini impurity for a set of items, suppose i ε {1, 2... m}, and let fi be the
fraction of items labeled with value i in the set.

Advantages of Decision Tree:

 Simple to understand and interpret. People are able to understand decision tree models
after a brief explanation.
 Requires little data preparation. Other techniques often require data normalization,
dummy variables need to be created and blank values to be removed.
 Able to handle both numerical and categorical data. Other techniques are usually
specialized in analysing datasets that have only one type of variable.
 Uses a white box model. If a given situation is observable in a model the explanation for
the condition is easily explained by Boolean logic.
 Possible to validate a model using statistical tests. That makes it possible to account for
the reliability of the model.
 Robust. Performs well even if its assumptions are somewhat violated by the true model
from which the data were generated.
 Performs well with large datasets. Large amounts of data can be analyzed using standard
computing resources in reasonable time.

NMREC, Hyderabad
Data Analytics

Tools used to make Decision Tree:

 Many data mining software packages provide implementations of one or more decision
tree algorithms.
 Several examples include:
o Salford Systems CART
o IBM SPSS Modeler
o Rapid Miner
o SAS Enterprise Miner
o Matlab
o R (an open source software environment for statistical computing which includes
several CART implementations such as rpart, party and random Forest packages)
o Weka (a free and open-source data mining suite, contains many decision tree
algorithms)
o Orange (a free data mining software suite, which includes the tree module
orngTree)
o KNIME
o Microsoft SQL Server
o Scikit-learn (a free and open-source machine learning library for the Python
programming language).

 Pruning
 After building the decision tree, a tree-pruning step can be performed to reduce the size
of the decision tree.
 Pruning helps by trimming the branches of the initial tree in a way that improves the
generalization capability of the decision tree.

 The errors committed by a classification model are generally divided into two types:
o training errors
o generalization errors.

 Training error
o also known as resubstitution error or apparent error.
o it is the number of misclassification errors committed on training records.
 generalization error
o is the expected error of the model on previously unseen records.
o A good classification model must not only fit the training data well, it must also
accurately classify records it has never seen before.
 A good model must have low training error as well as low generalization error.

NMREC, Hyderabad
Data Analytics

 Model overfitting
o Decision trees that are too large are susceptible to a phenomenon known as
overfitting.
o A model that fits the training data too well can have a poorer generalization error
than a model with a higher training error.
o Such a situation is known as model overfitting.

 Model underfitting
o The training and test error rates of the model are large when the size of the tree is
very small.
o This situation is known as model underfitting.
o Underfitting occurs because the model has yet to learn the true structure of the
data.

o Model complexity
o To understand the overfitting phenomenon, the training error of a model can be
reduced by increasing the model complexity.
o Overfitting and underfitting are two pathologies that are related to the model
complexity.

NMREC, Hyderabad
Data Analytics

ARIMA (Autoregressive Integrated Moving Average)

 ARIMA model is a generalization of an autoregressive moving average (ARMA) model,

in time series analysis,
 These models are fitted to time series data either to better understand the data or to
predict future points in the series (forecasting).
 They are applied in some cases where data show evidence of non-stationary, wherein
initial differencing step (corresponding to the "integrated" part of the model) can be
applied to reduce the non-stationary.

 Non-seasonal ARIMA models

o These are generally denoted ARIMA(p, d, q) where parameters p, d, and q are
non-negative integers, p is the order of the Autoregressive model, d is the degree
of differencing, and q is the order of the Moving-average model.

 Seasonal ARIMA models

o These are usually denoted ARIMA(p, d, q)(P, D, Q)_m, where m refers to the
number of periods in each season, and the uppercase P, D, Q refer to the
autoregressive, differencing, and moving average terms for the seasonal part of
the ARIMA model.

 ARIMA models form an important part of the Box-Jenkins approach to time-series

modeling.

 Applications
o ARIMA models are important for generating forecasts and providing
understanding in all kinds of time series problems from economics to health care
applications.
o In quality and reliability, they are important in process monitoring if observations
are correlated.
o designing schemes for process adjustment
o monitoring a reliability system over time
o forecasting time series
o estimating missing values
o finding outliers and atypical events
o understanding the effects of changes in a system

NMREC, Hyderabad
Data Analytics

Measure of Forecast Accuracy

 Forecast Accuracy can be defined as the deviation of Forecast or Prediction from the
actual results.
Error = Actual demand – Forecast
OR
ei = At – Ft
 We measure Forecast Accuracy by 2 methods :
 Mean Forecast Error (MFE)
o For n time periods where we have actual demand and forecast values:

o Ideal value = 0;
o MFE > 0, model tends to under-forecast
o MFE < 0, model tends to over-forecast

 Mean Absolute Deviation (MAD)

o For n time periods where we have actual demand and forecast values:

 While MFE is a measure of forecast model bias, MAD indicates the absolute size of the
errors

Uses of Forecast error:

 Forecast model bias
 Absolute size of the forecast errors
 Compare alternative forecasting models
 Identify forecast models that need adjustment

NMREC, Hyderabad
Data Analytics

ETL Approach

 Extract, Transform and Load (ETL) refers to a process in database usage and especially
in data warehousing that:
o Extracts data from homogeneous or heterogeneous data sources
o Transforms the data for storing it in proper format or structure for querying and
analysis purpose
o Loads it into the final target (database, more specifically, operational data store,
data mart, or data warehouse)
 Usually all the three phases execute in parallel since the data extraction takes time, so
while the data is being pulled another transformation process executes, processing the
already received data and prepares the data for loading and as soon as there is some data
ready to be loaded into the target, the data loading kicks off without waiting for the
completion of the previous phases.
 ETL systems commonly integrate data from multiple applications (systems), typically
developed and supported by different vendors or hosted on separate computer hardware.
 The disparate systems containing the original data are frequently managed and operated
by different employees.
 For example, a cost accounting system may combine data from payroll, sales, and
purchasing.

 Commercially available ETL tools include:

o Anatella
o Alteryx
o CampaignRunner
o ESF Database Migration Toolkit
o InformaticaPowerCenter
o Talend
o IBM InfoSphereDataStage
o Ab Initio
o Oracle Data Integrator (ODI)
o Oracle Warehouse Builder (OWB)
o Microsoft SQL Server Integration Services (SSIS)
o Tomahawk Business Integrator by Novasoft Technologies.
o
Stambia
o Diyotta DI-SUITE for Modern Data Integration
o FlyData
o Rhino ETL
o SAP Business Objects Data Services

NMREC, Hyderabad
Data Analytics

o SAS Data Integration Studio

o SnapLogic
o Clover ETL opensource engine supporting only basic partial functionality and not
server
o SQ-ALL - ETL with SQL queries from internet sources such as APIs
o North Concepts Data Pipeline

 Various steps involved in ETL.

o Extract
o Transform
o Load

o Extract
 The Extract step covers the data extraction from the source system and
makes it accessible for further processing.
 The main objective of the extract step is to retrieve all the required data
from the source system with as little resources as possible.
 The extract step should be designed in a way that it does not negatively
affect the source system in terms or performance, response time or any
kind of locking.

 There are several ways to perform the extract:

 Update notification - if the source system is able to provide a
notification that a record has been changed and describe the
change, this is the easiest way to get the data.
 Incremental extract - some systems may not be able to provide
notification that an update has occurred, but they are able to
identify which records have been modified and provide an extract
of such records. During further ETL steps, the system needs to
identify changes and propagate it down. Note, that by using daily
extract, we may not be able to handle deleted records properly.
 Full extract - some systems are not able to identify which data has
been changed at all, so a full extract is the only way one can get the
data out of the system. The full extract requires keeping a copy of
the last extract in the same format in order to be able to identify
changes. Full extract handles deletions as well.
 When using Incremental or Full extracts, the extract frequency is
extremely important. Particularly for full extracts; the data
volumes can be in tens of gigabytes.

NMREC, Hyderabad
Data Analytics

 Clean - The cleaning step is one of the most important as it ensures

the quality of the data in the data warehouse. Cleaning should
perform basic data unification rules, such as:
 Making identifiers unique (sex categories Male/Female/Unknown,
M/F/null, Man/Woman/Not Available are translated to standard
Male/Female/Unknown)
 Convert null values into standardized Not Available/Not Provided
value
 Convert phone numbers, ZIP codes to a standardized form
 Validate address fields, convert them into proper naming, e.g.
Street/St/St./Str./Str
 Validate address fields against each other (State/Country,
City/State, City/ZIP code, City/Street).

o Transform
 The transform step applies a set of rules to transform the data from the
source to the target.
 This includes converting any measured data to the same dimension (i.e.
conformed dimension) using the same units so that they can later be
joined.
 The transformation step also requires joining data from several sources,
generating aggregates, generating surrogate keys, sorting, deriving new
calculated values, and applying advanced validation rules.

o Load
 During the load step, it is necessary to ensure that the load is performed
correctly and with as little resources as possible.
 The target of the Load process is often a database.
 In order to make the load process efficient, it is helpful to disable any
constraints and indexes before the load and enable them back only after
the load completes.
 The referential integrity needs to be maintained by ETL tool to ensure
consistency.

 Managing ETL Process

o The ETL process seems quite straight forward.

o As with every application, there is a possibility that the ETL process fails.
o This can be caused by missing extracts from one of the systems, missing values in
one of the reference tables, or simply a connection or power outage.

NMREC, Hyderabad
Data Analytics

o Therefore, it is necessary to design the ETL process keeping fail-recovery in

mind.

 Staging
o It should be possible to restart, at least, some of the phases independently from
theothers.
o For example, if the transformation step fails, it should not be necessary to
restart the Extract step.
o We can ensure this by implementing proper staging. Staging means that the
data is simply dumped to the location (called the Staging Area) so that it can
then be read by the next processing phase.
o The staging area is also used during ETL process to store intermediate results
of processing.
o This is ok for the ETL process which uses for this purpose.
o However, the staging area should be accessed by the load ETL process only.
o It should never be available to anyone else; particularly not to end users as
it isnot intended for data presentation to the end-user.
o May contain incomplete or in-the-middle-of-the-processing data.

UNIT-5
Data Visualization

Data Visualization

 Data visualization is the art and practice of gathering, analyzing, and graphically
representing empirical information.
 They are sometimes called information graphics, or even just charts and graphs.
 The goal of visualizing data is to tell the story in the data.
 Telling the story is predicated on understanding the data at a very deep level, and
gathering insight from comparisons of data points in the numbers

Why data visualization?

 Gain insight into an information space by mapping data onto graphical primitives
Provide qualitative overview of large data sets
 Search for patterns, trends, structure, irregularities, and relationships among data.
 Help find interesting regions and suitable parameters for further quantitative analysis.
 Provide a visual proof of computer representations derived.

NMREC, Hyderabad