Module 3

This is a single, concatenated file, suitable for printing or saving as a PDF for offline viewing. Please
note that some animations or images may not work.

Module 3
March 25 - March 31

Topics: Lecture 5: Prediction and Classification Methods — Simple and Multiple Linear
Regression
Lecture 6: Prediction and Classification Methods — k-Nearest Neighbors (kNN)

Readings: Lecture 5:
Lecture Notes
Shmueli et al., Chapter 6

Lecture 6:
Lecture Notes
Shmueli et al., Chapter 7

Tutorials: Lecture 5:
Determining the Number of Variable Relationships (Regression)

Lecture 6:
Load, Partition, and Plot Riding Mowers
Normalizing Data and Finding Nearest Neighbors

Discussions: Discussion 3

Initial post due Thursday, March 27 at 11:59 PM ET

Respond to at least two of your classmates' posts by Monday, March 31 at
11:59 PM ET

Assignments: None this week

Assessments: None this week

Live Classroom: Module 3 Live Classroom Session: Monday, March 31 at 8:00pm ET

Lecture 5: Prediction and Classification Methods - Simple and Multiple

 Linear Regression

Learning Objectives

After successfully completing the lesson, students will be able to:

1. Identify when simple linear regression is used.

2. Explain how a simple linear regression model is built.
3. Interpret the results of a simple linear regression equation.
4. Identify when multiple linear regression is used.
5. Explain how a multiple linear regression model is built.
6. Interpret the results of a multiple linear regression equation.

Simple Linear Regression

Simple Linear Regression (SLR) is used when we want to understand the relationship between a single input
variable and a single response variable.

In an SLR model, the input variable is always plotted on the x-axis, whereas the response variable is plotted on
the y-axis. When building such a model, it is very important that we understand the cause-and-effect relationship
in advance. If, out of carelessness or out of a misunderstanding of the data, we were to switch the positioning of
the two variables, we would get very different results and would come to an incorrect interpretation.

SLR can be used for either explanatory or predictive purposes. When used for explanatory purposes, it attempts
to explain something that has already occurred; specifically, it measures the impact that the input variable has
upon the response variable. When used for predictive purposes, it attempts to predict the likely outcome of a
completely new situation.

An SLR model is typically built with a best-fit line, derived from using the Ordinary Least Squares (OLS) method.
To see how an OLS model is built, please check out the video titled How the Best-Fitted Line is Actually Built
video in the AD699 video library. The OLS methodology is also used when building Multiple Linear Regression
(MLR) models.

Please watch the video Getting the Regression Equation from a Model Summary (SLR) in the AD699 video
library to see an explanation for the regression equation that results from the SLR process.

Multiple Linear Regression

Multiple Linear Regression (MLR) involves the relationship between multiple independent variables and a single
response variable.

MLR is the most popular model for making predictions. In MLR, the outcome variable is always a continuous
numerical variable (for predicting categorical outcomes, we will use a technique called logistic regression, which
we’ll delve into in Lecture 9).

A large part of the process of building an MLR model involves selecting the input variables. When we choose
input variables from among a list of potential options, we are looking to include variables that are strongly
correlated with the outcome variable, but not with one another.

When input variables are highly correlated with one another, the problem of multicollinearity can result.
Multicollinearity is problematic because it makes it much harder for us to know which input variable truly had the
result that we found in the output variable. To make a cooking analogy, let’s say you made your recipe with equal
parts sea salt and table salt. When someone sampled your food and described it as “salty” it would be hard for
you to accurately attribute that result to either input.

Several methods can be used for examining the right mix of input variables in an MLR model. Among these are:

Exhaustive search. The idea behind exhaustive search is to evaluate all subsets of predictors in order to
determine the optimal mix. Some MLR functions do this by finding the greatest possible adjusted R-squared
value. An adjusted R-squared score is based on R-squared (a statistic that tells how much the input variables
explain the outcome variables), and it includes a penalty for the inclusion of too many input variables (in order to
favor overall model simplicity). Other metrics of model performance include Akaike Information Criterion (AIC)
and Bayesian Information Criterion (BIC). The derivation of AIC and BIC values is outside the scope of AD699,
but when interpreting the results, we need to know that lower AIC and BIC values are considered better.

Forward selection. In forward selection, we start with no independent variables, and add them one by one until
new additions do not improve the model.

Backward elimination. In backward elimination, we start with all of the potential independent variables included,
and then we eliminate them one-by-one, until all of the remaining predictors make significant contributions to the
model.

Stepwise regression. Stepwise regression is similar to forward selection, except that at each step, predictors
that lack statistical significance are dropped from the model.

To see each of the above methods in action, please check out the videos in the AD699 video library with the tags
Lecture 6 and MLR.

Once our MLR model is built, we can assess its performance by studying the prediction accuracy measures of
the validation data. A common metric used for this purpose is the root mean squared error (RMSE), which is
described on page 119 of the textbook.

MLR has many real-world use cases. As a quick summary of potential uses, an MLR model could be used to
determine the likelihood of insurance fraud for a particular set of predictors, the probable expiration date of a
pharmaceutical product, or the likely income of a bank customer when a bank is assessing lending suitability.

Note
 MLR can be an extremely dense topic, with an almost endless stream of buts, what-ifs, and nuances.
MLR alone could easily comprise an entire graduate-level course. One of the primary goals of AD699
is to expose you to a broad variety of data mining tools and algorithms; therefore, our discussion of
MLR will remain relatively high-level. Only the material covered in Chapter 6 of the textbook, in the
Lecture 5 notes here, and in the videos with the “Lecture 5” tag will be considered inside the scope of
the course.

Lecture 6: Prediction and Classification Methods - k-Nearest

Neighbors (kNN)

Learning Objectives

After successfully completing the lesson, students will be able to:

1. Identify when, and for what purposes, k-nearest neighbor analysis is used.
2. Given a data set, determine the Euclidean, Correlation, and Hamming Distance for the values of any
observation from any other observation in the set.
3. Identify the difference between content-based filtering and collaborative-based filtering.

K-Nearest Neighbors: Overview

K-nearest neighbors (k-nn) is a process that can be used in data mining for classification. Conceptually, k-nn is all
about determining the statistical distance between observations, and then assessing the likely classification of
new observations. By using data points that fall near the value that we are testing in order to make this
classification, we are essentially saying “birds of a feather flock together.”

Part of the process involves determining a value for k. If k = 1, that means that we will simply look at the status of
the nearest neighbor, and classify the test data accordingly. For instance, if we are trying to assess whether a
particular company called Newco is a default risk, and are using k =1, then we would look at our training data to
find the company most similar to Newco. If that company defaulted, then we’ll classify Newco as a default risk; if
that company has not defaulted, we will classify Newco as not being a default risk. If k = 3, that means that we
will look at the three nearest neighbors, and make a classification prediction with a “majority rules” system. If an
even number is used as a k-value, the knn() function will settle a tie with a random selection. Even-numbered
k values are discouraged.

If we use a k = all approach (using a k-value that is the same number as the number of records in the training
set), then our model will simply make a classification based on the most common classification in the training set.
This would be an application of the naïve rule.
In regression, we used m-1 pairs of binary dummies in our model whenever we had m pairs of binary splits. In k-
nn, however, we will use m pairs of binary dummies in such instances.

In nearly all cases, data will need to be normalized before a Euclidean distance can be found. When we build a k-
nn model, we will normalize the records of the training data, and then use the result to predict the normalized
values for our validation data and test data (to see how this works, check out the code narration for Table 7.2 in
the AD699 video library).

Measuring Distance: Euclidean Distance,

Correlation Distance, and Hamming Distance

To find the Euclidean distance between two observations, we take the sum of the squares of the differences
between those observations, and then find the square root of that number. A video that describes this process,
and shows the steps to take in R to find a Euclidean distance, is in the AD699 Video Library.

Another way to measure distance is to use correlation distance. While there are multiple ways in which
correlation distance can be measured, in AD699 we will use the method that involves taking (1 – correlation). For
a perfectly correlated data pair (correlation of 1.0), the correlation distance would be zero. The greatest possible
correlation distance between two values is 2. In the AD699 Video Library, you can find a narrated demonstration
involving correlation distance, coded in R.

Hamming distance comes from information theory, and was first used to compare sequences of binary values.
Hamming distance can be found by simply summing the number of times two variables are not in agreement with
one another. Again, you can see more information about Hamming distance, with an example calculation, in the
AD699 Video Library.

Content-Based Filtering vs. Collaborative-Based

Filtering

K-nearest neighbors can be very useful when building recommendation systems.

In a content-based filtering model, an entire corpus of material must first be reviewed and analyzed, in a very
“people-intensive” process, before similarity and difference can be determined. The best example of a well-known
content-based filtering system is the Music Genome Project, used by Pandora. Pandora ranks songs from 0 to 5
in hundreds of different categories, and can then use this data to make recommendations to listeners, after
learning about particular listeners’ expressed tastes (through their previous song selections).

In a collaborative-based filtering model, by contrast, recommendations can be made without any need to
understand the content itself. Instead, such a system can simply analyze the existing choices of users, and then
make recommendations to other users who have either liked or disliked similar items, or whose user profiles are
similar. In the former case, a model could analyze the rating patterns of various movie reviewers. If the model
finds that Alice and Bob tend to show a high degree of nearness in their reviews, it can then automatically
recommend movies to Bob if Alice rates them highly, or vice versa. In the latter case, a system that identifies
Alice and Bob as being similar to one another (maybe in terms of demographic information, expressed interests,
tastes, etc.) can use that similarity as a basis for making similar recommendations to both Alice and Bob.

Pros and Cons of k-nn

The major advantages of k-nn are its relative simplicity and its flexibility (it can be used with continuous input
data, categorical input data, etc.). As you can see from the riding mower scatterplot on page 176 in the textbook,
a nearest neighbor can be visualized, and the “nearness” can be intuitive even to someone who is not familiar
with the algorithm itself.

K-nn does, however, have some disadvantages. In a large training data set, it can take a long time to find the
nearest neighbors. For this reason, k-nn is discouraged for attempted simultaneous prediction of large numbers
of records. Also, as the number of predictors increases, the number of records required in the training set grows
exponentially.