DATA SCIENCE AND VISUALIZATION

21CS644
Module-3: Feature Generation and Feature Selection

2
 Feature Selection
1. The idea of feature selection is identifying the subset of data or transformed
data that you want to put into your model.
2. Feature selection is not only useful for winning competitions—it’s an important
part of building statistical models and algorithms in general. Just because you
have data doesn’t mean it all has to go into the model.
3. For example, it’s possible you have many redundancies or correlated variables in
your raw data
4. Similarly you might want to construct new variables by transforming the
variables with a logarithm, say, or turning a continuous variable into a binary
3
variable, before feeding them into the model.
 Feature Selection

1. Feature selection is the process of choosing a subset of relevant features

(variables) from a larger set of features to improve the performance of a model or
reduce its complexity.

4
 Feature Selection

1. Example: User Retention

2. Suppose you have an app that you designed where users pay a monthly
subscription fee to use it. The more users you have, the more money you make.
3. Suppose you realize that only 10% of new users ever come back after the first
month.
4. So you have two options to increase your revenue: find a way to increase the
retention rate of existing users, or acquire new users.
5. Generally it costs less to keep an existing customer around than to market and
advertise to new users. 5
 Feature Selection
1. Example: User Retention (cont…)
2. But setting aside that particular cost-benefit analysis of acquisition or retention,
let’s choose to focus on your user retention situation by building a model that
predicts whether or not a new user will come back next month based on their
behavior this month
3. You could build such a model in order to understand your retention situation, but
let’s focus instead on building an algorithm that is highly accurate at predicting.
4. You might want to use this model to give a free month to users who you predict
need the extra incentive to stick around, for example.
6
 Feature Selection

1. A good, crude, simple model you could start out with would be logistic regression
2. This would give you the probability the user returns their second month
conditional on their activities in the first month
3. You record each user’s behavior for the first 30 days after sign-up. You could log
every action the user took with timestamps
4. This would be the data collection phase.
5. Any action the user could take gets recorded.

7
 Feature Selection

1. Then need to process these logs down to a dataset with rows and columns,
where each row was a user and each column was a feature.
2. At this point, you shouldn’t be selective; you’re in the feature generation phase.
3. So your data science team (game designers, software engineers, statisticians,
and marketing folks) might sit down and brainstorm features.

8
 Feature Selection

1. Here are some examples:

9
 Feature Selection

1. Use your imagination and come up with as many features as possible.

2. Notice there are redundancies and correlations between these features;
that’s OK.
3. Let ci = 1 if user i returns to use Chasing Dragons any time in the subsequent
month.

10
 Feature Selection

1. Three categories of feature selection methods: filters, wrappers, and

embedded methods
2. Filters:
1. Filters order possible features with respect to a ranking based on a metric
or statistic, such as correlation with the outcome variable.
2. This is sometimes good on a first pass over the space of features,
because they then take account of the predictive power of individual features

11
 Feature Selection
1. Filters:
1. However, the problem with filters is that you get correlated features.
2. In other words, the filter doesn’t care about redundancy. And by treating
the features as independent, you’re not taking into account possible
interactions
3. Here’s an example of a filter: for each feature, run a linear regression with
only that feature as a predictor.
4. Each time, note either the p-value or R-squared, and rank order according to
the lowest p-value or highest R-squared
12
 Feature Selection

2. Wrappers:
1. Wrapper feature selection tries to find subsets of features, of some fixed
size, that will do the trick.
2. However, as anyone who has studied combinations and permutations knows,
the number of possible size k subsets of n things, called grows
exponentially
3. There are two aspects to wrappers that you need to consider: 1) selecting an
algorithm to use to select features and 2) deciding on a selection criterion or
filter to decide that your set of features is “good.” 13
 Feature Selection

2. Wrappers:
1. Selecting an algorithm
1. Let’s first talk about a set of algorithms that fall under the category of stepwise
regression, a method for feature selection that involves selecting features
according to some selection criterion by either adding or subtracting features
to a regression model in a systematic way.
2. There are three primary methods of stepwise regression: forward selection,
backward elimination, and a combined approach (forward and backward).
14
 Feature Selection

2. Wrappers:
1. Selecting an algorithm
1. Forward selection
1. In forward selection you start with a regression model with no features, and
gradually add one feature at a time according to which feature improves the
model the most based on a selection criterion.
2. This looks like this: build all possible regression models with a single
predictor.
15
 Feature Selection

2. Wrappers:
1. Selecting an algorithm
1. Forward selection
3. Pick the best. Now try all possible models that include that best predictor and
a second predictor.
4. Pick the best of those.
5. You keep adding one feature at a time, and you stop when your selection
criterion no longer improves, but instead gets worse.
16
 Feature Selection

2. Wrappers:
1. Selecting an algorithm
2. Backward elimination
1. In backward elimination you start with a regression model that includes all
the features, and you gradually remove one feature at a time according to
the feature whose removal makes the biggest improvement in the selection
criterion.
2. You stop removing features when removing the feature makes the selection
criterion get worse. 17
 Feature Selection

2. Wrappers:
1. Selecting an algorithm
3. Combined approach
1. Most subset methods are capturing some flavor of minimum-redundancy-
maximum-relevance.
2. So, for example, you could have a greedy algorithm that starts with the best
feature, takes a few more highly ranked, removes the worst, and so on.
3. This a hybrid approach with a filter method.
18
 Feature Selection
2. Wrappers:
1. Selection criterion
1. There are a number of selection criteria you could choose from.
2. You need to select the selection criterion
3. Despite the theoretical properties of these various criteria, the choice you
make is somewhat arbitrary.
4. One way to deal with this is to try different selection criteria and see how
robust your choice of model is
5. Different selection criterion might produce wildly different models, and it’s part
19
of your job to decide what to optimize for and why:
 Feature Selection

2. Wrappers:
1. R-squared

20
 Feature Selection

2. Wrappers:
2. p-values
1. In the context of regression where you’re trying to estimate coefficients (the
βs), to think in terms of p-values, you make an assumption of there being a
null hypothesis that the βs are zero
2. For any given β, the p-value captures the probability of observing the data that
you observed, and obtaining the test-statistic (in this case the estimated β)
that you got under the null hypothesis
21
 Feature Selection

2. Wrappers:
2. p-values
1. Specifically, if you have a low p-value, it is highly unlikely that you would
observe such a test-statistic if the null hypothesis actually held.
2. This translates to meaning that (with some confidence) the coefficient is highly
likely to be non-zero.

22
 Feature Selection

2. Wrappers:
2. p-values
1. Calculate the p-value:
2. The p-value is a measure of the probability of observing the observed test
statistic (or a more extreme value) under the assumption that the null
hypothesis is true.
3. A small p-value (typically below a chosen significance level, often 0.05)
suggests strong evidence against the null hypothesis. In other words, it
indicates that the feature may be relevant and should be retained. 23
 Feature Selection

1. Feature Selection Criteria:

2. Depending on the chosen significance level and the threshold for significance,
you can decide whether to include or exclude a feature in your model.
3. If the p-value is less than the significance level, you may choose to include the
feature as it suggests a significant relationship with the target variable.
4. If the p-value is greater than the significance level, you may choose to exclude
the feature as it suggests no significant relationship with the target variable.

24
 Feature Selection

1. Iterative Process:

2. Feature selection is often an iterative process.

3. You may calculate p-values for all features, and then either select the features
with p-values below the significance level or rank them by p-value and select the
top N features.

25
 Feature Selection

1. It’s important to note that while p-value-based feature selection can be a useful
technique, it has limitations and should be used with caution.
2. For example, it assumes that the data and model assumptions are met, and it
doesn’t necessarily capture complex interactions between features.
3. Additionally, in some cases, other techniques like regularization or domain
knowledge may be more effective for feature selection.

26
 Feature Selection

2. Wrappers:
1. Selection criterion
1. AIC (Akaike Infomation Criterion)
2. Given by the formula 2k−2ln(L), where k is the number of
parameters in the model and ln(L) is the “maximized value of the
log likelihood.”
3. The goal is to minimize AIC.

27
 Feature Selection

1. What Is an Akaike Information Criterion (AIC) Score?

2. An AIC score is a number used to determine which machine learning model is
best for a given data set in situations where one can’t easily test a data set.

3. An AIC test is most useful when you’re working with a small data set or time
series analysis. The lower the AIC score the better.

28
 Feature Selection

1. AIC is most frequently used in situations where one is not able to easily test the
model’s performance on a test set in standard machine learning practice (small
data or a time series.)
2. AIC is particularly valuable for time series, because time series analysis’ most
valuable data is often the most recent, which is stuck in the validation and test
sets.
3. As a result, training on all the data and using AIC can result in improved model
selection over traditional train/validation/test model selection methods.
29
 Feature Selection

1. AIC works by evaluating the model’s fit on the training data and adding a penalty
term for the complexity of the model (similar fundamentals to regularization.)
2. The desired result is to find the lowest possible AIC, which indicates the best
balance of model fit with generalizability.
3. This serves the eventual goal of maximizing fit on out-of-sample data.

30
 Feature Selection

1. BIC (Bayesian Information Criterion)

2. Given by the formula k*ln(n) −2ln(L) , where k is the number of parameters in the
model, n is the number of observations (data points, or users), and ln(L) is the
maximized value of the log likelihood.
3. The goal is to minimize BIC.

31
 Feature Selection
1. BIC (Bayesian Information Criterion)
2. In statistics, the Bayesian information criterion (BIC) or Schwarz criterion (also
SBC, SBIC) is a criterion for model selection among a finite set of models. It is
based, in part, on the likelihood function, and it is closely related to Akaike
information criterion (AIC).
3. When fitting models, it is possible to increase the likelihood by adding
parameters, but doing so may result in overfitting.
4. The BIC resolves this problem by introducing a penalty term for the number of
parameters in the model.
5. The penalty term is larger in BIC than in AIC 32
 Feature Selection

1. Entropy
1. Entropy represents order of randomness. In decision tree, it helps model in
selection of feature for splitting, at the node by measuring the purity of the
split.
2. If, Entropy = 0 means it is pure split i.e., all instances are of only 1 class.
3. Entropy=1 means Completely impure split i.e., equal instances (50%–50%)
of both class at node causing extreme disorder.

33
 Embedded Methods: Decision Trees

1. Decision trees have an intuitive appeal because outside the context of data
science in our every day lives, we can think of breaking big decisions down into a
series of questions.
2. Lets see the decision tree in Figure 7-3 about a college student facing the very
important decision of how to spend their time.

34
Embedded Methods: Decision Trees

35
 Embedded Methods: Decision Trees

1. This decision is actually dependent on a bunch of factors: whether or not there are
any parties or deadlines, how lazy the student is feeling, and what they care about
most (parties).
2. The interpretability of decision trees is one of the best features about them.
3. In the context of a data problem, a decision tree is a classification algorithm.
4. For the Chasing Dragons example, you want to classify users as “Yes, going to
come back next month” or “No, not going to come back next month.”
5. You know that the class of any given user is dependent on many factors (number of
dragons the user slew, their age, how many hours they already played the game). 36
 Embedded Methods: Decision Trees

1. And you want to break it down based on the data you’ve collected.
2. But how do you construct decision trees from data and what mathematical
properties can you expect them to have?
3. Ultimately you want a tree that is something like Figure 7-4
4. But you want this tree to be based on data and not just what you feel like

37
Embedded Methods: Decision Trees

38
 Embedded Methods: Decision Trees

1. Assume we break compound questions into multiple yes-or-no questions, and we

denote the answers by “0” or “1.”
2. Given a random variable X, we denote by p(X = 1) and p(X = 0) the probability that
X is true or false, respectively.

39
 Embedded Methods: Decision Trees

1. Entropy
2. To quantify what is the most “informative” feature, we define entropy – effectively a
measure for how mixed up something is—for X as follows:

3. Note when p X = 1 = 0 or p X = 0 = 0, the entropy vanishes, consistent with the fact

that:

40
 Embedded Methods: Decision Trees

1. In particular, if either option has probability zero, the entropy is 0.

2. Moreover, because p X = 1 = 1− p X = 0 , the entropy is symmetric about 0.5 and
maximized at 0.5, which we can easily confirm using a bit of calculus.
3. Figure 7-5 shows a picture of that.

41
 Embedded Methods: Decision Trees

1. But what does it mean in words, and why are we calling it entropy?
2. Earlier, we discussed that entropy is a measurement of how mixed up something is.
3. So, for example, if X denotes the event of a baby being born a boy, we’d expect it to
be true or false with probability close to 1/2, which corresponds to high entropy, i.e.,
the bag of babies from which we are selecting a baby is highly mixed.
4. But if X denotes the event of a rainfall in a desert, then it’s low entropy.
5. In other words, the bag of day-long weather events is not highly mixed in deserts.

42
 Embedded Methods: Decision Trees

1. Using this concept of entropy, we will be thinking of X as the target of our model.
2. So, X could be the event that someone buys something on our site.
3. We’d like to know which attribute of the user will tell us the most information about
this event X.
4. We will define the information gain, denoted IG(X,a), for a given attribute a, as the
entropy we lose if we know the value of that attribute:

43
 Embedded Methods: Decision Trees

1. To compute this we need to define

2. We can do this in two steps.

3. For any actual value of the attribute a we can compute the specific conditional
entropy as you might expect:

44
 Embedded Methods: Decision Trees
1. and then we can put it all together, for all possible values of a, to get the conditional
entropy

2. In words, the conditional entropy asks: how mixed is our bag really if we know the
value of attribute a?
3. And then information gain can be described as: how much information do we learn
about X (or how much entropy do we lose) once we know a?
4. Going back to how we use the concept of entropy to build decision trees: it helps us
decide what feature to split our tree on, or in other words, what’s the most informative 45
 The Decision Tree Algorithm

1. You build your decision tree iteratively, starting at the root.

2. You need an algorithm to decide which attribute to split on; e.g., which node should
be the next one to identify.
3. You choose that attribute in order to maximize information gain
4. You keep going until all the points at the end are in the same class or you end up with no
features left.
5. In this case, you take the majority vote.

46
 The Decision Tree Algorithm

1. Often people “prune the tree” afterwards to avoid overfitting.

2. This just means cutting it off below a certain depth.
3. After all, by design, the algorithm gets weaker and weaker as you build the tree,
and it’s well known that if you build the entire tree, it’s often less accurate (with new
data) than if you prune it.
4. This is an example of an embedded feature selection algorithm. (Why embedded?)
You don’t need to use a filter here because the information gain method is doing
your feature selection for you.
47
 Random Forests

1. Let’s turn to another algorithm for feature selection.

2. Random forests generalize decision trees with bagging, otherwise known as
bootstrap aggregating.
3. They’re conversely easy to specify, with two hyper-parameters: you just need to
specify the number of trees you want in your forest, say N, as well as the number
of features to randomly select for each tree, say F.

48
 Random Forests

1. Before we get into the weeds of the random forest algorithm, let’s review
bootstrapping.
2. A bootstrap sample is a sample with replacement, which means we might sample
the same data point more than once.
3. We usually take to the sample size to be 80% of the size of the entire (training) dataset,
but of course this parameter can be adjusted depending on circumstances.
4. This is technically a third hyper-parameter of our random forest algorithm.

49
 Random Forests
1. Now to the algorithm. To construct a random forest, you construct N decision trees
as follows:
1. For each tree, take a bootstrap sample of your data, and for each node you
randomly select F features, say 5 out of the 100 total features
2. Then you use your entropy-information-gain engine as described in the
previous section to decide which among those features you will split your tree
on at each stage.
3. Note that you could decide beforehand how deep the tree should get, or you could prune
your trees after the fact, but you typically don’t prune the trees in random forests,
because a great feature of random forests is that they can incorporate idiosyncratic
50
 Criticisms of Feature Selection

1. There’s a well-known bias-variance tradeoff: a model is “high bias” if it’s is too

simple (the features aren’t encoding enough information).
2. In this case, lots more data doesn’t improve our model.
3. On the other hand, if our model is too complicated, then “high variance” leads to
4. overfitting.
5. In this case we want to reduce the number of features we are using.

51
 Bias-Variance Trade Off

1. It is important to understand prediction errors (bias and variance) when it comes to

accuracy in any machine-learning algorithm.
2. There is a tradeoff between a model’s ability to minimize bias and variance which is
referred to as the best solution for selecting a value of Regularization constant.
3. A proper understanding of these errors would help to avoid the overfitting and
underfitting of a data set while training the algorithm.

52
 What is Bias?
1. The bias is known as the difference between the prediction of the values by the
Machine Learning model and the correct value.
2. Being high in biasing gives a large error in training as well as testing data.
3. It recommended that an algorithm should always be low-biased to avoid the
problem of underfitting.
4. By high bias, the data predicted is in a straight line format, thus not fitting accurately in
the data in the data set.
5. Such fitting is known as the Underfitting of Data.
6. This happens when the hypothesis is too simple or linear in nature.
53
What is Bias?

54
Fig. High Bias in the Model
 What is Variance?
1. The variability of model prediction for a given data point which tells us the spread of
our data is called the variance of the model.
2. The model with high variance has a very complex fit to the training data and thus is
not able to fit accurately on the data which it hasn’t seen before.
3. As a result, such models perform very well on training data but have high error rates
on test data.
4. When a model is high on variance, it is then said to as Overfitting of Data.
5. Overfitting is fitting the training set accurately via complex curve and high order
hypothesis but is not the solution as the error with unseen data is high.
6. While training a data model variance should be kept low. The high variance data looks as
55
 What is Variance?

56
Fig. High Variance in the Model
 Bias Variance Tradeoff
1. If the algorithm is too simple (hypothesis with linear equation) then it may be on
high bias and low variance condition and thus is error-prone.
2. If algorithms fit too complex (hypothesis with high degree equation) then it may be
on high variance and low bias.
3. In the latter condition, the new entries will not perform well.
4. Well, there is something between both of these conditions, known as a Trade-off or
Bias Variance Trade-off.
5. This tradeoff in complexity is why there is a tradeoff between bias and variance.
6. An algorithm can’t be more complex and less complex at the same time. For the
graph, the perfect tradeoff will be like this. 57
Bias Variance Tradeoff

58
 Bias Variance Tradeoff
1. The best fit will be given by the hypothesis on the tradeoff point. The error to
complexity graph to show trade-off is given as –

59
Bias Variance Tradeoff

60
 Bias Variance Tradeoff
1. This is referred to as the best point chosen for the training of the algorithm which
gives low error in training as well as testing data.
2. To summarise,

1. A model with a high bias error underfits data and makes very simplistic assumptions
on it
2. A model with a high variance error overfits the data and learns too much from it
3. A good model is where both Bias and Variance errors are balanced

61
 Bias Variance Tradeoff
1. To achieve a balance between the Bias error and the Variance error, we need a
value of k such that the model neither learns from the noise (overfit on data) nor
makes sweeping assumptions on the data(underfit on data).
2. To keep it simpler, a balanced model would look like this:

62
Recommendation Engines: Building a User-Facing Data
Product at Scale of Feature Selection

1. Recommendation engines, also called recommendation systems, are the

quintessential data product and are a good starting point when you’re explaining to
non–data scientists what you do or what data science really is.
2. This is because many people have interacted with recommendation systems when
they’ve been suggested books on [Link] or gotten recommended movies on
Netflix. B

63
Recommendation Engines: Building a User-Facing Data
Product at Scale of Feature Selection

1. Aside from being a clear example of a product that literally uses data as its fuel,
another reason we call recommendation systems “quintessential” is that building a
solid recommendation system end-to-end requires an understanding of linear
algebra and an ability to code; it also illustrates the challenges that Big Data poses
when dealing with a problem that makes intuitive sense, but that can get
complicated when implementing its solution at scale.

64
 A Real-World Recommendation Engine

1. Recommendation engines are used all the time—what movie would you like,
knowing other movies you liked? What book would you like, keeping in mind past
purchases? What kind of vacation are you likely to embark on, considering past
trips?
2. To set up a recommendation engine, suppose you have users, which form a set U; and
you have items to recommend, which form a set V.

65
 A Real-World Recommendation Engine

1. We can denote this as a bi‐partite graph (shown again in Figure 8-1) if each user
and each item has a node to represent it—there are lines from a user to an item if
that user has expressed an opinion about that item.
2. Note they might not always love that item, so the edges could have weights: they
could be positive, negative, or on a continuous scale

66
A Real-World Recommendation Engine

67
 A Real-World Recommendation Engine

1. Next up, you have training data in the form of some preferences—you know some
of the opinions of some of the users on some of the items.
2. From those training data, you want to predict other preferences for your users.
3. That’s essentially the output for a recommendation engine.
4. You represent a given user as a vector of features, sometimes including only
metadata—sometimes including only preferences

68
 Nearest Neighbor Algorithm Review
1. If you want to predict whether user A likes something, you look at a user B closest
to user A who has an opinion, then you assume A’s opinion is the same as B’s.
2. In other words, once you’ve identified a similar user, you’d then find something that
user A hadn’t rated (which you’d assume meant he hadn’t ever seen that movie or
bought that item), but that user B had rated and liked and use that as your
recommendation for user A
3. To implement this you need a metric so you can measure distance. One example
when the opinions are binary: Jaccard distance, cosine similarity or Euclidean
distance
69
 Some Problems with Nearest Neighbors
1. Curse of dimensionality: There are too many dimensions, so the closest neighbors
are too far away from each other to realistically be considered “close.”
2. Overfitting: Overfitting is also a problem. So one guy is closest, but that could be
pure noise
3. Correlated features: There are tons of features, moreover, that are highly correlated
with each other. This would lead to bad performance, because you’re using
redundant information and essentially placing double the weight on some variables.
4. Relative importance of features: Some features are more informative than others.
Weighting features may therefore be helpful
70
 Some Problems with Nearest Neighbors

1. Sparseness: If your vector (or matrix, if you put together the vectors) is too sparse,
or you have lots of missing data, then most things are unknown
2. Measurement errors: There’s measurement error (also called reporting error):
people may lie
3. Computational complexity: There’s a calculation cost—computational complexity
4. Preferences change over time: User preferences may also change over time
5. Cost to update: It’s also expensive to update the model as you add more data

71
 The Dimensionality Problem

1. Use both Singular Value Decomposition (SVD) and Principal Component Analysis
(PCA) to tackle
2. Our goal is to build a model that has a representation in a low dimensional
subspace

72
 Singular Value Decomposition (SVD)

1. Singular Value Decomposition (SVD) is a powerful mathematical tool used in a

variety of applications, such as data compression, image processing, and
recommendation systems.

73
 Singular Value Decomposition (SVD)

1. Given an m×n matrix X of rank k, it is a theorem from linear algebra that we can
always compose it into the product of three matrices as follows:

2. where U is m×k, S is k×k, and V is k×n, the columns of U and V are pairwise
orthogonal, and S is diagonal.
3. Note the standard statement of SVD is slightly more involved and has U and V both
square unitary matrices, and has the middle “diagonal” matrix a rectangular.
74
 Singular Value Decomposition (SVD)
1. Properties
1. SVD is a decomposition of any matrix into the product of three matrices, which
makes it useful for various matrix operations and data analysis tasks.
2. The SVD of a matrix is unique, which means that for any given matrix, there is
only one SVD that can be calculated.
3. The singular values in the SVD of a matrix describe the strength of the linear
relationships between the columns and rows of the matrix, making SVD a useful
tool for dimensionality reduction and data compression.
4. The orthogonal matrices U and V in the SVD provide a new basis for the matrix,
75
 Singular Value Decomposition (SVD)
1. Applications
1. SVD has many important applications in a variety of fields, including data
compression, image processing, and recommendation systems..

76
 Singular Value Decomposition (SVD)

1. X is our original dataset, which has users’ ratings of items.

2. We have m users, n items, and k would be the rank of X, and consequently would
also be an upper bound on the number d of latent variables
3. we choose d whereas m, n, and k are defined through our training dataset.
4. So just like in k-NN, where k is a tuning parameter (different k entirely—not trying to
confuse you!), in this case, d is the tuning parameter.

77
 Singular Value Decomposition (SVD)

1. Each row of U corresponds to a user, whereas V has a row for each item.
2. The values along the diagonal of the square matrix S are called the “singular
values.”
3. They measure the importance of each latent variable—the most important latent
variable has the biggest singular value.

78
 Important Properties of SVD

1. Because the columns of U and V are orthogonal to each other, you can order the
columns by singular values via a base change operation.
2. That way, if you put the columns in decreasing order of their corresponding singular
values, then the dimensions are ordered by importance from highest to lowest.
3. You can take lower rank approximation of X by throwing away part of S. In other
words, replace S by a submatrix taken from the upper-left corner of S.

79
 Important Properties of SVD

1. Of course, if you cut off part of S you’d have to simultaneously cut off part of U and
part of V, but this is OK because you’re cutting off the least important vectors.
2. This is essentially how you choose the number of latent variables d—you no longer
have the original matrix X anymore, only an approximation of it, because d is
typically much smaller than k, but it’s still pretty close to X
3. This is what people mean when they talk about “compression.”

80
 Important Properties of SVD

1. How would you actually use this for recommendation?

2. You’d take X, fill in all of the empty cells with the average rating for that item (you
3. don’t want to fill it in with 0 because that might mean something in the rating
system, and SVD can’t handle missing values), and then compute the SVD. Now
that you’ve decomposed it this way, it means that you’ve captured latent features
that you can use to compare users if you want to.
4. But that’s not what you want—you want a prediction
81
 Important Properties of SVD

1. SVD is extremely computationally expensive

82
 Principal Component Analysis (PCA)

1. Let’s look at another approach for predicting preferences.

2. With this approach, you’re still looking for U and V as before, but you don’t need S
anymore, so you’re just searching for U and V such that:

3. Your optimization problem is that you want to minimize the discrepency between the
actual X and your approximation to X via U and V measured via the squared error:

83
Principal Component Analysis (PCA)

84
 Principal Component Analysis (PCA)
1. So, you want to find the best choices of U and V that minimize the squared
differences between prediction and observation on everything you actually know,
and the idea is that if it’s really good on stuff you know, it will also be good on stuff
you’re guessing.
2. It’s mean squared error, like we used for linear regression
3. Now you get to choose a parameter, namely the number d defined as how may
latent features you want to use.
4. The matrix U will have a row for each user and a column for each latent feature,
85
and the matrix V will have a row for each item and a column for each latent feature.
 Theorem: The resulting latent features will be uncorrelated

1. But don’t forget, we’ve ignored V! However, it turns out that V’s rows will also be
mutually orthogonal when we force U’s columns to be.
2. The best choice of scalar (i.e., to minimize the sum of the squares of the entries of
86
U and of V) is in fact the geometric mean of those two quantities
 Build Your Own Recommendation System

1. GetGlue dataset (Chapter 6)

2. [Link]
tutorials/how-to-build-
recommendation-system-in-
python/#6

87
Build Your Own Recommendation System

88
Build Your Own Recommendation System

89