CSCI 6517 Recommender Systems

Model-based Collaborative Filtering

Ga Wu, Summer 2025

A bit Machine Learning
• Informally, we want to have a function fθ that can help us solve problems
without letting us de ning it manually. The data will help us approximate the
function with certain hypothesis.

I have a hypothesis!
fi
Example: house price prediction
• I think house price can be predicted with an unknown equation:
House Price = funknown (Size, Bedrooms, Location, etc)
<latexit sha1_base64="aW3BUaOJ591Sk0DwRh41yBGc2UI=">AAACSnicbZDPSxtBFMdnU2ttbDXq0ctgEFIIYVcFe6gg9eLBQ0SjQhLC7ORtHDI7s8y8rcZl/75eevLmH+HFQ0W8OMlGsKYPBr7zeT/mzTdMpLDo+3de6cPcx/lPC5/Li1++Li1XVlbPrE4NhxbXUpuLkFmQQkELBUq4SAywOJRwHg4PxvnzX2Cs0OoURwl0YzZQIhKcoUO9CusgXGN2qFMLtGkEh5zu0aiXFTxVQ6WvVJ7TWgFOxA3k9UL/hL7ROrav9yNdTM3rtACAPP/Wq1T9hj8JOiuCqaiSaTR7ldtOX/M0BoVcMmvbgZ9gN2MGBZeQlztu1YTxIRtA20nFYrDdbGJFTjcd6dNIG3cU0gl925Gx2NpRHLrKmOGlfZ8bw//l2ilG37uZUEmKoHjxUJRKipqOfaV9YYCjHDnBuBFuV8ovmWEcnftlZ0Lw/suz4myrEWw3to53qvs/pnYskHWyQWokILtknxySJmkRTn6Te/KXPHp/vAfvyXsuSkvetGeN/BOluRfzeLXL</latexit>

• Hypothesis: each factor contributes to the house price independently

House Price ⇡ Size ⇥ w1 + Bedrooms ⇥ w2 + Location ⇥ w3 + · · ·
<latexit sha1_base64="BjVG+v7OSGhxhWENGpxHvj2Ba8Q=">AAACW3icbZFLSwMxFIUz46vWV1VcuQkWQRDKTBV04UJ048JFRatCp5RM5lZDM5MhuaPWoX/SlS78K2Lajm8vBA7nu5ebnISpFAY978VxJyanpmdKs+W5+YXFpcryyqVRmebQ5EoqfR0yA1Ik0ESBEq5TDSwOJVyFveMhv7oDbYRKLrCfQjtmN4noCs7QWp2KDhAeMD9RmQHa0ILDgAYsTbV6oGN0Lh5hEKCIwdD7jk+3C/8IIq1UbGz/B6x/wlM1XvA1uDNkPFJoOpWqV/NGRf8KvxBVUlSjU3kKIsWzGBLkkhnT8r0U2znTKLiEQTmwd08Z77EbaFmZMLuxnY+yGdBN60S0q7Q9CdKR+30iZ7Ex/Ti0nTHDW/ObDc3/WCvD7n47F0maISR8vKibSYqKDoOmkdDAUfatYFwLe1fKb5lmHO13lG0I/u8n/xWX9Zq/U6uf7VYPD4o4SmSdbJAt4pM9ckhOSIM0CSfP5M2ZcUrOqzvhlt35cavrFDOr5Ee5a++GgbWv</latexit>

Problem: I don’t know the weights of the factors are But I have data
Example: house price prediction
• I think house price can be predicted with an unknown equation:
House Price = funknown (Size, Bedrooms, Location, etc)
<latexit sha1_base64="aW3BUaOJ591Sk0DwRh41yBGc2UI=">AAACSnicbZDPSxtBFMdnU2ttbDXq0ctgEFIIYVcFe6gg9eLBQ0SjQhLC7ORtHDI7s8y8rcZl/75eevLmH+HFQ0W8OMlGsKYPBr7zeT/mzTdMpLDo+3de6cPcx/lPC5/Li1++Li1XVlbPrE4NhxbXUpuLkFmQQkELBUq4SAywOJRwHg4PxvnzX2Cs0OoURwl0YzZQIhKcoUO9CusgXGN2qFMLtGkEh5zu0aiXFTxVQ6WvVJ7TWgFOxA3k9UL/hL7ROrav9yNdTM3rtACAPP/Wq1T9hj8JOiuCqaiSaTR7ldtOX/M0BoVcMmvbgZ9gN2MGBZeQlztu1YTxIRtA20nFYrDdbGJFTjcd6dNIG3cU0gl925Gx2NpRHLrKmOGlfZ8bw//l2ilG37uZUEmKoHjxUJRKipqOfaV9YYCjHDnBuBFuV8ovmWEcnftlZ0Lw/suz4myrEWw3to53qvs/pnYskHWyQWokILtknxySJmkRTn6Te/KXPHp/vAfvyXsuSkvetGeN/BOluRfzeLXL</latexit>

• Hypothesis: each factor contributes to the house price independently

House Price ⇡ Size ⇥ w1 + Bedrooms ⇥ w2 + Location ⇥ w3 + · · ·
<latexit sha1_base64="BjVG+v7OSGhxhWENGpxHvj2Ba8Q=">AAACW3icbZFLSwMxFIUz46vWV1VcuQkWQRDKTBV04UJ048JFRatCp5RM5lZDM5MhuaPWoX/SlS78K2Lajm8vBA7nu5ebnISpFAY978VxJyanpmdKs+W5+YXFpcryyqVRmebQ5EoqfR0yA1Ik0ESBEq5TDSwOJVyFveMhv7oDbYRKLrCfQjtmN4noCs7QWp2KDhAeMD9RmQHa0ILDgAYsTbV6oGN0Lh5hEKCIwdD7jk+3C/8IIq1UbGz/B6x/wlM1XvA1uDNkPFJoOpWqV/NGRf8KvxBVUlSjU3kKIsWzGBLkkhnT8r0U2znTKLiEQTmwd08Z77EbaFmZMLuxnY+yGdBN60S0q7Q9CdKR+30iZ7Ex/Ti0nTHDW/ObDc3/WCvD7n47F0maISR8vKibSYqKDoOmkdDAUfatYFwLe1fKb5lmHO13lG0I/u8n/xWX9Zq/U6uf7VYPD4o4SmSdbJAt4pM9ckhOSIM0CSfP5M2ZcUrOqzvhlt35cavrFDOr5Ee5a++GgbWv</latexit>

• Approximate the unknown function with historical data

350000 = 1000 ⇥ w1 + 2 ⇥ w2 · · · Approximated
parameters
470000 = 1200 ⇥ w1 + 2 ⇥ w2 · · · w1 = 10
240000 = 600 ⇥ w1 + 1 ⇥ w2 · · · w2 = 1.2
One equation for a row 650000 = 1300 ⇥ w1 + 3 ⇥ w2 · · · We cannot solve the equations, but we
in the database
<latexit sha1_base64="ttIJt8T/mcAkY5NWAxd0YF8yuYM=">AAACJ3icbZDLSgMxFIYzXut4G3XpJlgsrspMFXSjFN24rGAv0CklkzltQzOZIckoZejbuPFV3AgqokvfxPSCaOsPgS//OYfk/EHCmdKu+2ktLC4tr6zm1uz1jc2tbWdnt6biVFKo0pjHshEQBZwJqGqmOTQSCSQKONSD/tWoXr8DqVgsbvUggVZEuoJ1GCXaWG3nwg+gy0RGOOsKCIf2fdvDhXPsub5vuDTmYslcCj4NY61sH0T409528m7RHQvPgzeFPJqq0nZe/DCmaQRCU06UanpuolsZkZpRDkPbTxUkhPZJF5oGBYlAtbLxnkN8aJwQd2JpjtB47P6eyEik1CAKTGdEdE/N1kbmf7VmqjtnrYyJJNUg6OShTsqxjvEoNBwyCVTzgQFCJTN/xbRHJKHaRGubELzZleehVip6x8XSzUm+fDmNI4f20QE6Qh46RWV0jSqoiih6QE/oFb1Zj9az9W59TFoXrOnMHvoj6+sbRf2jJQ==</latexit>

···
480000 = 1100 ⇥ w1 + 2 ⇥ w2 · · · can approximate it with small error
<latexit sha1_base64="b8tqFIe9KZOzgMh64Beo6cZY+1c=">AAADAXicjZLLahsxFIY101s6vTnpIotsRE1DoWBGY+eyCYRmk2UCdRLwGKPRHDsiGs0gnUkwg7vJq2STRUvptm+RXd+m8qVu63iRA0K/zvmEfh0pKZS0GIa/PP/R4ydPn608D168fPX6TW117cTmpRHQFrnKzVnCLSipoY0SFZwVBniWKDhNLg7G9dNLMFbm+jMOC+hmfKBlXwqOLtVb9dbjBAZSV1zJgYZ0FDS3Qhd0c48yN8coM7D0qsfoRxrNVxGNRZqjjeOgtTPno4fwUesPv72As2X49l87zQW+udTO7pxnD7GzOVXUyRh0Ou9Dr1YPG+Ek6H3BZqJOZnHUq93FaS7KDDQKxa3tsLDAbsUNSqFgFMSlhYKLCz6AjpOaOyvdavKCI/reZVLaz40bGukk+++OimfWDrPEkRnHc7tYGyeX1Tol9ne7ldRFiaDF9KB+qSjmdPwdaCoNCFRDJ7gw0nml4pwbLtB9msA1gS1e+b44iRqs2YiOW/X9T7N2rJAN8o58IIzskH1ySI5Imwjvi3fjffW++df+rf/d/zFFfW+25y35L/yfvwGHKeGu</latexit>

···
Example: house price prediction
• When we want to estimate the house price for a new house on market
• Feed the request into our function with estimated parameters

770000 = 1800 ⇥ 10 + 3 ⇥ 1.2 · · ·

<latexit sha1_base64="9ICLQNj79MYElthlX8Y2HdthoU4=">AAACNHicdVDLSgMxFM3Ud31VXboJFkEQykwV6kYQ3QhuFOwDOqVk0ts2NJMZkjtCGeaj3PghbkRwoYhbv8G0dqGtHgg5nHMfyQliKQy67rOTm5tfWFxaXsmvrq1vbBa2tmsmSjSHKo9kpBsBMyCFgioKlNCINbAwkFAPBhcjv34H2ohI3eIwhlbIekp0BWdopXbhqlJxLegp9U7s5aMIwdDUH09OexpAZZ6b0UN69I9ZKmfU550ITbtQdEvuGHSWeBNSJBNctwuPfifiSQgKuWTGND03xlbKNAouIcv7iYGY8QHrQdNSxez6VjrentF9q3RoN9L2KKRj9WdHykJjhmFgK0OGfTPtjcS/vGaC3ZNWKlScICj+vaibSIoRHSVIO0IDRzm0hHEt7Fsp7zPNONqc8zYEb/rLs6RWLnlHpfLNcfHsfBLHMtkle+SAeKRCzsgluSZVwsk9eSKv5M15cF6cd+fjuzTnTHp2yC84n1/HA6iu</latexit>
 Machine Learning
• De nition of Machine Learning Train or t a model

A set of methods that can automatically detect patterns in data and then
use the uncovered patterns to predict future data .
Predict or inference

Categorical: Classi cation

Features Labels
1 Continuous: Regression

• Given a training
1 X
dataset Dtrain = {(xi , yi )} , we aim to learn a model fw such <latexit sha1_base64="p7j4gVT2OC/ll+6aLeXes9lgK/4=">AAAB9XicbVDLSsNAFL2pr1pfUZduBovgqiRV0GXRjcsK9gFtLJPppB06mYSZiaWE/IcbF4q49V/c+TdO2iy09cDA4Zx7uWeOH3OmtON8W6W19Y3NrfJ2ZWd3b//APjxqqyiRhLZIxCPZ9bGinAna0kxz2o0lxaHPacef3OZ+54lKxSLxoGcx9UI8EixgBGsjPQaDtB9iPfaDdJplA7vq1Jw50CpxC1KFAs2B/dUfRiQJqdCEY6V6rhNrL8VSM8JpVuknisaYTPCI9gwVOKTKS+epM3RmlCEKImme0Giu/t5IcajULPTNZB5RLXu5+J/XS3Rw7aVMxImmgiwOBQlHOkJ5BWjIJCWazwzBRDKTFZExlphoU1TFlOAuf3mVtOs196JWv7+sNm6KOspwAqdwDi5cQQPuoAktICDhGV7hzZpaL9a79bEYLVnFzjH8gfX5A0a+kwQ=</latexit>

that error is minimized.

<latexit sha1_base64="yB3Smlqj+5EQUQ/+vP1DPyfafsM=">AAACE3icbVBNS8NAEN34WetX1aOXxSKoSEmqoBehqAePFWwrNCFsthtdutmE3Ym0hPwHL/4VLx4U8erFm//GTduDXw8GHu/NMDMvSATXYNuf1tT0zOzcfGmhvLi0vLJaWVtv6zhVlLVoLGJ1HRDNBJesBRwEu04UI1EgWCfonxV+544pzWN5BcOEeRG5kTzklICR/MreuZ+5wAaQgSJc5jk+wW6240YEboMwG+Q+38dDn++6uV+p2jV7BPyXOBNSRRM0/cqH24tpGjEJVBCtu46dgJcRBZwKlpfdVLOE0D65YV1DJYmY9rLRTzneNkoPh7EyJQGP1O8TGYm0HkaB6Sxu1b+9QvzP66YQHnsZl0kKTNLxojAVGGJcBIR7XDEKYmgIoYqbWzG9JYpQMDGWTQjO75f/kna95hzU6peH1cbpJI4S2kRbaAc56Ag10AVqohai6B49omf0Yj1YT9ar9TZunbImMxvoB6z3L/9znjY=</latexit>

L(fw (xi ), yi )
|D| 2
i
3
<latexit sha1_base64="kk1CPlkNU32b6cZnq08pBq5USlo=">AAACJ3icbVDLSsNAFJ3UV62vqEs3wSJUkJJUQVdS1IULFxXsA5oSJtNJO3TyYGaihjR/48ZfcSOoiC79EydpBG09MMOZc+5l7j12QAkXuv6pFObmFxaXisulldW19Q11c6vF/ZAh3EQ+9VnHhhxT4uGmIILiTsAwdG2K2/boPPXbt5hx4ns3Igpwz4UDjzgEQSElSz01HQZRbCTx+GKcmDx0LWK6UAwRpPFVUnGs7GU78V1S+aH3iUX2DyJ5WWpZr+oZtFli5KQMcjQs9cXs+yh0sScQhZx3DT0QvRgyQRDFSckMOQ4gGsEB7krqQRfzXpztmWh7Uulrjs/k8YSWqb87YuhyHrm2rEwn5dNeKv7ndUPhnPRi4gWhwB6afOSEVBO+loam9QnDSNBIEogYkbNqaAhlcEJGW5IhGNMrz5JWrWocVmvXR+X6WR5HEeyAXVABBjgGdXAJGqAJEHgAT+AVvCmPyrPyrnxMSgtK3rMN/kD5+gZy8qd6</latexit>

•
Question: For house price
Murphy, K. P. (2012). Machine learning: a probabilistic perspective. MIT press.
prediction problem, what is
the type of the task?
Classi cation or Regression?
fi
fi
fi
fi
Linear Regression
• Linear projection from feature space to target space
• Example:
Preference level

Queens
Age

User info
Linear Regression
• Training Objective
SLIM / Linear Recommender
• Well, we don’t have user info
• User user rating matrix R as input X
• Predict user ratings matrix R as Y

Queens
Beetles

Why zero out diagonal matrix

SLIM / Linear Recommender
• Intuitively, what we are doing here is

1 ?
1 ? ? Features

1 ?
1 ? ?1 Labels

? 1 ?
1 ? Data point

? ? 1 ? ?
SLIM / Linear Recommender
• Well, we don’t have user info
• User user rating matrix A as input X
• Predict user ratings matrix A as Y

Queens
Beetles

Distributed training!

Question: How many models

you need for making
recommendation?
SLIM / Linear Recommender
SLIM / Linear Recommender
• Advantage: we can put something else into rating matrix
SLIM / Linear Recommender
SVD Decomposition
• We just spent a lot of me looking at interes ng consequences of the observa on that any
T
symmetric matrix A can be factorized as A = QDQ for some orthogonal matrix Q and some
diagonal matrix D

• Is there a generaliza on of this for arbitrary matrices?

• Yes! It is based on the same eigenvectors/eigenvalues construc on as above, but it isn’t
symmetric
ti
ti
ti
ti
ti
SVD Decomposition
(m×n) T (m×m)
• Every matrix A ∈ R can be factorized as A = UΣV where U ∈ R
(n×n) (m×n)
and V ∈ R are orthogonal matrices and Σ ∈ R is a diagonal matrix

• The diagonal entries of A are called the singular values of the matrix A and
T T
they are the square roots of the eigenvalues of A A and AA
T
• U are orthonormal eigenvectors of AA
T
• V are orthonormal eigenvectors of A A
Projected Linear Recommender
• Reduce feature dimension for linear recommender

Eigenvector Matrix from SVD

• You can also do user-based projection

AutoRec
• Using simple auto-encoder as bottleneck to generalize recommendation
(probably the rst neural recommender system)
fi
AutoRec
• Using simple auto-encoder as bottleneck to generalize recommendation
(probably the rst neural recommender system)

Encoder

Decoder
fi
Denoised Autoencoder

900+ citations
Variational Autoencoder
• Control the user representation to t gaussian distribution

fi
Variational Autoencoder
• Control the user representation to t gaussian distribution

fi
Performance comparison so far
Take home points
• Linear Recommender — Simple, interpretable
• Projected Linear Recommender — More e cient due to reducing feature dim
• Autoencoder based Recommender — The simplest neural recommender
• Denoise Autoencoder for Recommendation — High citation, doesn’t work
well in practice

• Variational Autoencoder for Recommendation — State of the art, but

computationally expensive

ffi