Topic Modeling You shall know a word by the company it keeps.

British Linguist [Link] (1957)

Shankar Venkatagiri
Given a huge collection of documents, discover what the documents are "about" without manually reading and labeling every document.
T he topics are not necessarily explicitly given.

For example, suppose we have 10,000 news articles. We don't tell the algorithm:

Topic 1 = politics
Topic 2 = sports
Topic 3 = business
Topic 4 = technology

Instead, the algorithm looks at the words that occur together and discovers patterns.
Context

❖ NLP tasks involve large “corpora”

❖ Applications: Topic modelling, entity extraction, question-answering, …
 Corpus
TDM
D 1: dog bites man
D 2: man bites dog
D 3: dog eats meat
D 4: man eats food

❖ To process text in NLP, we must represent words by numbers

❖ E.g. tf_idf = tf * idf
❖ Term Document Matrix has TF-IDF scores for every term across the corpus
❖ Applications: Semantic search, based on meaning of words, summarisation
❖ Technique - Latent Semantic Analysis
Drawback

❖ Words and topics in NLP are represented by numbers like TF-IDF scores
❖ Application: Relevance ranking of search results
❖ Problem: Two document texts with the same overall meaning (even synonyms!)
are likely to have different TF-IDF vectors
❖ Stemming and lemmatisation only help partially
❖ Cosine similarity may not detect this!
❖ Idea: Linear combinations of words that make up the dimensions of topic
vectors are powerful representations of meaning
Instead of representing a document directly using individual words, we want to represent it using latent topics.
Matrices A=
[ 1
3
2
3
]

❖ A matrix A is a rectangular array of numbers

❖ Matrices represent geometric transformations
❖ Multiplying the unit circle by A can typically results in an ellipse
❖ Certain matrices represent dilations, rotations, projections, …
❖ If A represents a rotation, then the images fall back on the circle
❖ If A represents a projection, then all images lie on an axis!
Eigen

❖ Video: EigValueVector
❖ Occasionally, the direction of x and Ax may coincide
❖ May be with a stretch/contraction/flip
❖ Such directions are called eigenvectors
❖ The stretch/compression factors are eigenvalues
SVD

❖ Singular value decomposition

❖ Data compression technique
❖ Let M be any matrix (dataset) of order m x n
❖ SVD helps express M as a product U Σ VT
Animation Courtesy:
❖ U contains eigenvectors of M MT Lucas Vieira, Wikipedia

❖ V contains eigenvectors of MT M
❖ Σ is a matrix of “singular” values in the same shape as M
❖ Geometrically, U and V are rotation matrices
So U contains important directions associated with the rows of M and V contains important directions

associated with the columns of M.

Σ contains the singular values. These singular values tell us how important the corresponding components are.
 Compression
Suppose, the full SVD has 20

components. Instead of keeping all 20,

we might keep only the first 5.

The components are not equally

important.

The larger singular values generally

correspond to the components that

capture more of the

structure/information in the matrix.

Expand M = u1 𝜎1 v1T + u2 𝜎2 v2T + …

❖ Video: SVDSong
❖

Each term represents one component of the matrix.

❖ Truncate for “lower rank” approximations
❖ SVD captures essence, ignores noise
Application
Image Courtesy: Malaika Shankar
Instead of storing all (352*297) 104,544 numbers for this image, SVD

lets us approximate the image using only a limited number of rank

components.

For one SVD component we need:

U(i): 352 numbers

Sigma(i): 1 number

V(i): 297 numbers

Total: 750 numbers

Increasing the rank improves the reconstruction because we retain

Grayscale Image: 352 x 297 more SVD components.

Application

Image Courtesy: Chetna Parihar, Deepashikha Panda, Peeyush Pandey

Image Courtesy: Malaika Shankar

Grayscale Image: 352 x 297

= 104,544 numbers

Each added rank involves

352 + 1 + 297 = 750 numbers
Collinearity Filename:
[Link]
Remedy

❖ When the predictors are highly correlated, a regression model fails

❖ Coefficients are insignificant a different sample may drive them to 0

❖ Can eliminate correlated predictor(s)

❖ E.g. Regress Y against (X2, X3)
❖ In reality, predictors can be multiply correlated!
❖ Fix: Carry out a PCA…
 X2
Z2
Z1
PCA
X2
X1

Z2

X1 Z1

❖ Q: Which re-orientiation best reduces dimensionality?

❖ Hint: Search for an axis that maximises the variance of projections
❖ Axes Z1 and Z2 are called principal axes
❖ The projections are called principal component scores
 Z2
Example Z1

❖ Principal Component Analysis

❖ Mean-center the columns to get matrix N
❖ Carry out an SVD and obtain N = U Σ VT
❖ Columns of V are along the principal axes (Z1, Z2)
❖ PC scores (projections) are given by U Σ
StackExchange: Further discussion
 Z2
Approach Z1

Image Courtesy: Athabasca University

❖ (Z1, Z2) are linear combinations of axes (X1, X2)

❖ Advantage: Can reduce dimension without much loss
❖ Q: Can we achieve the same for text data?
❖ Carry out a PCA of the Term-Document Matrix
❖ Compress original dimensions (words) into fewer (topics)
❖ Topics are like weighted colour mixes - weights can be +/-
A topic can be thought of as a combination of words with different weights.
 Z2
Interpretation Z1

❖ (X1, X2) denote single word frequencies

❖ New basis vectors (Z1, Z2) are analogous to topic vectors
❖ Combination of word frequencies
❖ E.g. “dog” and “cat” can be clubbed, in what we call “pets”
We, as humans, look at the important words/weights and decide
❖ The computer does not label it! what the component appears to represent.

❖ No need to label the topics - work directly with the representation

❖ Vector math reveals similarity of documents
 Result: Optimal when k = (a, b) 𝜶 difference vector of the group means
❖ Q: Which linear combination of X1 & X2 discriminates the two groups best?
❖

❖ Approach: With dataset X, we calculate “discriminant scores” t = X k

 Image Courtesy: Hobson, et al.

LDA

❖ Linear Discriminant Analysis

❖ Classifier, supervised ML
❖ Given: TF-IDF vectors for a collection of documents. Must predict the group
each document belongs to
1. Find out the group centroid that the document is closest to (X k)
2. Rescale the discriminant scores between 0 and 1, like a probability
3. Set a threshold for the scores and classify
Application

Accuracy = 97.75%

❖ Spam detector using LDA - open LDA_Spam.ipynb

❖ Corpus: 4,837 messages of which 638 are spam, labelled with !
❖ Adjust the threshold if FNs and FPs are out of balance
❖ Advantage: Semantic analysis does not rely on individual words
 Corpus

Topics D 1: I like to eat broccolis and bananas

D 2: I ate a banana and salad for breakfast
D 3: Puppies and kittens are cute
D 4: My sister adopted a kitten yesterday
D 5: Look at this cute hamster munching on
a piece of broccoli

❖ A topic model learned on this corpus can produce this output

❖ Topic A: 30% broccoli, 15% bananas, 10% breakfast, 10% munching
❖ Topic B: 20% puppies, 20% kittens, 20% cute, 15% hamster
❖ Likewise, each document is a mixture of topics
❖ D1 & D2: 100% Topic A, D3 & D4: 100% Topic B
❖ D5: 60% Topic A, 40% Topic B
 W1 W2 W3 … Wn T1 T2 … Tk
LDiA D1 D1
W1 W2 W3 … Wn

D2 D2 T1
D3 D = D3 M1 T2
M2
… … …
Dm Dm Tk
kxn
mxn mxk
m documents with n words m documents with k topics k topics with n words

❖ Latent Dirichlet Allocation

❖ Unsupervised ML technique that factorises the document-term matrix D of
the corpus to produce
❖ M1: Document-topic matrix
❖ M2: Topic-term matrix
❖ The number of topics k is “latent” - specified up-front while modelling
Corpus

❖ We work with the CMU Book Summary dataset

❖ Plot summaries for 16,559 books
Application

❖ We perform a Latent Dirichlet Allocation on the CMU Book Summaries

❖ Open up LDA_LSA.ipynb
❖ Must reverse-engineer the topic’s focus from the words and weights
LSA

❖ Latent Semantic Analysis (aka Latent Semantic Indexing)

❖ Principal Component Analysis applied to text data
❖ Apply SVD of the TF-IDF matrix to get M = UDVT
❖ U is the term-topic matrix
❖ Truncate the matrices to reduce dimensions
❖ New representation contains the “essence” of the documents
Summarisation

❖ Often, we want to summarise a large document

Extractive: Original document -> Find

❖ Result must contain key ideas of the text important sentences -> Copy those

sentences -> Summary

❖ Extractive versus abstractive

Abstractive: Original document ->

❖ Select important segments versus create an abstract Understand key ideas -> Generate

new sentences -> Summary

❖ Query-focused versus query-independent

❖ The first narrows the response to the query
❖ Single- versus multiple-document
autotldr

❖ Single-doc, query-independent, extractive

Control

❖ gensim library allows for better control

❖ word_count: maximum amount of words we want in the summary
❖ ratio: fraction of sentences in the original text that should be returned
Output

❖ word_count = 200, 100

Debate

❖ Noam Chomsky, the father of linguistics, vehemently disagrees with using

probability/statistics to make sense of language
❖ "It's true there's been a lot of work on trying to apply statistical models to
various linguistic problems. I think there have been some successes, but a lot
of failures. There is a notion of success ... which I think is novel in the history of
science. It interprets success as approximating unanalyzed data.”
❖ Video: Chomsky On Statistical NLP (upto 4:12), on Big Data
Counter

❖ Peter Norvig, a noted AI researcher, begs to differ

❖ Search engines: 100% of major players are trained and probabilistic. Their
operation cannot be described by a simple function.
❖ Speech recognition: 100% of major systems are trained and probabilistic,
mostly relying on probabilistic hidden Markov models.
❖ Machine translation: Of the 4000 language pairs covered by MT systems, a
statistical system is by far the best for every pair except Japanese-English…
❖ Question answering: … many systems build heavily on the statistical and
probabilistic approach used by search engines.
References

❖ Hobson, Howard and Hawke. NLP in Action. Manning Publications (2019)