See discussions, stats, and author profiles for this publication at: [Link]

net/publication/371922407

Neural Dynamic Focused Topic Model

Article in Proceedings of the AAAI Conference on Artificial Intelligence · June 2023

DOI: 10.1609/aaai.v37i11.26496

CITATIONS READS
0 8

3 authors:

Kostadin Cvejoski Ramsés J. Sánchez

Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS University of Bonn
34 PUBLICATIONS 91 CITATIONS 18 PUBLICATIONS 71 CITATIONS

SEE PROFILE SEE PROFILE

César Ojeda
Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS
39 PUBLICATIONS 143 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Resource-aware Learning View project

Lamarr Institute for Machine Learning and Artificial Intelligence View project

All content following this page was uploaded by Kostadin Cvejoski on 13 July 2023.

The user has requested enhancement of the downloaded file.

 The Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI-23)

Neural Dynamic Focused Topic Model

Kostadin Cvejoski1, 2 , Ramsés J. Sánchez1, 4 , César Ojeda3
1
Lamarr-Institute for Machine Learning and Artificial Intelligence
2
Fraunhofer-Institute for Intelligent Analysis and Information Systems (IAIS)
3
University of Potsdam
4
BIT University of Bonn
[Link]@[Link], sanchez@[Link], ojedamarin@[Link]

Abstract and social media content not only feature trends and themes
that change with time, but also employ their language differ-
Topic models and all their variants analyse text by learning ently as time evolves (Danescu-Niculescu-Mizil et al. 2013).
meaningful representations through word co-occurrences. As
pointed out by previous work, such models implicitly assume
The exchangeability assumption along the time component
that the probability of a topic to be active and its proportion is hence inappropriate in these cases and topic models have
within each document are positively correlated. This correla- been extended to account for changes in both topic (Blei and
tion can be strongly detrimental in the case of documents cre- Lafferty 2006; Wang, Blei, and Heckerman 2012; Jähnichen
ated over time, simply because recent documents are likely et al. 2018) and word (Bamler and Mandt 2017; Rudolph and
better described by new and hence rare topics. In this work Blei 2018; Dieng, Ruiz, and Blei 2019) distributions, among
we leverage recent advances in neural variational inference documents collected over long periods of time.
and present an alternative neural approach to the dynamic Fo- It is easy to imagine, however, that if one analyses the col-
cused Topic Model. Indeed, we develop a neural model for
lection’s content as one moves forward in time, one would
topic evolution which exploits sequences of Bernoulli ran-
dom variables in order to track the appearances of topics, find that (some of) the topics describing those documents
thereby decoupling their activities from their proportions. We appear, disappear or reappear with time. This simple intu-
evaluate our model on three different datasets (the UN gen- ition entails that one should not only model the time- and
eral debates, the collection of N EUR IPS papers, and the ACL document-dependent topic proportions, but also the proba-
Anthology dataset) and show that it (i) outperforms state- bilities for the topics to be active, and how such probabili-
of-the-art topic models in generalization tasks and (ii) per- ties change with time. Previous work has already pointed out
forms comparably to them on prediction tasks, while employ- that existing topic models implicitly assume that the proba-
ing roughly the same number of parameters, and converging bility of a topic being active and its proportion within each
about two times faster. document are positively correlated (Williamson et al. 2010;
Perrone et al. 2017). This assumption is generally unwanted,
Introduction simply because rare topics may account for a large part of
the words in the few documents in which they are active.
Probabilistic topic models, the likes of Latent Dirichlet Al- It is particularly detrimental (for both modelling and pre-
location (LDA) (Blei, Ng, and Jordan 2003), are generative diction) in a dynamic setting, because recent documents are
models of word co-occurrence that analyse large document likely better described by new and hence rare topics.
collections by learning latent representations (topics) encod-
Indeed, whenever the topic distribution over documents is
ing their themes. These models represent the documents of
strongly skewed, topic models tend to learn the more gen-
the collection as mixtures of latent topics, and group seman-
eral topics held by the big majority of documents in the col-
tically related words into single topics by means of word-
lection, rather than the rare topics contained only by fewer
pair frequency information within the collection. Such a
documents (Jagarlamudi, Daumé III, and Udupa 2012; Tang
generic generative structure has been successfully applied
et al. 2014; Zuo, Zhao, and Xu 2014). Document collections
to problems ranging from information retrieval, visualiza-
that reflect evolving content typically feature skew topic dis-
tion and multilingual modelling to linguistic understanding
tribution over its documents, with the newly added docu-
in fiction and non-fiction, scientific publications and politi-
ments being well described by new, rare topics. Dynamic
cal texts (see e.g. Boyd-Graber et al. (2017) for a review),
topic models that feature the topic proportion-activity cou-
and keeps being extended to new domains (Rezaee and Fer-
pling are then expected to perform badly, simply because
raro 2020; Zhao et al. 2021).
these will not be able to infer the new topics characteristic
Topic models implicitly assume that the documents within of recent documents. To properly model such recent docu-
a given collection are exchangeable. Yet document collec- ments one should therefore allow rarely seen topics to be
tions such as magazines, academic journals, news articles active with high proportion and frequently seem topics to be
Copyright © 2023, Association for the Advancement of Artificial active with low proportion.
Intelligence ([Link]). All rights reserved. In this work we seek to decouple the probability for a

12719
topic to be active from its proportion with the introduction
of sequences of Bernoulli random variables, which select
the active topics for a given document at a particular instant
of time. Earlier models attained such a decoupling via non-
parametric priors, such as the Indian Buffet Process prior
over infinite binary matrices, in both static (Williamson et al.
2010) and dynamic (Perrone et al. 2017) settings. Our con-
struction roughly follows a similar logic, but leverages the
reparametrization trick to perform neural variational infer-
ence (Kingma and Welling 2013). The result is a scalable
model that allows the instantaneous number of active topics
per document to fluctuate, and explicitly decouples the topic
proportion from its activity, thereby offering some novel lay-
ers of interpretability and transparency into the evolution of
topics over time.
We introduce the Neural Dynamic Focused Topic Model Figure 1: Graphical model representation of NDF-TM.
(NDF-TM) which builds on top of Neural Variational Topic
models (Miao, Yu, and Blunsom 2016) and uses Deep
Kalman Filters (Krishnan, Shalit, and Sontag 2015) to model the dynamic topic embeddings. As such, this model corre-
the independent dynamics of both topic proportion and topic sponds to the dynamic extension of Dieng, Ruiz, and Blei
activities. We train and test our model on three datasets, (2020).
namely the UN general debates, the collection of N EUR IPS Neural topic models. Another line of research leverages
papers and the ACL Anthology dataset. Our results show via neural networks to improve the performance of topic mod-
different metrics that NDF-TM outperforms state-of-the-art els, the so-called neural topic models (Miao, Yu, and Blun-
topic models in generalization tasks, and performs compara- som 2016; Srivastava and Sutton 2017; Zhang et al. 2018;
bly to them on prediction tasks. Very importantly, NDF-TM Dieng, Ruiz, and Blei 2020, 2019) which deploy neural vari-
does this while employing roughly the same number of pa- ational inference (Kingma and Welling 2013) for training.
rameters and converging two times faster than the strongest Decoupling topic activity from its proportion.
baseline. Williamson et al. (2010) noted the implicit and undesirable
correlation between topic activity and proportion assumed
Related Work by standard topic models and introduced the Focused Topic
The NDF-TM model merges concepts from dynamic topic Model (FTM). FTM uses the Indian Buffet Process (IBP) to
models, dynamic embeddings and neural topic models. decouple across-data prevalence and within-data proportion
Dynamic topic models. The seminal work of Blei in mixed membership models. Later Perrone et al. (2017)
and Lafferty (2006) introduced the Dynamic Topic Model extended FTM to a dynamic setting by using the Poisson
(DTM), which uses a state space model on the natural pa- Random Fields model from population genetics to generate
rameters of the distribution representing the topics, thus al- dependent IBPs, which allow them to model temporal
lowing the latter to change with time. The DTM methodol- correlations in data. Both of these models are trained using
ogy was first extended by Caron, Davy, and Doucet (2007) to complex sampling schemes, which can make the fast and
a nonparametric setting, via the correlation of Dirichlet pro- accurate inference of their model parameters difficult (Miao,
cess mixture models in time. Later Wang, Blei, and Hecker- Grefenstette, and Blunsom 2017).
man (2012) replaced the discrete state space model of DTM In what follows we propose an alternative neural ap-
with a Diffusion process, thereby extending the approach to proach to the dynamic Focused Topic model of Perrone et al.
a continuous time setting. Jähnichen et al. (2018) further ex- (2017), trainable via backpropagation, which learns to de-
tended DTM by introducing Gaussian process priors that al- couple the dynamic topic activity from its dynamic topic
lowed for a non-Markovian representation of the dynamics. proportion.
Other recent work on dynamic topic models is that of Hida
et al. (2018) Neural Dynamic Focused Topic Model
Dynamic embeddings. Rather than modelling the con-
tent evolution of document collections like DTM, other Suppose we are given an ordered collection of corpora D =
works focus on modelling how word semantics change with {D1 , D2 , . . . , DT }, so that the tth corpus Dt is composed
time (Bamler and Mandt 2017; Rudolph and Blei 2018). of Nt documents, all received within the tth time window.
These works use continuous representation of words captur- Let Wt denote the Bag-of-word (BoW) representation for
ing their semantics (as e.g. those of Pennington, Socher, and the whole document set within Dt and let wt,d denote the
Manning (2014)) and evolve such representation via diffu- BoW representation of the d-th document in Dt .
sion processes. More recently, Dieng, Ruiz, and Blei (2019) Let us now suppose that the corpora collection is de-
represent topics as dynamic embeddings, and model words scribed by a set of K unknown topics. We then assume
via categorical distributions whose parameters are given by there are two sequences of continuous hidden variables
the inner product between the static word embeddings and η 1 , . . . , η T ∈ Rdim(η) and ξ 1 , . . . , ξ T ∈ Rdim(ξ) which en-

12720
code, respectively, how the topic proportions and the topic where zt,d,n is the time-dependent topic assignment for
activities change among corpora as time evolves (i.e. as one wt,d,n , which labels the nth word in document d ∈ Dt , and
moves from Dt to Dt+1 ). That is, η t and ξ t encode the β ∈ RK×V is a learnable topic distribution over words. We
global dynamics of semantic content. We also assume there define the latter as
are two local hidden variables, conditioned on the global β = softmax(α ⊗ ρ), (9)
ones, namely a continuous variable ζ t,d ∈ RK which en-
codes the content of the dth document in Dt , in terms of the with α ∈ RK×E , ρ ∈ RV ×E learnable topic and word em-
available topics, and a binary variable bt,d ∈ {0, 1}K which beddings, respectively, for some embedding dimension E,
encodes which topics are active in the document in question. and ⊗ denoting tensor product.
We combine these local variables to compute the topic pro- NDF-TM is summarized in Figure 1.
portions θ t,d ∈ [0, 1]K from which the dth document in Dt
is generated. Inference
The generative model above involves two independent
Generation global hidden variables ξ t , η t , and two local hidden vari-
Let us denote with ψ the set of parameters of our genera- ables ζ t,d and bt,d . Our task is to infer the posterior distri-
tive model. We are first of all interested in modelling the butions of all these variables.1 Denoting with Γt,d the set
topic activity per document at each time step, directly from {ξ t , η t , ζ t,d , bt,d }, we approximate the true posterior distri-
the data. One could, for example, use a K-dimensional mask bution of the model with a variational (and structured) pos-
(i.e. a K-dimensional vector, whose kth entry is either 1 or 0 terior of the form
depending on whether the kth topic is active or inactive) for
each document d, at each time step t. To account for the vari- qφ (Γt,d |wt,d , W1:T ) =
ability of the data, one could also make this mask stochas- T
Y
tic. We thus introduce K time- and document-dependent qφ (η t |η 1:t−1 , W1:T ) qφ (ξ t |ξ 1:t−1 , W1:T )
Bernoulli variables bt,d ∈ {0, 1}K whose generation pro- t
cess is given by Nt
Y
  × qφ (ζ t,d |wt,d , η t ) qφ (bt,d |wt,d , ξ t ), (10)
ξ t ∼ N µξψ (ξ t−1 ), δ I , (1)
d
π t = α0 Sigmoid (Wξ ξ t + cξ ) , (2) where W1:T = (W1 , . . . , WT ) is the ordered sequence of
bt,d ∼ Bernoulli(π t ), (3) BoW representations for the corpus collection and φ labels
the variational parameters.
where α0 is a hyperparameter controlling the percentage of
Local variables. The posterior distribution over the local
active topics, and Wξ ∈ RK× dim(ξ) , cξ ∈ RK ⊂ ψ are variables ζ t,d , bt,d are chosen as Gaussian and Bernoulli,
trainable parameters. Also note that, just as in Deep Kalman respectively, each parametrized by neural networks taking
Filters (Krishnan, Shalit, and Sontag 2015), ξ t is Markovian as input their conditional variables.
and evolves under a Gaussian noise with mean µξψ , defined Global variables. The posterior distribution over the dy-
via a neural network with parameters in ψ, and variance δ. namic global variables ξ t , η t are also Gaussian, but now de-
The latter being a hyperparameter of the model. Finally, we pend not only on the latent variables at time t − 1, but also
choose ξ 1 ∼ N (0, 1). on the entire sequence of BoW representations W1:T . This
Analogously, we generate the topic proportions θt,d as follows directly from the graphical model in Figure 1, as
  noted by e.g. Krishnan, Shalit, and Sontag (2015). We shall
ηt ∼ N µηψ (η t−1 ), δ I , (4) use LSTM networks (Hochreiter and Schmidhuber 1997) to
model these dependencies. Specifically let
ζ t,d ∼ N (Wζ η t + cζ , 1) , (5) qφ (ξ t |ξ t−1 , W1:T ) = N (µξφ , σ ξφ ), (11)


bt,d ⊙ exp ζ t,d where µξφ , σ ξφ are neural networks which take as input the
θt,d = PK k  , (6)
k
k bt,d ⊙ exp ζt,d pair (ξ t−1 , hξt ), with hξt a hidden representation encoding
the sequence W1:T . Similarly
where bt,d is defined in (3) and ⊙ labels element-wise prod-
uct, Wζ ∈ RK× dim(η) , cζ ∈ RK ⊂ ψ are trainable, and qφ (η t |η t−1 , W1:T ) = N (µηφ , σ ηφ ), (12)
µηψ is modelled via a neural network. Here η t is also Marko- η η
where µφ , σ φ , again neural networks, take as input the pair
vian and we set η 1 ∼ N (0, 1). Note that the topic propor- (η t−1 , hηt ), with hηt a second hidden representation also en-
tion thus defined can be sparse vectors. That is, the model coding W1:T . These hidden representations hit , with i =
has the flexibility to completely mask some of the topics out {ξ, η}, correspond to the hidden states of LSTM networks
of a given document, at a given time. whose update equation read
Once we have θ t,d we generate the corpora sequence by hit = fφi (Wt , hit−1 ). (13)
sampling 1
Note in passing that we do not need to perform inference of
zt,d,n ∼ Categorical(θt,d ), (7) the latent topics zt,d,n , simply because these can be integrated out
wt,d,n ∼ Categorical(βzt,d,n ), (8) (aka marginalized).

12721
 UN N EUR IPS ACL
Models PPL-DC ↓ P-NLL ↓ PPL-DC ↓ P-NLL ↓ PPL-DC ↓ P-NLL ↓
DTM* 2393.5 - - - - -
DTM-REP 3 012 ± 14 8.331 ± 0.003 8 107±907 9.5 ± 0.4 8 503 ± 875 9.7 ± 0.5
D-ETM 1 748 ± 13 7.615 ± 0.005 7 746±699 8.983 ± 0.003 7 805±182 8.84 ± 0.02
NDF-LT-TM 1 578 ± 29 7.682± 0.080 6 549±21 8.923 ± 0.002 7 877±213 8.91 ± 0.03
NDF-TM 1 527 ± 36 7.640± 0.004 6 529±26 8.901 ± 0.001 7 690±215 8.88 ± 0.03

Table 1: Perplexity on document completion (PPL-DC) and predictive negative log likelihood (P-NLL). PPL-DC is calculated
by conditioning the model on the first half of the document and evaluating the perplexity on the second half of the document.
P-NLL is estimated using equation 16. The DTM* results are taken from (Dieng, Ruiz, and Blei 2019). All other results are
obtained by training the models on 5 different random splits of the datasets.

Training Objective Datasets

To optimize the model parameters {ψ, φ} we minimize We evaluate our model on three datasets, namely the collec-
the variational lower bound on the logarithm of the tion of UN speeches, N EUR IPS papers and the ACL An-
marginal likelihood pψ (wt,d,n |β). Following standard meth- thology. The UN dataset3 (Baturo, Dasandi, and Mikhaylov
ods (Bishop 2006), the latter can readily be shown to be 2017) contains the transcription of the speeches given at the
UN General Assembly during the period between the years
1970 and 2016. It consists of about 230950 documents. The
T X Nd
Nt X n o N EUR IPS dataset4 contains the collection of papers pub-
lished in between the years 1987 and 2016. It consists of
X
L[β, ψ, φ] = EΓ log pψ (wt,d,n |β, Γ)
t=1 d=1 n=1 about 6562 documents. Finally, the ACL Anthology (Bird
− KL [qφ (η 1 |W1:T ); p(η 1 )] − KL [qφ (ξ1 |W1:T ); p(ξ1 )] et al. 2008) contains a collection of computational linguistic
T
and natural language processing papers published between
−
X 
KL qφ (η t |η 1:t−1 , W1:T ); pψ (η t |η t−1 )
 1973 and 2006. It consists of about 10514 documents.
t=2
T
X  
Baselines
− KL qφ (ξt |ξ1:t−1 , W1:T ); pψ (ξt |ξt−1 ) Our main aim is to study the effect of the topic proportion-
t=2
activity coupling in the performance of dynamic topic mod-
Nt
T X
X n  o els5 on data collections displaying evolving content. To do
− Eηt KL qφ (ζ t,d |wt,d , η t ); pψ (ζ t,d |η t )
t=1 d=1
so we compare against three models:
n o
! (1) DTM-REP — the neural extension of DTM, fit-
+ Eξt KL [qφ (bt,d |wt,d , ξt ); pψ (bt,d |ξt )] , (14) ted using neural variational inference (Dieng, Ruiz, and
Blei 2019). This model uses a logistic-normal distribution,
parametrized with feedforward neural networks, as poste-
where KL labels the Kullback-Leibler divergence and β is rior for the topic proportion distribution; as in Miao, Grefen-
defined in Eq. 9. stette, and Blunsom (2017). It also uses Kalman Filters to
model the topic dynamics, but parametrizes the posterior
distribution over the dynamic latent variables with LSTM
Experiments networks, just as in Deep Kalman Filters (Krishnan, Shalit,
In this section we introduce our datasets and define our base- and Sontag 2015) (and just as NDF-TM too, see e.g. Eq. 13).
lines. Details about pre-processing and experimental setup As such, DTM-REP works as the dynamic extension of
can be found in the supplementary material, provided within Miao, Grefenstette, and Blunsom (2017)’s model. It follows
the repository of our code. Nevertheless, let us mention here that the DTM-REP model thus defined only differs from
that two important hyperparameters of the model are the NDF-TM in the way we model the topic proportions. Com-
maximum topic number K and the percentage of active paring our model against DTM-REP should therefore explic-
topics α0 . Both these hyperpameters are chosen via cross- itly show the effect of lifting the topic proportion-activity
validation, with K = 50 and α0 = 0.5 given the best re- coupling in dynamic neural topic models.
sults2 . Source code to reproduce our experiments is available (2) D-ETM — the Dynamic Embedded Topic Model (Di-
online.2 eng, Ruiz, and Blei 2019), which captures the evolution of
topics in such a way that both the content of topics and their
2
Specifically, K was chosen from the set 50, 100 and 200. We proportions evolve over time. This model adds complexity
found 50 to be the best value for all models, i.e. including the base-
3
lines. Similarly α0 was chosen from the set 0.1, 0.5, 1.0 [Link]
2 4
Source code: [Link] [Link]
5
Focused-Topic-Model This means we do not consider static topic models

12722
 UN N EUR IPS ACL
Models TC ↑ TD ↑ TC ↑ TD ↑ TC ↑ TD ↑
DTM* 0.1317 0.0799 - - - -
DTM-REP 0.11 ± 0.30 0.59 ± 0.10 -0.62±0.07 0.15±0.01 -0.82 ± 0.08 0.55 ± 0.02
D-ETM 0.43 ± 0.20 0.61 ±0.01 -0.54±0.09 0.82 ±0.01 -0.71±0.16 0.63±0.05
NDF-LT-TM 0.43 ± 0.18 0.56 ± 0.03 -0.53±0.02 0.90±0.01 -0.74±0.11 0.73±0.01
NDF-TM 0.46 ± 0.20 0.63 ± 0.01 -0.50±0.04 0.85±0.02 -0.64±0.12 0.74±0.01

Table 2: Topic coherence (TC) and Topic diversity (TD) for all models. TC is calculated by taking the average pointwise mutual
information between two words drawn randomly from the same topic. TD is the percentage of unique words in the top 25
words of all topics. The DTM* results are taken from (Dieng, Ruiz, and Blei 2019). All other results are obtained by training the
models on 5 different random splits of the datasets.

to DTM-REP by modelling words via categorical distribu- Models 0.5 0.6 0.7 0.8
tions whose parameters are given by the inner product be- WF-IBP 5.2 5.5 6.2 13.8
tween the static word embeddings and the dynamic topic D-ETM 27.2 26.8 26.8 25.1
embeddings. In this way, D-ETM does not (necessarily) suf- NDF-TM 35.3 27.8 27.8 27.3
fers from the topic proportion-activity coupling, for it can
implicitly model their decoupling via its additional degrees Table 3: Percentage (wrt. the score of the static model) of the
of freedom. PPL-DC difference between dynamic and static models on
(3) NDF-LT-TM — the Neural Dynamic Focused topic the N EUR IPS dataset, as the percentage of held-out words
model with linear transition. We introduce this last baseline was increased from 50% to 80%. Higher is better.
for the sake of ablation, viz. to investigate the effect of the
neural networks µξψ , µηψ in Eqs. 1 and 4. NDF-LT-TM is
We can now define the predictive log likelihood as
defined by replacing these neural networks with the identity
function. P-NLL = Epψ (ΓT +1 |ΓT ) Eqφ (Γ1:T |W1:T )
n o
Results log pψ (WT +1 |ΓT +1 ) . (16)
In order to quantify the performance of our models, we first
(2) To test generalization we use three metrics, namely
focus on two aspects, namely its prediction capabilities and
perplexity (PPL) on document completion, topic coherence
its ability to generalize to unseen data. Later we also (qual-
(TC) and topic diversity (TD). The document completion
itatively) discuss how the model actually performs the de-
PPL is calculated on the second half of the documents in
coupling between topic activities and proportions.
the test set, conditioned on their first half (Rosen-Zvi et al.
(1) To test how well our models perform on a prediction
2012). The TC is calculated by taking the average pointwise
task we compute the predictive negative log likelihood (P-
mutual information between two words drawn randomly
NLL). Since to our knowledge the latter does not appear
from the same topic (Lau, Newman, and Baldwin 2014) and
explicitly in the dynamic topic model literature, we briefly
measures the interpretability of the topic. In contrast, TD
revisit how to estimate it in what follows.
is the percentage of unique words in the top 25 words of all
In order to predict N steps into the future we rely on the
topics (Dieng, Ruiz, and Blei 2020). Note that one also often
generative process of our model, albeit conditioned on the
finds in the literature the topic quality metric (TQ), defined
past. Essentially, one must generate Monte Carlo samples
as the product of TC with TD.
from the posterior distribution and propagate the latent rep-
resentations (ξ t and η t in our model) into the future with Comparison With Baselines
the help of the prior transition function (Eqs. 1 and 4, re-
spectively)6 . This procedure is depicted on the conditional The results on both P-NLL and PPL tasks are shown in Ta-
predictive distribution of our model ble 1. Both our models (NDF-TM and NDF-LT-TM) outper-
form all baselines on the completion PPL metric, on all the
Z datasets. Similarly, our models outperform all baselines on
p(WT +1 |W1:T ) = pψ (WT +1 |ΓT +1 ) both the TC and TD metrics, on all datasets, as shown in
Table 2. These results (empirically) demonstrate that decou-
× pψ (ΓT +1 |ΓT )qφ (Γ1:T |W1:T )dΓ1:T , (15) pling the topic activity from the topic proportion generically
where we replaced the true (intractable) posterior with the improves the performance of topic models on generalization
approximate posterior qφ (Γ1:T |W1:T ), and where Γt,d la- tasks. In particular, we see that adding a non-linear transfor-
bels the set {ξ t , η t , ζ t,d , bt,d } as before. mation to the prior transition functions (Eqs. 1 and 4) over-
all improves the model performance (i.e. compare NDF-TM
6
Note that one is effectively performing a sequential Monte against NDF-LT-TM).
Carlo sample (Speekenbrink 2016), in which future steps are parti- Regarding the prediction task we first notice that NDF-TM
cles sampled from the posterior and propagated by the prior. outperforms DTM-REP in all datasets. As explained in the

12723
 0 0 0

10 10 10 0.9

0.8
20 20 20
Topic

Topic

Topic
0.7
30 30 30
0.6
40 40 40
0.5
1970 1980 1990 2000 2010 1987 1997 2007 2016 1973 1983 1993 2002
Year Year Year

Figure 2: Average time-dependent topic activity bt,d of all K = 50 topics in NDF-TM for UN (left), NIPS (middle) and ACL
(right) datasets.

ACL UN model of Perrone et al. (2017). In their work they evalu-

1014
1026 NDF-TM NDF-TM ated the PPL-DC on four splits of a N EUR IPS datasets.7
D-ETM D-ETM The splits differ from each other on the percentage of held-
1010
1018
PPL

PPL

out words used to define their test sets. Intuitively, the larger
1010 106 the percentage of held-out words, the more a dynamic topic
model has to rely on its inferred temporal representations.
0 100 200 0 5 10 15 20 The reported results seem however to be in a completely
Epochs Epochs different scale from those we get (e.g. their simplest, static
model yields PPL-DC values of the order of 1000, whereas
Figure 3: Learning curves for D-ETM and NDF-TM (100 our best models yield results twice as large). We there-
topics each) on the ACL and UN evaluation datasets. The fore decided to compare the difference in performance be-
mean and the 2x std are obtained by training the models on tween their dynamic WF-IBP model and their static base-
5 different random splits of the data. line, against the difference in performance between our neu-
ral dynamic models and a static LDA model (LDA-REP),
fitted with the reparametrization trick. Table 3 shows our re-
Baselines subsection, DTM-REP and NDF-TM only differ sults.
in the topic proportion-activity coupling, from which one
can infer that lifting the coupling explicitly helps when pre- Qualitative Results
dicting the content of future documents. Yet NDF-TM only
One of our main claims is that decoupling topic activity from
performs comparably to D-ETM, the strongest baseline, on
topic proportion helps the model better describe sequentially
this task. Note that D-ETM learns different embeddings for
collected data. We have seen above this is indeed the case
each topic at each time step (i.e. K ∗ T embeddings in to-
from a quantitative point of view. Nevertheless, one could
tal). One can argue that the flexibility to change the semantic
ask whether (or how) this decoupling is effectively taking
content of topics as time evolves gives D-ETM the possibil-
place as time evolves. To study how the model encodes the
ity to implicitly model rare yet relevant topics. In compari-
temporal aspects of the data, we track the time evolution of
son, NDF-TM learns only K topic embeddings, and has only
both (i) the probability for topics to be active and (ii) the
about α0 K active embeddings (in average), at each time
topic proportions. Figure 2 shows the first of these. Imme-
step. The number of parameter for both models is about the
diately we notice there is much more structure on the topic
same however, because NDF-TM embeds the (fairly large)
activities in both the N EUR IPS and ACL datasets, as com-
BoW vectors for the inference of its two global variables.
pared to the UN dataset. We can understand these findings
Learning a single, global embedding for these BoW vectors
by arguing (a posteriori) that N EUR IPS and ACL feature
would lower the number of needed parameters in NDF-TM,
more emergent and volatile topics (wrt. their activity) as
way below those needed in D-ETM, and we shall explore
compared to those characteristic of the UN dataset. Typi-
such an approach in the future. Nevertheless, in practice, and
cally, (dynamic) topic models fitted on the UN dataset tend
as shown in Figure 3, NDF-TM converges ∼2.8x faster than
D-ETM in the ACL dataset (left figure). It also converges 7
Note that this dataset is different from the N EUR IPS dataset
∼2x faster than D-ETM in the UN dataset (right figure), and
in our main experiments. We only used this new one to com-
this is the worst case we have observed. Thus, ultimately, pare against Perrone et al. (2017). The dataset is available at
NDF-TM is more efficient than D-ETM. [Link]
We have also tried to compare against the non-parametric 1987-2015.

12724
 UN 0.75
0.035
3

θ - Probability

b - Probability
0.030 0.70
Entropy

2
0.025
0.65
1 0.020
1970 1980 1990 2000 2010 1970 1980 1990 2000 2010
Year Year
0
1970 1980 1990 2000 2010
Year Figure 5: Evolution of topic proportion and activity proba-
NeurIPS bility for the topic middle east inferred from the UN dataset
via NDF-TM.
3
pling) for UN is close to zero, meaning that DTM-REP usu-
Entropy

2 ally describes the documents with few topics, whereas for

N EUR IPS the entropy of the average topic distribution is
1 close to its maximum value (log(K = 50) ≈ 3.9), mean-
ing that it allocates almost equal probability for all K topics
0 (that is, the model needs all topics to fit the data well), as ex-
1987 1992 1997 2002 2007 2012 2017 pected for a skew topic distributions. In contrast, NDF-TM
Year uses the additional Bernoulli variable sequences to redis-
tribute the noise in the topic dynamics. Note also how the
Figure 4: Entropy of topic distribution inferred by DTM-REP topic entropy of D-ETM is often similar to that of NDF-TM,
(squares), D-ETM (triangles) and NDF-TM (circles), aver- meaning D-ETM does in fact implicitly lift the proportion-
aged over documents as time evolves. Values shown with activity coupling.
one standard deviation for both UN (above) and N EUR IPS Figure 5 shows our results for one topic inferred from the
(below) datasets. Note that the maximum entropy value is UN dataset, namely middle east. Note, for example, that the
log(K = 50) ≈ 3.9. The inset shows the details of the time- topic proportion for this topic peaks in the year 1990, which
dependent topic-proportion entropy featured by NDF-TM. coincides with the Gulf War, to then drop right after. Such
Note how the entropy decreases with time in the UN dataset a drop is also reflected in the topic activity. Later, in 2011,
(leaving aside the pick around the window 2005-2010) but the Syrian Civil War started. This event is captured by the
fluctuates strongly for the (skewed) N EUR IPS dataset. topic activity which peaks at 2011, even though the topic
proportion probability is decreasing. That is, even when the
proportion of the middle east topic is low within the docu-
to infer topics which circle about e.g. war, peace or climate. ments of that year, it must remain active to properly describe
In contrast, topic models trained on, say, N EUR IPS, generi- the data.
cally infer more varied topics, ranging from e.g. Neural Net-
works and their training to Reinforcement Learning. See, Conclusion
for example, Table 6 in the supplementary material provided We have introduced the Neural Dynamic Focused Topic
within the repository of our code, which shows six randomly Model for sequentially collected data, which explicitly de-
sampled topics from each dataset as inferred by NDF-TM. couples the dynamic topic proportions from the topic activi-
It is easy to imagine that the more generic topics in ties through the addition of sequences of Bernoulli variables.
the UN dataset (like war, climate, etc) have reached some We have shown that our approach consistently yields co-
type of equilibrium and thus display overall a less skewed herent and diverse topics, which correctly capture historical
distribution over the document collection. If this were the events. Future work includes using NDF-TM together with
case, topic models featuring the proportion-activity cou- Variational Autoencoders for topic-guided text generation.
pling would fit well the data by only inferring the more
generic topics. Figure 4 shows the (Shannon) entropy of the
topic distribution, averaged over documents as time evolves, Acknowledgments
as inferred by all models.8 Note how the entropy inferred This research has been funded by the Federal Ministry of
by DTM-REP (which features the proportion-activity cou- Education and Research of Germany and the state of North-
Rhine Westphalia as part of the Lamarr-Institute for Ma-
8
The Shannon entropy of the topic distribution per document
P (i)
chine Learning and Artificial Intelligence, LAMARR22B.
(i) (i)
and time is defined here by Ht,d = − Ki θt,d log θt,d , where θt,d César Ojeda is supported by Deutsche Forschungsge-
is the ith component of θ t,d . meinschaft (DFG) - Project-ID 318763901 - SFB1294.

12725
 References Krishnan, R. G.; Shalit, U.; and Sontag, D. 2015. Deep
Bamler, R.; and Mandt, S. 2017. Dynamic word embed- Kalman Filters. arXiv:1511.05121.
dings. In Proceedings of the 34th International Conference Lau, J. H.; Newman, D.; and Baldwin, T. 2014. Machine
on Machine Learning-Volume 70, 380–389. JMLR. org. reading tea leaves: Automatically evaluating topic coherence
Baturo, A.; Dasandi, N.; and Mikhaylov, S. J. 2017. Un- and topic model quality. In Proceedings of the 14th Confer-
derstanding state preferences with text as data: Introducing ence of the European Chapter of the Association for Com-
the UN General Debate corpus. Research & Politics, 4(2): putational Linguistics, 530–539.
2053168017712821. Miao, Y.; Grefenstette, E.; and Blunsom, P. 2017. Discov-
Bird, S.; Dale, R.; Dorr, B. J.; Gibson, B.; Joseph, M. T.; ering discrete latent topics with neural variational inference.
Kan, M.-Y.; Lee, D.; Powley, B.; Radev, D. R.; and Tan, In Proceedings of the 34th International Conference on Ma-
Y. F. 2008. The ACL anthology reference corpus: A ref- chine Learning-Volume 70, 2410–2419. JMLR. org.
erence dataset for bibliographic research in computational Miao, Y.; Yu, L.; and Blunsom, P. 2016. Neural variational
linguistics. inference for text processing. In International conference on
Bishop, C. M. 2006. Pattern recognition and machine learn- machine learning, 1727–1736.
ing. springer. Pennington, J.; Socher, R.; and Manning, C. D. 2014. GloVe:
Blei, D. M.; and Lafferty, J. D. 2006. Dynamic topic mod- Global Vectors for Word Representation. In Empirical Meth-
els. In Proceedings of the 23rd international conference on ods in Natural Language Processing (EMNLP), 1532–1543.
Machine learning, 113–120.
Perrone, V.; Jenkins, P. A.; Spano, D.; and Teh, Y. W. 2017.
Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003. Latent dirich- Poisson random fields for dynamic feature models. Journal
let allocation. Journal of machine Learning research, 3(Jan): of Machine Learning Research, 18.
993–1022.
Rezaee, M.; and Ferraro, F. 2020. A Discrete Variational Re-
Boyd-Graber, J. L.; Hu, Y.; Mimno, D.; et al. 2017. Appli- current Topic Model without the Reparametrization Trick.
cations of topic models, volume 11. Now Publishers Incor- In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M. F.;
porated. and Lin, H., eds., Advances in Neural Information Process-
Caron, F.; Davy, M.; and Doucet, A. 2007. Generalized ing Systems, volume 33, 13831–13843. Curran Associates,
Polya Urn for Time-Varying Dirichlet Process Mixtures. Inc.
In Proceedings of the Twenty-Third Conference on Uncer-
Rosen-Zvi, M.; Griffiths, T.; Steyvers, M.; and Smyth, P.
tainty in Artificial Intelligence, UAI’07, 33–40. Arlington,
2012. The author-topic model for authors and documents.
Virginia, USA: AUAI Press. ISBN 0974903930.
arXiv preprint arXiv:1207.4169.
Danescu-Niculescu-Mizil, C.; West, R.; Jurafsky, D.;
Leskovec, J.; and Potts, C. 2013. No Country for Old Rudolph, M.; and Blei, D. 2018. Dynamic embeddings for
Members: User Lifecycle and Linguistic Change in On- language evolution. In Proceedings of the 2018 World Wide
line Communities. In Proceedings of the 22nd Interna- Web Conference, 1003–1011.
tional Conference on World Wide Web, 307–318. New York, Speekenbrink, M. 2016. A tutorial on particle filters. Jour-
NY, USA: Association for Computing Machinery. ISBN nal of Mathematical Psychology, 73: 140–152.
9781450320351. Srivastava, A.; and Sutton, C. 2017. Autoencoding
Dieng, A. B.; Ruiz, F. J.; and Blei, D. M. 2019. The dynamic variational inference for topic models. arXiv preprint
embedded topic model. arXiv preprint arXiv:1907.05545. arXiv:1703.01488.
Dieng, A. B.; Ruiz, F. J.; and Blei, D. M. 2020. Topic mod- Tang, J.; Meng, Z.; Nguyen, X.; Mei, Q.; and Zhang, M.
eling in embedding spaces. Transactions of the Association 2014. Understanding the Limiting Factors of Topic Model-
for Computational Linguistics, 8: 439–453. ing via Posterior Contraction Analysis. In Xing, E. P.; and
Hida, R.; Takeishi, N.; Yairi, T.; and Hori, K. 2018. Dy- Jebara, T., eds., Proceedings of the 31st International Con-
namic and Static Topic Model for Analyzing Time-Series ference on Machine Learning, volume 32 of Proceedings of
Document Collections. CoRR, abs/1805.02203. Machine Learning Research, 190–198. PMLR.
Hochreiter, S.; and Schmidhuber, J. 1997. Long Short-Term Wang, C.; Blei, D.; and Heckerman, D. 2012. Con-
Memory. Neural Computation, 9(8): 1735–1780. tinuous time dynamic topic models. arXiv preprint
Jagarlamudi, J.; Daumé III, H.; and Udupa, R. 2012. Incor- arXiv:1206.3298.
porating Lexical Priors into Topic Models. In Proceedings Williamson, S.; Wang, C.; Heller, K. A.; and Blei, D. M.
of the 13th Conference of the European Chapter of the As- 2010. The IBP compound Dirichlet process and its applica-
sociation for Computational Linguistics, 204–213. Avignon, tion to focused topic modeling. In Proceedings of the 27th
France: Association for Computational Linguistics. international conference on machine learning (ICML-10),
Jähnichen, P.; Wenzel, F.; Kloft, M.; and Mandt, S. 2018. 1151–1158.
Scalable generalized dynamic topic models. arXiv preprint Zhang, H.; Chen, B.; Guo, D.; and Zhou, M. 2018. WHAI:
arXiv:1803.07868. Weibull Hybrid Autoencoding Inference for Deep Topic
Kingma, D. P.; and Welling, M. 2013. Auto-encoding varia- Modeling. In International Conference on Learning Rep-
tional bayes. arXiv preprint arXiv:1312.6114. resentations.

12726
 Zhao, H.; Phung, D.; Huynh, V.; Le, T.; and Buntine, W.
2021. Neural Topic Model via Optimal Transport. In Inter-
national Conference on Learning Representations.
Zuo, Y.; Zhao, J.; and Xu, K. 2014. Word Network Topic
Model: A Simple but General Solution for Short and Imbal-
anced Texts. arXiv:1412.5404.

12727

View publication stats