Text Classification Using Label Names Only: A Language Model

Self-Training Approach
Yu Meng1 , Yunyi Zhang1 , Jiaxin Huang1 , Chenyan Xiong2 ,
Heng Ji1 , Chao Zhang3 , Jiawei Han1
1
University of Illinois at Urbana-Champaign, IL, USA
2
Microsoft Research, WA, USA 3 Georgia Institute of Technology, GA, USA
1
{yumeng5, yzhan238, jiaxinh3, hengji, hanj}@[Link]
2
[Link]@[Link] 3 chaozhang@[Link]

Abstract 2016) have been developed and achieved great

success when trained on large-scale labeled doc-
Current text classification methods typically
uments (usually over tens of thousands), thanks
require a good number of human-labeled doc-
uments as training data, which can be costly to their strong representation learning power that
and difficult to obtain in real applications. Hu- effectively captures the high-order, long-range se-
mans can perform classification without see- mantic dependency in text sequences for accurate
ing any labeled examples but only based on classification.
a small set of words describing the categories Recently, increasing attention has been paid to
to be classified. In this paper, we explore semi-supervised text classification which requires a
the potential of only using the label name of
much smaller amount of labeled data. The success
each class to train classification models on un-
labeled data, without using any labeled doc-
of semi-supervised methods stems from the usage
uments. We use pre-trained neural language of abundant unlabeled data: Unlabeled documents
models both as general linguistic knowledge provide natural regularization for constraining the
sources for category understanding and as rep- model predictions to be invariant to small changes
resentation learning models for document clas- in input (Chen et al., 2020; Miyato et al., 2017;
sification. Our method (1) associates semanti- Xie et al., 2019), thus improving the generalization
cally related words with the label names, (2) ability of the model. Despite mitigating the annota-
finds category-indicative words and trains the
tion burden, semi-supervised methods still require
model to predict their implied categories, and
(3) generalizes the model via self-training. We manual efforts from domain experts, which might
show that our model achieves around 90% ac- be difficult or expensive to obtain especially when
curacy on four benchmark datasets including the number of classes is large.
topic and sentiment classification without us- Contrary to existing supervised and semi-
ing any labeled documents but learning from supervised models which learn from labeled docu-
unlabeled data supervised by at most 3 words ments, a human expert will just need to understand
(1 in most cases) per class as the label name1 .
the label name (i.e., a single or a few representative
1 Introduction words) of each class to classify documents. For
example, we can easily classify news articles when
Text classification is a classic and fundamental task given the label names such as “sports”, “business”,
in Natural Language Processing (NLP) with a wide and “politics” because we are able to understand
spectrum of applications such as question answer- these topics based on prior knowledge.
ing (Rajpurkar et al., 2016), spam detection (Jin- In this paper, we study the problem of weakly-
dal and Liu, 2007) and sentiment analysis (Pang supervised text classification where only the label
et al., 2002). Building an automatic text classifica- name of each class is provided to train a classifier
tion model has been viewed as a task of training on purely unlabeled data. We propose a language
machine learning models from human-labeled doc- model self-training approach wherein a pre-trained
uments. Indeed, many deep learning-based clas- neural language model (LM) (Devlin et al., 2019;
sifiers including CNNs (Kim, 2014; Zhang et al., Peters et al., 2018; Radford et al., 2018; Yang et al.,
2015) and RNNs (Tang et al., 2015a; Yang et al., 2019) is used as both the general knowledge source
1
Source code can be found at [Link] for category understanding and feature represen-
yumeng5/LOTClass. tation learning model for classification. The LM

9006
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9006–9017,
November 16–20, 2020. c 2020 Association for Computational Linguistics
creates contextualized word-level category super- ELMo (Peters et al., 2018), GPT (Radford et al.,
vision from unlabeled data to train itself, and then 2018) and XLNet (Yang et al., 2019) and autoen-
generalizes to document-level classification via a coding LMs such as BERT (Devlin et al., 2019) and
self-training objective. its variants (Lan et al., 2020; Lewis et al., 2020; Liu
Specifically, we propose the LOTClass model et al., 2019b), has brought astonishing performance
for Label-Name-Only Text Classification built in improvement to a wide range of NLP tasks, mainly
three steps: (1) We construct a category vocabu- for two reasons: (1) LMs are pre-trained on large-
lary for each class that contains semantically corre- scale text corpora, which allow the models to learn
lated words with the label name using a pre-trained generic linguistic features (Tenney et al., 2019) and
LM. (2) The LM collects high-quality category- serve as knowledge bases (Petroni et al., 2019);
indicative words in the unlabeled corpus to train and (2) LMs enjoy strong feature representation
itself to capture category distinctive information learning power of capturing high-order, long-range
with a contextualized word-level category predic- dependency in texts thanks to the Transformer ar-
tion task. (3) We generalize the LM via document- chitecture (Vaswani et al., 2017).
level self-training on abundant unlabeled data.
LOTClass achieves around 90% accuracy on 2.2 Semi-Supervised and Zero-Shot Text
four benchmark text classification datasets, AG Classification
News, DBPedia, IMDB and Amazon corpora, with- For semi-supervised text classification, two lines
out learning from any labeled data but only using of framework are developed to leverage unlabeled
at most 3 words (1 word in most cases) per class data. Augmentation-based methods generate new
as the label name, outperforming existing weakly- instances and regularize the model’s predictions
supervised methods significantly and yielding even to be invariant to small changes in input. The
comparable performance to strong semi-supervised augmented instances can be either created as real
and supervised models. text sequences (Xie et al., 2019) via back transla-
The contributions of this paper are as follows: tion (Sennrich et al., 2016) or in the hidden states
of the model via perturbations (Miyato et al., 2017)
• We propose a weakly-supervised text classifi- or interpolations (Chen et al., 2020). Graph-based
cation model LOTClass based on a pre-trained methods (Tang et al., 2015b; Zhang et al., 2020)
neural LM without any further dependencies2 . build text networks with words, documents and la-
LOTClass does not need any labeled documents bels and propagate labeling information along the
but only the label name of each class. graph via embedding learning (Tang et al., 2015c)
• We propose a method for finding category- or graph neural networks (Kipf and Welling, 2017).
indicative words and a contextualized word-level Zero-shot text classification generalizes the clas-
category prediction task that trains LM to pre- sifier trained on a known label set to an unknown
dict the implied category of a word using its one without using any new labeled documents.
contexts. The LM so trained generalizes well to Transferring knowledge from seen classes to un-
document-level classification upon self-training seen ones typically relies on semantic attributes and
on unlabeled corpus. descriptions of all classes (Liu et al., 2019a; Pushp
and Srivastava, 2017; Xia et al., 2018), correlations
• On four benchmark datasets, LOTClass outper- among classes (Rios and Kavuluru, 2018; Zhang
forms significantly weakly-supervised models et al., 2019) or joint embeddings of classes and
and has comparable performance to strong semi- documents (Nam et al., 2016). However, zero-shot
supervised and supervised models. learning still requires labeled data for the seen label
set and cannot be applied to cases where no labeled
2 Related Work
documents for any class is available.
2.1 Neural Language Models
2.3 Weakly-Supervised Text Classification
Pre-training deep neural models for language
Weakly-supervised text classification aims to cat-
modeling, including autoregressive LMs such as
egorize text documents based only on word-level
2
Other semi-supervised/weakly-supervised methods usu- descriptions of each category, eschewing the need
ally take advantage of distant supervision like Wikipedia
dump (Chang et al., 2008), or augmentation systems like of any labeled documents. Early attempts rely on
trained back translation models (Xie et al., 2019). distant supervision such as Wikipedia to interpret

9007
the label name semantics and derive document- produced by the BERT encoder to the MLM head,
concept relevance via explicit semantic analy- which will output a probability distribution over
sis (Gabrilovich and Markovitch, 2007). Since the entire vocabulary V , indicating the likelihood
the classifier is learned purely from general knowl- of each word w appearing at this position:
edge without even requiring any unlabeled domain-
p(w | h) = Softmax (W2 σ (W1 h + b)) , (1)
specific data, these methods are called dataless
classification (Chang et al., 2008; Song and Roth, where σ(·) is the activation function; W1 ∈ Rh×h ,
2014; Yin et al., 2019). Later, topic models (Chen b ∈ Rh , and W2 ∈ R|V |×h are learnable param-
et al., 2015; Li et al., 2016) are exploited for seed- eters that have been pre-trained with the MLM
guided classification to learn seed word-aware top- objective of BERT.
ics by biasing the Dirichlet priors and to infer pos- Table 1 shows the pre-trained MLM prediction
terior document-topic assignment. Recently, neu- for the top words (sorted by p(w | h)) to replace
ral approaches (Mekala and Shang, 2020; Meng the original label name “sports” under two different
et al., 2018, 2019) have been developed for weakly- contexts. We observe that for each masked word,
supervised text classification. They assign docu- the top-50 predicted words usually have similar
ments pseudo labels to train a neural classifier by meanings with the original word, and thus we use
either generating pseudo documents or using LMs the threshold of 50 words given by the MLM to
to detect category-indicative words. While achiev- define valid replacement for each occurrence of the
ing inspiring performance, these neural approaches label names in the corpus. Finally, we form the cat-
train classifiers from scratch on the local corpus egory vocabulary of each class using the top-100
and fail to take advantage of the general knowledge words ranked by how many times they can replace
source used by dataless classification. In this pa- the label name in the corpus, discarding stopwords
per, we build our method upon pre-trained LMs, with NLTK (Bird et al., 2009) and words that ap-
which are used both as general linguistic knowl- pear in multiple categories. Tables 2, 3, 4 and 9
edge sources for understanding the semantics of (Table 9 is in Appendix A) show the label name
label names, and as strong feature representation used for each category and the obtained category
learning models for classification. vocabulary of AG News, IMDB, Amazon and DB-
Pedia corpora, respectively.
3 Method
3.2 Masked Category Prediction
In this section, we introduce LOTClass with Like how humans perform classification, we want
BERT (Devlin et al., 2019) as our backbone model, the classification model to focus on category-
but our method can be easily adapted to other pre- indicative words in a sequence. A straightforward
trained neural LMs. way is to directly highlight every occurrence of the
category vocabulary entry in the corpus. However,
3.1 Category Understanding via Label Name
this approach is error-prone because: (1) Word
Replacement
meanings are contextualized; not every occurrence
When provided label names, humans are able to of the category keywords indicates the category.
understand the semantics of each label based on For example, as shown in Table 1, the word “sports”
general knowledge by associating with it other cor- in the second sentence does not imply the topic
related keywords that indicate the same category. “sports”. (2) The coverage of the category vocabu-
In this section, we introduce how to learn a cate- lary is limited; some terms under specific contexts
gory vocabulary from the label name of each class have similar meanings with the category keywords
with a pre-trained LM, similar to the idea of topic but are not included in the category vocabulary.
mining in recent studies (Meng et al., 2020a,b). To address the aforementioned challenge, we
Intuitively, words that are interchangeable most introduce a new task, Masked Category Predic-
of the time are likely to have similar meanings. We tion (MCP), as illustrated in Fig. 1, wherein a pre-
use the pre-trained BERT masked language model trained LM creates contextualized word-level cat-
(MLM) to predict what words can replace the la- egory supervision for training itself to predict the
bel names under most contexts. Specifically, for implied category of a word with the word masked.
each occurrence of a label name in the corpus, we To create contextualized word-level category su-
feed its contextualized embedding vector h ∈ Rh pervision, we reuse the pre-trained MLM method in

9008
 Sentence Language Model Prediction
The oldest annual US team sports competition that sports, baseball, handball, soccer,
includes professionals is not in baseball, or football or basketball, football, tennis, sport,
basketball or hockey. It’s in soccer. championship, hockey, . . .
Samsung’s new SPH-V5400 mobile phone sports a built-in has, with, features, uses, includes,
1-inch, 1.5-gigabyte hard disk that can store about 15 times had, is, contains, featured, have,
more data than conventional handsets, Samsung said. incorporates, requires, offers, . . .

Table 1: BERT language model prediction (sorted by probability) for the word to appear at the position of “sports”
under different contexts. The two sentences are from AG News corpus.

Label Name Category Vocabulary

politics, political, politicians, government, elections, politician, democracy,
politics democratic, governing, party, leadership, state, election, politically, affairs, issues,
governments, voters, debate, cabinet, congress, democrat, president, religion, . . .
sports, games, sporting, game, athletics, national, athletic, espn, soccer, basketball,
sports stadium, arts, racing, baseball, tv, hockey, pro, press, team, red, home, bay, kings,
city, legends, winning, miracle, olympic, ball, giants, players, champions, boxing, . . .
business, trade, commercial, enterprise, shop, money, market, commerce, corporate,
business global, future, sales, general, international, group, retail, management, companies,
operations, operation, store, corporation, venture, economic, division, firm, . . .
technology, tech, software, technological, device, equipment, hardware, devices,
technology infrastructure, system, knowledge, technique, digital, technical, concept, systems,
gear, techniques, functionality, process, material, facility, feature, method, . . .

Table 2: The label name used for each class of AG News dataset and the learned category vocabulary.

Section 3.1 to understand the contextualized mean- parameters of the linear layer (K is the number of
ing of each word by examining what are valid re- classes).
placement words. As shown in Table 1, the MLM We note that it is crucial to mask out the category-
predicted words are good indicators of the original indicative word for category prediction, because
word’s meaning. As before, we regard the top-50 this forces the model to infer categories based on
words given by the MLM as valid replacement of the word’s contexts instead of simply memoriz-
the original word, and we consider a word w as ing context-free category keywords. In this way,
“category-indicative” for class cw if more than 20 the BERT encoder will learn to encode category-
out of 50 w’s replacing words appear in the cate- discriminative information within the sequence into
gory vocabulary of class cw . By examining every the contextualized embedding h that is helpful for
word in the corpus as above, we will obtain a set of predicting the category at its position.
category-indicative words and their category labels
Sind as word-level supervision. 3.3 Self-Training
For each category-indicative word w, we mask it After training the LM with the MCP task, we pro-
out with the [MASK] token and train the model pose to self-train the model on the entire unlabeled
to predict w’s indicating category cw via cross- corpus for two reasons: (1) There are still many
entropy loss with a classifier (a linear layer) on unlabeled documents not seen by the model in the
top of w’s contextualized embedding h: MCP task (due to no category keywords detected)
that can be used to refine the model for better gen-
X
LM CP = − log p(cw | hw ), (2)
(w,cw )∈Sind
eralization. (2) The classifier has been trained on
top of words to predict their categories with them
p(c | h) = Softmax (Wc h + bc ) , (3)
masked, but have not been applied on the [CLS]
where Wc ∈ RK×h and bc ∈ RK are learnable token where the model is allowed to see the entire

9009
 Label Name Category Vocabulary
good, excellent, fair, wonderful, sound, high, okay, positive, sure, solid, quality,
good smart, normal, special, successful, quick, home, brilliant, beautiful, tough, fun,
cool, amazing, done, interesting, superb, made, outstanding, sweet, happy, old, . . .
bad, badly, worst, mad, worse, sad, dark, awful, rotten, rough, mean, dumb,
bad negative, nasty, mixed, thing, much, fake, guy, ugly, crazy, german, gross, weird,
sorry, like, short, scary, way, sick, white, black, shit, average, dangerous, stuff, . . .

Table 3: The label name used for each class of IMDB dataset and the learned category vocabulary.

Label Name Category Vocabulary

good, excellent, fine, right, fair, sound, wonderful, high, okay, sure, quality, smart,
good positive, solid, special, home, quick, safe, beautiful, cool, valuable, normal,
amazing, successful, interesting, useful, tough, fun, done, sweet, rich, suitable, . . .
bad, terrible, horrible, badly, wrong, sad, worst, worse, mad, dark, awful, mean,
bad rough, rotten, much, mixed, dumb, nasty, sorry, thing, negative, funny, far, go, crazy,
weird, lucky, german, shit, guy, ugly, short, weak, sick, gross, dangerous, fake, . . .

Table 4: The label name used for each class of Amazon dataset and the learned category vocabulary.

sequence to predict its category. than hard labeling, probably because hard labeling
The idea of self-training (ST) is to iteratively treats high-confident predictions directly as ground-
use the model’s current prediction P to compute truth labels and is more prone to error propagation.
a target distribution Q which guides the model for Another advantage of soft labeling is that the target
refinement. The general form of ST objective can distribution is computed for every instance and no
be expressed with the KL divergence loss: confidence thresholds need to be preset.
N X
K
X qij We update the target distribution Q via Eq. (5)
LST = KL(QkP ) = qij log , (4)
pij every 50 batches and train the model via Eq. (4).
i=1 j=1
The overall algorithm is shown in Algorithm 1.
where N is the number of instances.
There are two major choices of the target distri-
bution Q: Hard labeling and soft labeling. Hard
labeling (Lee, 2013) converts high-confidence pre- Algorithm 1: LOTClass Training.
dictions over a threshold τ to one-hot labels, i.e., Input: An unlabeled text corpus D; a set of
qij = 1(pij > τ ), where 1(·) is the indicator func- label names C; a pre-trained neural
tion. Soft labeling (Xie et al., 2016) derives Q language model M .
by enhancing high-confidence predictions while Output: A trained model M for classifying
demoting low-confidence ones via squaring and the K classes.
normalizing the current predictions: Category vocabulary ← Section 3.1;
p2ij /fj X Sind ← Section 3.2;
qij = P   , fj = pij , (5) Train M with Eq. (2);
2
j 0 pij 0 /fj 0 i B ← Total number of batches;
for i ← 0 to B − 1 do
where the model prediction is made by applying the
if i mod 50 = 0 then
classifier trained via MCP (Eq. (3)) to the [CLS]
Q ← Eq. (5);
token of each document, i.e.,
Train M on batch i with Eq. (4);
pij = p(cj | hdi :[CLS] ). (6) Return M ;
In practice, we find that the soft labeling strategy
consistently gives better and more stable results

9010
 Category 1 Vocabulary: Category 2 Vocabulary: Category 3 Vocabulary:
politics, political, politicians, government… sports, soccer, game, baseball, sport… business, trade, commercial, enterprise…

> 20/50 matched

Word-Level
Probable Words (Top 50):
MLM sports, baseball, handball, soccer…
Category Prediction: MCP
[0, 1, 0]

···
<latexit sha1_base64="AcPHc2/GmseUvYw39tetylSl3A8=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPJnN/WWvcFHGU4QRO4Rw8uIIG3EETWsDgEZ7hFd4c5bw4787HorXkFDPH8AfO5w+uBY8u</latexit>

···
<latexit sha1_base64="AcPHc2/GmseUvYw39tetylSl3A8=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPJnN/WWvcFHGU4QRO4Rw8uIIG3EETWsDgEZ7hFd4c5bw4787HorXkFDPH8AfO5w+uBY8u</latexit>

··· <latexit sha1_base64="AcPHc2/GmseUvYw39tetylSl3A8=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPJnN/WWvcFHGU4QRO4Rw8uIIG3EETWsDgEZ7hFd4c5bw4787HorXkFDPH8AfO5w+uBY8u</latexit>

···
<latexit sha1_base64="AcPHc2/GmseUvYw39tetylSl3A8=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPJnN/WWvcFHGU4QRO4Rw8uIIG3EETWsDgEZ7hFd4c5bw4787HorXkFDPH8AfO5w+uBY8u</latexit>

BERT Encoder BERT Encoder

(Pre-trained, not fine-tuned, as general knowledge) (Pre-trained, fine-tuned, as classification model)

[CLS]
<latexit sha1_base64="hLexQ2ue+kyjKjaV8rYXfkjAhMU=">AAAB9XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMdALh48RDQP2KxhdjKbDJl9MNOrhiX/4cWDIl79F2/+jZNkD5pY0FBUddPd5SdSaLTtb6uwsrq2vlHcLG1t7+zulfcPWjpOFeNNFstYdXyquRQRb6JAyTuJ4jT0JW/7o/rUbz9wpUUc3eE44V5IB5EIBKNopPsu8idEzNz69a036ZUrdtWegSwTJycVyNHolb+6/ZilIY+QSaq169gJehlVKJjkk1I31TyhbEQH3DU0oiHXXja7ekJOjNInQaxMRUhm6u+JjIZaj0PfdIYUh3rRm4r/eW6KwaWXiShJkUdsvihIJcGYTCMgfaE4Qzk2hDIlzK2EDamiDE1QJROCs/jyMmmdVR2TzM15pWbncRThCI7hFBy4gBpcQQOawEDBM7zCm/VovVjv1se8tWDlM4fwB9bnD6/uko4=</latexit>

···
<latexit sha1_base64="AcPHc2/GmseUvYw39tetylSl3A8=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPJnN/WWvcFHGU4QRO4Rw8uIIG3EETWsDgEZ7hFd4c5bw4787HorXkFDPH8AfO5w+uBY8u</latexit>

US team sports competition ···

<latexit sha1_base64="AcPHc2/GmseUvYw39tetylSl3A8=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPJnN/WWvcFHGU4QRO4Rw8uIIG3EETWsDgEZ7hFd4c5bw4787HorXkFDPH8AfO5w+uBY8u</latexit>

[CLS]
<latexit sha1_base64="hLexQ2ue+kyjKjaV8rYXfkjAhMU=">AAAB9XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMdALh48RDQP2KxhdjKbDJl9MNOrhiX/4cWDIl79F2/+jZNkD5pY0FBUddPd5SdSaLTtb6uwsrq2vlHcLG1t7+zulfcPWjpOFeNNFstYdXyquRQRb6JAyTuJ4jT0JW/7o/rUbz9wpUUc3eE44V5IB5EIBKNopPsu8idEzNz69a036ZUrdtWegSwTJycVyNHolb+6/ZilIY+QSaq169gJehlVKJjkk1I31TyhbEQH3DU0oiHXXja7ekJOjNInQaxMRUhm6u+JjIZaj0PfdIYUh3rRm4r/eW6KwaWXiShJkUdsvihIJcGYTCMgfaE4Qzk2hDIlzK2EDamiDE1QJROCs/jyMmmdVR2TzM15pWbncRThCI7hFBy4gBpcQQOawEDBM7zCm/VovVjv1se8tWDlM4fwB9bnD6/uko4=</latexit>

··· <latexit sha1_base64="AcPHc2/GmseUvYw39tetylSl3A8=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPJnN/WWvcFHGU4QRO4Rw8uIIG3EETWsDgEZ7hFd4c5bw4787HorXkFDPH8AfO5w+uBY8u</latexit>

US team [MASK]
<latexit sha1_base64="DTQHaOsJ5gwF+zQectHbaT/ajcg=">AAAB+HicbVBNS8NAEJ3Ur1o/WvXoJVgETyURQY8VL4IIFe0HpKFstpt26WYTdidiDf0lXjwo4tWf4s1/47bNQasPBh7vzTAzL0gE1+g4X1ZhaXllda24XtrY3NouV3Z2WzpOFWVNGotYdQKimeCSNZGjYJ1EMRIFgrWD0cXUb98zpXks73CcMD8iA8lDTgkaqVcpd5E9IGLmXZ/fXvmTXqXq1JwZ7L/EzUkVcjR6lc9uP6ZpxCRSQbT2XCdBPyMKORVsUuqmmiWEjsiAeYZKEjHtZ7PDJ/ahUfp2GCtTEu2Z+nMiI5HW4ygwnRHBoV70puJ/npdieOZnXCYpMknni8JU2Bjb0xTsPleMohgbQqji5labDokiFE1WJROCu/jyX9I6rrkmmZuTat3J4yjCPhzAEbhwCnW4hAY0gUIKT/ACr9aj9Wy9We/z1oKVz+zBL1gf38I1kxM=</latexit>
competition ···
<latexit sha1_base64="AcPHc2/GmseUvYw39tetylSl3A8=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPJnN/WWvcFHGU4QRO4Rw8uIIG3EETWsDgEZ7hFd4c5bw4787HorXkFDPH8AfO5w+uBY8u</latexit>

US team sports competition US team sports competition

Input Tokens Contextualized Embeddings Neural Network Modules

Figure 1: Overview of Masked Category Prediction (MCP). The Masked Language Model (MLM) head first
predicts what are probable words to appear at each token’s position. A token is considered as “category-indicative”
if its probable replacement words highly overlap with the category vocabulary of a certain class. The MCP head is
trained to predict the implied categories of the category-indicative words with them masked.

Dataset Classification Type # Classes # Train # Test

AG News News Topic 4 120,000 7,600
DBPedia Wikipedia Topic 14 560,000 70,000
IMDB Movie Review Sentiment 2 25,000 25,000
Amazon Product Review Sentiment 2 3,600,000 400,000

Table 5: Dataset statistics. Supervised models are trained on the entire training set. Semi-supervised models use
10 labeled documents per class from the training set and the rest as unlabeled data. Weakly-supervised models are
trained by using the entire training set as unlabeled data. All models are evaluated on the test set.

4 Experiments unlabeled data. All methods are evaluated on the

test set.
4.1 Datasets
Weakly-supervised methods:
We use four benchmark datasets for text classi-
fication: AG News (Zhang et al., 2015), DBPe- • Dataless (Chang et al., 2008): Dataless classi-
dia (Lehmann et al., 2015), IMDB (Maas et al., fication maps label names and each document
2011) and Amazon (McAuley and Leskovec, 2013). into the same semantic space of Wikipedia con-
The dataset statistics are shown in Table 5. All cepts. Classification is performed based on vec-
datasets are in English language. tor similarity between documents and classes us-
ing explicit semantic analysis (Gabrilovich and
4.2 Compared Methods Markovitch, 2007).

We compare LOTClass with a wide range of • WeSTClass (Meng et al., 2018): WeSTClass
weakly-supervised methods and also state-of-the- generates pseudo documents to pre-train a CNN
art semi-supervised and supervised methods. The classifier and then bootstraps the model on unla-
label names used as supervision on each dataset beled data with self-training.
for the weakly-supervised methods are shown in
• BERT w. simple match: We treat each docu-
Tables 2, 3, 4 and 9. (Table 9 can be found in Ap-
ment containing the label name as if it is a la-
pendix A.) Fully supervised methods use the entire
beled document of the corresponding class to
training set for model training. Semi-supervised
train the BERT model.
method UDA uses 10 labeled documents per class
from the training set and the rest as unlabeled data. • LOTClass w/o. self train: This is an ablation
Weakly-supervised methods use the training set as version of our method. We train LOTClass

9011
 only with the MCP task, without performing Supervised BERT and show its corresponding per-
self-training on the entire unlabeled data. formance in Fig. 2(a). The performance of LOT-
Class is equivalent to that of Supervised BERT
Semi-supervised method: with 48 labeled documents per class.
• UDA (Xie et al., 2019): Unsupervised data aug- 4.5 Study of Category Understanding
mentation is the state-of-the-art semi-supervised We study the characteristics of the method intro-
text classification method. Apart from using a duced in Section 3.1 from the following two as-
small amount of labeled documents for super- pects. (1) Sensitivity to different words as label
vised training, it uses back translation (Sennrich names. We use “commerce” and “economy” to
et al., 2016) and TF-IDF word replacing for aug- replace “business” as the label name on AG News
mentation and enforces the model to make con- dataset. Table 7 shows the resulting learned cat-
sistent predictions over the augmentations. egory vocabulary. We observe that despite the
change in label name, around half of terms in the
Supervised methods:
resulting category vocabulary overlap with the orig-
• char-CNN (Zhang et al., 2015): Character-level inal one (Table 2 “business” category); the other
CNN was one of the state-of-the-art supervised half also indicate very similar meanings. This guar-
text classification models before the appearance antees the robustness of our method since it is the
of neural LMs. It encodes the text sequences into category vocabulary rather than the original label
characters and applies 6-layer CNNs for feature name that is used in subsequent steps. (2) Ad-
learning and classification. vantages over alternative solutions. We take the
pre-trained 300-d GloVe (Pennington et al., 2014)
• BERT (Devlin et al., 2019): We use the pre- embeddings and use the top words ranked by co-
trained BERT-base-uncased model and fine-tune sine similarity with the label names for category
it with the training data for classification. vocabulary construction. On Amazon dataset, we
use “good” and “bad” as the label names, and the
4.3 Experiment Settings category vocabulary built by LOTClass (Table 4)
We use the pre-trained BERT-base-uncased model accurately reflects the sentiment polarity, while the
as the base neural LM. For the four datasets AG results given by GloVe (Table 8) are poor—some
News, DBPedia, IMDB and Amazon, the maximum words that are close to “good”/“bad” in the GloVe
sequence lengths are set to be 200, 200, 512 and embedding space do not indicate sentiment, or even
200 tokens. The training batch size is 128. We the reversed sentiment (the closest word to “bad” is
use Adam (Kingma and Ba, 2015) as the optimizer. “good”). This is because context-free embeddings
The peak learning rate is 2e − 5 and 1e − 6 for only learn from local context windows, while neu-
MCP and self-training, respectively. The model is ral LMs capture long-range dependency that leads
run on 4 NVIDIA GeForce GTX 1080 Ti GPUs. to accurate interpretation of the target word.

4.4 Results 4.6 Effect of Self-Training

The classification accuracy of all methods on the We study the effect of self-training with two sets of
test set is shown in Table 6. LOTClass consistently experiments: (1) In Fig. 2(b) we show the test accu-
outperforms all weakly-supervised methods by a racy and self-training loss (Eq. (4)) when training
large margin. Even without self-training, LOT- LOTClass on the first 1, 000 steps (batches) of un-
Class’s ablation version performs decently across labeled documents. It can be observed that the loss
all datasets, demonstrating the effectiveness of our decreases within a period of 50 steps, which is the
proposed category understanding method and the update interval for the target distribution Q—when
MCP task. With the help of self-training, LOT- the self training loss approximates zero, the model
Class’s performance becomes comparable to state- has fit the previous Q and a new target distribution
of-the-art semi-supervised and supervised models. is computed based on the most recent predictions.
With the model refining itself on unlabeled data iter-
How many labeled documents are label names atively, the performance gradually improves. (2) In
worth? We vary the number of labeled docu- Fig. 2(c) we show the performance of LOTClass
ments per class on AG News dataset for training vs. BERT w. simple match with the same self-

9012
 Supervision Type Methods AG News DBPedia IMDB Amazon
Dataless (Chang et al., 2008) 0.696 0.634 0.505 0.501
WeSTClass (Meng et al., 2018) 0.823 0.811 0.774 0.753
Weakly-Sup. BERT w. simple match 0.752 0.722 0.677 0.654
LOTClass w/o. self train 0.822 0.860 0.802 0.853
LOTClass 0.864 0.911 0.865 0.916
Semi-Sup. UDA (Xie et al., 2019) 0.869 0.986 0.887 0.960
char-CNN (Zhang et al., 2015) 0.872 0.983 0.853 0.945
Supervised
BERT (Devlin et al., 2019) 0.944 0.993 0.945 0.972

Table 6: Test accuracy of all methods on four datasets.

Label Name Category Vocabulary

commerce, trade, consumer, retail, trading, merchants, treasury, currency, sales,
commerce commercial, market, merchant, economy, economic, marketing, store, exchange,
transactions, marketplace, businesses, investment, markets, trades, enterprise, . . .
economy, economic, economies, economics, currency, trade, future, gdp, treasury,
economy sector, production, market, investment, growth, mortgage, commodity, money,
markets, commerce, economical, prosperity, account, income, stock, store, . . .

Table 7: Different label names used for class “business” of AG News dataset and the learned category vocabulary.

Label Name Category Vocabulary

good, better, really, always, you, well, excellent, very, things, think, way, sure,
good thing, so, n’t, we, lot, get, but, going, kind, know, just, pretty, i, ’ll, certainly, ’re,
nothing, what, bad, great, best, something, because, doing, got, enough, even, . . .
bad, good, things, worse, thing, because, really, too, nothing, unfortunately, awful,
bad n’t, pretty, maybe, so, lot, trouble, something, wrong, got, terrible, just, anything,
kind, going, getting, think, get, ?, you, stuff, ’ve, know, everything, actually, . . .

Table 8: GloVe 300-d pre-trained embedding for category understanding on Amazon dataset.

training strategy. BERT w. simple match does not 5 Discussions

seem to benefit from self-training as our method
does. This is probably because documents con- The potential of weakly-supervised classifica-
taining label names may not be actually about the tion has not been fully explored. For the sim-
category (e.g., the second sentence in Table 1); the plicity and clarity of our method, (1) we only use
noise from simply matching the label names causes the BERT-base-uncased model rather than more ad-
the model to make high-confidence wrong predic- vanced and recent LMs; (2) we use at most 3 words
tions, from which the model struggles to extract per class as label names; (3) we refrain from using
correct classification signals for self-improvement. other dependencies like back translation systems
This demonstrates the necessity of creating word- for augmentation. We believe that the performance
level supervision by understanding the contextu- will become better with the upgrade of the model,
alized word meaning and training the model via the enrichment in inputs and the usage of data aug-
MCP to predict the category of words instead of mentation techniques.
directly assigning the word’s implied category to
Applicability of weak supervision in other NLP
its document.
tasks. Many other NLP problems can be for-
mulated as classification tasks such as named en-

9013
 0.88 0.20
(50, 0.867) 0.86 Acc. 0.90 LOTClass
Loss 0.15 BERT w. simple match
0.84 0.85

Test Acc.
Test Acc.

Test Acc.
0.85
(20, 0.822)

Loss
0.10
0.80 0.84
Sup. BERT 0.80
0.05
LOTClass 0.83
0.76
0.00 0.75
20 40 60 80 100 0 200 400 600 800 0 200 400 600 800
# Labeled Documents / Class Steps Steps

(a) Supervised BERT: Test acc. vs. (b) LOTClass: Test accuracy and self- (c) LOTClass vs. BERT w. sim-
number of labeled documents. training loss. ple match during self-training.

Figure 2: (On AG News dataset.) (a) The performance of LOTClass is close to that of Supervised BERT with
48 labeled documents per class. (b) The self-training loss of LOTClass decreases in a period of 50 steps; the
performance of LOTClass gradually improves. (c) BERT w. simple match does not benefit from self-training.

tity recognition and aspect-based sentiment analy- fication with label names as the only supervision
sis (Huang et al., 2020). Sometimes a label name in three steps: Category understanding via label
could be too generic to interpret (e.g., “person”, name replacement, word-level classification via
“time”, etc). To apply similar methods as intro- masked category prediction, and self-training on
duced in this paper to these scenarios, one may unlabeled corpus for generalization. The effective-
consider instantiating the label names with more ness of LOTClass is validated on four benchmark
concrete example terms like specific person names. datasets. We show that label names is an effective
supervision type for text classification but has been
Limitation of weakly-supervised classification.
largely overlooked by the mainstreams of litera-
There are difficult cases where label names are
ture. We also point out several directions for future
not sufficient to teach the model for correct classi-
work by generalizing our methods to other tasks or
fication. For example, some review texts implicitly
combining with other techniques.
express sentiment polarity that goes beyond word-
level understanding: “I find it sad that just because Acknowledgments
Edward Norton did not want to be in the film or
have anything to do with it, people automatically Research was sponsored in part by US DARPA
think the movie sucks without even watching it or KAIROS Program No. FA8750-19-2-1004 and So-
giving it a chance.” Therefore, it will be interesting cialSim Program No. W911NF-17-C-0099, Na-
to improve weakly-supervised classification with tional Science Foundation IIS 19-56151, IIS 17-
active learning where the model is allowed to con- 41317, IIS 17-04532, IIS 16-18481, and III-
sult the user about difficult cases. 2008334, and DTRA HDTRA11810026. Any opin-
ions, findings, and conclusions or recommenda-
Collaboration with semi-supervised classifica-
tions expressed herein are those of the authors
tion. One can easily integrate weakly-supervised
and should not be interpreted as necessarily repre-
methods with semi-supervised methods in differ-
senting the views, either expressed or implied, of
ent scenarios: (1) When no training documents
DARPA or the U.S. Government. The U.S. Gov-
are available, the high-confidence predictions of
ernment is authorized to reproduce and distribute
weakly-supervised methods can be used as ground-
reprints for government purposes notwithstanding
truth labels for initializing semi-supervised meth-
any copyright annotation hereon. We thank anony-
ods. (2) When both training documents and la-
mous reviewers for valuable and insightful feed-
bel names are available, a joint objective can be
back.
designed to train the model with both word-level
tasks (e.g., MCP) and document-level tasks (e.g.,
augmentation, self-training). References
6 Conclusions Steven Bird, Ewan Klein, and Edward Loper. 2009.
Natural language processing with Python: analyz-
In this paper, we propose the LOTClass model ing text with the natural language toolkit. ” O’Reilly
built upon pre-trained neural LMs for text classi- Media, Inc.”.

9014
Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Hongmei Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu,
Vivek Srikumar. 2008. Importance of semantic rep- Qimai Li, Xiao ming Wu, and Albert Y. S. Lam.
resentation: Dataless classification. In AAAI. 2019a. Reconstructing capsule networks for zero-
shot intent classification. In EMNLP.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix-
text: Linguistically-informed interpolation of hid- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-
den space for semi-supervised text classification. In dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
ACL. Luke Zettlemoyer, and Veselin Stoyanov. 2019b.
Roberta: A robustly optimized bert pretraining ap-
Xingyuan Chen, Yunqing Xia, Peng Jin, and John A. proach. ArXiv, abs/1907.11692.
Carroll. 2015. Dataless text classification with de-
scriptive lda. In AAAI. Andrew L. Maas, Raymond E. Daly, Peter T. Pham,
Dan Huang, Andrew Y. Ng, and Christopher Potts.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and 2011. Learning word vectors for sentiment analysis.
Kristina Toutanova. 2019. Bert: Pre-training of deep In ACL.
bidirectional transformers for language understand-
ing. In NAACL-HLT. Julian J. McAuley and Jure Leskovec. 2013. Hidden
factors and hidden topics: understanding rating di-
Evgeniy Gabrilovich and Shaul Markovitch. 2007. mensions with review text. In RecSys ’13.
Computing semantic relatedness using wikipedia-
based explicit semantic analysis. In IJCAI. Dheeraj Mekala and Jingbo Shang. 2020. Contextu-
alized weak supervision for text classification. In
Jiaxin Huang, Yu Meng, Fang Guo, Heng Ji, and Ji- ACL.
awei Han. 2020. Weakly-supervised aspect-based
sentiment analysis via joint aspect-sentiment topic Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan
embedding. In EMNLP. Wang, Chao Zhang, Yu Zhang, and Jiawei Han.
2020a. Discriminative topic mining via category-
Nitin Jindal and Bing Liu. 2007. Review spam detec- name guided text embedding. In WWW.
tion. In WWW.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han.
Yoon Kim. 2014. Convolutional neural networks for 2018. Weakly-supervised neural text classification.
sentence classification. In EMNLP. In CIKM.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han.
method for stochastic optimization. In ICLR. 2019. Weakly-supervised hierarchical text classifi-
cation. In AAAI.
Thomas Kipf and Max Welling. 2017. Semi-
supervised classification with graph convolutional Yu Meng, Yunyi Zhang, Jiaxin Huang, Yu Zhang, Chao
networks. In ICLR. Zhang, and Jiawei Han. 2020b. Hierarchical topic
mining via joint spherical tree and text embedding.
Zhen-Zhong Lan, Mingda Chen, Sebastian Goodman, In KDD.
Kevin Gimpel, Piyush Sharma, and Radu Soricut.
2020. Albert: A lite bert for self-supervised learn- Takeru Miyato, Andrew M. Dai, and Ian J. Goodfel-
ing of language representations. In ICLR. low. 2017. Adversarial training methods for semi-
supervised text classification. In ICLR.
Dong-Hyun Lee. 2013. Pseudo-label : The simple and
efficient semi-supervised learning method for deep Jinseok Nam, Eneldo Loza Mencı́a, and Johannes
neural networks. In Workshop on challenges in rep- Fürnkranz. 2016. All-in text: Learning document,
resentation learning, ICML. label, and word representations jointly. In AAAI.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan.
Dimitris Kontokostas, Pablo N. Mendes, Sebastian 2002. Thumbs up? sentiment classification using
Hellmann, Mohamed Morsey, Patrick van Kleef, machine learning techniques. In EMNLP.
Sören Auer, and Christian Bizer. 2015. Dbpedia -
a large-scale, multilingual knowledge base extracted Jeffrey Pennington, Richard Socher, and Christopher D.
from wikipedia. Semantic Web, 6:167–195. Manning. 2014. Glove: Global vectors for word rep-
resentation. In EMNLP.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar-
jan Ghazvininejad, Abdelrahman Mohamed, Omer Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt
Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. Gardner, Christopher Clark, Kenton Lee, and Luke
Bart: Denoising sequence-to-sequence pre-training Zettlemoyer. 2018. Deep contextualized word repre-
for natural language generation, translation, and sentations. In NAACL-HLT.
comprehension. In ACL.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton
Chenliang Li, Jian Xing, Aixin Sun, and Zongyang Ma. Bakhtin, Yuxiang Wu, Alexander H. Miller, and Se-
2016. Effective document labeling with very few bastian Riedel. 2019. Language models as knowl-
seed words: A topic model approach. In CIKM. edge bases? In EMNLP.

9015
Pushpankar Kumar Pushp and Muktabh Mayank Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He,
Srivastava. 2017. Train once, test anywhere: Alexander J. Smola, and Eduard H. Hovy. 2016. Hi-
Zero-shot learning for text classification. ArXiv, erarchical attention networks for document classifi-
abs/1712.05972. cation. In NAACL-HLT.

Alec Radford, Karthik Narasimhan, Tim Salimans, and Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019.
Ilya Sutskever. 2018. Improving language under- Benchmarking zero-shot text classification:
standing by generative pre-training. Datasets, evaluation and entailment approach.
In EMNLP.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and
Percy Liang. 2016. Squad: 100, 000+ questions for Jingqing Zhang, Piyawat Lertvittayakumjorn, and Yike
machine comprehension of text. In EMNLP. Guo. 2019. Integrating semantic knowledge to
tackle zero-shot text classification. In NAACL-HLT.
Anthony Rios and Ramakanth Kavuluru. 2018. Few-
shot and zero-shot multi-label learning for structured Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.
label spaces. In EMNLP. Character-level convolutional networks for text clas-
sification. In NIPS.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Improving neural machine translation models Yu Zhang, Yu Meng, Jiaxin Huang, Frank F. Xu, Xuan
with monolingual data. In ACL. Wang, and Jiawei Han. 2020. Minimally supervised
categorization of text with metadata. In SIGIR.
Yangqiu Song and Dan Roth. 2014. On dataless hierar-
chical text classification. In AAAI. A Label Names Used and Category
Vocabulary Obtained for DBPedia
Duyu Tang, Bing Qin, and Ting Liu. 2015a. Docu-
ment modeling with gated recurrent neural network We show the label names used for DBPedia corpora
for sentiment classification. In EMNLP. and the obtained category vocabulary in Table 9. In
Jian Tang, Meng Qu, and Qiaozhu Mei. 2015b. Pte: most cases, only one word as the label name will
Predictive text embedding through large-scale het- be sufficient; however, sometimes the semantics
erogeneous text networks. In KDD. of the label name might be too general so we in-
stead use 2 or 3 keywords of the class to represent
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun
Yan, and Qiaozhu Mei. 2015c. Line: Large-scale the label name. For example, we use “school” and
information network embedding. In WWW. “university” to represent the class “educational insti-
tution”; we use “river”, “lake” and “mountain” to
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang,
represent the class “natural place”; we use “book”,
Adam Poliak, R. Thomas McCoy, Najoung Kim,
Benjamin Van Durme, Samuel R. Bowman, Dipan- “novel” and “publication” to represent the class “pa-
jan Das, and Ellie Pavlick. 2019. What do you learn per work”.
from context? probing for sentence structure in con-
textualized word representations. In ICLR.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob

Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In NIPS.

Congying Xia, Chenwei Zhang, Xiaohui Yan,

Yi Chang, and Philip S. Yu. 2018. Zero-shot user
intent detection via capsule neural networks. In
EMNLP.

Junyuan Xie, Ross B. Girshick, and Ali Farhadi. 2016.

Unsupervised deep embedding for clustering analy-
sis. In ICML.

Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang

Luong, and Quoc V. Le. 2019. Unsupervised data
augmentation. ArXiv, abs/1904.12848.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Car-

bonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.
Xlnet: Generalized autoregressive pretraining for
language understanding. In NeurIPS.

9016
Label Name Category Vocabulary
companies, co, firm, concern, subsidiary, brand, enterprise, division, partnership,
company manufacturer, works, inc, cooperative, provider, corp, factory, chain, limited,
holding, consortium, industry, manufacturing, entity, operator, product, giant . . .
academy, college, schools, ecole, institution, campus, university, secondary,
school
form, students, schooling, standard, class, educate, elementary, hs, level,
university
student, tech, academic, universities, branch, degree, universite, universidad, . . .
artists, painter, artistic, musician, singer, arts, poet, designer, sculptor, composer,
artist star, vocalist, illustrator, architect, songwriter, entertainer, cm, painting,
cartoonist, creator, talent, style, identity, creative, duo, editor, personality, . . .
athletes, athletics, indoor, olympian, archer, events, sprinter, medalist, olympic,
athlete runner, jumper, swimmer, competitor, holder, mile, ultra, able, mark, hurdles,
relay, amateur, medallist, footballer, anchor, metres, cyclist, shooter, athletic, . . .
politics, political, government, politicians, politician, elections, policy, party,
politics affairs, legislature, politically, democracy, democratic, governing, history,
leadership, cabinet, issues, strategy, election, religion, assembly, law, . . .
transportation, transport, transit, rail, travel, traffic, mobility, bus, energy,
transportation railroad, communication, route, transfer, passenger, transported, traction,
recreation, metro, shipping, railway, security, transports, infrastructure, . . .
buildings, structure, tower, built, wing, hotel, build, structures, room,
building courthouse, skyscraper, library, venue, warehouse, block, auditorium, location,
plaza, addition, museum, pavilion, landmark, offices, foundation, headquarters, . . .
river river, lake, bay, dam, rivers, water, creek, channel, sea, pool, mountain,
lake stream, lakes, flow, reservoir, hill, flowing, mountains, basin, great, glacier,
mountain flowed, pond, de, valley, peak, drainage, mount, summit, brook, mare, head, . . .
village, villages, settlement, town, east, population, rural, municipality, parish,
village na, temple, commune, pa, ha, north, pre, hamlet, chamber, settlements, camp,
administrative, lies, township, neighbourhood, se, os, iran, villagers, nest, . . .
animal, animals, ape, horse, dog, cat, livestock, wildlife, nature, lion, human,
animal owl, cattle, cow, wild, indian, environment, pig, elephant, fauna, mammal,
beast, creature, australian, ox, land, alligator, eagle, endangered, mammals, . . .
shrub, plants, native, rose, grass, herb, species, jasmine, race, vine, hybrid,
plant
bamboo, hair, planted, fire, growing, flame, lotus, sage, iris, perennial, variety,
tree
palm, cactus, trees, robert, weed, nonsense, given, another, stand, holly, poppy, . . .
lp, albums, cd, ep, effort, recording, disc, compilation, debut, appearance,
album soundtrack, output, genus, installation, recorded, anthology, earth, issue, imprint, ex,
era, opera, estate, single, outing, arc, instrumental, audio, el, song, offering, . . .
films, comedy, drama, directed, documentary, video, language, pictures,
film miniseries, negative, movies, musical, screen, trailer, acting, starring, filmmaker,
flick, horror, silent, screenplay, box, lead, filmmaking, second, bond, script, . . .
book novel, books, novels, mystery, memoir, fantasy, fiction, novelist, reader, read, cycle,
novel romance, writing, written, published, novella, play, narrative, trilogy, manga,
publication autobiography, publication, literature, isbn, write, tale, poem, year, text, reading, . . .

Table 9: The label name used for each class of DBPedia dataset and the learned category vocabulary.

9017