Psychological Methods

The Text-Package: An R-Package for Analyzing and Visualizing Human

Language Using Natural Language Processing and Transformers
Oscar Kjell, Salvatore Giorgi, and H. Andrew Schwartz
Online First Publication, May 1, 2023. [Link]

CITATION
Kjell, O., Giorgi, S., & Schwartz, H. A. (2023, May 1). The Text-Package: An R-Package for Analyzing and Visualizing Human
Language Using Natural Language Processing and Transformers. Psychological Methods. Advance online publication.
[Link]
 Psychological Methods
© 2023 American Psychological Association
ISSN: 1082-989X [Link]

The Text-Package: An R-Package for Analyzing and Visualizing Human

Language Using Natural Language Processing and Transformers

Oscar Kjell1, 2, Salvatore Giorgi3, and H. Andrew Schwartz2

1
Department of Psychology, Lund University
2
Department of Computer Science, Stony Brook University
3
Department of Computer and Information Science, University of Pennsylvania
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

Abstract
The language that individuals use for expressing themselves contains rich psychological information. Recent
significant advances in Natural Language Processing (NLP) and Deep Learning (DL), namely transformers,
have resulted in large performance gains in tasks related to understanding natural language. However, these
state-of-the-art methods have not yet been made easily accessible for psychology researchers, nor designed
to be optimal for human-level analyses. This tutorial introduces text ([Link] a new R-package for
analyzing and visualizing human language using transformers, the latest techniques from NLP and DL. The
text-package is both a modular solution for accessing state-of-the-art language models and an end-to-end
solution catered for human-level analyses. Hence, text provides user-friendly functions tailored to test
hypotheses in social sciences for both relatively small and large data sets. The tutorial describes methods for
analyzing text, providing functions with reliable defaults that can be used off-the-shelf as well as providing a
framework for the advanced users to build on for novel pipelines. The reader learns about three core meth-
ods: (1) textEmbed(): to transform text to modern transformer-based word embeddings; (2) textTrain() and
textPredict(): to train predictive models with embeddings as input, and use the models to predict from; (3)
textSimilarity() and textDistance(): to compute semantic similarity/distance scores between texts. The reader
also learns about two extended methods: (1) textProjection()/textProjectionPlot() and (2) textCentrality()/
textCentralityPlot(): to examine and visualize text within the embedding space.
Translational Abstract
Natural language is the fundamental way individuals communicate their thoughts and emotions to others.
Recent advances in Artificial Intelligence (AI), referred to as transformers, have resulted in large
increases in performance at most tasks related to understanding natural language. This tutorial introduces
how to use these state-of-the-art AI techniques in both custom research analyses as well as in completely
end-to-end analytic processes. We describe text, a software package which provides transformer-based
techniques intended to be easily accessible for social scientists. The text-package is open-source, written
for the statistical programming language R, and it is free to use or alter. It comprises user-friendly func-
tions to transform text to numeric representations, that are used for examining their relationship to other
variables or for visualizing statistically significant features of texts. Transformers can facilitate analyses
of natural language for gaining psychological insights with unprecedented accuracy and provide a more
detailed understanding of the human condition.

Keywords: Natural Language Processing, machine learning, computational language assessments,

transformers, #Rtext

Supplemental materials: [Link]

dgczt/. Oscar Kjell has co-founded a start-up using computational language

Oscar Kjell [Link] assessments to diagnose mental health problems.
Salvatore Giorgi [Link] The data are available at [Link]
H. Andrew Schwartz [Link] The experiment materials are available at [Link]
Oscar Kjell was funded by the Swedish Research Council (2019-06305) Correspondence concerning this article should be addressed to Oscar
and H. Andrew Schwartz was funded by a National Institutes of Health- Kjell, Department of Psychology, Lund University, Box 117, 221 00 Lund,
NIAAA (R01 AA028032). Tutorial data, models, and code: [Link] Sweden. Email: [Link]@[Link]

1
 2 KJELL, GIORGI, AND SCHWARTZ

How individuals express themselves and their state of mind assessments as a complement to traditional rating scales (Kjell
with natural language constitutes a wealth of information for et al., 2019, 2022), and analyze written narratives of traumatic life
understanding them psychologically and socially (e.g., see Kern events to predict health related outcomes (Campbell & Penne-
et al., 2016; Kjell et al., 2019). “Language is the most common baker, 2003; Son et al., 2020). Probed language may also involve
and reliable way for people to translate their internal thoughts and asking participants to recall various memories, describe them-
emotions into a form that others can understand” (Tausczik & Pen- selves in various ways, partake in stream-of-consciousness tasks
nebaker, 2010, p. 25). This tutorial explains how psychology and so on.
researchers can use recent advances in Artificial Intelligence (AI), In addition, language may be generated within an experimental
including Natural Language Processing (NLP) and Deep Learning, context. Computational language methods can, for example, be
to quantitatively analyze natural language. used to enhance experimental control by matching word stimuli
Researchers in NLP have turned to open-vocabulary methods according to semantic similarity (Dougal & Rotello, 2007; Gagné
that rely on patterns in linguistic data to derive models of lan- et al., 2005), examining the text generated from experimental
guage. These methods leverage the idea that words may be repre- manipulations (Garcia & Sikström, 2013) and using semantic sim-
sented by values based on how they co-occur in languages (e.g., ilarity of the names of objects to find significant correlations with
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Firth, 1957), and allow for modeling words according to the con-
This document is copyrighted by the American Psychological Association or one of its allied publishers.

neural organizations in the brain (Carlson et al., 2014). The multi-

texts in which they appear, rather than relying on a priori assump- tude of applications of computational methods highlights its poten-
tions about word-category relations resulting in extremely tial and flexibility as a research method.
powerful models of language. In fact, the latest version of such This tutorial covers methods in an R-package called text
models, the deep learning-based transformers, have led to nothing ([Link] to carry out the use of transformers for psy-
short of a transformation in the AI field concerned with language: chological research. The text-package is an open-source library
Natural Language Processing. Researchers have called for the use comprising tools for analyzing and visualizing various features of
of these language models to gain psychological insights (e.g., texts; both in relation to other texts as well as numerical variables.
Eichstaedt et al., 2021). The package makes state-of-the-art NLP, statistics, and ML tech-
The transformers method has universally increased the accuracy niques available to R users, with functions particularly targeting
of AI techniques for processing language. For example, the trans-
social scientific applications. To map words to numeric represen-
former-based model GPT-3, by OpenAI, is writing documents
tations, text-functions transform texts to word embeddings using
believed to be authored by humans (Brown et al., 2020); and
transformer-based pretrained language models. This tutorial
BERT (Bidirectional Encoder Representations from Transformers;
assumes very basic knowledge of R (R Core Team, 2022); so, for
Devlin et al., 2019), the first widely used, general purpose trans-
a free, beginners tutorial see R for Data Science by (Grolemund &
former network developed by Google, has already been referenced
Wickham, 2018).
in over 16,000 scholarly works in a period of only 2 years.1
The key to transformers’ success is that such models can repre-
sent words differently according to the context they are in. BERT Objectives and Aims
was released by Google and has been integrated into its Search The text-package incorporates two main objectives. First, to serve R-
function; now it actually understands the difference between users as a point solution for converting text to contextual word embed-
“travel from Sweden to New York” and “travel from New York to dings—numeric representations of words—using a state-of-the-art AI
Sweden” (Nayak, 2019). This tutorial aims to make these methods technique called transformers. These word embeddings may be used
easily available to a broad audience within social and behavioral for a large variety of tasks in the user’s own analyses (pipeline). The sec-
sciences; as well as further developing and optimizing them for
ond objective is to serve as an end-to-end solution, where text provides
human-level analyses. These state-of-the-art word embeddings
powerful and accessible functions for analyzing and visualizing text in
may be used to examine their relationships to (i.e., predict) numer-
relation to other text and numerical variables. This tutorial describes
ical variables, compute semantic similarity to other texts, or visu-
both core and extended functionalities. Core functionalities are based on
alize (statistically significant) words in various dimensions within
standard methods with empirical support, including: (a) transforming
the word embedding space.
text to word embeddings, (b) predictive modeling with word embed-
For this discussion, the type of data may broadly be categorized
dings, and (c) computing semantic similarity scores between text.
into two (overlapping) types: First, everyday occurring language,
The extended functionalities are less well-established, novel meth-
which, for example, include gaining insights from analyzing
ods, aimed at supporting the core functionalities, including (a) visu-
recordings of spoken language in relation to emotional fluctuations
alizing words’ position in the embedding space. The objective of the
throughout the day (Sun et al., 2020); and social media text (Park
et al., 2015) to predict both physical (Eichstaedt et al., 2015) and text-package is to balance user-friendliness and flexibility while
psychological (Eichstaedt et al., 2018) outcomes. Examining natu- simultaneously empowering with advanced analyses. Accessibility
rally occurring language may also include analyzing emails, let- is reflected in functions with reliable default settings selected by
ters, blogs, text messages, medical journals, speeches, diaries, experts in the NLP fields so that those with no experience can use out
song texts, voice recordings etc., though one must consider ethical of the box settings to test their research hypotheses. Further, visual-
and privacy issues when analyzing such text. izations aim to make it easy to understand what is going on in the
Second, probed language involves probing or asking partici- analyses; for example, displaying actual word embeddings and
pants to answer questions with spoken or written language. These tokens, plotting words, or viewing cross-validated predictive results.
types of data have, for example, been used to measure and
1
describe psychological constructs through probed language-based Google Scholar accessed March 11, 2021.
 THE TEXT-PACKAGE 3

Current Alternatives If you experience problems with the installation, see the
extended installation guide for up-to-date information ([Link]
There are few alternatives in R (R Core Team, 2022) that focus
[Link]/articles/Extended_Installation_Guide.html).
on getting state-of-the-art word embeddings; and that have func-
tionalities tailored for analyzing embeddings in downstream tasks
relevant for social sciences and psychological research. Computer Core Functionality
scientists have predominantly used Python (Van Rossum & Drake,
1995), and Python-libraries such as the Differential Language The text-package has three core functionalities: (1) transforming
Analysis ToolKit (DLATK; Schwartz et al., 2017), PyTorch text to word embeddings, (2) predictive modeling with word
(Paszke et al., 2019), spaCy (Honnibal, & Montani, 2017), and embeddings, and (3) comparing words or texts for semantic simi-
NLTK (Bird et al., 2009). Further, even though computer scientists larity. These are described below, followed by the introduction of
use Python for standard NLP/ML tasks, few python packages two extended functionalities.
attempt to bridge this line of work with psychological research
such as DLATK. The text-package comprises an interface with Transforming Text to State-of-the-Art Word
Python to get the state-of-the-art language models while also con-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Embeddings
This document is copyrighted by the American Psychological Association or one of its allied publishers.

ceptually building on DLATK, and drawing on experiences

learned from that project (here also see the web-application NLP is concerned with automatically processing and making sense
Semantic Excel; Sikström et al., 2018). of digitized human language (for a broad and detailed treatment of
There are some R-packages that allow for text mining (e.g., NLP methods see Jurafsky & Martin, 2020). The idea that each word
see tm [Feinerer & Hornik, 2019], tidytext [Silge & Robinson, in human languages may be represented by numeric vectors dates
2016], and quanteda [Benoit et al., 2018]) and for various auto- back at least to the 1950s (e.g., see Firth, 1957; Osgood et al., 1957).
matic coding procedures of text responses (see shinyReCoR; Despite its long history, the field of NLP has undergone a sort of rev-
Andersen & Zehner, 2021). To our knowledge, there is only one olution over the past 3 years, seeing substantial advances in the state-
other R-package that enables the transformation of text to word of-the-art. These advances span over nearly every common task that
embeddings (RBERT; an R implementation of BERT; Bratt &
NLP attempts to solve including syntactic parsing, sentiment analysis,
Harmon, 2020); however, it does not come with functions to use
question answers, and machine translation. All advances are tied to a
these word embeddings including ML techniques. Additionally,
single method: transformers (Devlin et al., 2019; Rogers et al., 2020;
text comprises functions tailored for small data as well as big
data. Whereas other R-packages focus on text mining of big Vaswani et al., 2017).
data, text enables transformer-based embeddings as well as anal- The text-package implements these advances to enable the user
yses of relatively small data sets, which is often common in to access many state-of-the-art pretrained language models. In this
social and behavioral sciences. next part, word embeddings are described in more detail, followed
by a demonstration on how the text-package transforms text data
into word embeddings. These word embeddings are later used in
Installation of Text
down-stream tasks to provide examples of how they may be used,
The text-package can be downloaded and installed from CRAN including predicting psychology related outcomes (e.g., psycho-
or Github. The text-package uses an R-package called reticulate logical rating scale scores), and examining similarity in meaning
(Ushey et al., 2020) as an interface to Python (Van Rossum & between texts. Hence, word embeddings may be seen as the back-
Drake, 1995) and the Python packages torch (Paszke et al., 2019), bone of most text-functions.
transformers (Wolf et al., 2019), nltk (Bird, Klein, & Loper,
2009), and numpy (Oliphant, 2006). It is important to get versions Word Embeddings
of these libraries to work together. The text-package includes
A word embedding is a list of values (an ordered vector) that
functions that help you set up an environment with the correct ver-
aim to numerically represent the meaning of a word. A word
sions of these packages for your specific OS (that is tested for
Linux, MacOS, and Windows). embedding normally comprises several hundred numbers; so that
This tutorial was written for text Version 1.0. Tutorial data, a word is represented by many dimensions. The numbers may be
models and code can be retrieved at the open science framework seen as coordinates in a geometric space that comprises several
at: [Link] where we will update the code for future hundred dimensions (a high dimensional space). The closer two
versions of the text-package. To install the text-package run: words are in this space (i.e., the more similar their vector embed-
dings are), the more similar the words are expected to be in
meaning. In other words, embeddings capture the relationships
# Install text from CRAN between words, where proximity in the high dimensional embed-
[Link]("text") ding space signifies similarity in meaning. To represent several
library(text) words, sentences and paragraphs, word embeddings may be
aggregated. The aggregation can, for example, be through the
# Set-up an environment with text-required python packages mean, maximum or minimum of each dimension of the word
textrpp_install() embedding.
To train word embeddings with high quality requires a lot of
# Initialize the environment — and save the settings
text data; where the approaches use the statistical patterns of how
for next time
words are used. “You shall know a word by the company it keeps”
textrpp_initialize(save_profile = TRUE)
(Firth, 1957, p. 11) is a core rationale underlying the creation of
 4 KJELL, GIORGI, AND SCHWARTZ

word embeddings. Words within natural language are not ran- static (or decontextualized), so that the embedding for “happy” is
domly distributed, instead contextual words are predictable and always the same even if it appeared in the context of “I am happy”
define its meaning (Iliev et al., 2015). Hence, it is possible to use versus “I am not happy.” Examples of commonly used decontextual-
this distribution to represent “the meaning of a word through the ized approaches include word2vec (Mikolov et al., 2013), Latent
contexts in which it has been observed in a corpus” (Erk, 2012, Semantic Analysis (LSA; Deerwester et al., 1990) and Latent
p. 635). This may be achieved by constructing a table of word co- Dirichlet allocation (LDA; Blei et al., 2003).
occurrence counts, where the dimensions are extracted using a
Contextualized Word Embeddings and Word Order
dimension reduction technique such as the Singular Value Decom-
position (Golub & Kahan, 1965). Practically, the first column of Contemporary NLP algorithms aim to extract the latent mean-
the frequency table may hold the words in a natural language, the ings in text. These algorithms are based on deep neural net (or
first row may hold word contexts (such as documents of texts) and deep learning) architectures to construct contextualized word
the rest of the cells hold the co-occurrence counts/frequency. Ulti- embeddings, using decontextualized word embeddings as input
mately, the words are represented by a vector containing a number (e.g., see BERT; Devlin et al., 2019). When using these algo-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

for each extracted dimension. Note that these types of approaches rithms, a word’s embedding is different depending on the context
This document is copyrighted by the American Psychological Association or one of its allied publishers.

are referred to as unsupervised, as no predefined categories or it was in and the order of the words in the context—this is
judges are used to produce the numeric representations. achieved by enabling the word embedding to be influenced by
other word embeddings in the context through a mechanism
Decontextualized Word Embeddings: A Bag of Words referred to as self-attention.
Decontextualized word embeddings (or semantic representations The Deep Neural Network Architecture: The Layers and
as they were called) do not account for the context a word was in.
Hidden States
Earlier methods were unable to capture the order that words appeared
in, so that words in a text were treated as a bag of scrambled words: As a deep learning transformer model, BERT comprises several
Bag of words (BOW) models (see Figure 1). The word embedding is layers. Hidden states refer to the output of each layer; these may

Figure 1
An Illustration of How Information is Connected Within the Output of Different Language Model Architectures

1.4 0.5 1.1 0.4 1.4 0.7

1.6 0.3 0.8 2.3 1.1 0.2
1.3 1.2 0.8 1.2 1.0 1.1

1.2 0.2 1.5 1.2 0.2 1.5 1.2 0.2 1.5

0.9 1.7 0.3 0.9 1.7 0.3 0.9 1.7 0.3
1.4 2.3 1.5 1.4 2.3 1.5 1.4 2.3 1.5

LU LU LU LU LU LU LU LU LU

w1 w2 w3 w1 w2 w3 w1 w2 w3

I run fast I run fast I run fast

BOW model LSTM model Self-attention model

Recurrent Neural Network Transformers
Note. The BOW model is how traditional word embeddings, such as those from latent semantic analysis (LSA) or word2vec, are typically applied:
there is no notion of capturing word order or adjusting the individual word representations in light of the other words. LU = Look-up; the circles
describe a mechanism that looks-up words’ non-contextualized embeddings. BOW = bag of words (i.e., no transformations beyond non-contextualised
embeddings); LSTM = Long Short-Term Memory; w = word; squared symbols = word embeddings; gray = decontextualized word embeddings; stron-
ger shades of blue indicate more contextualized word embeddings. See the online article for the color version of this figure.
 THE TEXT-PACKAGE 5

be aggregated across layers to a word embedding that represents introducing deep learning with a focus on Natural Language Proc-
each word (or sentence). The use of multiple layers enables the essing (e.g., see Lauriola et al., 2022), in length discussing the
models to capture nonlinear relationships. BERT-base comprises opportunities and risks of using these models (Bommasani et al.,
12 layers, whereas BERT-large comprises 24 layers; where each 2021) as well as reviews/surveys providing a broader overview
layer comprises numeric values for each dimension, which for and deeper understanding about various model architectures of
BERT is 768 dimensions. Hence, with BERT-large one token can neural networks such as recurrent, recursive, convolutional, and
be represented with 18,432 (24 3 768) values. There are several attention models (Babi
c et al., 2020; Otter et al., 2019; Qiu et al.,
ways to make use of these layers: including only using one of the 2020), and specifically focusing on the BERT models (Rogers
layers or concatenating them to one longer embedding. et al., 2020).
It is not yet fully understood how the various layers differ; and
Performance of BERT
the advice on how to best use them differs. In short, empirical
examinations indicate that “BERT’s intermediate layers encode a Devlin et al. (2019) demonstrated that BERT obtains substantial
rich hierarchy of linguistic information, with surface features at improvement on a wide range of NLP tasks. For example, it
the bottom, syntactic features in the middle and semantic features
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

achieved a 7.7% point absolute improvement on the language

This document is copyrighted by the American Psychological Association or one of its allied publishers.

at the top” (Jawahar et al., 2019, p. 1). Further, it has been demon- understanding tasks called GLUE (General Language Understand-
strated that later BERT layers are more context specific (Ethayar- ing Evaluation); and a 4.6% absolute improvement accuracy on
ajh, 2019). Later we discuss how to use these layers in human the MultiNLI (the Multi-Genre Natural Language Inference
level tasks. corpus, which comprises several hundred thousands of sentence
pairs).
Self-Attention
Accessible Pretrained Language Models in Text
Transformer’s deep neural net architecture has the self-attention
mechanism at its core (Vaswani et al., 2017). It is self-attention To get contextualized word embeddings for your text data, the
that enables the model to combine information about surrounding text-package connects with HuggingFace’s Transformers library
words at varying degrees. Thus, each surrounding word can influ- (Wolf et al., 2019) in Python. This enables the users to imple-
ence the word embedding of the target word. In other words, self- ment many state-of-the-art pretrained language models, includ-
attention enables the algorithm to incorporate the embeddings of ing XLnet (Yang et al., 2019), RoBERTa (Liu et al., 2019), and
other words’ embeddings in so far as it produces a more robust ALBERT (Lan et al., 2019). As the name suggests, a pretrained
representation of the target word. The self-attention mechanism model has already been trained on other data; so that the model
puts different attention or weight to different word embeddings can be applied to your text to retrieve high-quality embeddings.
from the context depending on which appear to be most relevant. This is good considering that training a high-quality deep learn-
During training, the transformer-based models have learned how ing language model requires a lot of computational resources;
to parameterize the attention layers to amplify the influence of the Wolf et al. (2019) point out that RoBERTa was trained on 160
most relevant parts of a context. GB text, and that training this on a typical cloud computing serv-
The transformers’ self-attention mechanism may be compared ice would cost around 100K USD. Consequently, some models
with previous models such as Long Short-Term Memory models have been developed with the focus of being smaller and requir-
(LSTM; Hochreiter & Schmidhuber, 1997). These models are based ing less computational resources to be trained (e.g., see Destil-
on recurrent neural networks, where the influence of the context BERT; Sanh et al., 2019), whereas others have had the focus on
mechanism is less pervasive than in transformers. In LSTM the achieving improved performance (e.g., XLNet; Yang et al.,
influence of the context is only based on the previous state—the 2019).
embedding of the word just to the left of the target word, which in Some models include multiple languages such as multilin-
turn was influenced by its surrounding words, and so on. The model gual BERT (mBERT), which is trained on text from the top 104
can also be run across the sequence backward, such that the previ- languages with the largest Wikipedia entries. Hence, the same
ous state is an embedding of the word to the right. However, it still model includes several languages, where similar languages
requires for nonadjacent pairs of words to pass information about (e.g., Germanic, Slavic) are found close to each other in the
each other through the embeddings between them rather than embedding space (Libovickỳ et al., 2019). mBERT is found to
directly. With self-attention all words are connected with the poten- learn cross-lingual word alignment with high-quality (Libov-
tial to influence each other’s meaning through the word embed- ickỳ et al., 2019). The text-package implements these multilin-
dings. The transformers can model the dependency of every two gual models.
words in a text sequence and are also suitable for long range
Model and Word Embedding Specifics
dependencies. The architecture of the transformers also reduces the
amount of sequential computational steps compared with previous Different models may be based on different types of (domain)
contemporary models such as LSTMs; and this results in decreased text, and use different tokenization (described in detail below),
information loss and faster training times through parallel process- number of layers and hidden states. It is useful to consider these
ing (Wolf et al., 2019). aspects when selecting an appropriate model and its settings,
These descriptions are at a very high level aimed to provide the although default settings in text will often achieve apt results.
reader with a sufficient understanding to be able to make use of Type of Domain Text Used in Pretraining. Word embed-
the key functionality of transformers: turning text into feature vec- dings become better when there is high domain similarity between
tors (embeddings). There are many recent papers specifically the text used to create the language model and the text that should
 6 KJELL, GIORGI, AND SCHWARTZ

be interpreted/used for down-stream tasks. To create models for a generally been found to yield good results; but recently Matero
narrow domain can however be challenging because pretrained et al. (2021) more carefully examined which layer(s) that pro-
models require large quantities of text data (and computational duce most accurate results in predicting depression related
resources). Nevertheless, it is worth thinking about the type of text measures, which suggested that “layer 19 (sixth-to last) is the
used to train the models, and how that might influence your most ideal by itself” (p. 1).
results. BERT was originally trained using Wikipedia text; how- Limitations of Transformers. Transformer models are very
ever, other BERT models have been fine-tuned on clinical text large. While researchers have developed ways to shrink mod-
(Alsentzer et al., 2019) and scientific text data (Beltagy et al., els, the best versions of these models are still scaling exponen-
2019). These and many more models are available through the tially. That is, the size of the files defining the models that
text-package. users need is increasing more rapidly than the performance
Uncased Versus Cased Models. Uncased models convert all gains. While BERT-base comprises 110M parameters and
words to lowercase, whereas cased models keep the casing of the achieves a Spearman rho of .86 on a standard semantic simi-
text. Select a cased model if you think that letter casing will be larity task, BERT-large is approximately three times that size
helpful for your analyses. If you do not think that casing will be comprising 336M and achieves only a modest improvement in
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

the same task: q = .87. With costs of storage and memory

This document is copyrighted by the American Psychological Association or one of its allied publishers.

important for your task, it is likely better to select an uncased

model because it comprises fewer words (e.g., happy and Happy fairly inexpensive today, it may often be more ideal to use the
are not two different words); thus, has had the chance to learn larger models at the expense of needing better computing
more high-quality embeddings. equipment, but this could exclude users with more limited
Sequence Length of the Input. You can only impute a lim- computing resources. Further, it is very expensive to create
ited amount of text (tokens) to many language models at the time. your own model.
For the BERT model the standard limit is 512 tokens at a time.
This means that each word can be contextually influenced by a Functions: Mapping Text to Numbers (i.e., Word
maximum of 511 tokens. If longer sequences of text are submitted Embeddings)
to the text-package function, the text will be split up in smaller The text-package includes three functions to map text to word
chunks; and then the word embeddings may be aggregated to rep- embeddings. The textEmbed() is a high-level function that encompasses
resent the entire text. both textEmbedRawLayers() and textEmbedLayerAggregation(). The
Tokens and Tokenizers. A token is a string of characters textEmbedRawLayers() function retrieves layers and hidden states from
with a meaning; and a tokenizer is a function that splits up senten- the language model; and the textEmbedLayerAggregation() function
ces and words to tokens. Tokenizers may differ across models. aggregates these layers to form word embeddings. It is possible to get
Each token gets its own embedding. Common words are tokenized both contextualized and decontextualized embeddings for individual
as themselves, whereas uncommon words that are not part of a words (i.e., where only one word at a time has been sent to the model).
pretrained model will be tokenized into smaller parts each having
their own embedding. Punctuation characters (e.g., “.,” “!,” and textEmbed()
“?”) also tend to get tokenized as individual tokens. BERT also The textEmbed() function automatically transforms charac-
uses tokens to indicate delimiter to each input: [CLS] and [SEP]. ter variables in a given dataset (i.e., a dataframe or a tibble)
The [CLS] token is always put to indicate the start of the sequence to word embeddings. It is the main embedding function in
and it is often used to represent the whole sequence. The [SEP] to- text; and can output contextualized (and decontextualized)
ken stands for separator, and this tag is used to separate sentences embeddings for tokens (i.e., the embeddings for each single
during the pretraining tasks of BERT that include predicting the word instance of each text) and texts (i.e., single embeddings
next sentence. per text taken from aggregating all token embeddings of the
As an example, the sentence: “I’m feeling relatedness with text). 2
others” is tokenized to “[CLS] I’m feeling related ##ness with
others [SEP]”; where “##” indicates that “ness” was originally
# Example text
attached to “related.” “Relatedness” was tokenized this way
texts <- c("I’m feeling relatedness with
because it was not present within (this version of) BERT. Instead, others", "That’s great!")
the two-word embeddings for “related” and “ness” are aggregated
to represent relatedness. # Defaults
Layers. To select the type of model and number of layers embeddings <- textEmbed(texts)
may be based on systematic, empirical research. Two systemic
studies evaluating different transformers models found that # Output
RoBERTa (Liu et al., 2019) consistently performs better than embeddings$tokens

BERT on human-level tasks (e.g., predicting characteristics of

the person that authored a text) including predicting the demo-
2
graphics (age, gender), personality (extraversion, openness) and “Words” are defined by how the tokenizer breaks up the text string into
mental health (suicide risk) of individual based on their text individual units, often called unigrams. These may not always reflect what
are conventionally called “words.” See [Link]
(Ganesan et al., 2021; Matero et al., 2021). For human level tokenizers/index for a description of the different tokenizers available from
tasks, using the four last layers or the second to last layer have HuggingFace’s transformers package.
 THE TEXT-PACKAGE 7

consuming functions (note that it is possible to speed up the embed-

$texts
$texts[[1]]
ding process with access to graphics processing units, GPU).
A tibble: 10 × 771
The Layers
tokens Dim1 Dim2 Dim3
<chr> <dbl> <dbl> <dbl> The textEmbed() function has the layers parameter for selecting
[CLS] 0.467 0.0459
0.222 the layer(s) to extract (default = the second to last layer). The func-
i 0.593 0.198
0.384 tion also provides parameters to aggregate the layers in different
‘ 0.0611 0.0160
0.0248
ways: the aggregation_from_layers_to_tokens parameter controls
m 0.236 0.231
0.552
feeling
0.402 0.0777 0.252
how to aggregate layers representing the same token (default =
related 0.230
0.400 0.479 “concatenate”). The aggregation_from_tokens_to_texts parameter
#ness
0.695
0.328 0.782 controls how embeddings from different tokens should be aggregated
with 0.264 0.219 1.11 to represent a text (default = “mean”). These defaults follow conven-
others
0.298 0.0784 0.139 tions from Natural Language Processing literature using transformers
[SEP] 0.0446 0.0103
0.0345 for regression and classification (Ganesan et al., 2021).
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

# . . . with 765 more variables: Dim4 <dbl>, Dim5 <dbl>,

This document is copyrighted by the American Psychological Association or one of its allied publishers.

Dim6 <dbl>, . . . # Look at example data included in the text-package

comprising both text and numerical variables (note that
... there are only 40 participants in this example).
# Output Language_based_assessment_data_8
embeddings$texts
# Transform the text/word data to word embeddings
$texts (see help(textEmbed) to see the default settings).
$texts[[1]]
word_embeddings <- textEmbed(
A tibble: 10 × 771 Language_based_assessment_data_8,
model = "bert-base-uncased",
Dim1_texts <dbl> Dim2_texts <dbl> Dim3_texts <dbl> aggregation_from_layers_to_tokens = "concatenate",
0.0501 0.0149 0.155 aggregation_from_tokens_to_texts = "mean",
0.0942
0.104
0.0625
keep_token_embeddings = FALSE
# . . . with 765 more variables: Dim4_texts <dbl>,
)
Dim5_texts <dbl>, Dim6_text <dbl>, . . .
...
# See how the word embeddings are structured
word_embeddings
The Language Model
# Save the word embeddings to avoid having to embed the
The pretrained language model is set with the model setting text again. It is good practice to save output from
in textEmbed() (or textEmbedRawLayers()). The default model analyses that take a lot of time to compute, which is
is “bert-base-uncased,” and other models can be specified by often the case when analyzing text data.
using their HuggingFace identifier ([Link] saveRDS(word_embeddings, "word_embeddings.rds")
models) such as “roberta-base” (Liu et al., 2019), “distilbert-
# Get the saved word embeddings (again)
base-uncase” (Sanh et al., 2019), or “gpt2” (Radford et al.,
word_embeddings <- readRDS("word_embeddings.rds")
2019). When running a language model for the first time, the
text-package automatically downloads it using the python
package transformers. The files necessary to run a model are
cached in a subdirectory of the user’s home directory called Note that one of the output layers is referred to as layer 0, which
.cache. Subsequent calls to text methods that use the same is the original input embedding to BERT; and hence, it is not
model will directly access the model in cache rather than contextualized.
downloading it again. The textModels() function lists models textEmbedRawLayers(). The textEmbedRawLayers() takes
and tokenizers that you have downloaded, and textModelsRe- text as input, and returns the hidden states for each token of the
move(“model-name”) enables you to delete specific models text, including the [CLS] and the [SEP]. This gives you more con-
trol over the embedding process compared with the textEmbed().
and its accompanying files.
The output below shows the hidden states of layer 11 and 12 for
This tutorial uses example data that is accessible through the each word/token of “I am fine” (i.e., textEmbedRawLayers()
text-package. It is a subset from a study (see Kjell et al., 2019) returns the layers in its raw form rather than aggregating them,
where participants have described their satisfaction with life and which is the default in textEmbed()).
harmony in life (or lack thereof) with a text response, 10 descriptive
words or rating scales including the Satisfaction with Life Scale
(SWLS; Diener et al., 1985) and the Harmony in Life Scale (HILS; # Get hidden states for "I am fine"
Kjell et al., 2016). To embed text and build predictive models takes imf_embeddings_11_12 <- textEmbedRawLayers(
time. Embedding the included text examples takes approximately "I am fine",
10 min using a standard laptop. Word embeddings and models for layers = 11:12
the tutorial have been saved as part of the open tutorial data, so it is )
possible to follow the tutorial without running the most time- imf_embeddings_11_12
 8 KJELL, GIORGI, AND SCHWARTZ

These word embeddings can be used for many different pur-

# OUTPUT poses and kinds of analyses. As such, the text-package serves R-
users as a point solution for transforming text to state-of-the-art
$context_tokens word embeddings that are ready to be used for their specific tasks.
$context_tokens$texts
$context_tokens$texts[[1]] The next part of the tutorial exemplifies how word embeddings
# A tibble: 10 × 771
layer_ may be used and how the text-package may serve as an end-to-end
tokens token_id number Dim1 Dim2 Dim3
<chr> <int> <int> <dbl> <dbl> <dbl> solution that provides AI techniques tailored for social and behav-
[CLS] 1 11 0.266
0.0910
0.0521 ioral scientists.
i 2 11 0.178
0.296 0.188
am 3 11
0.114
0.542 0.0757
fine 4 11 1.18
0.434
0.317
[SEP] 5 11 0.0461 0.0134
0.0311 Predictive Modeling With Word Embeddings
[CLS] 1 12 0.0609 0.377 0.213
i 2 12
0.173 0.0227 0.0647 A common use of word embeddings is as input for statistical
am 3 12
0.406
0.319 0.128 learning based predictive modeling (also known as machine learn-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

fine 4 12 0.763
0.373
0.009

ing). In such work for psychology, one often attempts to “predict”
This document is copyrighted by the American Psychological Association or one of its allied publishers.

[SEP] 5 12 0.798
0.0405
0.324

# . . . with 765 more variables: Dim4 <dbl>, Dim5 <dbl>, Dim6 a score for a standard psychological assessment from language use
<dbl>, . . . (Kjell et al., 2021, Park et al., 2015). While there are endless types
of predictive models from statistical learning in which one could
textEmbedLayerAggregation(). The textEmbedLayerAg- input embeddings (see James et al., 2021, for a good introduction)
gregation() function aggregates the hidden states from textEm- numerous recent empirical studies have demonstrated that when
bedRawLayers(). It is possible to select different methods for using modern contextual embeddings, L2-penalized linear models
how layers and tokens are aggregated. The returned object is a yield state-of-the-art results (Ganesan et al., 2021; Kjell et al.,
word embedding that represents the entire text. Below we 2022; Matero et al., 2019). For example, Matero et al. (2019) used
show how to (a) aggregate hidden states by concatenating
L2 regularized logistic regression on top of transformer-based con-
layers 11 and 12, and computing the mean of each dimension
across tokens (yielding a word embedding with 1,536 dimen- textual embeddings to achieve top results on the computational lin-
sions); and (b) only selecting layer 11 and computing the mean guistics for psychology 2019 Shared task B (“CLPsych-2019”;
across tokens (yielding a word embedding with 768 dimen- Zirikly et al., 2019), which compared 28 predictive systems, using
sions). techniques varying from convolutional neural networks (CNNs) to
random forests or multilayer perceptrons, from 11 participating
# 1. Concatenate layers(results in 1,536 dimensions).
teams. Shared tasks are particularly robust evaluations because the
textEmbedLayerAggregation( system designers are not giving the true labels but rather, they
imf_embeddings_11_12$context_tokens, send their predictions based on text without labels to a third party
layers = 11:12, that runs the evaluation and reports the accuracies. Many other
aggregation_from_layers_to_tokens = "concatenate",
recent shared tasks have similarly confirmed systems with L2
aggregation_from_tokens_to_texts = "mean"
penalized linear models on top of contextual embeddings to be
)
# OUTPUT among the best performing, such as a BERT-based system for the
$x
SemEval-2020 task on sentiment analysis (Palomino & Ochoa-
# A tibble: 1 × 1,536 Luna, 2020) or a system combining multiple contextual embed-
Dim1 Dim2 Dim3 Dim4 Dim5 Dim6
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl> dings via an L2 penalized linear regression (“ridge regression”)
0.311
0.270
0.0274
0.408 0.202 0.113 layer for the SemEval-2020 task on humor detection (Hossain
. . . with 1,530 more variables: Dim7, Dim8, Dim9, . . .
et al., 2020; Morishita et al., 2020).3 These works support the idea
# 2. Only select layer 11 (768 dimensions).
that because transformer-based contextual embeddings produce
textEmbedLayerAggregation( state-of-the-art semantic representations of text itself, sophisti-
imf_embeddings_11_12$context_tokens, cated models on top of them are not necessary (and sometimes
layers = 11, overfit) for producing accurate predictions (Bommasani et al.,
aggregation_from_tokens_to_texts = "mean"
2021). Next, we describe how these techniques are used in more
)
detail.
# OUTPUT

$x
# A tibble: 1 × 768
Dim1 Dim2 Dim3 Dim4 Dim5 Dim6
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
0.311
0.270
0.0274
0.408 0.202 0.113
3
. . . with 762 more variables: Dim7, Dim8, Dim9, . . . Note that in many modern NLP papers and deep learning packages, an
L2 penalization is referred to as “weight decay.”
 THE TEXT-PACKAGE 9

Word Embeddings as Input to a Predictive Model model’s predictive ability while simultaneously accounting for
its ability to generalize (e.g., see Browne, 2000). The simplest
The word embeddings may be used in predictive models such as form of a cross validation method randomly divides a dataset
multiple linear regression: into a training-set and a testing-set. The parameters, that is,
the β…βm in the specified regression above, are estimated
y = b0 + b1 ∗x1 ... + bm ∗xm + 1 in the training-set; and then applied to the test-set
to predict values (ŷ) that can be compared with the observed
where y denotes the observed values (in which ŷ represents the values (y). Using this method results in an out-of-sample per-
formance, where the trained model is applied on new data from
predicted value), x1 … xm the predictors, i.e., the dimensions of
the same population.
the word embeddings (where the subscript refers to the mth dimen-
sion), β0 the constant, β1 … βm the coefficients that define the
relationship between embeddings and the outcome, and ε the error
K-Fold Cross Validation
term.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Considering that the word embeddings often comprise many K-fold cross validation is a procedure to divide the training
This document is copyrighted by the American Psychological Association or one of its allied publishers.

dimensions, it is often useful to first carry out a dimension reduction and testing sets efficiently. If data are scarce, it may be consid-
technique. To achieve this, Principal Component Analysis (e.g., see ered wasteful to not use all data, for example, to only use 50%
Wold et al., 1987) is implemented in text because we have noticed of the data for training and 50% only for testing. First, it is pos-
that it is particularly efficient for reducing the dimensionality of sible to run the procedure again by alternating the purpose of the
word embeddings. It is possible to select other statistical prediction data sets, so that the original testing-set becomes the training-set,
models in text, including using ridge (L2 penalized) regression and the original training-set becomes the testing-set: In this way
by itself (Hoerl & Kennard, 1970), lasso regression (also called all data will have predictions that can be correlated with the
L1-penalized model; Tibshirani, 1996), or a mix of L1 and observed values. Second, to make better parameter estimations
L2-penalized regression (i.e., “elastic-net” regression; Zou, & it is possible to get more power in the statistical models by mak-
Hastie, 2005) and for categorical variables ridge-penalized logistic ing sure that the training set is larger. One common way is to
regression or random forests (e.g., see Ho, 1995). Although text sup- select 90% of the data to the training set and 10% for the testing;
ports lasso and elastic-net (the latter is a composite of both a lasso and then alter the purpose of the data as previously described.
and ridge penalty), it is worth noting that the top systems in the pre- This is done ten times, so that all cases get a prediction.
viously mentioned tasks use only a ridge penalty (Matero et al., Hence, k in k-fold refers to the number of groups (or folds);
2019), which is also the default in the most popular contextual so, the described procedure is called 10-fold cross validation.
embedding library, transformers (controlled with the “weight_de- It is also possible to train on all but one case (that is called
cay” in the pytorch library it utilizes). This lack of utilization for leave-one-out cross validation). Leave-one-out cross validation
is good for very small dataset, whereas for larger dataset it
dense embeddings is probably because any regularization involving
requires a lot of computational power and time (for rather
a lasso is designed for sparse input information (Ganguli &
small gains).
Sompolinsky, 2012), which is not present in most embeddings.
Embeddings are typically dense and rotationally invariant. In other Cross Validation With Test, Train, and Development Sets. To
words, singular dimensions do not necessarily correspond to distinct fit models with hyperparameters (e.g., the penalty in ridge regres-
sets of words. On the other hand, for nondense input vectors, such as sion), a development set (also called assessment set) can be used
with dimensions corresponding to counts of particular words—a to evaluate on. Figure 2 shows how the training set in the outside
multihot encoding or word count vector—the lasso has been fold is split into a training (or analysis) set and development
found beneficial (Schwartz et al., 2013).
(assessment) set. Hence, the training set is used to fit models with
different penalties; these are then assessed in the development set.
The model with the best result (e.g., evaluated using the correla-
Cross Validation
tions between observed and predicted scores) is subsequently
To evaluate the predictions made by a statistical model, it is applied in the test set of the outside fold. This procedure is
possible to correlate the predicted values with the observed val- repeated for all folds.
ues (i.e., cor[y, ŷ]) using cross-validation. Cross validation is a
technique that may be used to evaluate and select a statistical Nested K-Fold Cross Validation. To use nested cross valida-
model with minimal overfit (for a more detailed discussion tion, including train and test loops in the inner fold of each outside
on cross validation in psychology, e.g., see Mosier, 1951; fold, is also a way to make better use of the data (see Figure 3). The
Yarkoni & Westfall, 2017). Overfitting arises when data- above cross validation method with test, train and development sets
specific errors or noise are modeled, which results in a focuses on achieving more accurate estimates of the machine learning
model that is bad at generalizing to other data. The cross- approach, which is good when comparing machine learning algo-
validation methods aim to limit this risk by estimating a rithms. The nested k-fold cross validation focuses on achieving a
 10 KJELL, GIORGI, AND SCHWARTZ

Figure 2
Cross Validation With Three Outside Folds; and ¾ Analysis and ¼ Development Set

inside fold of outside fold 1 Deve

Development/Assessment Train/Analysis
1

outside fold 1 Test Train

outside fold 22 Train Test Train
outside fold 33 Train Test
Settings in textTrain()
1
The best performing model with hyperparameters is applied on the test data outside_folds = 3
2-3
Inner folds of outside folds 2 and 3 are not shown inside_folds = 3/44
cv_method = "validation_split"
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Note. See the online article for the color version of this figure.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

more accurate model; but tends to take longer time because more to get predictions for words that were not in the original model/
models need to be fitted and evaluated. words list, since the estimated parameters are applied to the dimen-
sions of the word embeddings.
Applying Predictive Models to New Data
Functions
Language-based predictive models can be applied to new data.
Independently trained language models may be applied to new data textTrain(). The textTrain() function is used to examine
to examine semantic-psychological features such as valence or how well the word embeddings from a text can predict a numeric
arousal. It is also easy to share predictive models between projects or categorical variable, based on the different cross-validation
and researchers. For example, it is possible to develop predictive methods described above. The textTrain() function is a wrapper
models for valence and arousal, based on the Affective norms for for textTrainRegression() and textTrainRandomForest(). In the
English Words (ANEW; Bradley & Lang, 1999). The ANEW is a example below we examine how well the satisfaction text-responses
list of more than 1,000 words that have been rated according to va- can predict the rating scale scores from the Satisfaction with life
lence, dominance, and arousal by participants. A valence model can scale. We advise researchers to share their predictive models,
be created by predicting valence from the word embeddings of the for example, on their Open Science Framework account (https://
words in the list. First the model can be evaluated using cross vali- [Link]/) or GitHub. For this purpose, it is possible to attach a descrip-
dation to see how good it is; and then it can be saved without using tion of the data and model by describing it in the describe_model set-
cross validation to use all the data. This model can now be used to ting. For example, use the following format:
estimate the valence of any word or set of text represented by a
word embedding based on the same language model (and settings)
originally used to create the word embeddings in the predictive model_description = “author(s): XXX; data: N = XXX, population =
model. Hence, it is a flexible method considering that it is possible XXX; publication: title = XXX; description: e.g., measure details etc.”

Figure 3
Example of Nested Cross Validation With Two Outside Folds and Four Inside Folds

inside fold 1 Test Train

inside fold 2 Train Test Train
The best performing model with
hyperparameters is applied on inside fold 3 Train Test Train
the test data
inside fold 4 Train Test

outside fold 1 Test Train

outside fold 2 Train Test

inside fold 1 Test Train

inside fold 2 Train Test Train Settings in textTrain()
outside_folds = 2
inside fold 3 Train Test Train inside_folds = 4
inside fold 4 Train Test cv_method = "cv_folds"

Note. Prioritizing more outside than inside folds makes often more sense. See the online article for the color version of this figure.
 THE TEXT-PACKAGE 11

textTrainLists(). The textTrainLists() runs through several

# Examine the relationship between satisfactiontext and
the corresponding rating scale
text variables (i.e., lists of word embeddings) and/or numeric vari-
model_satisfactiontext_swls <- textTrain(
ables at the same time.
x = word_embeddings$texts$satisfactiontexts, # the predictor
# Predicting several outcomes from several word embeddings
variables (i.e., the word embeddings)
models_words_ratings <- textTrainLists(
y = Language_based_assessment_data_8$swlstotal, # the criterion
word_embeddings$texts[1:2],
variable (i.e., the rating scale score.
Language_based_assessment_data_8[5:6]
model_description = "author(s): Kjell, Giorgi, & Schwartz; )
data: N = 40, population = Online, Mechanical Turk; publication:
title = Example for demo; description: swls = the satisfaction # See results
with life scale" models_words_ratings$results
)
# OUTPUT
# Examine the correlation between predicted and observed
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

p_value_
Harmony in life scale scores descriptions correlation df p_value t_statistics alternative corrected
This document is copyrighted by the American Psychological Association or one of its allied publishers.

<chr> <chr> <chr> <chr> <chr> <chr> <dbl>

model_satisfactiontext_swls$results
satisfactiontexts_hilstotal 0.45 38 0.00,172 3.12 greater 3.44e-03
harmonytexts_hilstotal 0.69 38 5.36e-07 5.80 greater 2.15e-06
satisfactiontexts_swlstotal 0.54 38 0.00,017 3.94 greater 5.07e-04
# OUTPUT: harmonytexts_swlstotal 0.43 38 0.00,281 2.94 greater 3.44e-03

Pearson’s product-moment correlation

data: predy_y$predictions and predy_y$y
textPredict(). Trained models created by textTrain() can be
applied to new data sets. In the next example we download and
t = 3.9396; df = 38, p-value = .0001688
apply a model that has been trained to predict valence. This model
alternative hypothesis: true correlation is
was based on a list of nearly 14,000 words that were rated on va-
greater than 0
lence, arousal and dominance (Warriner et al., 2013). To achieve
95% confidence interval:
the prediction we use the textPredict() function.
0.3199934 1.0000000
sample estimates:
cor # Read a valence trained prediction model (download it from
0.5385082 [Link]
valence_Warriner_L11 <- readRDS(
# Examine the names in the object returned from "valence_Warriner_L11.rds"
training )
names(model_satisfactiontext_swls)
# Examine the model
#OUTPUT: valence_Warriner_L11

[1] "predictions" # PART OF THE OUTPUT

[2] "final_recipe"
Pearson’s product-moment correlation
[3] "final_model"
data: predy_y$predictions and predy_y$y
[4] "model_description"
t = 126.76, df = 13913, p-value < 2.2e-16
[5] "results"
alternative hypothesis: true correlation is
greater than 0
95 percent confidence interval:
The returned output from training includes: (a) the cross- 0.7255292 1.0000000
validated predictions with the observed values and row id; (b) sample estimates:
cor
the final “recipe” with information about how to preprocess
0.7320673
the data before using it for training/prediction; (c) the final # Apply the model to the satisfaction text
model with model information, which can be used for predic- satisfaction_text_valence <- textPredict(
tion, (d) the model description contain information about the valence_Warriner_L11,
options used to create the model as well as the information word_mbeddings$texts$satisfactiontexts
added by the user; and (e) the results including a comparison )
between cross-validated predictions and observed values/cate-
gories. The result for numeric values is by default a Pearson # Examine the correlation between the predicted valence and
correlation, which can be changed with method_cor. The the Satisfaction with life scale score
psych::[Link](
results for a classification task include a Fisher’s Exact test for
satisfaction_text_valence$word_embeddings__ypred,
Count Data, Pearson’s chi-square test with Yates’ continuity cor- Language_based_assessment_data_8$swlstotal
rection, a plotted ROC curve and metrics from the R-package )
yardstick (Kuhn & Vaughan, 2020b) including accuracy, balanced
accuracy (bal_accuracy), specificity (spec), sensitivity (sens), and # OUTPUT
kappa (kap).
 12 KJELL, GIORGI, AND SCHWARTZ

Figure 4
Call:psych::[Link](x = satisfaction_text_valence$.pred, y = Illustration of the Semantic Similarity of Two Words in a
Language_based_assessment_data_8$swlstotal) Simplified Two-Dimensional Embedding Space
Correlation matrix
[1] 0.64
Sample Size
[1] 40
Probability values adjusted for multiple tests.
[1] 0

Computing Semantic Similarity and Distance

Word embeddings may be used to measure how similar two
words/texts are in meaning. The values composing a word
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

embedding may be seen as coordinates in a high dimensional

This document is copyrighted by the American Psychological Association or one of its allied publishers.

space; and the more similar two embedding vectors are the closer
they are positioned in the space. There are many ways to capture
how closely positioned two vectors are; in text many common
similarity and distance measures can be used, including “cosine,”
“euclidean,” “maximum,” “manhattan,” “canberra,” “binary,” or
“minkowski” (for a review of similarity measures see Chandra- Note. This also holds when adding many more dimensions.
sekaran & Mago, 2022). The default similarity measure in text,
is to measure the cosine of the angle between them (see a simpli-
fied illustration in Figure 4). used to measure the semantic similarity between a person’s answer
Similar to a correlation, the cosine can range from
1 to 1 (see to whether they experience harmony in their life to the word norm
Wickens, 2014, for a discussion on how correlation and cosine describing harmony in life: If the score is high the person is seen to
have high harmony in life. Below this analysis is described.
are mathematically related); but in the embedding space the co-
sine between two words is typically not much below 0. The
smaller the angle, the higher the cosine is, and thus, the more # Compute semantic similarity scores between two text columns,
similar are the embeddings (this cosine measure is referred to as using the previously created word_embeddings.
a semantic similarity score). semantic_similarity_scores <- textSimilarity(
The cosine may be seen as an unstandardized effect size indicat- word_embeddings$texts$harmonytexts,
ing the strength of a relationship (Charikar, 2002); however, it word_embeddings$texts$satisfactiontexts
should be noted that its absolute value is not comparable between )

different models and model specification setups. For example,

# Look at the first scores
Ethayarajh (2019) demonstrated that word embeddings have differ- head(semantic_similarity_scores)
ent distributions in different BERT layers, so that two random words
on average tend to have higher cosine similarity in higher as # OUTPUT
opposed to lower layers. Thus, comparing two cosine based seman-
tic similarity scores between two different models or layers is not in- [1] .9281080 .9211980 .8998845 .8469789 .9453016 .9075929
terpretable; instead examine the relative difference in scores from # Read word norms text (later we will use these for the semantic
centrality plot)
the same model/layer(s).
word_norms <- [Link] (
"Word_Norms_Mental_Health_Kjell2018_text.csv"
Functions: textSimilarity() and textDistance() )

The textSimilarity() function computes the semantic similarity

# Read the word embeddings for the word norms
between two embeddings, and the textSimilarityNorm() computes word_norms_embeddings <- readRDS(
the semantic similarity between one text variable and a word norm. "Word_Norms_Mental_Health_Kjell2018_ text_embedding_L11
We also provide corresponding textDistance() and textDistance- .rds"
Norm() functions that compute the distance between the vectors )
rather than the similarity (i.e., greater value indicates more differ-
ence rather than more similarity). For textDistance() the default is # Examine which word norms there are.
names(word_norms_embeddings$texts)
euclidean distance, and for textSimilarity() the default is cosine sim-
ilarity. Semantic similarity scores can, for example, be used to mea-
# OUTPUT
sure psychological constructs independent from rating scales by
using word norms that represent the to-be-measured construct [1] "harmonynorm” “disharmonynorm” "satisfactionnorm"
(Kjell et al., 2019). A word norm can be created by asking partici- [4] "dissatisfactionnorm” “worrynorm" "depressionnorm"
pants to describe a psychological construct. The word norm can be
 THE TEXT-PACKAGE 13

vector; see Figure 5 for a visualization). More precisely, the plot is

[7] "notatallworried” “notatalldepressed" "singlewords_we" based on the following steps:
# Compute semantic similarity score between the harmony Preprocessing.
answers and the harmony norm
# Note that the descriptive word answers are used instead of 1. Responses are divided into two groups (G1 and G2;
text answers to correspond with how the word norm was created. where a scale variable is split according to mean or lower
norm_similarity_scores_harmony <- textSimilarityNorm( and higher quartile).
word_embeddings$texts$harmonywords,
word_norms_embeddings$texts$harmonynorm 2. The aggregated word embeddings of the two groups are
) computed: The G1 split aggregation embedding and the
G2 split aggregation embedding.
# Correlating the semantic measure with the corresponding
rating scale Supervised dimension projection statistics.
psych::[Link](
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

norm_similarity_scores_harmony, 3. An Aggregated direction embedding is computed:

This document is copyrighted by the American Psychological Association or one of its allied publishers.

Language_based_assessment_data_8$hilstotal Aggregated direction embeddings = G2 split aggregation

)
embedding
G1 split aggregation embedding.
# OUTPUT
Call:psych::[Link](x = norm_similarity_scores_harmony, y =
So, for example, the direction of harmony = Group(high
Language_based_assessment_data_8$hilstotal) harmony)
Group(low harmony); where the direction is
Correlation matrix seen to go through the origin and the aggregated direction
[1] .59 embedding.
Sample Size
[1] 40
4. All individual word embeddings are positioned (or anch-
Probability values adjusted for multiple tests.
[1] 0
ored) to the same point. So, for each word: Anchored
embedding = original embedding
aggregation embed-
ding of G1 and G2 from Step 2.
Extended Functionality
5. To project onto the Aggregated direction embedding (i.e.,
In addition to implementations of standard analyses of text (the from Step 3), the dot product is computed between the
functions listed above), text comes with several less established Anchored embedding of all individual words (i.e., from Step
novel functions. These are intended to support the core functional- 4) and the Aggregated direction embedding (i.e., Step 3).
ities (and overtime become more empirically evaluated). That is, dot product(Anchored we, Aggregated direction
we) = a point on the direction.
Words’ Position in the Embedding Space
Computing p-values with a permutation procedure.
There are many different ways to visualize the words in a data
set such as multidimensional scaling (MDS; e.g., Borg & Groenen, 6. A permuted null distribution of Supervised Dimension
2005), t-Distributed Stochastic Neighbor Embedding (t-SNE; Van Projections is created by computing the dot product
der Maaten, & Hinton, 2008) and Principal Component Analysis between randomly selected word embeddings from G1 and
(PCA; see Appendix for the text functions textPCA() and textPCA- G2, and a Permuted aggregated direction embedding (i.e.,
Plot()). Here we will briefly demonstrate two plot functions in text the direction embedding is also created by randomly swap-
for demonstrating potential basic uses of the word embeddings: ping words from G1 and G2).
The Supervised Dimension Projection Plot shows words that are
significantly related to one compared with another group, where 7. The p-values for each word are computed by comparing
the test statistics are tested in permutation test procedures. The their Supervised Dimension Projection with the permuted
Semantic Centrality Plot shows the words that are most semanti- null distribution of Supervised Dimension Projections from
cally central to a set of texts/words. Both functions plot the words’ Step 6, as in previous descriptions; while correcting for
position based on their word embeddings. multiple comparisons according to selected methods.
The Supervised Dimension Projection
The Supervised Dimension Projection compares two groups
Functions
(e.g., intervention versus nonintervention participant), responses to The plotting is made with two functions, in two steps: First, text-
different questions (e.g., harmony versus satisfaction responses), or Projection() analyzes the data, including Supervised Dimension Pro-
low versus high scorers on a rating scale using median split or jections, null distributions, p-values, and frequencies. Second,
lower/higher quartiles. In short, we construct an embedding that textProjectionPlot() uses the output from textProjection() to plot the
captures the difference between the two groups, a vector that forms words, including providing design options for the figure. Dividing the
a line through the origin (the aggregated direction embedding); plotting procedure into these two steps makes the process more transpar-
then all individual words are “projected” onto that direction line ent (i.e., the user naturally sees the output according to which the words
(where dot product is computed to “project” a vector onto another will be plotted) as well as being more flexible by enabling the user to try
 14 KJELL, GIORGI, AND SCHWARTZ

Figure 5
Illustrations of the Supervised Dimension Projection Method
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

Note. The illustration exemplifies the reduction of two dimensions to one dimension, where in practice there
are many more dimensions. See the online article for the color version of this figure.

out different visual settings without the need to rerun the heavy compu-
tational processes. # Pre-processing data for plotting
To plot words, each individual word type needs to be repre- projection_results <- textProjection(
words = Language_based_assessment_data_8$harmonywords,
sented by an embedding: a word type embedding (i.e., a single
word_embeddings = harmony_words_embeddings$texts,
embedding per word type taken from aggregating token embed-
word_types_embeddings = harmony_words_embeddings$word_types,
dings across all instances of all word types in the given data).4 In
x = Language_based_assessment_data_8$hilstotal,
textEmbed() the optional setting aggregation_from_tokens_to_-
y = Language_based_assessment_data_8$age
word_types, controls how the word types embeddings are aggre- )
gated (default = “mean”). projection_results$word_data

textProjection().

# Extract word type embeddings and text embeddings for

harmony words
harmony_words_embeddings <- textEmbed(
4
texts = Language_based_assessment_data_8["harmonywords"], Word “types” refers to a word in general as opposed to a word “token”
aggregation_from_layers_to_tokens = "concatenate", that refers to a specific instance of a word. For example, in “The horse
aggregation_from_tokens_to_texts = "mean",
raced past the other horse” the word type “horse” appears twice: as the
second- and seventh-word tokens. Each token for horse can have a different
aggregation_from_tokens_to_word_types = "mean",
embedding so to get a single embedding for the word type, “horse,” they
keep_token_embeddings = FALSE must be aggregated in some way. For noncontextual embeddings that are
) always the same for the same word, all one has, technically, are
embeddings of the word type and not the specific instance (tokens).
 THE TEXT-PACKAGE 15

# PART OF OUTPUT plot_n_word_extreme = 10,

words dot.x p_values_dot.x plot_n_word_frequency = 5,
<chr> <dbl> <dbl> plot_n_words_middle = 5,
accepting 1.21 0.885 y_axes = FALSE,
agreeing 15.1 0.0000247
p_alpha = .05,
alcohol
10.2 0.0366
amazed
0.678 0.818 p_adjust_method = "fdr",
title_top = "Harmony words (Supervised
# . . . with 288 more rows, and 6 more variables
Dimension Projection"),
x_axes_label = "Low vs. High Harmony in Life
Scale Score",
textProjectionPlot(). y_axes_label = "",
bivariate_color_codes = c(
"#FFFFFF", "#FFFFFF", "#FFFFFF",
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

# Supervised Dimension Projection Plot

"#E07F6A", "#EAEAEA", "#85DB8E",
This document is copyrighted by the American Psychological Association or one of its allied publishers.

# To avoid warnings—and that words do not get plotted, first

"#FFFFFF", "#FFFFFF", "#FFFFFF"
increase the [Link] for the entire session:
)
options([Link] = 1000)
# View plot (see Figure 6)
plot_projection$final_plot
plot_projection <- textProjectionPlot(
word_data = projection_results,
min_freq_words_plot = 1,

Figure 6
A One-Dimensional Plot From textProjectionPlot()

Harmony Words Responses (Supervised Dimension Projection)

humiliated
hopeless impatient warm happy
extended unstable
anxious sad amazed brotherhood
driven
love content fair sunshine being
incompatible rattled
ashamed unsure
overwhelmed kinship peaceful respectfulcaring well good
accepting positive
scattered
peace
−30 −20 −10 0 10 20 30
Low to High Harmony in Life Scale Score

SDP Frequency
71 142 82 a a a
x
2.5 5.0 7.5

Note. DPP = Dot Product Projection. The dots represent the point for visible words. See the online article for the color version of this
figure.

# Supervised Dimension Projection Plot (Supervised Dimension Projection)",

plot_projection_2D <- textProjectionPlot( x_axes_label = "Low vs. High Harmony in Life Scale Score",
word_data = projection_results, y_axes_label = "Low [Link] Age",
min_freq_words_plot = 1, bivariate_color_codes = c(
plot_n_word_extreme = 10, "#E07F6a", "#60A1F7", "#85DB8E",
plot_n_word_frequency = 5, "#FF0000", "#EAEAEA", "#5dc688",
plot_n_words_middle = 5, "#E07F6a", "#60A1F7", "#85DB8E"
y_axes = TRUE, )
p_alpha = .05, )
p_adjust_method = "fdr", # View plot (see Figure 7)
title_top = "Harmony Words Responses plot_projection_2D$final_plot
 16 KJELL, GIORGI, AND SCHWARTZ

Figure 7
A Two-Dimensional Plot From textProjectionPlot()
Harmony Words Responses (Supervised Dimension Projection)
fear sadness guilt
nature pleasure being
loss
anger depression feelings caring happy well
10 brotherhood
love fair warm good
Low [Link] Age

incompatible unstable dislike sad kinship content positive

0
humiliatedashamed anxious tired
amused
accepting present
agreeing peace sunshine
respectful
hopeless hatred compatible
impatient overwhelmed amazed quiet peaceful
−10 extended
open
rattled confused clash driven balance
disconnected connected cold
unsure orderly grounded
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

unfit
scattered
This document is copyrighted by the American Psychological Association or one of its allied publishers.

−20 proud communion

− 30 −20 −10 0 10 20 30
Low vs. High Harmony in Life Scale Score

SDP Frequency
2 37 0

33 44 64 a a a
y

36 37 18
2.5 5.0 7.5
x

Note. A supervised dimension projection plot of words significantly differing between low versus high harmony in life scale score (x-axis) and low versus high age
(y-axis). The font size of the words indicates their frequency. The color indicates whether a word is significant or not significant (gray) when correcting for multiple
comparisons using the false discovery rate (FDR). The color-legend in the lower left corner indicates the color and number of significant words in each part of the fig-
ure (e.g., there are 18 light green words that are significantly high on the x-axis and low on the y-axis). The dots represent the point for visible words. The values on
the x- and y-axes represent the dot product projection value (these should be compared with caution between figures); DPP = Dot product projection. See the online
article for the color version of this figure.

Words’ Semantic Centrality

word_types_embeddings = word_norms_
The Semantic Centrality Plot aims to highlight words that are embeddings$word_types
semantically similar to the aggregated word embeddings of all )
words in the given text variable. Hence, it describes the psychologi- centrality_plot <- textCentralityPlot(
cal construct or latent meaning of a text under investigation in the word_data = centrality_results,
word embedding space. The aim is to highlight words from one min_freq_words_test = 2,
type of response, rather than comparing two groups/dimensions. plot_n_word_extreme = 10,
The statistics are computed with the textCentrality() function. This plot_n_word_frequency = 5,
is achieved in the following steps: plot_n_words_middle = 5,
title_top = "Satisfaction with life word norm:
Semantic Centrality Plot",
1. Computing the Aggregated word embedding (Awe) based
x_axes_label = "Satisfaction with Life
on all words.
Semantic Centrality"
)
2. Computing the observed semantic similarity scores
between individual word’s embeddings (Iwe) and the # View plot (see Figure 8)
aggregated word embedding (Awe). centrality_plot$final_plot
cos(Iwe, Awe) = semantic similarity score

Using the textCentralityPlot() it is possible to select the most

extreme semantic centrality scores, middle scores as well as select-
ing words based on their frequency.
Summary
Functions
The text-package has two main aims: First, to work as a modular
solution for transforming text to state-of-the-art word embeddings
# Computing words’ centrality (semantic similarity) score
for R-users. Second, to work as an end-to-end solution focusing
to the aggregated embedding of all words
on relevant functions for social sciences and human-level analyses.
centrality_results <- textCentrality(
words = word_norms$satisfactionnorm,
It is our hope that the text-package can increase the ability of psycho-
word_embeddings = word_norms_embeddings$texts logical scientists to analyze, with state-of-the-art computational
$satisfactionnorm, techniques, one of the fundamental and key behaviors of people: nat-
ural language.
 THE TEXT-PACKAGE 17

Figure 8 Benoit, K., Watanabe, K., Wang, H., Nulty, P., Obeng, A., Müller, S., &
A Plot From textCentralityPlot() Matsuo, A. (2018). quanteda: An R package for the quantitative analysis
Satisfaction with life word norm: Semantic Centrality Plot
of textual data. Journal of Open Source Software, 3(30), Article 774.
[Link]
friendship family happiness Bird, S., Klein, E., & Loper, E. (2009). Natural Language Processing with
generous purpose education excellent mind strong
comfortable
friends success
warmth
clothes prepared tv plants food happy fun
love excitement
security Python: Analyzing text with the natural language toolkit. O’Reilly
money health Media, Inc.
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation.
0.5 0.6 0 .7 0 .8 0.9
Journal of Machine Learning Research, 3, 993–1022.
Satisfaction with Life Semantic Centrality
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S.,
Frequency
Bernstein, M. S., Bohg, J., Bosselut, A., & Brunskill, E. (2021). On the
a a a a opportunities and risks of foundation models. arXiv, Preprint
10 20 30 40

ArXiv:2108.07258.
Note. A Semantic Centrality Plot illustrating the words composing the
Borg, I., & Groenen, P. J. (2005). Modern multidimensional scaling: Theory
Satisfaction with life word norm. The size of the words represents their fre-
and applications. Springer Science & Business Media.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

quency, the color their position. The values on the x-axes represent the
Bradley, M. M., & Lang, P. J. (1999). Affective norms for English words
This document is copyrighted by the American Psychological Association or one of its allied publishers.

semantic similarity scores (measured as the cosine score) between each

word’s embedding and the aggregated word embedding of all words in the (ANEW): Instruction manual and affective ratings. Technical Report
word norm. The dots represent the point for visible words. See the online arti- C-1, The Center for Research in Psychophysiology, University of Florida.
cle for the color version of this figure. Bratt, J., & Harmon, J. (2020). RBERT: R Implementation of BERT. R pack-
age version 0.1.11. [Link]
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P.,
R-Packages Neelakantan, A., Shyam, P., Sastry, G., & Askell, A. (2020). Language
R-packages used in text include: models are few-shot learners. arXiv, Preprint ArXiv:2005.14165.
Browne, M. W. (2000). Cross-validation methods. Journal of Mathematical
dplyr (Wickham et al., 2020), tokenizers (Mullen et al., 2018), psych Psychology, 44(1), 108–132. [Link]
(Revelle, 2019), stringr (Wickham, 2019), tidyr (Wickham & Henry, Campbell, R. S., & Pennebaker, J. W. (2003). The secret life of pronouns:
2020), ggplot2 (Wickham, 2016), ggrepel (Slowikowski, 2020), cow- Flexibility in writing style and physical health. Psychological Science,
plot (Wilke, 2019), scales (Wickham & Seidel, 2020), rlang (Henry &
14(1), 60–65. [Link]
Wickham, 2020b), purrr (Henry & Wickham, 2020a), Matrix (Bates
Carlson, T. A., Simmons, R. A., Kriegeskorte, N., & Slevc, L. R. (2014). The
& Maechler, 2019), stringi (Gagolewski, 2020), [Link] (Dowle &
emergence of semantic meaning in the ventral temporal pathway. Journal
Srinivasan, 2019), magrittr (Bache & Wickham, 2014), parsnip (Kuhn
of Cognitive Neuroscience, 26(1), 120–131. [Link]
& Vaughan, 2020a), recipes (Kuhn & Wickham, 2020), reticulate
a_00458
(Ushey et al., 2020), rsample (Kuhn et al., 2020), tune (Kuhn, 2020),
Chan, C., Chan, G. C., Leeper, T. J., & Becker, J. (2018). rio: A Swiss-army
workflows (Vaughan, 2020), yardstick (Kuhn & Vaughan, 2020b),
knife for data file I/O. R package version 0.5.29. [Link]
broom (Robinson & Hayes, 2020), knitr (Xie, 2014), rmarkdown (Xie
web/packages/rio/[Link]
et al., 2018), testthat (Wickham, 2011), tibble (Müller & Wickham,
Chandrasekaran, D., & Mago, V. (2022). Evolution of semantic similarity—
2020), and rio (Chan et al., 2018).
A survey. [CSUR]. ACM Computing Surveys, 54(2), 1–37. [Link]
10.1145/3440755
Python Libraries
Charikar, M. S. (2002). Similarity estimation techniques from rounding algo-
Python packages integrated within the text-package include: rithms. In Proceedings of the Thirty-Fourth Annual ACM Symposium on
PyTorch (Paszke et al., 2019), transformers (Wolf et al., 2019), nltk Theory of Computing (pp. 380–;388;). Association for Computing
(Bird et al., 2009), and numpy (Oliphant, 2006). Machinery. [Link]
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman,
R. (1990). Indexing by latent semantic analysis. Journal of the American
References Society for Information Science, 41(6), 391–407. [Link]
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., (SICI)1097-4571(199009)41:6,391::[Link];2-9
& McDermott, M. (2019). Publicly available clinical BERT embeddings. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT:
arXiv, Preprint ArXiv:1904.03323. Pre-training of Deep Bidirectional Transformers for Language
Andersen, N., & Zehner, F. (2021). shinyReCoR: A shiny application for Understanding. In Proceedings of the 2019 Conference of the North
automatically coding text responses using R. Psych, 3(3), 422–446. American Chapter of the Association for Computational Linguistics:
[Link] Human Language Technologies, Volume 1 (Long and Short Papers)
Babić , K., Martinčić -Ipšić , S., & Meštrović , A. (2020). Survey of neural (pp. 4171–4186). Association for Computational Linguistics. https://
text representation models. Information, 11(11), Article 511. https:// [Link]/10.18653/v1/N19-1423
[Link]/10.3390/info11110511 Diener, E., Emmons, R. A., Larsen, R. J., & Griffin, S. (1985). The satisfac-
Bache, S. M., & Wickham, H. (2014). magrittr: A forward-pipe operator for tion with life scale. Journal of Personality Assessment, 49(1), 71–75.
R. [Link] [Link]
Bates, D., & Maechler, M. (2019). Matrix: Sparse and dense matrix classes Dougal, S., & Rotello, C. M. (2007). “Remembering” emotional words is
and methods. [Link] based on response bias, not recollection. Psychonomic Bulletin &
Beltagy, I., Lo, K., & Cohan, A. (2019). SciBERT: A pretrained language Review, 14(3), 423–429. [Link]
model for scientific text. In Proceedings of the 2019 Conference on Dowle, M., & Srinivasan, A. (2019). [Link]: Extension of ‘[Link]’.
Empirical Methods in Natural Language Processing and the 9th [Link]
International Joint Conference on Natural Language Processing Eichstaedt, J. C., Kern, M. L., Yaden, D. B., Schwartz, H. A., Giorgi, S.,
(EMNLP-IJCNLP) (pp. 3606–3611). Association for Computational Park, G., Hagan, C. A., Tobolsky, V., Smith, L. K., & Buffone, A.
Linguistics. [Link] (2021). Closed-and open-vocabulary approaches to text analysis: A
 18 KJELL, GIORGI, AND SCHWARTZ

review, quantitative comparison, and recommendations. Psychological Hossain, N., Krumm, J., Gamon, M., & Kautz, H. (2020). Semeval-2020
Methods, 26(4), 398–427. [Link] Task 7: Assessing humor in edited news headlines. arXiv, Preprint
Eichstaedt, J. C., Schwartz, H. A., Kern, M. L., Park, G., Labarthe, D. R., arXiv:2008.00304. [Link]
Merchant, R. M., Jha, S., Agrawal, M., Dziurzynski, L. A., Sap, M., Iliev, R., Dehghani, M., & Sagi, E. (2015). Automated text analysis in psy-
Weeg, C., Larson, E. E., Ungar, L. H., & Seligman, M. E. (2015). chology: Methods, applications, and future developments. Language
Psychological language on Twitter predicts county-level heart disease and Cognition, 7(2), 265–290. [Link]
mortality. Psychological Science, 26(2), 159–169. [Link] James, G., Witten, D., Hastie, T., Tibshirani, R., Sohil, F., Sohali, M. U., &
.1177/0956797614557867 Shabbir, J. (2021). An introduction to statistical learning with applications
Eichstaedt, J. C., Smith, R. J., Merchant, R. M., Ungar, L. H., Crutchley, P., in R. [Link]
Preoţ iuc-Pietro, D., Asch, D. A., & Schwartz, H. A. (2018). Facebook Jawahar, G., Sagot, B., & Seddah, D. (2019). What does BERT learn about
language predicts depression in medical records. Proceedings of the the structure of language? In ACL 2019-57th Annual Meeting of the
National Academy of Sciences of the United States of America, 115(44), Association for Computational Linguistics, July 2019, Florence, Italy
11203–11208. [Link] (pp. 3651–3657). Association for Computational Linguistics.
Erk, K. (2012). Vector space models of word meaning and phrase meaning: Jurafsky, D., & Martin, J. (2020). Speech and language processing: An intro-
A survey. Language and Linguistics Compass, 6(10), 635–653. https:// duction to Natural Language Processing, computational linguistics, and
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

[Link]/10.1002/lnco.362 speech recognition. [Link]

This document is copyrighted by the American Psychological Association or one of its allied publishers.

Ethayarajh, K. (2019). How contextual are contextualized word representa- Kern, M. L., Park, G., Eichstaedt, J. C., Schwartz, H. A., Sap, M., Smith, L.
tions? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. K., & Ungar, L. H. (2016). Gaining insights from social media language:
arXiv, Preprint ArXiv:1909.00512. [Link] Methodologies and challenges. Psychological Methods, 21(4), 507–525.
1006 [Link]
Feinerer, I., & Hornik, K. (2019). tm: Text mining package. [Link] Kjell, K., Johnsson, P., & Sikström, S. (2021). Freely generated word
[Link]/web/packages/tm/ responses analyzed with artificial intelligence predict self-reported symp-
Firth, J. R. (1957). A synopsis of linguistic theory 1930–1955. In F. Palmer toms of depression, anxiety, and worry. Frontiers in Psychology, 12,
(Ed.), Studies in linguistic analysis: Philological Society (reprinted in
Article 602581. [Link]
1968: Selected Papers of J. R. Firth). Longman. Kjell, O. N. E., Daukantaitė, D., Hefferon, K., & Sikström, S. (2016). The
Gagné, C. L., Spalding, T. L., & Ji, H. (2005). Re-examining evidence for the
Harmony in Life Scale Complements the Satisfaction with Life Scale:
use of independent relational representations during conceptual combina-
Expanding the conceptualization of the cognitive component of subjective
tion. Journal of Memory and Language, 53(3), 445–455. [Link]
well-being. Social Indicators Research, 126(2), 893–919. [Link]
10.1016/[Link].2005.03.006
10.1007/s11205-015-0903-z
Gagolewski, M. (2020). R package stringi: Character string processing facil-
Kjell, O. N. E., Kjell, K., Garcia, D., & Sikström, S. (2019). Semantic mea-
ities. [Link]
sures: Using Natural Language Processing to measure, differentiate, and
Ganesan, A. V., Matero, M., Ravula, A. R., Vu, H., & Schwartz, H. A.
describe psychological constructs. Psychological Methods, 24(1), 92–
(2021). Empirical evaluation of pre-trained transformers for human-level
115. [Link]
NLP: The role of sample size and dimensionality. arXiv, Preprint
Kjell, O. N. E., Sikström, S., Kjell, K., & Schwartz, H. A. (2022). Natural
ArXiv:2105.03484. [Link]
language analyzed with AI-based transformers predict traditional subjec-
Ganguli, S., & Sompolinsky, H. (2012). Compressed sensing, sparsity, and
dimensionality in neuronal information processing and data analysis. tive well-being measures approaching the theoretical upper limits in accu-
Annual Review of Neuroscience, 35(1), 485–508. [Link] racy. Scientific Reports, 12(1), Article 3918. [Link]
.1146/annurev-neuro-062111-150410 s41598-022-07520-w
Garcia, D., & Sikström, S. (2013). Quantifying the semantic representations Kuhn, M. (2020). tune: Tidy tuning tools. [Link]
of adolescents’ memories of positive and negative life events. Journal of package=tune
Happiness Studies, 14(4), 1309–1323. [Link] Kuhn, M., & Vaughan, D. (2020a). parsnip: A common API to modeling and
012-9385-8 analysis functions. [Link]
Golub, G., & Kahan, W. (1965). Calculating the singular values and Kuhn, M., & Vaughan, D. (2020b). yardstick: Tidy characterizations of
pseudo-inverse of a matrix. Journal of the Society for Industrial and model performance. [Link]
Applied Mathematics, Series B. Numerical Analysis, 2(2), 205–224. Kuhn, M., & Wickham, H. (2020). recipes: Preprocessing tools to create
[Link] design matrices. [Link]
Grolemund, G., & Wickham, H. (2018). R for data science. O’Reilly Media. Kuhn, M., Chow, F., & Wickham, H. (2020). rsample: General resampling
Henry, L., & Wickham, H. (2020a). purrr: Functional programming tools. infrastructure. [Link]
[Link] Lauriola, I., Lavelli, A., & Aiolli, F. (2022). An introduction to Deep
Henry, L., & Wickham, H. (2020b). rlang: Functions for base types and core Learning in Natural Language Processing: Models, techniques, and
R and “Tidyverse” features. [Link] tools. Neurocomputing, 470, 443–456. [Link]
Ho, T. K. (1995). Random decision forests. In Proceedings of 3rd .2021.05.103
International Conference on Document Analysis and Recognition, Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R.
August 1995 (Vol. 1, pp. 278–282). IEEE Computer Society. (2019). Albert: A lite bert for self-supervised learning of language repre-
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural sentations. arXiv preprint arXiv:1909.11942.
Computation, 9(8), 1735–1780. [Link] Libovickỳ , J., Rosa, R., & Fraser, A. (2019). How language-neutral is mul-
.1735 tilingual BERT? arXiv, Preprint ArXiv:1911.03310.
Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M.,
for nonorthogonal problems. Technometrics, 12(1), 55–67. [Link] Zettlemoyer, L., & Stoyanov, V. (2019). Roberta: A robustly optimized
10.1080/00401706.1970.10488634 BERT pretraining approach. arXiv, Preprint ArXiv:1907.11692.
Honnibal, M., & Montani, I. (2017). spaCy 2: Natural language understand- Matero, M., Hung, A., & Schwartz, H. A. (2021). Understanding RoBERTa’s
ing with Bloom embeddings, convolutional neural networks and incre- mood: The role of contextual-embeddings as user-representations for
mental parsing. Advance online publication. depression prediction. arXiv, Preprint ArXiv:2112.13795.
 THE TEXT-PACKAGE 19

Matero, M., Idnani, A., Son, Y., Giorgi, S., Vu, H., Zamani, M., Limbachiya, Robinson, D., & Hayes, A. (2020). broom: Convert statistical analysis
P., Guntuku, S. C., & Schwartz, H. A. (2019). Suicide risk assessment with objects into tidy tibbles. [Link]
multi-level dual-context language and BERT. In Proceedings of the sixth Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A primer in Bertology:
workshop on computational linguistics and clinical psychology (pp. 39– What we know about how bert works. arXiv, Preprint ArXiv:2002.
44). Association for Computational Linguistics. [Link] 12327.
.18653/v1/W19-3005 Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). version of BERT: Smaller, faster, cheaper and lighter. ArXiv, Preprint
Distributed representations of words and phrases and their compositional- ArXiv:1910.01108.
ity. In NIPS’13: Proceedings of the 26th International Conference on Schwartz, H., Eichstaedt, J., Kern, M., Dziurzynski, L., Lucas, R., Agrawal,
Neural Information Processing Systems (pp. 3111–3119). Curran M., Park, G., Lakshmikanth, S., Jha, S., Seligman, M., & Ungar, L. (2013).
Associates, Inc. Characterizing geographic variation in well-being using tweets.
Morishita, T., Morio, G., Ozaki, H., & Miyoshi, T. (2020). Hitachi at Proceedings of the International AAAI Conference on Web and Social
SemEval-2020 task 7: Stacking at scale with heterogeneous language Media, 7(1), 583–591.
models for humor recognition. In Proceedings of the Fourteenth Schwartz, H. A., Giorgi, S., Sap, M., Crutchley, P., Ungar, L., & Eichstaedt,
Workshop on Semantic Evaluation (pp. 791–803). International J. (2017, September). Dlatk: Differential language analysis toolkit. In
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Proceedings of the 2017 conference on empirical methods in Natural

This document is copyrighted by the American Psychological Association or one of its allied publishers.

Committee for Computational Linguistics. [Link]

[Link]-1.101 Language Processing: System demonstrations (pp. 55–60). ACL.
Mosier, C. I. (1951). I. Problems and designs of cross-validation 1. Sikström, S., Kjell, O. N. E., & Kjell, K. (2018). Semantic excel: An introduc-
Educational and Psychological Measurement, 11(1), 5–11. https:// tion to a user-friendly online software application for statistical analyses
[Link]/10.1177/001316445101100101 of text data. PsyArXiv, [Link]
Mullen, L. A., Benoit, K., Keyes, O., Selivanov, D., & Arnold, J. (2018). Silge, J., & Robinson, D. (2016). tidytext: Text mining and analysis using
Fast, consistent tokenization of natural language text. Journal of Open tidy data principles in R. Journal of Open Source Software, 1(3), Article
37. [Link]
Source Software, 3(23), Article 655. [Link]
Slowikowski, K. (2020). ggrepel: Automatically position non-overlapping
Müller, K., & Wickham, H. (2020). tibble: Simple data frames. [Link]
text labels with “ggplot2.” [Link]
.[Link]/package=tibble
Son, Y., Clouston, S. A., Kotov, R., Eichstaedt, J. C., Bromet, E. J.,
Nayak, P. (2019). Understanding searches better than ever before. https://
Luft, B. J., & Schwartz, H. A. (2020). World Trade Center responders
[Link]/products/search/search-language-understanding-bert/.
in their own words: Predicting PTSD symptom trajectories with
Oliphant, T. E. (2006). A guide to NumPy (Vol. 1). Trelgol Publishing USA.
Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1957). The measurement of AI-based language analyses of interviews. arXiv, Preprint ArXiv:2011.
06457.
meaning. University of Illinois Press.
Sun, J., Schwartz, H. A., Son, Y., Kern, M. L., & Vazire, S. (2020). The lan-
Otter, D. W., Medina, J. R., & Kalita, J. K. (2019). A survey of the usages of
guage of well-being: Tracking fluctuations in emotion experience through
deep learning in Natural Language Processing. arXiv, 1807.10854 [Cs].
everyday speech. Journal of Personality and Social Psychology, 118(2),
[Link]
364–387. [Link]
Palomino, D., & Ochoa-Luna, J. (2020). Palomino-Ochoa at SemEval-2020
Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of
Task 9: Robust system based on transformer for code-mixed sentiment
words: LIWC and computerized text analysis methods. Journal of
classification. arXiv, Preprint arXiv:2011.09448. [Link]
Language and Social Psychology, 29(1), 24–54. [Link]
.18653/v1/[Link]-1.124
0261927X09351676
Park, G., Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Kosinski, M.,
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso.
Stillwell, D. J., Ungar, L. H., & Seligman, M. E. P. (2015). Automatic per-
Journal of the Royal Statistical Society. Series B. Methodological, 58(1),
sonality assessment through social media language. Journal of Personality
267–288. [Link]
and Social Psychology, 108(6), 934–952. [Link]
Ushey, K., Allaire, J. J., & Tang, Y. (2020). reticulate: Interface to “Python.”
pspp0000020
[Link]
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen,
Van der Maaten, L., & Hinton, G. (2008). Visualizing data using
T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., t-SNE. Journal of Machine Learning Research, 9(11), 2579–
DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., 2605.
. . . Chintala, S. (2019). PyTorch: An imperative style, high-performance Van Rossum, G., & Drake, F. L., Jr. (1995). Python reference manual.
deep learning library. In E. Fox & R. Garnett (Eds.), Advances in neural Centrum voor Wiskunde en Informatica Amsterdam.
information processing systems: Vol. 32 (pp. 8024–8035). Curran Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N.,
Associates, Inc. [Link] Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances
[Link] in Neural Information Processing Systems (pp. 5998–6008). MIT Press.
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020). Pre-trained Vaughan, D. (2020). workflows: Modeling workflows. [Link]
models for Natural Language Processing: A survey. Science China. [Link]/package=workflows
Technological Sciences, 63(10), 1872–1897. [Link] Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013). Norms of valence,
s11431-020-1647-3 arousal, and dominance for 13,915 English lemmas. Behavior Research
R Core Team. (2022). R: A Language and Environment for Statistical Methods, 45(4), 1191–1207. [Link]
Computing. R Foundation for Statistical Computing. [Link] Wickens, T. D. (2014). The geometry of multivariate statistics. Psychology
[Link]/ Press. [Link]
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Wickham, H. (2011). testthat: Get Started with Testing. The R Journal, 3(1),
Language models are unsupervised multitask learners. OpenAI Blog, 5–10. [Link]
1(8), 9. Wickham, H. (2016). ggplot2: Elegant graphics for data analysis.
Revelle, W. (2019). psych: Procedures for psychological, psychometric, and Springer-Verlag New York. [Link]
personality research. Northwestern University. [Link] Wickham, H. (2019). stringr: Simple, consistent wrappers for common string
.org/package=psych operations. [Link]
 20 KJELL, GIORGI, AND SCHWARTZ

Wickham, H., & Henry, L. (2020). tidyr: Tidy messy data. [Link] Xie, Y., Allaire, J. J., & Grolemund, G. (2018). R markdown: The definitive
[Link]/package=tidyr guide. CRC Press. [Link]
Wickham, H., & Seidel, D. (2020). scales: Scale functions for visualization. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., & Le, Q. V.
[Link] (2019). Xlnet: Generalized autoregressive pretraining for language under-
Wickham, H., François, R., Henry, L., & Müller, K. (2020). dplyr: A gram- standing. Advances in neural information processing systems, 32.
mar of data manipulation. [Link] Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in
Wilke, C. O. (2019). cowplot: Streamlined plot theme and plot annotations psychology: Lessons from machine learning. Perspectives on
for “ggplot2.” [Link] Psychological Science, 12(6), 1100–1122. [Link]
Wold, S., Esbensen, K., & Geladi, P. (1987). Principal component analysis. 1745691617693393
Chemometrics and Intelligent Laboratory Systems, 2(1–3), 37–52. https:// Zirikly, A., Resnik, P., Uzuner, O., & Hollingshead, K. (2019). CLPsych
[Link]/10.1016/0169-7439(87)80084-9 2019 shared task: Predicting the degree of suicide risk in Reddit posts.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, In Proceedings of the sixth workshop on computational linguistics and
P., Rault, T., Louf, R., & Funtowicz, M. (2019). Huggingface’s transform- clinical psychology (pp. 24–33). ACL.
ers: State-of-the-art Natural Language Processing. ArXiv:1910.03771v1. Zou, H., & Hastie, T. (2005). Regularization and variable selection via the
Xie, Y. (2014). knitr: A comprehensive tool for reproducible research in R. In elastic net. Journal of the Royal Statistical Society. Series B, Statistical
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

V. Stodden, F. Leisch, & R. D. Peng (Eds.), Implementing reproducible Methodology, 67(2), 301–320. [Link]
This document is copyrighted by the American Psychological Association or one of its allied publishers.

computational research (pp. 3–32). Chapman and Hall. .00503.x

(Appendix follows)
 THE TEXT-PACKAGE 21

Appendix
PCA Plot

To create a traditional two-dimensional Principal

Component Analysis plot, it is possible to use textPCA and
textPCAPlot.

# PCA results to be plotted

textPCA_results <- textPCA(
words = Language_based_assessment_data_8$satisfactionwords,
word_types_embeddings = harmony_words_embeddings$word_types
)
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

# Plotting the PCA results

plot_PCA <- textPCAPlot(
word_data = textPCA_results,
min_freq_words_test = 2,
plot_n_word_extreme = 5,
plot_n_word_frequency = 5,
plot_n_words_middle = 5
)
# View plot (see Figure A1)
plot_PCA$final_plot

Figure A1
A plot from textPCA()

Principal Component (PC) Plot

frustration

worried anxious
intelligence
angry depressed

unsure
PC2

love
money family happy cheerful lost
food content positive loved
friends
healthy driven
strong
good

loving
−15 −10 −5 0 5 10
PC1

PC Frequency
0 3 2

a a a a
PC2

5 41 2

0 4 1
4 8 12 16
PC1

Note. See the online article for the color version of this figure.

Received March 17, 2021

Revision received September 19, 2022
Accepted October 4, 2022 n