Temporal Knowledge Graph Forecasting

Without Knowledge Using In-Context Learning

Dong-Ho Lee∗, Kian Ahrabian∗ , Woojeong Jin, Fred Morstatter, Jay Pujara

Department of Computer Science and Information Sciences Institute

University of Southern California
{[Link],ahrabian,[Link]}@[Link], {fredmors,jpujara}@[Link]

Abstract Which team will win the Super Bowl in 2023?

Temporal knowledge graph (TKG) forecasting 2000: [Superbowl, Champion, St Louis]

benchmarks challenge models to predict future 2001: [Superbowl, Champion, Baltimore]
2002: [Superbowl, Champion, New England]
arXiv:2305.10613v3 [[Link]] 20 Oct 2023

facts using knowledge of past facts. In this pa-

2003: [Superbowl, Champion, Tampa Bay]
per, we develop an approach to use in-context ...
learning (ICL) with large language models 2019: [Superbowl, Champion, New England]
(LLMs) for TKG forecasting. Our extensive 2020: [Superbowl, Champion, Kansas City]
evaluation compares diverse baselines, includ- 2021: [Superbowl, Champion, Tampa Bay]
2022: [Superbowl, Champion, Los Angeles]
ing both simple heuristics and state-of-the-art 2023: [Superbowl, Champion,
(SOTA) supervised models, against pre-trained
LLMs across several popular benchmarks and
experimental settings. We observe that naive Model # Params. Prediction G.T Rank

LLMs perform on par with SOTA models, EleutherAI gpt-j-6b 6B Los Angeles 3
EleutherAI gpt-neox-20b 20B Kansas City 1
which employ carefully designed architectures OpenAI text-ada-001 350M Carolina >5
and supervised training for the forecasting task, OpenAI text-babbage-001 1.3B Indianapolis >5
OpenAI text-curie-001 6.7B New England >5
falling within the (-3.6%, +1.5%) Hits@1 mar-
OpenAI text-davinci-003 175B TBD >5
gin relative to the median performance. To bet- OpenAI gpt-3.5-turbo - Sorry, I cannot predict future events.
ter understand the strengths of LLMs for fore-
casting, we explore different approaches for Table 1: Model predictions for an example struc-
selecting historical facts, constructing prompts, tured prompt. Given a structured prompt (top), large
controlling information propagation, and pars- language models forecast the missing fact using next
ing outputs into a probability distribution. A token prediction (bottom). G.T Rank indicates the rank
surprising finding from our experiments is that of the ground truth response (“Kansas City”) within the
LLM performance endures (±0.4% Hit@1) token probability distribution.
even when semantic information is removed
by mapping entities/relations to arbitrary num- of knowledge (Leblay and Chekol, 2018; García-
bers, suggesting that prior semantic knowledge Durán et al., 2018; Goel et al., 2020; Lacroix et al.,
is unnecessary; rather, LLMs can leverage the
2020). Recent endeavors in the area of TKGs
symbolic patterns in the context to achieve such
a strong performance. Our analysis also reveals have been focused on predicting future missing
that ICL enables LLMs to learn irregular pat- links (i.e., forecasting), given a query quadruple
terns from the historical context, going beyond q = (si , pj , ?, tT ) and a set of associated historical
frequency and recency biases1 . facts Eq = {(s, p, o, t) |t < tT } ⊆ E (Gastinger
et al., 2022). An illustrative example of this task
1 Introduction is the question “Which team will win the Super
Bowl in 2023?” that can be expressed as q = (Su-
Knowledge Graphs (KGs) are prevalent resources per bowl, Champion, ?, 2023) and Eq = {(Super
for representing real-world facts in a structured way. bowl, Champion, Los Angeles, 2022), (Super bowl,
While traditionally, KGs have been utilized for rep- Champion, Tampa Bay, 2021), ...} (See Table 1).
resenting static snapshots of “current” knowledge, The ultimate objective is to identify the most suit-
recently, temporal KGs (TKGs) have gained popu- able entity among all the football teams E (e.g., St
larity to preserve the complex temporal dynamics Louis, Baltimore) to fill the missing field.
∗
Authors contributed equally. Prior research on TKG forecasting has been
1
[Link] primarily focused on developing supervised ap-
proaches such as employing graph neural networks +1.5%) Hits@1 margin, relative to the median ap-
to model interrelationships among entities and proach for each dataset; (2) LLMs perform almost
relations (Jin et al., 2020; Li et al., 2021; Han identically when we replace entities’ and relations’
et al., 2021b,a), using reinforcement learning tech- lexical names with numerically mapped indices,
niques (Sun et al., 2021), and utilizing logical suggesting that the prior semantic knowledge is not
rules (Zhu et al., 2021; Liu et al., 2022). How- a critical factor for achieving such a high perfor-
ever, these techniques have prominent limitations, mance; and (3) LLMs outperform the best heuristic
including the need for large amounts of training rule-based baseline on each dataset (i.e., the most
data that include thorough historical information frequent or the most recent, given the historical
for the entities. Additionally, model selection is a context) by (+10%, +28%) Hits@1 relative margin,
computationally expensive challenge as the state- indicating that they do not simply select the output
of-the-art approach differs for each dataset. using frequency or recency biases in ICL (Zhao
et al., 2021).
In this paper, we develop a TKG forecasting
approach by casting the task as an in-context learn- 2 Problem Formulation
ing (ICL) problem using large language models
(LLMs). ICL refers to the capability of LLMs to In-Context Learning. ICL is an emergent capa-
learn and perform an unseen task efficiently when bility of LLMs that aims to induce a state in the
provided with a few examples of input-label pairs model to perform a task by utilizing contextual
in the prompt (Brown et al., 2020). Prior works input-label examples, without requiring changes
on ICL usually leverage few-shot demonstrations, to its internal parameters (Brown et al., 2020).
where a uniform number of examples are provided Formally, in ICL for classification, a prompt is
for each label to solve a classification task (Min constructed by linearizing a few input-output pair
et al., 2022; Wei et al., 2023). In contrast, our examples (xi , yi ) from the training data. Subse-
work investigates what the model learns from ir- quently, when a new test input text xtest is provided,
regular patterns of historical facts in the context. ICL generates the output ytest ∼ PLLM (ytest |
We design a three-stage pipeline to control (1) the x1 , y1 , . . . , xk , yk , xtest ) where ∼ refers to decod-
background knowledge selected for context, (2) the ing strategy.
prompting strategy for forecasting, and (3) decod- Temporal Knowledge Graph Forecasting. For-
ing the output into a prediction. The first stage uses mally, a TKG, G = (V, R, E, T ), is comprised
the prediction query to retrieve a set of relevant of a set of entities V, relations R, facts E, and
past facts from the TKG that can be used as context timestamps T . Moreover, since time is sequen-
(Section 3.1). The second stage transforms these tial, G can be split into a sequence of time-stamped
contextual facts into a lexical prompt representing snapshots, G = {G1 , G2 , . . . , Gt , . . .}, where each
the prediction task (Section 3.3). The third stage de- snapshot, Gt = (V, R, Et ), contains the facts at
codes the output of the LLM into a probability dis- a specific point in time t. Each fact f ∈ Et is a
tribution over the entities and generates a response quadruple (s, p, o, t) where s, o ∈ V, p ∈ R, and
to the prediction query (Section 3.4). Our exper- t ∈ T . The TKG forecasting task involves pre-
imental evaluation performs competitively across dicting a temporally conditioned missing entity in
a diverse collection of TKG benchmarks without the future given a query quadruple, (?, p, o, t) or
requiring the time-consuming supervised training, (s, p, ?, t), and previous graph snapshots G1:t−1 =
or custom-designed architectures. {G1 , G2 , . . . , Gt−1 }. Here, the prediction typically
We present extensive experimental results involves ranking each entity’s assigned score.
on common TKG benchmark datasets such as
3 In-context Learning for Temporal
WIKI (Leblay and Chekol, 2018), YAGO (Mahdis-
Knowledge Graph Forecasting
oltani et al., 2014), and ICEWS (García-Durán
et al., 2018; Jin et al., 2020). Our findings are In this work, we focus on 1) modeling appropriate
as follows: (1) LLMs demonstrate the ability to history Eq for a given query quadruple q, 2) con-
make predictions about future facts using ICL with- verting {Eq , q} into a prompt θq , and 3) employing
out requiring any additional training. Moreover, ICL to get prediction yq ∼ PLLM (yq | θq ) in a
these models show comparable performance to zero-shot manner. Here, the history Eq is mod-
supervised approaches, falling within the (-3.6%, eled on the facts from the previous graph snapshots
G1:t−1 = {G1 , G2 , . . . , Gt−1 }, and we employ to- Category Prompt
ken probabilities for yq to get ranked scores of 2000: [Superbowl, Champion, 0. St Louis]
Lexical 2001: [Superbowl, Champion, 1. Baltimore]
candidate entities in a zero-shot manner. In the rest L(·) ...
of this section, we study history modeling strate- 2023: [Superbowl, Champion,
gies (Sec 3.1), response generation approaches 2000: [0, 0, 0. 0]
(Sec 3.2), prompt construction templates (Sec 3.3), Index 2001: [0, 0, 1. 1]
I(·) ...
and common prediction settings (Sec 3.4). 2023: [0, 0,

Table 2: Prompt Example.

3.1 History Modeling
To model the history Eq , we filter facts that the 3.3 Prompt Construction
known entity or relation in the query q has been Given the history Eq and query q, we construct a
involved in. Specifically, given the query quadruple prompt using a pre-defined template θ. Specifically,
q = (s, p, ?, t) under the object entity prediction given the query quadruple q = (s, p, ?, t) under
setting, we experiment with two different aspects the object entity prediction setting, we present two
of historical facts: versions of the template θ with varying levels of
information. Our assumption is that each entity or
Entity vs. Pair. Entity includes past facts that relation has an indexed I(·) (e.g., 0) and a lexical
contain s, e.g., all historical facts related to Super- L(·) (e.g., Superbowl) form (See Table 2).
bowl. In contrast, Pair includes past facts that
contain both s and p, e.g., a list of (Superbowl, Index. Index displays every fact, (fs , fp , fo , ft )
Champion, Year) as shown in Table 1. ∈ E using the “ft :[I(fs ), I(fp ), nfo . I(fo )]” tem-
plate where fs , fo ∈ V, fp ∈ R, ft ∈ T , nfo
Unidirectional vs. Bidirectional. Unidirec- denotes an incrementally assigned numerical label
tional includes past facts F wherein s (Entity) (i.e., indirect logit), and I is a mapping from enti-
or (s, p) (Pair) is in the same position as it is in q ties to unique indices. For example, in Table 1, we
(e.g., Unidirectional & Pair – s and p served as can use the following mappings are for the entities
subject and predicate in f ∈ F). Bidirectional and relations, respectively: {Superbowl → 0, St
includes past facts F wherein s (Entity) or (s, p) Louis → 1, Baltimore → 2} and {Champion → 0}.
(Pair) appear in any valid position (e.g., Bidirec- The query q is then represented as “t:[I(s), I(p),”,
tional & Entity – s served as subject or object concatenated to the end of the prompt. For subject
in f ∈ F). As an example of the Bidirectional entity prediction, we follow the same procedure
setting, given q = (Superbowl, Champion, ?, 2023), from the other side.
we include f = (Kupp, Played, Superbowl, 2022)
because s (i.e., Superbowl) is present as the object Lexical. Lexical follows the same process as
in f . Moreover, in the Bidirectional setting, to Index but uses lexical form L(·) of entity and rela-
preserve the semantics of the facts in the Eq , we tion. Each fact in (fs , fp , fo , ft ) ∈ E is represented
transform the facts where s appears as an object by as “ft :[L(fs ), L(fp ), nfo . L(fo )]” and the query
1) swapping the object and subject and 2) replac- q is represented as “t:[L(s), L(p),”, concatenated
ing the relation with its uniquely defined inverse to the end of the prompt.
relation (e.g., (fs , fp , fo , ft ) → (fo , fp−1 , fs , ft )).
3.4 Prediction Setting
3.2 Response Generation All the historical facts in the dataset are split into
three subsets, Dtrain , Dvalid , and Dtest , based on
Given a prompt θq , we pass it to an LLM to ob- the chronological order with train < valid < test.
tain the next token probabilities. Then, we use the Given this split, during the evaluation phase, the
obtained probabilities to get a ranked list of enti- TKG forecasting task requires models to predict
ties. However, obtaining scores for entities based over Dtest under the following two settings:
on these probabilities is challenging as they may
be composed of several tokens. To address this Single Step. In this setting, for each test query,
challenge, we utilize a mapped numerical label as the model is provided with ground truth facts from
an indirect logit to estimate their probabilities (Lin past timestamps in the test period. Hence, after
et al., 2022). making predictions for a test query in a specific
 # of Facts Model Family Model Name # Params Instruction-tuned
Dataset # Ents # Rels Interval
Train Valid Test GPT2 gpt2 124M ✗
gpt2-medium 355M ✗
WIKI 12,554 24 539,286 67,538 63,110 1 year
gpt2-large 774M ✗
YAGO 10,623 10 161,540 19,523 20,026 1 year
gpt2-xl 1.5B ✗
ICEWS14 6,869 230 74,845 8,514 7,371 1 day
ICEWS18 23,033 256 373,018 45,995 49,995 1 day GPT-J gpt-j-6b 6B ✗
ACLED-CD22 243 6 1,788 216 222 1 day GPT-NeoX gpt-neox-20b 20B ✗
InstructGPT gpt-3.5-turbo - ✓
Table 3: Data statistics. Each dataset consists of enti-
ties, relations, and historical facts, with the facts within Table 4: Language Models used in the paper. Exact
the same time interval identified by the same timestamp. model size of gpt-3.5-turbo is unknown.
The facts are divided into three subsets based on time,
where train < valid < test.
4.2 Evaluation
timestamp, the ground truth fact for that query We evaluate the models on well-known metrics for
is added to the history before moving to the test link prediction: Hits@k, with k = 1, 3, 10. Follow-
queries in the next timestamp. ing (Gastinger et al., 2022), we report our results
in two evaluation settings: 1) Raw retrieves the
Multi Step. In this setting, the model is not pro- sorted scores of candidate entities for a given query
vided with ground truth facts from past timestamps quadruple and calculates the rank of the correct
in the test period and has to rely on its noisy pre- entity; and 2) Time-aware filter also retrieves the
dictions. Hence, after making predictions for a sorted scores but removes the entities that are valid
test query in a specific timestamp, instead of the predictions before calculating the rank, preventing
ground truth fact for that query, we add the pre- them from being considered errors. To illustrate, if
dicted response to the history before moving to the the test query is (NBA, Clinch Playoff, ?, 2023) and
test queries in the next timestamp. This setting is the true answer is Los Angeles Lakers, there may
considered more difficult as the model is forced exist other valid predictions such as (NBA, Clinch
to rely on its own noisy predictions, which can Playoff, Milwaukee Bucks, 2023) or (NBA, Clinch
lead to greater uncertainty with each successive Playoff, Boston Celtics, 2023). In such cases, the
timestamp. time-aware filter removes these valid predictions,
allowing for accurate determination of the rank of
4 Experimental Setup the “Los Angeles Lakers.” In this paper, we present
performance with the time-aware filter.
4.1 Datasets
4.3 Models.
For our experiments, we use the WIKI (Leblay
and Chekol, 2018), YAGO (Mahdisoltani et al., As shown in Table 4, we perform experiments
2014), ICEWS14 (García-Durán et al., 2018), and on four language model families. Among those,
ICEWS18 (Jin et al., 2020) benchmark datasets three are open-sourced: GPT2 (Radford et al.,
with the unified splits introduced in previous stud- 2019), GPT-J (Wang, 2021), and GPT-NeoX (Black
ies (Gastinger et al., 2022). Additionally, we ex- et al., 2022). All models employ the GPT-2 byte
tract a new temporal forecasting dataset from the level BPE tokenizer (Radford et al., 2019) with
Armed Conflict Location & Event Data Project nearly identical vocabulary size. In addition, we
(ACLED) project2 which provides factual data of use the gpt-3.5-turbo model to analyze the per-
crises in a particular region. We specifically focus formance of the instruction-tuned models. How-
on incidents of combat and violence against civil- ever, we do not directly compare this model to
ians in Cabo Delgado from January 1900 to March other models in terms of size since the actual
2022, using data from October 2021 to March 2022 model size is unknown. As for the TKG baselines,
as our test set. This dataset aims to investigate (i.e., RE-Net (Jin et al., 2020), RE-GCN (Li et al.,
whether LLMs leverage prior semantic knowledge 2021), TANGO (Han et al., 2021b), xERTE (Han
to make predictions and how effective they are et al., 2021a), TimeTraveler (Sun et al., 2021),
when deployed in real-world applications. Table 3 CyGNet (Zhu et al., 2021), and TLogic (Liu et al.,
presents the statistics of these datasets. 2022)), we report the numbers presented in prior
research (Gastinger et al., 2022). Appendix A.4
2
[Link] provides more details on baseline models.
 YAGO WIKI ICEWS14 ICEWS18 ACLED-CD22
Single-Step Train
H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10
RE-GCN ✓ 0.787 0.842 0.884 0.747 0.817 0.846 0.313 0.473 0.626 0.223 0.367 0.525 0.446 0.545 0.608
xERTE ✓ 0.842 0.902 0.912 0.703 0.785 0.801 0.330 0.454 0.570 0.209 0.335 0.462 0.320 0.445 0.497
TLogic ✓ 0.740 0.789 0.791 0.786 0.860 0.870 0.332 0.476 0.602 0.204 0.336 0.480 0.009 0.045 0.094
TANGO ✓ 0.590 0.646 0.677 0.483 0.514 0.527 0.272 0.408 0.550 0.191 0.318 0.462 0.327 0.482 0.599
Timetraveler ✓ 0.845 0.908 0.912 0.751 0.820 0.830 0.319 0.454 0.575 0.212 0.325 0.439 0.240 0.315 0.457
GPT-NeoX (Entity) ✗ 0.784 0.891 0.927 0.694 0.804 0.844 0.324 0.460 0.565 0.192 0.313 0.414 0.324 0.492 0.604
GPT-NeoX (Pair) ✗ 0.787 0.892 0.926 0.721 0.812 0.847 0.297 0.408 0.482 0.196 0.307 0.402 0.317 0.440 0.566

YAGO WIKI ICEWS14 ICEWS18 ACLED-CD22

Multi-Step Train
H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10
RE-GCN ✓ 0.717 0.776 0.817 0.594 0.648 0.678 0.278 0.421 0.575 0.195 0.326 0.475 0.421 0.464 0.502
RE-Net ✓ 0.534 0.613 0.662 0.472 0.507 0.530 0.278 0.408 0.549 0.184 0.314 0.461 0.238 0.445 0.563
CyGNet ✓ 0.613 0.742 0.834 0.525 0.624 0.675 0.266 0.402 0.545 0.166 0.295 0.444 0.408 0.500 0.588
TLogic ✓ 0.631 0.706 0.715 0.613 0.663 0.682 0.265 0.395 0.531 0.155 0.272 0.412 0.009 0.045 0.094
GPT-NeoX (Entity) ✗ 0.686 0.793 0.840 0.543 0.622 0.655 0.247 0.363 0.471 0.136 0.224 0.321 0.319 0.417 0.500
GPT-NeoX (Pair) ✗ 0.688 0.793 0.839 0.570 0.625 0.652 0.236 0.324 0.395 0.155 0.245 0.331 0.289 0.410 0.464

Table 5: Performance (Hits@K) comparison between supervised models and ICL for single-step (top) and multi-
step (bottom) prediction. The first group in each table consists of supervised models, whereas the second group
consists of ICL models, i.e., GPT-NeoX with a history length of 100. The best model for each dataset in the first
group is shown in bold, and the second best is underlined.

ICEWS14 ICEWS18 ICEWS14 ICEWS18

Single-Step Multi-Step
H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10
frequency 0.243 0.387 0.532 0.141 0.265 0.409 frequency 0.222 0.349 0.460 0.121 0.207 0.307
recency 0.228 0.387 0.536 0.120 0.242 0.403 recency 0.151 0.268 0.423 0.074 0.149 0.266
GPT-NeoX (Entity) 0.324 0.460 0.565 0.192 0.313 0.414 GPT-NeoX (Entity) 0.247 0.363 0.471 0.136 0.224 0.321
GPT-NeoX (Pair) 0.297 0.408 0.482 0.196 0.307 0.402 GPT-NeoX (Pair) 0.236 0.324 0.395 0.155 0.245 0.331
(a) Single-step (b) Multi-step
Table 6: Performance (Hits@K) with rule-based predictions. The best model for each dataset is shown in bold.

4.4 ICL Implementation Details. the rank of the actual value within the predictions
is assigned a value of 100, which is considered
We implement our frameworks using Py- incorrect according to our evaluation metric.
Torch (Paszke et al., 2019) and Huggingface (Wolf For instruction-tuned model, we use the manual
et al., 2020). We first collate the facts f ∈ Dtest curated system instructions in Appendix A.3.
based on the identical test query to eliminate any
repeated inference. To illustrate, suppose there 5 Experimental Results
exist two facts in the test set denoted as (s, p, a, t) 5.1 In-context learning for TKG Forecasting
and (s, p, b, t) in the object prediction scenario.
We consolidate these facts into (s, p, [a, b], t) and In this section, we present a multifaceted perfor-
forecast only one for (s, p, ?, t). Subsequently, we mance analysis of ICL under Index & Unidirec-
proceed to generate an output for each test query tion prompt strategy for both Entity and Pair
with history by utilizing the model, obtaining history.
the probability for the first generated token in Q1: How well does ICL fare against the super-
a greedy approach, and sorting the probability. vised learning approaches? We present a com-
The outputs are deterministic for every iteration. parative analysis of the top-performing ICL model
We retain the numerical tokens corresponding to against established supervised learning methodolo-
the numerical label n that was targeted, selected gies for TKG reasoning, which are mostly based on
from the top 100 probability tokens for each test graph representation learning. As evident from the
query. To facilitate multi-step prediction, we results in Table 5, GPT-NeoX with a history length
incorporate the top-k predictions of each test query of 100 shows comparable performance to super-
as supplementary reference history. In this paper, vised learning approaches, without any fine-tuning
we present results with k = 1. It is important on any TKG training dataset.
to acknowledge that the prediction may contain
minimal or no numerical tokens as a result of Q2: How do frequency and recency biases affect
inadequate in-context learning. This can lead to ICL’s predictions? To determine the extent to
problems when evaluating rank-based metrics. To which LLMs engage in pattern analysis, beyond
mitigate this, we have established a protocol where simply relying on frequency and recency biases,
 0.8 Single-step, Unidirection, Entity, History Length = 100 ICEWS14
Original Single-Step Prompt
0.7
Time-removed H@1
0.6 Time-removed & Shuffle
0.5 gpt-3.5-turbo index 0.1615
0.4 gpt-3.5-turbo lexical 0.1858
0.3
0.2 Table 7: Performance (Hits@1) between index and
0.1 YAGO WIKI ICEWS14 ICEWS18
lexical for gpt-3.5-turbo.

Figure 1: Performance (Hit@1) with and without time,

and with shuffling Q4: How does instruction-tuning affect ICL’s
performance? To investigate the impact of
we run a comparative analysis between GPT-NeoX instruction-tuning on ICL, we employ the gpt-
and heuristic-rules (i.e., frequency & recency) 3.5-turbo model with manually curated system
on the ICEWS14 dataset, with history length set to instruction detailed in Appendix 4.4. Since the
100. frequency identifies the target that appears size of this model is not publicly disclosed, it is
most frequently in the provided history while re- challenging to make direct comparisons with other
cency selects the target associated with the most models featured in this paper. Moreover, since this
recent fact in the provided history. The reason for model does not provide output probabilities, we are
our focus on ICEWS is that each quadruple repre- only able to report the Hit@1 metric.
sents a single distinct event in time. In contrast, the Table 7 showcases that the performance of
process of constructing YAGO and WIKI involves the lexical prompts exceeds that of the index
converting durations to two timestamps to display prompts by 0.024, suggesting that instruction-tuned
events across the timeline. This step has resulted models can make better use of semantic priors.
in recency heuristics outperforming all of the ex- This behavior is different from the other foundation
isting models, showcasing the shortcoming of ex- LLMs, where the performance gap between the two
isting TKG benchmarks (See Appendix A.5). The prompt types was insignificant (See Figure 4 (a)).
experimental results presented in Table 6 demon-
strate that ICL exhibits superior performance to Q5: How does history length affect ICL’s per-
rule-based baselines. This finding suggests that formance? To evaluate the impact of the history
ICL does not solely rely on specific biases to make length provided in the prompt, we conduct a set
predictions, but rather it actually learns more so- of experiments using varying history lengths. For
phisticated patterns from historical data. this purpose, we use the best performing prompt
format for each benchmark, i.e., Entity for WIKI,
YAGO, ICEWS18, and Pair for ICEWS14. Our
Q3: How does ICL use the sequential and tempo- results, as shown in Figure 2, indicate a consistent
ral information of events? To assess the ability improvement in performance as the history length
of LLMs to comprehend the temporal information increases. This suggests that the models learn bet-
of historical events, we compare the performance of ter as additional historical facts are presented. This
prompts with and without timestamps. Specifically, observation is connected to few-shot learning in
we utilize the original prompt format, “ft :[I(fs ), other domains, where performance improves as the
I(fr ), nfo . I(fo )]”, and the time-removed prompt number of examples per label increases. However,
format, “[I(fs ), I(fr ), nfo . I(fo )]”, make the in our case, the historical patterns presented in the
comparison (See Appendix A.2). Additionally, prompt do not explicitly depict the input-label map-
we shuffle the historical facts in the time-removed ping but rather aid in inferring the next step.
prompt format to see how the model is affected by
the corruption of sequential information. Figure 1 Q6: What is the relation between ICL’s perfor-
shows that the absence of time reference can lead mance and model size? Here, we analyze the
to a deterioration in performance, while the ran- connection between model size and performance.
dom arrangement of historical events may further Our results, as presented in Figure 3, conform to the
exacerbate this decline in performance. This obser- expected trend of better performance with larger
vation implies that the model has the capability to models. This finding aligns with prior works show-
forecast the subsequent event by comprehending ing the scaling law of in-context learning perfor-
the sequential order of events. mance. Our findings are still noteworthy since they
0.9 Single Step 0.9 Multi Step 0.9 Single Step 0.8 Multi Step
0.8 YAGO 0.8 0.8 YAGO 0.7 YAGO
0.7 WIKI 0.7 YAGO 0.7 WIKI 0.6 WIKI
0.6 0.6 WIKI 0.6 0.5
0.5 0.5 0.5 0.4
0.4 0.4 0.4
0.3 ICEWS14 0.3 ICEWS14 0.3 ICEWS14
ICEWS14 0.3 0.2
0.2 ICEWS18 0.2 ICEWS18 0.1 ICEWS18
ICEWS18 0.2
0.1 0.1
0.0 135 10 15 20 25 0.0 135 10 15 20 25 0.1124M 335M 774M 1.5B 6B 20B 0.0124M 335M 774M 1.5B 6B 20B
50 50

Figure 2: Performance (Hit@1) adheres to the scaling Figure 3: Performance (Hit@1) adheres to the scaling
law based on the history length. law based on the model size.

show how scaling model size can facilitate more Entity vs. Pair Finally, we examine the impact
powerful pattern inference for forecasting tasks. of the history retrieval query on performance. Our
hypothesis posits that when the query is limited to
5.2 Prompt Construction for TKG a single entity, we can incorporate more diverse his-
Forecasting torical facts. Conversely, when the query is a pair,
To determine the most effective prompt variation, we can acquire a more focused set of historical facts
we run a set of experiments on all prompt variations, related to the query. Our results (Figure 4 (c)) indi-
using GPT-J (Wang, 2021) and under the single- cate that the performance of the model is dependent
step setting. Comprehensive results for prompt on the type of data being processed. Specifically,
variations can be found in Appendix A.5. the WIKI and ICEWS18 benchmarks perform bet-
ter when the query is focused on the entity, as a
Index vs. Lexical Our first analysis compares the broader range of historical facts is available. In
performance of index and lexical prompts. This contrast, the ICEWS14 benchmark performs better
investigation aims to determine whether the model when the query is focused on pairs, as the historical
relies solely on input-label mappings or if it also facts present a more focused pattern.
incorporates semantic priors from pre-training to
make predictions. Our results (Figure 4 (a)) show
that the performance is almost similar (±4e − 3 on 6 Related Works
average) across the datasets. This finding is aligned
with previous studies indicating that foundation Event Forecasting. Forecasting is a complex
models depend more on input-label mappings and task that plays a crucial role in decision-making and
are minimally impacted by semantic priors (Wei safety across various domains (Hendrycks et al.,
et al., 2023). 2021). To tackle this challenging task, researchers
have explored various approaches, including sta-
Unidirectional vs. Bidirectional We next ana- tistical and judgmental forecasting (Webby and
lyze how the relation direction in the history model- O’Connor, 1996; Armstrong, 2001; Zou et al.,
ing impacts the performance. This analysis aims to 2022). Statistical forecasting involves leveraging
ascertain whether including historical facts, where probabilistic models (Hyndman and Khandakar,
the query entity or pair appears in any position, can 2008) or neural networks (Li et al., 2018; Sen et al.,
improve performance by offering a diverse array 2019) to predict trends over time-series data. While
of historical facts. Our results (Figure 4 (b)) show this method works well when there are many past
that there is a slight decrease in performance when observations and minimal distribution shifts, it is
Bidirectional history is employed, with a signif- limited to numerical data and may not capture the
icant drop in performance observed particularly in underlying causal factors and dependencies that
the ICEWS benchmarks. These observations may affect the outcome. On the other hand, judgmen-
be attributed to the considerably more significant tal forecasting involves utilizing diverse sources
number of entities placed in both subject and ob- of information, such as news articles and exter-
ject positions in ICEWS benchmarks than YAGO nal knowledge bases, to reason and predict future
and WIKI benchmarks (See Appendix A.5). This events. Recent works have leveraged language
finding highlights the necessity of having robust models to enhance reasoning capabilities when ana-
constraints on the historical data for ICL to com- lyzing unstructured text data to answer forecasting
prehend the existing pattern better. inquiries (Zou et al., 2022; Jin et al., 2021).
 0.8 Unidirection, Entity, History Length = 25 0.8 Index, Entity, History Length = 25 0.8 Index, Unidirection, History Length = 100
0.7 Index 0.7 Unidirection 0.7 Entity
Lexical Bidirection 0.6 Pair
0.6 0.6
0.5 0.5 0.5
0.4
0.4 0.4 0.3
0.3 0.3 0.2
0.2 0.2 0.1
0.1 YAGO WIKI ICEWS14 ICEWS18 0.1 YAGO WIKI ICEWS14 ICEWS18 0.0 YAGO WIKI ICEWS14 ICEWS18

(a) Index vs. Lexical (b) Unidirection vs. Bidirection (c) Entity vs. Pair

Figure 4: Performance (Hit@1) Analysis on Prompt Variation. The comparable performance exhibited by
both the Index and Lexical models indicates that these models rely heavily on learning patterns and are less
influenced by semantic priors. Moreover, the Unidirectional model typically outperforms the Bidirectional
model, suggesting that the robust constraints on historical data enable the model to comprehend observed patterns
better. Finally, the performance of the Entity and Pair models varies depending on the dataset.

Temporal Knowledge Graph. Temporal knowl- edge by examining their correlation with training
edge graph (TKG) reasoning models are commonly examples (Min et al., 2022; Razeghi et al., 2022;
employed in two distinct settings, namely interpo- Xie et al., 2022), data distribution (Chan et al.,
lation and extrapolation, based on the facts avail- 2022), and language compositionality (Hahn and
able from t0 to tn . (1) Interpolation aims to pre- Goyal, 2023) in the pre-training corpus. Other
dict missing facts within this time range from t0 recent works show that LLMs can actually learn
to tn , and recent works have utilized embedding- input-label mappings from in-context examples by
based algorithms to learn low-dimensional repre- showing the transformer models trained on specific
sentations for entities and relations to score candi- linear function class is actually predicting accu-
date facts (Leblay and Chekol, 2018; García-Durán rately on new unseen linear functions (Garg et al.,
et al., 2018; Goel et al., 2020; Lacroix et al., 2020); 2022). More recently, there is a finding that large-
(2) Extrapolation aims to predict future facts be- enough models can still do ICL using input-label
yond tn . Recent studies have treated TKGs as a mappings when semantic prior knowledge is not
sequence of snapshots, each containing facts corre- available (Wei et al., 2023).
sponding to a timestamp ti , and proposed solutions
by modeling multi-relational interactions among 7 Conclusion
entities and relations over these snapshots using In this paper, we examined the forecasting capabil-
graph neural networks (Jin et al., 2020; Li et al., ities of in-context learning in large language mod-
2021; Han et al., 2021b,a), reinforcement learn- els. To this end, we experimented with temporal
ing (Sun et al., 2021) or logical rules (Zhu et al., knowledge graph forecasting benchmarks. We pre-
2021; Liu et al., 2022). In our work, we focus on sented a framework that converts relevant histor-
the extrapolation setting. ical facts into prompts and generates ranked link
predictions through token probabilities. Our exper-
In-context Learning. In-context learning (ICL) imental results demonstrated that without any fine-
has enabled LLMs to accomplish diverse tasks in a tuning and only through ICL, LLMs exhibit com-
few-shot manner without needing parameter adjust- parable performance to current supervised TKG
ments (Brown et al., 2020; Chowdhery et al., 2022). methods that incorporate explicit modules to cap-
In order to effectively engage in ICL, models can ture structural and temporal information. We also
leverage semantic prior knowledge to accurately discovered that using numerical indices instead of
predict labels following the structure of in-context entity/relation names does not significantly affect
exemplars (Min et al., 2022; Razeghi et al., 2022; the performance, suggesting that prior semantic
Xie et al., 2022; Chan et al., 2022; Hahn and Goyal, knowledge is not critical for overall performance.
2023), and learn the input-label mappings from the Additionally, our analysis indicated that ICL helps
in-context examples presented (Wei et al., 2023). the model learn irregular patterns from historical
To understand the mechanism of ICL, recent stud- facts, beyond simply making predictions based on
ies have explored the ICL capabilities of LLMs the most common or the most recent facts in the
with regards to the impact of semantic prior knowl- given context. Together, our results and analyses
demonstrated that ICL can be a valuable tool for 2022. Gpt-neox-20b: An open-source autoregressive
predicting future links using historical patterns, and language model. arXiv preprint arXiv:2204.06745.
also prompted further inquiry into the potential of Tom Brown, Benjamin Mann, Nick Ryder, Melanie
ICL for additional capabilities. Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
8 Limitations Askell, et al. 2020. Language models are few-shot
learners. Advances in neural information processing
There are certain limitations to our experiments. systems, 33:1877–1901.
First, computing resource constraints restrict our Stephanie C.Y. Chan, Adam Santoro, Andrew Kyle
experiments to small-scale open-source models. Lampinen, Jane X Wang, Aaditya K Singh,
Second, our methodologies have constraints re- Pierre Harvey Richemond, James McClelland, and
garding models where the tokenizer vocabulary Felix Hill. 2022. Data distributional properties drive
emergent in-context learning in transformers. In Ad-
comprises solely of single-digit numbers as tokens, vances in Neural Information Processing Systems.
such as LLAMA (Touvron et al., 2023). The per-
formance of such models exhibits a similar trend in Aakanksha Chowdhery, Sharan Narang, Jacob Devlin,
Maarten Bosma, Gaurav Mishra, Adam Roberts,
terms of scaling law concerning model size and his-
Paul Barham, Hyung Won Chung, Charles Sutton,
tory length, but these models demonstrate inferior Sebastian Gehrmann, et al. 2022. Palm: Scaling
performance compared to other models of the same language modeling with pathways. arXiv preprint
model size. Third, our methodologies have certain arXiv:2204.02311.
limitations with respect to link prediction settings. Alberto García-Durán, Sebastijan Dumančić, and Math-
While real-world forecasting can be performed in ias Niepert. 2018. Learning sequence encoders for
the transductive setting, where the answer can be an temporal knowledge graph completion. In Proceed-
unseen history, our approach is constrained to the ings of the 2018 Conference on Empirical Methods
in Natural Language Processing, pages 4816–4821,
inductive setting, where the answer must be one of Brussels, Belgium. Association for Computational
the histories observed. There are further directions Linguistics.
that can be pursued. The first is to explore trans-
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gre-
ductive extrapolation link prediction using LLMs. gory Valiant. 2022. What can transformers learn
The second is to analyze the effects of fine-tuning in-context? a case study of simple function classes.
on the results. Lastly, there is the opportunity to Advances in Neural Information Processing Systems,
investigate the new capabilities of ICL. 35:30583–30598.
Julia Gastinger, Timo Sztyler, Lokesh Sharma, and
9 Acknowledgement Anett Schuelke. 2022. On the evaluation of meth-
ods for temporal knowledge graph forecasting. In
This work was funded in part by the Defense NeurIPS 2022 Temporal Graph Learning Workshop.
Advanced Research Projects Agency (DARPA)
and Army Research Office (ARO) under Con- Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker,
and Pascal Poupart. 2020. Diachronic embedding for
tract No. W911NF-21-C-0002 and Contract No. temporal knowledge graph completion. In Proceed-
HR00112290106, and with support from the Ke- ings of the AAAI conference on artificial intelligence,
ston Exploratory Research Award and Amazon. volume 34, pages 3988–3995.
The views and conclusions contained herein are Michael Hahn and Navin Goyal. 2023. A theory of
those of the authors and should not be interpreted as emergent in-context learning as implicit structure
necessarily representing the official policies, either induction. arXiv preprint arXiv:2303.07971.
expressed or implied, of DARPA, ARO or the U.S.
Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp.
Government. 2021a. Explainable subgraph reasoning for forecast-
ing on temporal knowledge graphs. In International
Conference on Learning Representations.
References
Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and
Jon Scott Armstrong. 2001. Principles of forecasting: Volker Tresp. 2021b. Learning neural ordinary equa-
a handbook for researchers and practitioners, vol- tions for forecasting future links on temporal knowl-
ume 30. Springer. edge graphs. In Proceedings of the 2021 Conference
on Empirical Methods in Natural Language Process-
Sid Black, Stella Biderman, Eric Hallahan, Quentin ing, pages 8352–8364, Online and Punta Cana, Do-
Anthony, Leo Gao, Laurence Golding, Horace He, minican Republic. Association for Computational
Connor Leahy, Kyle McDonell, Jason Phang, et al. Linguistics.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe,
Jacob Steinhardt. 2021. Unsolved problems in ml Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle-
safety. arXiv preprint arXiv:2109.13916. moyer. 2022. Rethinking the role of demonstrations:
What makes in-context learning work? In Proceed-
Rob J Hyndman and Yeasmin Khandakar. 2008. Auto- ings of the 2022 Conference on Empirical Methods in
matic time series forecasting: the forecast package Natural Language Processing, pages 11048–11064,
for r. Journal of statistical software, 27:1–22. Abu Dhabi, United Arab Emirates. Association for
Computational Linguistics.
Woojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee,
Fred Morstatter, Aram Galstyan, and Xiang Ren. Adam Paszke, Sam Gross, Francisco Massa, Adam
2021. ForecastQA: A question answering challenge Lerer, James Bradbury, Gregory Chanan, Trevor
for event forecasting with temporal text data. In Killeen, Zeming Lin, Natalia Gimelshein, Luca
Proceedings of the 59th Annual Meeting of the Asso- Antiga, Alban Desmaison, Andreas Köpf, Edward
ciation for Computational Linguistics and the 11th Yang, Zachary DeVito, Martin Raison, Alykhan Te-
International Joint Conference on Natural Language jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang,
Processing (Volume 1: Long Papers), pages 4636– Junjie Bai, and Soumith Chintala. 2019. Pytorch: An
4650, Online. Association for Computational Lin- imperative style, high-performance deep learning li-
guistics. brary. In Advances in Neural Information Processing
Systems 32: Annual Conference on Neural Informa-
Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. tion Processing Systems 2019, NeurIPS 2019, De-
2020. Recurrent event network: Autoregressive struc- cember 8-14, 2019, Vancouver, BC, Canada, pages
ture inferenceover temporal knowledge graphs. In 8024–8035.
Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP), Alec Radford, Jeffrey Wu, Rewon Child, David Luan,
pages 6669–6683, Online. Association for Computa- Dario Amodei, Ilya Sutskever, et al. 2019. Language
tional Linguistics. models are unsupervised multitask learners. OpenAI
blog, 1(8):9.
Timothée Lacroix, Guillaume Obozinski, and Nicolas
Usunier. 2020. Tensor decompositions for temporal Yasaman Razeghi, Robert L Logan IV, Matt Gardner,
knowledge base completion. In International Con- and Sameer Singh. 2022. Impact of pretraining term
ference on Learning Representations. frequencies on few-shot numerical reasoning. In
Findings of the Association for Computational Lin-
Julien Leblay and Melisachew Wudage Chekol. 2018. guistics: EMNLP 2022, pages 840–854, Abu Dhabi,
Deriving validity time in knowledge graph. In Com- United Arab Emirates. Association for Computa-
panion proceedings of the the web conference 2018, tional Linguistics.
pages 1771–1776.
Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. 2019.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Think globally, act locally: A deep neural network
2018. Diffusion convolutional recurrent neural net- approach to high-dimensional time series forecasting.
work: Data-driven traffic forecasting. In Interna- Advances in neural information processing systems,
tional Conference on Learning Representations. 32.
Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and
Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Kun He. 2021. TimeTraveler: Reinforcement learn-
Cheng. 2021. Temporal knowledge graph reason- ing for temporal knowledge graph forecasting. In
ing based on evolutional representation learning. In Proceedings of the 2021 Conference on Empirical
Proceedings of the 44th International ACM SIGIR Methods in Natural Language Processing, pages
Conference on Research and Development in Infor- 8306–8319, Online and Punta Cana, Dominican Re-
mation Retrieval, pages 408–417. public. Association for Computational Linguistics.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Teaching models to express their uncertainty in Martinet, Marie-Anne Lachaux, Timothée Lacroix,
words. Transactions on Machine Learning Research. Baptiste Rozière, Naman Goyal, Eric Hambro,
Faisal Azhar, et al. 2023. Llama: Open and effi-
Yushan Liu, Yunpu Ma, Marcel Hildebrandt, Mitchell cient foundation language models. arXiv preprint
Joblin, and Volker Tresp. 2022. Tlogic: Temporal arXiv:2302.13971.
logical rules for explainable link forecasting on tem-
poral knowledge graphs. In Proceedings of the AAAI Ben Wang. 2021. Mesh-Transformer-JAX: Model-
Conference on Artificial Intelligence, volume 36, Parallel Implementation of Transformer Lan-
pages 4120–4127. guage Model with JAX. [Link]
kingoflolz/mesh-transformer-jax.
Farzaneh Mahdisoltani, Joanna Biega, and Fabian
Suchanek. 2014. Yago3: A knowledge base from Richard Webby and Marcus O’Connor. 1996. Judge-
multilingual wikipedias. In 7th biennial conference mental and statistical time series forecasting: a re-
on innovative data systems research. CIDR Confer- view of the literature. International Journal of fore-
ence. casting, 12(1):91–118.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert
Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu,
Da Huang, Denny Zhou, et al. 2023. Larger language
models do in-context learning differently. arXiv
preprint arXiv:2303.03846.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi, Pier-
ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-
icz, Joe Davison, Sam Shleifer, Patrick von Platen,
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
Teven Le Scao, Sylvain Gugger, Mariama Drame,
Quentin Lhoest, and Alexander Rush. 2020. Trans-
formers: State-of-the-art natural language processing.
In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing: System
Demonstrations, pages 38–45, Online. Association
for Computational Linguistics.
Sang Michael Xie, Aditi Raghunathan, Percy Liang,
and Tengyu Ma. 2022. An explanation of in-context
learning as implicit bayesian inference. In Interna-
tional Conference on Learning Representations.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and
Sameer Singh. 2021. Calibrate before use: Improv-
ing few-shot performance of language models. In In-
ternational Conference on Machine Learning, pages
12697–12706. PMLR.
Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan
Cheng, and Yan Zhang. 2021. Learning from history:
Modeling temporal knowledge graphs with sequen-
tial copy-generation networks. In Proceedings of
the AAAI Conference on Artificial Intelligence, vol-
ume 35, pages 4732–4740.
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas
Mazeika, Richard Li, Dawn Song, Jacob Steinhardt,
Owain Evans, and Dan Hendrycks. 2022. Forecast-
ing future world events with neural networks. In
Thirty-sixth Conference on Neural Information Pro-
cessing Systems Datasets and Benchmarks Track.
A Appendix A.4 Baseline Models

A.1 Prompt Example RE-Net (Jin et al., 2020) leverages an auto-

regressive architecture that employs a two-step pro-
Given the test query at timestamp 571, prompt ex-
cess for learning temporal dependency from a se-
amples for Index and Lexical are shown in Fig-
quence of graphs and local structural dependency
ure 5. Here, we assume the entity dictionary con-
from the vicinity. The model represents the likeli-
tains “Islamist Militia (Mozambique)” as index 0,
hood of a fact occurring as a probability distribution
“Meluco” as 10, “Namatil” as 36, “Muatide” as 53,
that is conditioned on the sequential history of past
“Limala” as 54, and “Nacate” as 55, while relation
snapshots.
dictionary contains “Battles” as index 1 and “Vi-
olence against civilians” as 4. Also, the history RE-GCN (Li et al., 2021) also employs auto-
setting is unidirectional entity setting where the regressive architecture while it utilizes multi-layer
history length is set to 5. relation-aware GCN on each graph snapshot to cap-
ture the structural dependencies among concurrent
568: [0, 4, 1. 55] 568: [Islamist Militia (Mozambique), Violence against civilians, 1. Nacate]
568: [0, 4, 3. 10]
568: [0, 1, 2. 36]
568: [Islamist Militia (Mozambique), Violence against civilians, 3. Meluco]
568: [Islamist Militia (Mozambique), Battles, 2. Namatil]
facts. Furthermore, the static properties of entities
569: [0, 1, 0. 54] 569: [Islamist Militia (Mozambique), Battles, 0. Limala]
570: [0, 1, 4. 53] 570: [Islamist Militia (Mozambique), Battles, 4. Muatide]
such as entity types, are also incorporated via a
571: [0, 1, 571: [Islamist Militia (Mozambique), Battles,

(a) Index (b) Lexical

static graph constraint component to obtain better
entity representations.
Figure 5: Prompt examples for Index and Lexical
settings. TANGO (Han et al., 2021b) employs auto-
regressive architecture as well but the use of
continuous-time embedding in encoding tempo-
A.2 Prompt Example for Analysis ral and structural information is a distinguishing
To assess the ability of LLMs to comprehend the feature of the proposed method, as opposed to RE-
sequential information of historical events, we com- Net (Jin et al., 2020) (Li et al., 2021) and RE-GCN
pare the performance of prompts with and without which operate on a discrete level with regards to
timestamps (See Section 5.1 Q3). Figure 6 shows time.
the prompt examples for time-removed and shuf-
fled version of prompts. xERTE (Han et al., 2021a) employs an atten-
tion mechanism that can effectively capture the
568: [0, 4, 1. 55] [0, 4, 1. 55] [0, 1, 0. 54] relevance of important aspects by selectively fo-
568: [0, 4, 3. 10] [0, 4, 3. 10] [0, 4, 3. 10]
568: [0, 1, 2. 36]
569: [0, 1, 0. 54]
[0, 1, 2. 36]
[0, 1, 0. 54]
[0, 4, 1. 55]
[0, 1, 4. 53]
cusing on them. It employs a sequential reasoning
570: [0, 1, 4. 53] [0, 1, 4. 53] [0, 1, 2. 36]
571: [0, 1, [0, 1, [0, 1,
approach over local subgraphs. This process begins
(a) Index (b) Time-removed (c) Time-removed + Shuffle with the query and iteratively selects relevant edges
of entities within the subgraph, subsequently prop-
Figure 6: Prompt examples for time-removed and
shuffled version. agating attention along these edges. After multiple
rounds of expansion, the final subgraph represents
the interpretable reasoning path towards the pre-
A.3 System Instruction for Instruction-tuned dicted outcomes.
models.
TimeTraveler (Sun et al., 2021) employs rein-
For the instruction-model, we use the manual cu- forcement learning for forecasting. The approach
rated system instructions to provide task descrip- involves the use of an agent that navigates through
tions and constraint the output format as follow: historical knowledge graph snapshots, commenc-
ing from the query subject node. Thereafter, it se-
You must be able to correctly predict the next
{object_label} from a given text consisting of
quentially moves to a new node by leveraging tem-
multiple quadruplets in the form of poral facts that are linked to the current node, with
"{time}:[{subject}, {relation}, {object_label}. the ultimate objective of halting at the answer node.
{object}]" and the query in the form of
"{time}:[{subject}, {relation}," in the end. To accommodate the issue of unseen-timestamp,
the approach incorporates a relative time encod-
You must generate only the single number for ing function that captures time-related information
{object_label} without any explanation.
when making decisions.
CyGNet (Zhu et al., 2021) leverages the statis-
tical relevance of historical facts, acknowledging
the recurrence of events in the temporal knowl-
edge graph datasets. It incorporates two inference
modes, namely Copy and Generation. The Copy
mode determines the likelihood of the query being
a repetition of relevant past facts. On the other
hand, the Generation mode estimates the probabil-
ity of each potential candidate being the correct
prediction, using a linear classifier. The final fore-
cast is obtained by aggregating the outputs of both
modes.
TLogic (Liu et al., 2022) mines cyclic temporal
logical rules by extracting temporal random walks
from a graph. This process involves the extraction
of temporal walks from the graph, followed by a
lift to a more abstract, semantic level, resulting in
the derivation of temporal rules that can generalize
to new data. Subsequently, the application of these
rules generates answer candidates, with the body
groundings in the graph serving as explicit and
easily comprehensible explanations for the results
obtained.

A.5 Full Experimental Results

 YAGO WIKI ICEWS14 ICEWS18 ACLED
Prompt History
H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10
Index Unidirectional Entity 0.777 0.880 0.904 0.610 0.724 0.775 0.293 0.427 0.533 0.160 0.267 0.395 0.364 0.497 0.613
Index Unidirectional Pair 0.778 0.880 0.904 0.646 0.731 0.777 0.294 0.400 0.471 0.187 0.294 0.385 0.331 0.457 0.564
Index Bidirectional Entity 0.778 0.879 0.904 0.607 0.721 0.773 0.274 0.404 0.527 0.142 0.245 0.382 0.336 0.497 0.613
Index Bidirectional Pair 0.780 0.879 0.904 0.647 0.733 0.777 0.291 0.398 0.471 0.185 0.291 0.384 0.324 0.455 0.564
Lexical Unidirectional Entity 0.777 0.874 0.904 0.607 0.722 0.775 0.285 0.406 0.532 0.149 0.257 0.394 0.293 0.469 0.601
Lexical Unidirectional Pair 0.781 0.874 0.904 0.645 0.730 0.777 0.280 0.392 0.470 0.171 0.278 0.381 0.277 0.426 0.549
Lexical Bidirectional Entity 0.773 0.872 0.904 0.601 0.714 0.771 0.270 0.399 0.526 0.141 0.238 0.371 0.343 0.464 0.601
Lexical Bidirectional Pair 0.777 0.874 0.904 0.643 0.728 0.777 0.278 0.390 0.470 0.182 0.288 0.381 0.293 0.426 0.554

Table 8: Performance Comparison (Hits@k) in single-step inference with time-aware filter.

YAGO WIKI ICEWS14 ICEWS18 ACLED

Prompt History
H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10
Index Unidirectional Entity 0.693 0.790 0.820 0.489 0.555 0.587 0.209 0.303 0.377 0.101 0.161 0.223 0.348 0.433 0.478
Index Unidirectional Pair 0.694 0.790 0.820 0.518 0.567 0.596 0.228 0.315 0.381 0.143 0.224 0.298 0.317 0.400 0.438
Index Bidirectional Entity 0.692 0.970 0.820 0.483 0.548 0.580 0.174 0.247 0.318 0.082 0.126 0.178 0.312 0.433 0.478
Index Bidirectional Pair 0.694 0.790 0.820 0.518 0.567 0.595 0.226 0.313 0.379 0.142 0.223 0.298 0.308 0.400 0.438
Lexical Unidirectional Entity 0.697 0.789 0.820 0.487 0.557 0.588 0.218 0.306 0.379 0.102 0.161 0.223 0.341 0.428 0.481
Lexical Unidirectional Pair 0.698 0.789 0.820 0.524 0.570 0.597 0.230 0.317 0.382 0.143 0.226 0.299 0.303 0.393 0.445
Lexical Bidirectional Entity 0.693 0.789 0.820 0.479 0.547 0.579 0.178 0.255 0.327 0.091 0.122 0.168 0.338 0.428 0.481
Lexical Bidirectional Pair 0.697 0.789 0.820 0.521 0.568 0.596 0.227 0.315 0.379 0.141 0.212 0.287 0.312 0.393 0.445

Table 9: Performance Comparison (Hits@k) in multi-step inference with time-aware filter.

YAGO WIKI ICEWS14 ICEWS18

Single-Step
H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10
frequency 0.766 0.859 0.921 0.549 0.712 0.818 0.243 0.387 0.532 0.141 0.265 0.409
recency 0.886 0.927 0.928 0.701 0.831 0.849 0.228 0.387 0.536 0.120 0.242 0.403
GPT-NeoX (Entity) 0.784 0.891 0.927 0.694 0.804 0.844 0.324 0.460 0.565 0.192 0.313 0.414
GPT-NeoX (Pair) 0.787 0.892 0.926 0.721 0.812 0.847 0.297 0.408 0.482 0.196 0.307 0.402

Table 10: Performance Comparison (Hits@k) between rule-based prediction and ICL in single-step inference
with time-aware filter.

YAGO WIKI ICEWS14 ICEWS18

Single-Step
H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10 H@1 H@3 H@10
frequency 0.691 0.789 0.837 0.484 0.603 0.652 0.222 0.349 0.460 0.121 0.207 0.307
recency 0.785 0.840 0.842 0.540 0.637 0.661 0.151 0.268 0.423 0.074 0.149 0.266
GPT-NeoX (Entity) 0.686 0.793 0.840 0.543 0.622 0.655 0.247 0.363 0.471 0.136 0.224 0.321
GPT-NeoX (Pair) 0.688 0.793 0.839 0.570 0.625 0.652 0.236 0.324 0.395 0.155 0.245 0.331

Table 11: Performance Comparison (Hits@k) between rule-based prediction and ICL in multi-step inference with
time-aware filter.