Enhancing Temporal Knowledge Graph Forecasting with Large Language

Models via Chain-of-History Reasoning

Yuwei Xia1,2 , Ding Wang3,4 , Liu Qiang3,4 , Liang Wang3,4 , Shu Wu3,4 , Xiaoyu Zhang1,2
1
Institute of Information Engineering, Chinese Academy of Sciences
2
School of Cyber Security, University of Chinese Academy of Sciences
3
School of Artificial Intelligence, University of Chinese Academy of Sciences
4
Institute of Automation, Chinese Academy of Sciences
xiayuwei@[Link], wangding2024@[Link], [Link]@[Link]
Liang Wang@[Link], [Link]@[Link], zhangxiaoyu@[Link]

Abstract Russia Ukraine Germany Denmark Russia Germany

have a war
Temporal Knowledge Graph (TKG) forecast- Condemn Sign communicate Support
Sanction agreement
ing aims to predict future facts based on given ?
arXiv:2402.14382v1 [[Link]] 22 Feb 2024

histories. Most recent graph-based models ex- European Union Russia Germany
cel at capturing structural information within �� �� �� �� time
TKGs but lack semantic comprehension abil- Germany will support whom at �� ?
ities. Nowadays, with the surge of LLMs, Germany sign agreement with Denmark at �3 Possible answers:
Germany communicate with Russia at �3 Denmark Russia
the LLM-based TKG prediction model has Germany condemn Russia at �2
emerged. However, the existing LLM-based (a) Provide only first-order history for LLMs to infer
model exhibits three shortcomings: (1) It only Germany will support whom at �� ?
focuses on the first-order history for predic- Germany sign agreement with Denmark at �3 Possible answers:
tion while ignoring high-order historical infor- Germany communicate with Russia at �3 Denmark Russia
Germany condemn Russia at �2 Ukraine
mation, resulting in the provided information Russia has a war with Ukraine at �1
for LLMs being extremely limited. (2) LLMs
(b) Provide high-order history for LLMs to infer
struggle with optimal reasoning performance
under heavy historical information loads. (3) Figure 1: An example of reasoning over TKG with
For TKG prediction, the temporal reasoning LLMs. In Figures (a) and (b), we provide LLMs with
capability of LLM alone is limited. To address different histories, which prompt LLMs to reason differ-
the first two challenges, we propose Chain-of- ent answers for the predicted fact.
History (CoH) reasoning which explores high-
order histories step-by-step, achieving effective
utilization of high-order historical information
capability on various tasks has been gradually ex-
for LLMs on TKG prediction. To address the
third issue, we design CoH as a paly-and-plug
plored (Jain et al., 2023; Yuan et al., 2023). Lee et
module to enhance the performance of graph- al. take the first attempt on TKG reasoning with
based models for TKG prediction. Extensive LLMs (Lee et al., 2023), providing LLMs with
experiments on three datasets and backbones histories in textual form. Despite some achieved
demonstrate the effectiveness of CoH. advancements, we argue there are still significant
issues that need to be addressed.
1 Introduction Firstly, the existing TKG prediction model with
As a carrier of facts with temporal information, LLMs only focuses on the first-order histories, ig-
Temporal Knowledge Graphs (TKGs) hold signifi- noring important high-order historical information.
cant practical value across various applications (Xi- Taking Figure 1 as an example, LLMs aim to infer
ang et al., 2022; Chen et al., 2023). Most advanced “Germany will support whom at t4 ” with provided
research on TKGs mainly focuses on predicting fu- histories. The existing model provides LLMs with
ture facts occur at time tn based on given historical only first-order histories. In this case, LLMs are
facts occur at time t with t < tn . constrained to infer wrong answers to “Denmark”
Recent supervised methods (Jin et al., 2020; Li and “Russia” because given histories fail to encom-
et al., 2021b, 2022) mainly rely on Graph Neural pass the correct answer. When supplied with more
Networks (GNNs) to capture structural dependen- high-order histories, LLMs can utilize the history
cies within TKGs, yet they all fall short in semantic chain “Germany→Russia→Ukraine” to reason the
modeling. Nowadays, with the surge of Large Lan- correct answer “Ukraine” more possibly.
guage Models (LLMs), their temporal reasoning Secondly, LLMs struggle to maintain reasoning

1
 to multiple-step reasoning for complex situations.
Mistral-7B (%)
LLMs can continue inferring important second-
Mixtral-8x7B (%)
37
order history chains in Step 2, then infer answers
with third-order history chains in Step 3, and so
on. In this way, LLMs only need to process a lim-
33 ited quantity of histories at each step, preventing
an overwhelming influx of complex information
50 75 100 125 150
while effectively leveraging a more comprehensive
set of high-order information. This approach en-
Figure 2: The performance (MRR (%)) of LLMs of two ables LLMs to perform more accurate reasoning
sizes based on different history lengths on TKG pre- with higher-order information for TKG prediction.
diction. The provided histories contain both first- and Moreover, we design CoH as a play-and-plug mod-
second-order histories. The y-axis represents the MRR ule for TKG reasoning. As shown in Figure 3, we
(%) value, and the x-axis denotes the total length of pro- fuse the predicted results obtained by LLMs and
vided first- and second-order histories. The results are graph-based TKG models to make the final pre-
based on the commonly used TKG dataset ICEWS14.
diction more comprehensively. To summarize, the
contributions of this paper can be listed as follows:
• We are the first to explore the necessity and
performance under heavy historical information
difficulty of providing numerous high-order
loads. Intuitively, we provide more comprehensive
histories for LLMs on TKG prediction. And
high-order histories for LLMs to infer. However, as
we propose the Chain-of-History reasoning
shown in Figure 2, the performance of LLMs does
method which adopts LLMs to explore the
not necessarily improve or remain stable with the
history chains step-by-step.
increase in history length, instead experiencing a
• We are the first to propose enhancing the per-
steep decline beyond a certain threshold of history
formance of graph-based TKG models with
length regardless of the model size. This indicates
LLMs, utilizing the semantic understanding
that over-complicated historical information may
advantage of LLM to compensate for the
confuse LLMs (Shi et al., 2023), making LLMs
shortcoming of graph-based models.
hard to reason correct answers. Thus, exploring
• We conduct extensive experiments on three
ways to offer higher-order histories for LLMs ef-
commonly used TKG datasets and three
fectively is a worthwhile investigation.
graph-based TKG models, the results demon-
Thirdly, relying solely on the reasoning capabil- strate the effectiveness of CoH.
ities of LLMs still remains limited on TKG pre-
diction. Though LLMs possess unique semantic 2 Problem Formulation
comprehension advantages in reasoning, they still
struggle to achieve the same level of ability in Temporal Knowledge Graph Prediction. Let E
capturing complex structural information as graph- and R represent a set of entities and relations. A
based models. However, this unique advantage of Temporal Knowledge Graph (TKG) G can be de-
LLMs precisely compensates for the shortcomings fined as G = {G1 , G2 , · · · , Gn }. Each Gt ∈ G con-
of graph-based models in modeling semantic in- tains facts that occur at time t. Each fact is repre-
formation, thereby enhancing the performance of sented as a quadruple (s, r, o, t), in which s, o ∈ E
graph-based models on TKG prediction. and r ∈ R. Given a query q = (sq , rq , ?, tq ) or q =
(?, rq , oq , tq ), TKG prediction task aims to predict
To address the above issues, we propose a Chain-
the missing object entity or subject entity with his-
of-History (CoH) reasoning method for TKG pre-
torical KG sequence G<tq = {G1 , G2 , · · · , Gtq −1 }.
diction. Instead of providing LLMs with all his-
The candidate answers for q are all entities in E,
tories at once, CoH provides LLMs with high-
each candidate ei ∈ E will be estimated with a
order histories step-by-step. Specifically, CoH
score by TKG prediction models.
adopts LLMs to explore important high-order his-
tory chains step-by-step, and reason the answers High-order History Chains in TKGs. For each
to the query only based on inferred history chains query q = (sq , rq , ?, tq ) to be predicted, we denote
in the last step. A two-step CoH reasoning proce- {(sq , r, o, t)|(sq , r, o, t) ∈ G<tq } as the first-order
dure is shown in Figure 3. This also can extend histories of q. If (sq , r, o, t) is the first-order history

2
 Predict
Query: Germany will support whom at �� ?

Reasoning Procedure with LLMs

Step 1 Step 2
First-order histories Second-order history chains
Germany condemn Russia at �2
Germany condemn Russia at �2 ,
Germany meeting with European Union at �2 Russia has a war with Ukraine at �1
Germany hold a visit to China at �2
Germany sign agreement with Denmark at �3 Score
Germany hold meeting with Albania at �3 ,
Germany communicate with Russia at �3 Albania provide aid for Afghanistan at �1
Fusion
Germany hold meeting with Albania at �3
Germany consult with Japan at �3 Germany consult with Japan at �3 ,
...
Japan hold meeting with Vietnam at �1
...
Large Language Models
Large Language Models
Inferred first-order histories
Germany condemn Russia at �2 Graph-based
Inferred possible answers model
Germany holds meeting with Albania at �3
1. Ukraine 2. Vietnam 3. Afghanistan
Germany consult with Japan at �3
... 4. Russia ... Entity
Score 0.7 0.5 0.3
…

Figure 3: An illustration of a two-step CoH reasoning procedure. In the first step, LLMs are provided with only
first-order histories and asked to infer the most important histories. In the second step, LLMs are provided with
second-order history chains based on the inferred first-order histories and asked to infer possible answers to the
given query. Then the answers inferred by LLMs and graph-based models are adaptively fused to make the final
prediction. Note that this only serves as a two-step reasoning example, more steps can be executed with CoH.

of q, we denote histories in the form of (o, r, o′ , t) Denmark, 2023-06-02)”. To make each quadru-
as a set of second-order histories of q. The higher- ple more linguistically comprehensible for LLMs,
order histories of q are deduced in this way. And we we introduce prepositions to transform the quadru-
denote [(sq , r, o, t), (o, r, o′ , tj )] as a second-order ple into a more fluent sentence. Most importantly,
history chain of q, which consists of a first-order considering the prior knowledge of LLMs, we pro-
history of q and its associated second-order history cess the time “2023-06-02” in each quadruple into
of q. The higher-order history chains of q can be a more abstract form like “153rd day” to prevent
deduced in this way. LLMs from directly exploiting prior knowledge
for predictions. Consequently, we provide LLMs
3 Chain-of-History Reasoning over with each quadruple in the form of “(Germany Sign
Temporal Knowledge Graph agreement with Denmark on the 153rd day)”.

For a given query q = (sq , rq , ?, tq ), CoH predicts 3.2 Reasoning Steps

the answers by exploring history chains related to q
In a k-step reasoning procedure of CoH, LLMs are
step-by-step, then answers are fused with predicted
instructed to explore the most significant history
results by graph-based TKG models to make the
chains related to the given query q from Step 1 to
final prediction for q. In this section, we illustrate
Step k-1. Subsequently, LLMs reason the possible
CoH shown in Figure 3 in detail. Section 3.1 ex-
answers to q in Step k with k-order history chains.
plains how to convert quadruples in TKGs into text
formats suitable for LLMs. Section 3.2 demon- Step 1 to Step k-1 Reasoning. In Step 1, LLMs
strates how to properly provide histories from G<tq are provided with only first-order histories of q, and
for LLMs and how to instruct LLMs to reason in are instructed to reason n first-order histories that
each step. Section 3.3 shows how to transform an- mostly contribute to answering q. From Step 2 to
swers predicted by LLMs into scores and fuse them Step k-1, LLMs are provided with i-order history
with the results from graph-based TKG models. chains in Step i (i = {2, 3, · · · , k − 1}), and are
instructed to infer n most significant history chains.
3.1 History Processing The instruction design is shown in Table 1. Within
Each fact in TKGs is presented as a quadruple this sequence of steps, the outputs of LLMs in Step
(s, r, o, t), such as “(Germany, Sign agreement, i-1 are the inferred (i-1)-order history chains, then

3
 Step i Instruction
Step 1 There is a given text consisting of multiple historical events in the form of “{id}:[{subject} {relation}
{object} {time}];”. And there is a query in the form of: “{subject} {relation} {whom} time}?” If you
must infer several {object} that you think may be the answer to the given query based on the given
historical events, what important historical events do you base your predictions on? Please list the top n
most important histories and output their {id}.
Step 2 to Step k-1 There is a given text consisting of multiple history chains in the form of “{id}:[{subject} {relation}
{object} {time}, {subject} {relation} {object} {time}, ...];”. And there is a query in the form of: “{subject}
{relation} {whom} time}?” If you must infer several {object} that you think may be the answer to
the given query based on the given historical events, what important history chains do you base your
predictions on? Please list the top n most important history chains and output their {id}.
Step k You must be able to correctly predict the {whom} of the given query from a given text consisting of
multiple historical events in the form of “{subject} {relation} {object} {time}” and the query in the form
of “{subject} {relation} {whom} {time}?” You must output several {object} that you think may be the
answer to the given query based on the given historical events. Please list all possible {object} which may
be answers to the query. Please assign each answer a serial number to represent its probability of being
the correct answer. Note that answers with a high probability of being correct should be listed first.

Table 1: Instruction design for each step in a k-step CoH reasoning procedure.

each of which is supplied with corresponding i- which also constitutes a significant part of TKGs.
order histories to consist of i-order history chains. Considering the importance of both structural and
These history chains subsequently serve as input for semantic information within TKGs, we propose to
the next Step i. As shown in Figure 3, “Germany fuse the predicted results of LLMs and graph-based
condemn Russia at t2 ” is one of the outputs in Step models to obtain more comprehensive results for
1. Then it is supplied with corresponding second- predicting over TKGs more accurately.
order history “Russia has a war with Ukraine at t1 ” Firstly, for a given q, we need to obtain the score
to consist of the second-order history chain, which of each entity ei in the LLMs’ predicted answer set
serves as the input for Step 2. The prompt example AqLLM . As we mentioned each answer predicted by
for Step 1 and Step i (i = {2, 3, · · · , k − 1}) can LLMs in Step k is assigned an index, which repre-
refer to Appendix A. sents the probability of the answer being correct.
We convert the index of each answer ei ∈ AqLLM
Step k Reasoning. In Step k, LLMs are provided
into its corresponding score with an exponential
with k-order history chains and instructed to reason
decay function as follows:
possible answers for the given query q. The instruc-
tion design for Step k is shown in Table 1. Espe- ei 1
SLLM = e , (1)
cially, we instruct LLMs to prioritize outputting 1 + eα·idx i
the entity with a higher probability of being the ei
where SLLM denotes the score of the entity ei ob-
correct answer. As shown in Figure 3, the output of
tained with LLMs for being the answer to q, idxei
Step 2 includes several possible answers to q. Each
represents the numerical index of the answer ei in
answer is assigned a numerical index (1,2,3,· · · ),
the outputs of LLMs, and α is a hyper-parameter
with a lower index indicating a higher probability
to control the score disparity among answers with
of the answer being correct. The prompt example
different indexes. Note that since the outputs of
for Step k can refer to Appendix A.
LLMs can not include all candidate entities in E
3.3 Results Processing and Fusion like graph-based models, we assign the score of ei
as 0 where ei ∈ E but ei ∈/ AqLLM .
In graph-based TKG models, each entity and re-
Then, we can fuse the score of each candidate
lation in a quadruple are denoted with an id like
entity ei ∈ E obtained with LLMs and graph-based
“(30, 13, 8, 2023-06-02)” instead of “(Germany,
models as follows:
Sign agreement, Denmark, 2023-06-02)”. The lack
of semantic modeling of histories makes graph- ei
S ei = w · SGraph ei
+ (1 − w) · SLLM , (2)
based models mainly depend on structural informa-
ei
tion within TKGs for prediction. However, entities where SGraph denotes the score of ei obtained with
and relations inherently carry semantic information, graph-based models, and w is a hyper-parameter

4
to determine the weight of different scores. S ei first-order histories under all evaluation metrics
represents the comprehensive score of the candi- on three datasets. This indicates the usefulness
date ei . Finally, the ranked candidate list based on of the higher-order histories provided step-by-step.
comprehensive scores is used for predicting q. And we observe that the relative improvements of
CoH over the existing method are more obvious
4 Experiments on ICEWS18 than other datasets, which implies
In this section, we conduct extensive experiments that ICEWS18 may contain more and complex in-
to evaluate and analyze CoH on three typical formation of history chains. Despite the progress
datasets and three backbones for TKG prediction. achieved by CoH, the performance of only utiliz-
Details of datasets and backbones can be referred ing LLMs on TKG prediction is still pretty limited
to Appendix B and C, respectively. compared with graph-based models.
On the other hand, though the temporal reason-
4.1 Experimental Settings ing capability of LLMs on TKG prediction is rela-
4.1.1 Evaluation tively limited, they can be flexibly used as a play-
For evaluation, we adopt widely used metrics MRR and-plug module to enhance the performance of
and Hits@{1, 3, 10} in experiments. Without loss graph-based models. From the results of plugging
of generality (Li et al., 2021b), we only report the CoH and the ICL-based model into existing graph-
experimental results under the raw setting. Note based models, we can see that the two LLM-based
that different from (Lee et al., 2023), we fully align models can effectively improve their performance.
the evaluation mechanism for LLMs in TKG pre- The gains introduced by CoH to the performance of
diction with those used in graph-based models to graph-based models far surpass those achieved by
ensure a more fair comparison. Specifically, dur- the ICL-based model, which further demonstrates
ing the testing phase of graph-based models, the the effectiveness of our proposed model. More-
test set is typically augmented by doubling its size over, we analyze the rationale behind these gains
through reversing (s, r, o, t) into (o, r−1 , s, t), to may be attributed to the distinct reasoning mecha-
assess the model’s performance more comprehen- nisms of LLMs and graph-based models, each of
sively. Correspondingly, we also evaluate LLMs which possesses unique strengths. In this case, the
on TKG prediction with reversed test sets. powerful semantic understanding ability of LLMs
may be capable of compensating to some extent
4.1.2 CoH Implementation Details for the inherent limitations in semantic information
In this paper, we implement CoH with two-step rea- modeling of graph-based models.
soning based on an open-sourced language model
4.3 Ablation Study
Mixtral-8x7B (Jiang et al., 2024). In Step 1, we
provide the LLM with 100 first-order histories and In this section, we conduct experiments to investi-
set n to 30, allowing the LLM to infer the most gate the effectiveness of the high-order historical
important 30 first-order histories from the given information, step-by-step reasoning mechanism,
ones. In Step 2, we do not strictly limit the number and score ranking procedure in CoH. The results
of answers output from the LLM. For more details are shown in Table 3.
on implementation please refer to Appendix D. Analysis of high-order historical information.
4.2 Performance Comparison To verify the usefulness of the high-order historical
information for TKG prediction with LLMs, we
In this section, we present a comprehensive eval-
use the inferred first-order histories in Step 1 and
uation of the proposed CoH. We first evaluate the
the second-order history chains in Step 2 to predict
performance of only utilizing LLMs on TKG pre-
answers, respectively. From the results of CoH
diction with CoH reasoning. Then we plug CoH
shown in Table 3 we can observe that, the predicted
on three existing state-of-the-art graph-based TKG
results of LLMs based on the second-order history
prediction models to see the potential gains it can
chains are notably superior to the results derived
yield. The results are shown in Table 2, from which
solely from first-order histories. This indicates the
we have the following observations.
usefulness of second-order histories.
On the one hand, from the results of LLMs, the
two-step CoH reasoning outperforms ICL (Lee Analysis of step-by-step reasoning mechanism.
et al., 2023) which solely provides LLMs with Specifically, “LR” in Table 3 denotes the step of

5
 ICEWS14 ICEWS18 ICEWS05-15
Model Type Model
MRR Hit@1 Hit@3 Hit@10 MRR Hit@1 Hit@3 Hit@10 MRR Hit@1 Hit@3 Hit@10
ICL (Lee et al., 2023)* 31.79 22.38 37.67 47.70 21.51 14.77 26.08 40.57 35.34 25.18 43.92 56.24
LLMs CoH 34.51 24.20 39.67 51.21 23.94 16.81 28.15 42.68 37.51 27.72 47.17 59.58
∆Improve 8.56% 8.13% 5.31% 7.36% 11.30% 13.81% 7.94% 5.20% 6.14% 10.09% 7.40% 5.94%
RE-NET 38.75 28.96 43.64 57.61 28.72 18.84 32.66 48.18 44.05 33.22 51.23 65.02
RE-NET + ICL* 39.39 29.12 44.37 58.25 29.01 18.98 33.11 48.78 45.12 33.98 52.09 66.23
RE-NET + CoH 40.43 30.34 45.78 60.42 29.77 19.96 34.14 49.59 46.37 34.99 53.13 67.71
∆Improve* 1.65% 0.56% 1.67% 1.11% 1.00% 0.73% 1.39% 1.25% 2.42% 2.29% 1.67% 1.87%
∆Improve 4.34% 4.77% 4.90% 4.87% 3.66% 5.94% 4.53% 2.93% 5.26% 5.33% 3.71% 4.14%
RE-GCN 41.33 30.61 46.66 62.31 31.08 20.44 35.39 52.06 46.89 35.5 53.33 68.4
RE-GCN + ICL* 41.84 30.84 47.27 62.97 31.31 20.65 35.79 52.61 47.87 36.16 54.24 69.43
LLMs + Graph
RE-GCN + CoH 42.41 31.77 47.85 63.80 32.10 21.75 36.51 53.37 47.98 37.53 54.94 70.68
∆Improve* 1.23% 0.75% 1.31% 1.06% 0.74% 1.03% 1.13% 1.06% 2.09% 1.86% 1.71% 1.51%
∆Improve 2.61% 3.79% 2.55% 2.39% 3.28% 6.41% 3.16% 2.52% 2.32% 5.72% 3.02% 3.33%
TiRGN 42.93 32.1 48.53 63.6 31.97 20.95 36.67 53.66 48.5 36.87 55.19 70.27
TiRGN + ICL* 43.27 32.28 49.04 64.17 32.18 21.07 36.98 54.04 49.15 37.25 55.73 70.9
TiRGN + CoH 43.94 33.07 49.64 64.90 32.98 21.83 37.79 54.92 49.71 38.01 56.40 71.25
∆Improve* 0.79% 0.56% 1.05% 0.90% 0.66% 0.57% 0.85% 0.71% 1.34% 1.03% 0.98% 0.90%
∆Improve 2.35% 3.02% 2.29% 2.04% 3.16% 4.20% 3.05% 2.35% 2.49% 3.09% 2.19% 1.39%

Table 2: Performance comparison of CoH on TKG prediction on three datasets in terms of MRR (%), Hit@1 (%),
Hit@3 (%), and Hit@10 (%). All results are obtained under raw metrics. The highest performance is highlighted
in bold. And * represents the reproduced model with the same evaluation and LLM used in this paper. ∆Improve
and ∆Improve* indicate the relative improvements of CoH and the ICL-based model plugged into the graph-based
models over the original graph-based backbones in percentage, respectively.

ICEWS14 ICEWS18 ICEWS05-15

Model Step i
MRR Hit@1 Hit@3 Hit@10 MRR Hit@1 Hit@3 Hit@10 MRR Hit@1 Hit@3 Hit@10
Step 1 32.31 23.18 37.92 49.15 20.97 13.88 24.83 36.52 34.5 22.24 42.01 56.18
CoH w/o LR
Step 2 32.68 23.74 38.12 50.09 21.89 14.17 25.32 37.98 34.89 22.48 43.26 57.34
Step 1 33.97 23.86 39.03 49.96 22.03 15.34 26.82 40.57 36.48 24.84 46.44 58.51
CoH
Step 2 34.51 24.20 39.67 51.21 23.94 16.81 28.15 42.68 37.51 27.72 47.17 59.58
CoH w/o IS Step 2 24.57 12.31 34.75 51.09 13.8 9.14 20.57 35.67 29.61 18.55 37.64 56.87

Table 3: Ablation studies to investigate the effectiveness of the high-order historical information, step-by-step
mechanism, and score ranking procedure of CoH in terms of MRR (%), Hit@1 (%), Hit@3 (%), and Hit@10 (%).
And all results are obtained under raw metrics.

reasoning important first-order histories by LLMs we can see that the shuffled index order leads to
in two-step CoH reasoning. And we implement a huge drop in performance. This indicates that
“CoH w/o LR” by replacing the inferred n first- the indexes outputted by LLMs can be helpful for
order histories by LLMs with n first-order histories score ranking on TKG prediction.
in the latest timestamps. In this way, we can find
out whether LLMs can deduce meaningful histori- 4.4 Case Study
cal information within the step-by-step reasoning
mechanism. From the results we can see that CoH In this section, we visualize the reasoning process
outperforms the one without LR under all evalua- of two queries to understand the reasoning mech-
tion metrics on three datasets, which illustrates the anism of CoH. Moreover, to further understand
effectiveness of achieving the step-by-step reason- how CoH benefits the graph-based TKG prediction
ing mechanism with LLMs. models, we show the difference between the results
predicted by CoH and those inferred by the graph-
Analysis of the score ranking procedure. In based model RE-NET (Jin et al., 2020). Details of
the last step of CoH reasoning, we instruct LLMs the two cases are shown in Table 4.
to output possible answers in order based on their From the reasoning process of the two cases,
possibilities of being correct. To verify whether we can see that LLMs possess the capability of
the output index of each answer is related to its inferring important histories related to the given
correctness, we shuffle the index order of the an- query. And in the way of CoH reasoning, LLMs
swers, which is denoted as “CoH w/o IS” in Table can accurately infer the answers like “Thailand”
3. Compared the results of CoH w/o IS with CoH, in the second-order history chains. Furthermore,

6
 Model Query 1: Military_(Myanmar) Express_intent_to_meet_or_negotiate to whom on the 351th day? (GT: Thailand)
[Military_(Myanmar) Fight_with_small_arms_and_light_weapons Myanmar 338]→ [Myanmar Make_a_visit to Thailand 328]
[Military_(Myanmar) Charge_with_legal_action Media_Personnel_(Myanmar) 305]→ [Media_Personnel_(Myanmar) Make_statement Detainee_(Myanmar) 302]
[Military_(Myanmar) Make_an_appeal_or_request Citizen_(Thailand) 272]→[Citizen_(Thailand) Use_violence to Thailand 271]
CoH [Military_(Myanmar) Make_an_appeal_or_request Citizen_(Thailand) 271] →[Citizen_(Thailand) Release_person(s) Activist_(Thailand) 267]
[Military_(Myanmar) Praise_or_endorse Military_(Thailand) 185]
[Military_(Myanmar) Use_military_force to Rebel_Group_(Myanmar) 174]
··· ···
[Military_(Myanmar) Use_military_force to National_Liberation_Army 62]→[National_Liberation_Army Make_statement to Guerrilla_(Colombia) 58]
Answer: 1. Thailand 2. Citizen_Thailand 3. Activist_Thailand 4. National_Liberation_Army 5. Media_Personnel_Myanmar
Graph Answer: 1. Myanmar 2. Malaysia 3. Cambodia 4. Citizen_(Thailand) 5. Thailand
Query 2: Saudi_Army Use_military_force to whom on the 3744th day? (GT: Armed_Rebel_(Yemen))
[Saudi_Army Make_an_appeal_or_request Zillur_Rahman 2581] →[Zillur_Rahman Make_empathetic_comment Citizen_(North_Korea) 2543]
[Saudi_Army Consult Zillur_Rahman 2581] → [Zillur_Rahman Make_empathetic_comment citizen_(Nerth_Kerea) 2543]
[Saudi_Army Express_intent_to_meet_or_negotiate with Thailand 2581] → [Thailand Use_military_force Military_(Cambodia) 2580]
CoH [Saudi_Army Investigate Armed_Gang_(Saudi_Arabia) 1798] → [Armed_Gang_(Saudi_Arabia) Use_unconventional_violence Citizen_(Saudi_Arabia) 1773]
[Saudi_Army Employ_aerial_weapons Yemen 1769] → [Yemen Charge_with_legal_action Armed_Rebel_(Yemen) 1764]
Answer: 1. Military_(Cambodia) 2. Armed_Rebel_(Yemen) 3. Armed_Gang_(Saudi_Arabia) 4. Citizen_(Saudi_Arabia) 5. Citizen_(North_Korea)
Graph Answer: 1. Yemen 2. Citizen_(Saudi_Arabia) 3. Saudi_Arabian_Defence_Forces 4. Police_(Saudi_Arabia) 5. Armed_Rebel_(Yemen)

Table 4: Case studies with two queries for showing the procedure of CoH reasoning. The bold histories denote the
inferred first-order histories by LLMs in Step 1, and the histories in blue color denote corresponding second-order
histories. These two kinds of histories consist of second-order history chains for LLMs to infer answers in Step 2.
And the answers in red color represent the ground truth of the given query.

Datasets ICEWS14 ICEWS05-15 ICEWS18 ICEWS14 ICEWS18 ICEWS05-15

Model
Ratio of known facts 3.89% 5.67% 5.87% Hit@1 Hit@10 Hit@1 Hit@10 Hit@1 Hit@10
Anon-CoH 17.00 48.18 13.81 38.39 23.21 51.62
CoH 24.2 51.21 16.81 42.68 25.61 59.58
Table 5: Known facts checking results.
Table 6: Anonymization experimental results.
comparing the answers of CoH with the graph-
based model, we can see that CoH can infer the 4.5 Analysis of Data Leakage
correct answer more accurately in some scenarios.
We analyze the potential reason is that the semantic Due to the absence of publicly disclosed temporal
reasoning capability can allow LLMs to identify horizons for the pre-trained data of Mixtral-8x7B
crucial historical information. (Jiang et al., 2024), we consider whether this LLM
For example, from the numerous histories in may have covered some data within ICEWS14,
query 1, LLMs can more precisely infer facts ICEWS18, and ICEWS05-15. To verify this prob-
involved with relations like “Fight with small lem, we have a conversation with Mixtral-8x7B to
arms”, “Charge with legal action”, or “Make check whether it knows about the facts within the
an appeal” that are more likely to result in the three datasets following (Shi et al., 2023). The ratio
occurrence of “Express intent to meet or nego- of known facts in each dataset is shown in Table
tiate” based on semantic comprehension. For 5, which indicates that Mixtral-8x7B only knows a
query 2, understanding the semantic meaning dif- very limited subset of facts. And the conversations
ferences and correlations between entity “Yemen” are presented in Appendix E. To avoid the problem
and “Armed_Rebel_(Yemen)” is a very crucial clue. of data leakage, we exclude this subset of known
However, the two entities are just regarded as two facts from the testing set for CoH reasoning.
different IDs in graph-based models, and avail-
4.6 Analysis on the effect of Prior Knowledge
able histories for query 2 are too limited for them
within LLMs
to aggregate abundant information. In this case,
the semantic reasoning characteristics of CoH may In this section, we conduct experiments to inves-
be more advantageous. Consequently, LLMs ex- tigate how the prior knowledge within LLMs af-
hibit semantic comprehension capability, while the fects the performance of CoH on TKG prediction.
graph-based models possess a powerful ability to Specifically, we anonymize the TKG data by rep-
capture structural information. Thus, the reasoning resenting each entity and relation with numerical
capability of CoH based on LLMs may potentially IDs. The anonymized results of CoH are presented
complement the reasoning performed by graph- in Table 6, which is denoted as Anon-CoH. From
based models in certain scenarios. Table 6 we can see that the anonymization leads

7
 5 Related works
48
42
RE-NET + CoH (%) 31 RE-NET + CoH (%) RE-NET + CoH (%) Temporal Knowledge Graph Forecasting with
RE-GCN + CoH (%) RE-GCN + CoH (%) RE-GCN + CoH (%)

40
47 Supervised Models. TKG prediction aims to
30

0.1 0.3 0.5 0.7 0.9 0.1 0.3 0.5 0.7 0.9 0.1 0.3 0.5 0.7 0.9
infer future facts based on given histories. Re-
cently, most supervised models (Jin et al., 2020; Li
(a) ICEWS14s (b) ICEWS18 (c) ICWS05-15
et al., 2021b, 2022; Zhang et al., 2023; Liang et al.,
Figure 4: Performance of graph-based models plugged 2023; Zhang et al., 2023) for TKG prediction adopt
with CoH under different α-values in terms of MRR (%). GNNs to capture the structural information within
The x-axis denotes different α-values, and the y-axis TKGs. For further details about supervised models
shows MRR (%) values. on TKG prediction, please refer to Appendix F.

Temporal Knowledge Graph with Large Lan-

42
48 guage Models (LLMs). Recently, there are sev-
32
RE-NET + CoH (%) RE-NET + CoH (%) RE-NET + CoH (%) eral works (Han et al., 2022; Gao et al., 2023;
RE-GCN + CoH (%) RE-GCN + CoH (%) RE-GCN + CoH (%)
40
30
46 Xu et al., 2023a) have attempted to leverage Pre-
0.3 0.4 0.5 0.6 0.7 0.3 0.4 0.5 0.6 0.7 0.3 0.4 0.5 0.6 0.7
trained Language Models (PLMs) on TKG reason-
ing, which mainly input histories in textual form
(a) ICEWS14s (b) ICEWS18 (c) ICWS05-15
into PLMs to obtain contextualized knowledge em-
Figure 5: Performance of graph-based models plugged beddings. Nowadays, with the surge of LLMs,
with CoH under different w-values in terms of MRR their reasoning capabilities on structural and tempo-
(%). The x-axis denotes different w-values, and the y- ral data are progressively under exploration (Jiang
axis shows MRR (%) values. et al., 2023; Jain et al., 2023; Yuan et al., 2023;
Aghzal et al., 2023; Wang and Zhao, 2023; Tan
et al., 2023). In the area of TKGs, Ding et al. input
to a certain decline in the performance of CoH, relations in textual form into LLMs to generate cor-
which indicates that the prior knowledge provides responding descriptions, which are then introduced
a certain degree of assistance for reasoning. Note into embed-based models as a supplement to the
that since the possible leakage data is filtered (Sec- semantic information of zero-shot relations (Ding
tion 4.5), we analyze that the usefulness of prior et al., 2023). And Lee et al. take the first attempt
knowledge is primarily attributed to certain static on TKG prediction using LLMs, which is the most
semantic knowledge. closely related work to our paper. They covert TKG
prediction into an In-context Learning (ICL) prob-
4.7 Sensitivity Analysis lem, providing LLMs with the first-order histories
of the query in textual form to predict the possible
For converting the indexes outputted by LLM into
answers (Lee et al., 2023).
corresponding scores, α determines the score gap
for different indexes. we conduct two graph-based
6 Conclusion
models plugged with CoH when α is in the range
of {0.1, 0.3, 0.5, 0.7, 0.9}. The results are shown In this paper, we first analyze the shortcomings
in Figure 4, from which we can see that variations and challenges of the existing LLM-based model
in the value of α within a very narrow range have a about how to effectively provide comprehensive
minimal impact on the model’s performance. high-order historical information for LLM. Then
Moreover, w determines the score weight for we point out that relying solely on the reasoning ca-
fusing the predicted results of graph-based mod- pability of LLMs is still limited for TKG prediction.
els and CoH. we conduct two graph-based models To resolve these issues, we propose CoH reasoning
plugged with CoH under various w-values. The which achieves effective utilization of high-order
results are shown in Figure 5, from which we can histories for LLM. And we design CoH as play-
see that in the optimal outcomes, the results of and-plug, serving to complement and enhance the
CoH contribute slightly more to the final score. We performance of graph-based models. Extensive ex-
analyze the underlying reason leading to the afore- perimental results demonstrate the superiority of
mentioned observations may be related to the score CoH, and its effectiveness in enhancing the perfor-
distribution of the graph-based models. mance of graph-based models for TKG prediction.

8
7 Limitations Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp.
2021a. Explainable subgraph reasoning for forecast-
In this section, we discuss the limitations of our ing on temporal knowledge graphs. In ICLR.
model. Since CoH reasoning is conducted in multi-
step, LLMs need to be invoked multiple times, re- Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and
Volker Tresp. 2021b. Learning neural ordinary equa-
sulting in an increased complexity of the inference tions for forecasting future links on temporal knowl-
process. Moreover, we design CoH as play-and- edge graphs. In EMNLP, pages 8352–8364.
plug, fusing its answers with predicted results of
graph-based models. As this entire process does Zhen Han, Ruotong Liao, Beiyan Liu, Yao Zhang,
Zifeng Ding, Jindong Gu, Heinz Koeppl, Hinrich
not involve any training, the fusion weight can only
Schuetze, and Volker Tresp. 2022. Enhanced tem-
be controlled by the hyper-parameter w, making it poral knowledge embeddings with contextualized
impossible to achieve adaptive fusion which can au- language representations.
tomatically learn the weight allocation of the scores
obtained from CoH for different queries. Moving Zhen Han, Yunpu Ma, Yuyi Wang, Stephan Günnemann,
and Volker Tresp. 2020. Graph hawkes neural net-
forward, how to design an adaptive fusion strat- work for forecasting on temporal knowledge graphs.
egy that optimally enhances the performance of In AKBC.
graph-based models with results of CoH without
compromising efficiency is worth exploring. Raghav Jain, Daivik Sojitra, Arkadeep Acharya, Sri-
parna Saha, Adam Jatowt, and Sandipan Dandapat.
2023. Do language models have a common sense
regarding time? revisiting temporal commonsense
References reasoning in the era of large language models. In Pro-
Mohamed Aghzal, Erion Plaku, and Ziyu Yao. 2023. ceedings of the 2023 Conference on Empirical Meth-
Can large language models be good path planners? ods in Natural Language Processing, pages 6750–
a benchmark and investigation on spatial-temporal 6774.
reasoning. arXiv preprint arXiv:2310.03249.
Albert Q Jiang, Alexandre Sablayrolles, Antoine
Elizabeth Boschee, Jennifer Lautenschlager, Sean Roux, Arthur Mensch, Blanche Savary, Chris Bam-
O’Brien, Steve Shellman, James Starz, and Michael ford, Devendra Singh Chaplot, Diego de las Casas,
Ward. 2015. Icews coded event data. Harvard Data- Emma Bou Hanna, Florian Bressand, et al. 2024.
verse, 12. Mixtral of experts. arXiv preprint arXiv:2401.04088.

Ziyang Chen, Jinzhi Liao, and Xiang Zhao. 2023. Multi- Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye,
granularity temporal question answering over knowl- Wayne Xin Zhao, and Ji-Rong Wen. 2023. Structgpt:
edge graphs. In Proceedings of the 61st Annual Meet- A general framework for large language model to
ing of the Association for Computational Linguistics reason over structured data. arXiv e-prints, pages
(Volume 1: Long Papers), pages 11378–11392. arXiv–2305.

Zifeng Ding, Heling Cai, Jingpei Wu, Yunpu Ma, W. Jin, M. Qu, X. Jin, and X. Ren. 2020. Recur-
Ruotong Liao, Bo Xiong, and Volker Tresp. 2023. rent event network: Autoregressive structure infer-
Zero-shot relational learning on temporal knowledge enceover temporal knowledge graphs. In EMNLP,
graphs with large language models. arXiv preprint pages 6669–6683.
arXiv:2311.10112.
Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and 2019. Recurrent event network: Autoregressive
Dan Alistarh. 2022. Gptq: Accurate post-training structure inference over temporal knowledge graphs.
quantization for generative pre-trained transformers. arXiv preprint arXiv:1904.05530.
arXiv preprint arXiv:2210.17323.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying
Yifu Gao, Yongquan He, Zhigang Kan, Yi Han, Linbo Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E.
Qiao, and Dongsheng Li. 2023. Learning joint struc- Gonzalez, Hao Zhang, and Ion Stoica. 2023. Effi-
tural and temporal contextualized knowledge embed- cient memory management for large language model
dings for temporal knowledge graph completion. In serving with pagedattention. In Proceedings of the
Findings of the Association for Computational Lin- ACM SIGOPS 29th Symposium on Operating Systems
guistics: ACL 2023, pages 417–430. Principles.

Alberto García-Durán, Sebastijan Dumančić, and Math- Dong-Ho Lee, Kian Ahrabian, Woojeong Jin, Fred
ias Niepert. 2018. Learning sequence encoders Morstatter, and Jay Pujara. 2023. Temporal knowl-
for temporal knowledge graph completion. arXiv edge graph forecasting without knowledge using in-
preprint arXiv:1809.03202. context learning. arXiv preprint arXiv:2305.10613.

9
Yujia Li, Shiliang Sun, and Jing Zhao. 2022. Tirgn: Yuqing Wang and Yun Zhao. 2023. Tram: Benchmark-
Time-guided recurrent graph network with local- ing temporal reasoning for large language models.
global historical patterns for temporal knowledge arXiv preprint arXiv:2310.00835.
graph reasoning. In Proceedings of the Thirty-First
International Joint Conference on Artificial Intelli- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
gence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, Chaumond, Clement Delangue, Anthony Moi, Pier-
pages 2152–2158. ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz,
et al. 2019. Huggingface’s transformers: State-of-
Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng the-art natural language processing. arXiv preprint
Guo, Yuanzhuo Wang, and Xueqi Cheng. 2021a. arXiv:1910.03771.
Search from history and reason for future: Two-stage
reasoning on temporal knowledge graphs. In Pro- Yuwei Xia, Mengqi Zhang, Qiang Liu, Shu Wu, and
ceedings of the 59th Annual Meeting of the Asso- Xiao-Yu Zhang. 2022. Metatkg: Learning evolution-
ciation for Computational Linguistics and the 11th ary meta-knowledge for temporal knowledge graph
International Joint Conference on Natural Language reasoning. In EMNLP, pages 7230–7240.
Processing (Volume 1: Long Papers), pages 4732–
4743. Sheng Xiang, Dawei Cheng, Chencheng Shang, Ying
Zhang, and Yuqi Liang. 2022. Temporal and het-
Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng erogeneous graph neural network for financial time
Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi series prediction. In Proceedings of the 31st ACM In-
Cheng. 2021b. Temporal knowledge graph reason- ternational Conference on Information & Knowledge
ing based on evolutional representation learning. In Management, pages 3584–3593.
SIGIR, pages 408–417.
Wenjie Xu, Ben Liu, Miao Peng, Xu Jia, and Min Peng.
Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenx- 2023a. Pre-trained language model with prompts
uan Tu, Siwei Wang, Sihang Zhou, and Xinwang Liu. for temporal knowledge graph completion. arXiv
2023. Learn from relational correlations and periodic preprint arXiv:2305.07912.
events for temporal knowledge graph reasoning. In
Proceedings of the 46th International ACM SIGIR Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. 2023b. Tem-
Conference on Research and Development in Infor- poral knowledge graph reasoning with historical con-
mation Retrieval, pages 1559–1568. trastive learning. In AAAI.

Yushan Liu, Yunpu Ma, Marcel Hildebrandt, Mitchell Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia
Joblin, and Volker Tresp. 2022. Tlogic: Tempo- Ananiadou. 2023. Back to the future: Towards ex-
ral logical rules for explainable link forecasting on plainable temporal reasoning with large language
temporal knowledge graphs. In Proceedings of the models. arXiv preprint arXiv:2310.01074.
AAAI conference on artificial intelligence, volume 36,
pages 4120–4127. Mengqi Zhang, Yuwei Xia, Qiang Liu, Shu Wu, and
Liang Wang. 2023. Learning latent relations for tem-
Adam Paszke, Sam Gross, Francisco Massa, Adam poral knowledge graph reasoning. In Proceedings
Lerer, James Bradbury, Gregory Chanan, Trevor of the 61st Annual Meeting of the Association for
Killeen, Zeming Lin, Natalia Gimelshein, Luca Computational Linguistics (Volume 1: Long Papers),
Antiga, et al. 2019. Pytorch: An imperative style, pages 12617–12631.
high-performance deep learning library. Advances in
neural information processing systems, 32. Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan
Cheng, and Yan Zhang. 2021. Learning from his-
Xiaoming Shi, Siqiao Xue, Kangrui Wang, Fan Zhou, tory: Modeling temporal knowledge graphs with se-
James Y Zhang, Jun Zhou, Chenhao Tan, and quential copy-generation networks. In AAAI, pages
Hongyuan Mei. 2023. Language models can im- 4732–4740.
prove event prediction by few-shot abductive reason-
ing. arXiv preprint arXiv:2305.16646. A Prompt Examples for CoH Reasoning
Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Taking the query (Government_(Nigeria),
Kun He. 2021. TimeTraveler: Reinforcement learn- Make_an_appeal_or_request, ?, 340) as
ing for temporal knowledge graph forecasting. In
EMNLP, pages 8306–8319. an example, whose ground-truth is “Mem-
ber_of_the_Judiciary_(Nigeria)”. The prompts of
Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. CoH reasoning are shown as follows.
Towards benchmarking and improving the temporal
reasoning capability of large language models. arXiv
preprint arXiv:2306.08952.
Rakshit Trivedi, Hanjun Dai, Yichen Wang, and
Le Song. 2017. Know-evolve: Deep temporal reason-
ing for dynamic knowledge graphs. In ICML, pages
3462–3471.

10
 Prompt for Step 1

There is a given text consisting of multiple historical events in the form of “{id}:[{subject}
{relation} {object} {time}];”. And there is a query in the form of: “{subject} {relation} {whom}
time}?” If you must infer several {object} that you think may be the answer to the given query
based on the given historical events, what important historical events do you base your predictions
on? Please list the top 30 most important histories and output their {id}.

Here are the given historical events:

0:[Government_(Nigeria) Engage_in_diplomatic_cooperation with Indepen-
dent_Corrupt_Practices_Commission on the 339th day];
1:[Government_(Nigeria) Threaten Education_(Nigeria) on the 338th day];
2:[Government_(Nigeria) Criticize_or_denounce Boko_Haram on the 337th day];
3:[Government_(Nigeria) Threaten Education_(Nigeria) on the 337th day];
4:[Government_(Nigeria) Provide_aid for Citizen_(Nigeria) on the 337th day];
5:[Government_(Nigeria) Make_optimistic_comment on Citizen_(Nigeria) on the 336th day];
6:[Government_(Nigeria) Use_conventional_military_force to Boko_Haram on the 335th day];
7:[Government_(Nigeria) Make_statement to Barack_Obama on the 333rd day];
8:[Government_(Nigeria) Accuse Boko_Haram on the 332nd day];
9:[Government_(Nigeria) Express_intent_to_engage_in_diplomatic_cooperation to Citi-
zen_(Nigeria) on the 332nd day];
10:[Government_(Nigeria) Engage_in_diplomatic_cooperation with Citizen_(Nigeria) on the
331st day];
··· ···

Here is the query:

Government_(Nigeria), Make_an_appeal_or_request to, whom, on the 340th day?

Please only output {id} of the historical events that your inferred answers are based on.
Note that you must only output no more than 30 {id} without any explanation. Please strictly
follow the above demands for output.

Answer:

0, 1, 2, 5, 7, 8, 9, 10, 11, 14, 17, 21, 24, 25, 26, 27, 28, 30, 31, 32, 33, 34, 36, 37, 38,
39, 45, 49, 50, 56

11
 Prompt for Step 2

You must be able to correctly predict the {whom} of the given query from a given text consisting
of multiple historical events in the form of “{subject} {relation} {object} {time}” and the query in
the form of “{subject} {relation} {whom} {time}?” You must output several {object} that you
think may be the answer to the given query based on the given historical events. Please list all
possible {object} which may be answers to the query. Please assign each answer a serial number
to represent its probability of being the correct answer. Note that answers with a high probability
of being correct should be listed first.

Here are the given historical events:

Government_(Nigeria), Engage_in_diplomatic_cooperation with, Inde-
pendent_Corrupt_Practices_Commission, on the 339th day; Indepen-
dent_Corrupt_Practices_Commission, Arrest_or_detain_or_charge_with_legal_action to,
Citizen_(Nigeria), on the 308th day;
Government_(Nigeria), Criticize_or_denounce, Boko_Haram, on the 337th day; Boko_Haram,
Use_conventional_military_force to, Citizen_(Nigeria), on the 336th day;
Government_(Nigeria), Threaten, Education_(Nigeria), on the 337th day; Education_(Nigeria),
Make_statement to, Muslim_(Nigeria), on the 332nd day;
Government_(Nigeria), Make_optimistic_comment on, Citizen_(Nigeria), on the 336th day;
Citizen_(Nigeria), Make_an_appeal_or_request to, Member_of_the_Judiciary_(Nigeria), on the
331st day;
··· ···

Here is the query:

Government_(Nigeria), Make_an_appeal_or_request to, whom, on the 340th day?

lease list all possible {object} which may be answers (one per line) without explanations.
Note that answers with high probability should be listed first.
For example:
"""
Possible answers:
1. XXX
2. XXX
3. XXX
··· ···
"""
Please strictly follow the above demands for output.

Answer:

1. Citizen_(Nigeria)
2. Education_(Nigeria)
3. Member_of_the_Judiciary_(Nigeria) 4. Barack_Obama
5. Xi_Jinping
6. Boko_Haram
7. Head_of_Government_(Nigeria)
8. Court_Judge_(Nigeria)

12
 Datasets ICEWS14 ICEWS05-15 ICEWS18 ing functionalities from the HuggingFace’s Trans-
#E 6,869 10,094 23,033 formers library (Wolf et al., 2019) and the effi-
#R 230 251 256 cient LLM inference framework vLLM (Kwon
# Train 74,845 368,868 373,018 et al., 2023). All experiments were executed on
# Valid 8,514 46,302 45,995 an NVIDIA A100 GPU with AMD EPYC 7763
# Test 7,371 46,159 49,545 CPU processor. Our experimental setup primar-
Time gap 24 hours 24 hours 24 hours ily utilized the TheBloke/Mixtral-8x7B-Instruct-
v0.1-GPTQ model with a parameter size of 6.07B.
Table 7: The statistics of the datasets.
This model represents a quantized variant of the
Mixtral-8x7B MoE model, employing the GPTQ
B Datasets technique (Frantar et al., 2022) to achieve a bal-
anced compromise between computational speed
In this paper, we utilize three representative TKG and model performance. Regarding the generation
datasets for experimental analysis: ICEWS14 hyper-parameters, we set the Maximum tokens as
(García-Durán et al., 2018), ICEWS18 (Jin et al., 8000, Top-p sampling as 1, and Temperature as 0.
2019), and ICEWS05-15 (García-Durán et al.,
2018), all sourced from the Integrated Crisis Early Moreover, the three graph-based TKG prediction
Warning System (Boschee et al., 2015), document- models are all implemented with the most optimal
ing events in 2014, 2018, and from 2005 to 2015, hyper-parameters reported in their corresponding
respectively. Detailed statistics of the three datasets papers. Especially, for evaluating CoH on reversed
are shown in Table 7. testing sets like graph-based models, we manually
process relations in datasets into reversed ones. For
C Backbones example, we reverse “Express intent to cooperate”
into “Receive intent to cooperate”, and “threaten”
Since CoH is plug-and-play, we plug it into several into “be threatened”. For the score fusion module,
following state-of-the-art TKG reasoning models to the hyper-parameter α is set to 0.3 for all datasets.
evaluate the effectiveness of our proposed model. And the weights w of ICEWS14s, ICWES18, and
• RE-NET (Jin et al., 2020) deals with TKGs as ICEWS05-15 are set to 0.35, 0.45, and 0.35, re-
KG sequences. RE-NET utilizes the RGCN to spectively.
capture the structural dependencies of entities
and relations within each KG. Then RNN is
adopted to associate KGs with different time
stamps for capturing the temporal dependen- E Prompt Examples for Data Leakage
cies of entities and relations. Analysis
• RE-GCN (Li et al., 2021b) proposes a re-
current evolution module based on relational
GNNs to obtain embeddings that contain dy- To avoid the possible data leakage issue, we directly
namic information for entities and relations. ask the LLM Mixtral-8x7B regarding the facts con-
In particular, RE-GCN designs a static mod- tained in our used datasets, to check whether it
ule that utilizes the static properties of entities knows the knowledge. And we remove all queries
to enrich the embeddings for prediction. whose answer is “Yes” from the testing set for all
• TiRGN (Li et al., 2022) utilizes a recurrent three datasets. A few prompt examples are shown
graph encoder to capture local temporal de- as follows.
pendencies, and designs a history encoder net-
work to capture global temporal dependencies
by collecting repeated facts in history. TiRGN Prompt for Example 1
fuses the obtained local and global temporal
dependencies for final prediction. Do you know the fact that United Arab Emi-
rates reduced or broke diplomatic relations
D Implementation Details with Qatar on 2014-12-04?

In this study, we developed CoH using the Py- Answer No.

Torch framework (Paszke et al., 2019), integrat-

13
 Prompt for Example 2

Do you know the fact that Police (Egypt)

used tactics of violent repression against
Protesters (Egypt) on 2014-12-02?

Answer Yes.

Prompt for Example 3

Do you know the fact that Abdullah Ab-

dullah met at a ‘third’ location with Jens
Stoltenberg on 2014-12-02?

Answer: No.

F Related works
Previous classic methods include GHNN (Han
et al., 2020) and Know-Evolve (Trivedi et al.,
2017), which model the temporal information
within TKGs by temporal point process (TTP).
And CyGNet (Zhu et al., 2021) proposes a copy-
generation mechanism to explore patterns among
repetitive histories. Recently, most supervised mod-
els (Jin et al., 2020; Li et al., 2021b, 2022; Zhang
et al., 2023; Liang et al., 2023; Zhang et al., 2023)
for TKG prediction adopt GNNs to capture the
structural information within TKGs. Based on
these, TANGO (Han et al., 2021b) employs Neural
Ordinary Differential Equations to build up con-
tinuous temporal information, CENET (Xu et al.,
2023b) adopts contrastive learning to identify im-
portant non-historical entities, MetaTKG (Xia et al.,
2022) explores the evolution patterns of events with
meta-learning, and xERTE (Han et al., 2021a) pro-
poses an explainable model by searching sub-graph
in TKGs. Besides, some works (Sun et al., 2021;
Li et al., 2021a) search significant paths with rein-
forcement learning, and Tlogic (Liu et al., 2022)
extracts paths via temporal logic rules for TKG
prediction.

14