KoLA: Carefully Benchmarking World Knowledge

of Large Language Models

Jifan Yu∗, Xiaozhi Wang∗, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv,
Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, Chunyang Li,
Zheyuan Zhang, Yushi Bai, Yantao Liu, Amy Xin, Nianyi Lin, Kaifeng Yun,
Linlu Gong, Jianhui Chen, Zhili Wu, Yunjia Qi, Weikai Li, Yong Guan,
arXiv:2306.09296v2 [[Link]] 6 Jul 2023

Kaisheng Zeng, Ji Qi, Hailong Jin, Jinxin Liu, Yu Gu, Yuan Yao, Ning Ding,
Lei Hou, Zhiyuan Liu, Bin Xu, Jie Tang, Juanzi Li†
Tsinghua University
Beijing, China 100084
kola-benchmark@[Link]

Abstract
The unprecedented performance of large language models (LLMs) necessitates
improvements in evaluations. Rather than merely exploring the breadth of LLM
abilities, we believe meticulous and thoughtful designs are essential to thorough,
unbiased, and applicable evaluations. Given the importance of world knowledge to
LLMs, we construct a Knowledge-oriented LLM Assessment benchmark (KoLA),
in which we carefully design three crucial factors: (1) For ability modeling, we
mimic human cognition to form a four-level taxonomy of knowledge-related abil-
ities, covering 19 tasks. (2) For data, to ensure fair comparisons, we use both
Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously col-
lected emerging corpora, aiming to evaluate the capacity to handle unseen data and
evolving knowledge. (3) For evaluation criteria, we adopt a contrastive system,
including overall standard scores for better numerical comparability across tasks
and models and a unique self-contrast metric for automatically evaluating knowl-
edge hallucination. We evaluate 21 open-source and commercial LLMs and obtain
some intriguing findings. The KoLA dataset and open-participation leaderboard are
publicly released at [Link] and will be continuously updated
to provide references for developing LLMs and knowledge-related systems.

1 Introduction
Recent remarkable breakthroughs achieved by large language models (LLMs) like GPT-4 [1] have
elicited widespread astonishment. Considering the extensive and profound natural language un-
derstanding and generation abilities exhibited by LLMs [2], the conventional benchmarks [3, 4]
focusing on relatively narrow and superficial abilities are no longer as helpful for testing them. It has
become necessary to construct better benchmarks for effectively comparing LLMs and providing
valuable diagnostic results. To this end, various benchmarks are proposed, focusing on extending the
evaluation scope to cover broader abilities [5, 6, 7] or more challenging tasks [8, 9].
In addition to broadening the evaluation scope to explore the breadth of LLM abilities, we believe
meticulous designs are also necessary to build evaluations that facilitate in-depth insights, maintain
impartiality towards different LLMs, and have high applicability for audiences interested in selecting
* Equal Contribution.
†
Corresponding author.

37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.
 Data: Ability: Evaluation:
Known and Evolving Cognitive Taxonomy Contrastive Metrics
SEASON 1 :
st

RANKING
STANDARDIZED
KC Overall
KM KU KA

Hig
KC

h-le
CREATING 1st
62.3 54.3 49.7
GPT4 53.8

vel
Contrastive Generation ...
Considering both familiar ...

Kno
...
8th
and unfamiliar Knowledge KA 25.0 25.1 18.7

wle
ChatGLM 36.1

dge
of LLMs APPLYING 10th
21.7 20.6 37.5
Multihop Knowledge Reasoning LLaMa 14.0

Abil
ws E 12th
NoveNl e
Wikipe ICIPAT
PART 18.1 17.6 28.1

ity
dia LL FOR Alpaca 13.4
KU CA
N2
nd

Evolving Known UNDERSTANDING SEASO model or

Entity, Relation, Concept, Event… ng the
Providi your APIs …
e
Prepar
KM
MEMORIZATION
Frequent and Long-tail World Knowledge…

Figure 1: KoLA’s careful design on three key factors for LLM evaluation.

and enhancing LLMs. Designing a benchmark requires careful consideration of three key factors:
(1) Ability Modeling. A benchmark should not only define the scope of desired abilities but also
model the inherent connections between the evaluated abilities, which allows for diagnostic insights
on how to acquire and improve these abilities. (2) Data. Given the extremely broad range of
training data for LLMs, which might include annotated data for certain tasks and is sometimes
undisclosed, ensuring that differences in training data do not impact the evaluation fairness is critical
and challenging. (3) Evaluation Criteria. For high applicability, evaluation metrics should enable
audiences to easily understand and gain helpful observations. Moreover, there are many well-known
issues [10, 11, 12, 13] awaiting resolutions for evaluating tasks with large search spaces like the
generative tasks. Evaluations for related abilities still heavily rely on human evaluation, which is
time-consuming and not easily reproducible [14, 15].
In this paper, we propose a Knowledge-oriented LLM Assessment benchmark (KoLA), which
aims at carefully benchmarking the world knowledge of LLMs by undertaking meticulous designs
considering the aforementioned three factors:
For ability modeling, we evaluate world knowledge of LLMs and design a four-level cognitive ability
taxonomy. We chose world knowledge as our evaluation scope because: (i) World Knowledge is
widely recognized as playing a fundamental role in the impressive performance of LLMs [5, 6, 7],
and a deeper grasp of knowledge enables LLMs to better assist humans; (ii) Recent work has shown
that understanding and generating structural world knowledge remain challenging for LLMs. Unlike
previous work focusing on expanding the evaluation breadth by covering diver tasks and disciplinary
knowledge to test the knowledge boundaries of LLMs [5, 6, 7], we focus more on the “depth” of
evaluation, i.e., modeling the intrinsic connections between knowledge-related abilities and ensuring
reliable evaluation results. Inspired by the human cognitive processes in learning theory, such as
Bloom’s taxonomy [16], we organize evaluated abilities into four levels: Knowledge Memorization,
Knowledge Understanding, Knowledge Applying, and Knowledge Creating. This taxonomy helps
to provide more specific and helpful evaluation results, detailing which aspect of knowledge the
evaluated models may be deficient in. It also facilitates a preliminary exploration of the similarities
and differences between the learning mechanisms of LLMs and humans. To coordinate with our data
design considerations introduced later, we selected 19 tasks, primarily focusing on world knowledge
about entities, concepts, and events.
For data, we obtain both known and evolving data sources. Some studies adopt unpublished or
machine-unreadable data [6, 7] to reduce the possibility that the test data has been learned by LLMs.
However, considering the intense competition between LLMs and their hunger for high-quality data,
those data may also be trained by LLMs in the near future. We believe the ideal approach is to do
evaluations on newly emerging data and maintain a continuously evolving benchmark. In KoLA,
we host a new competition season every three months. For each season, we crawl and annotate 500
recently published articles as the evolving data. The evolving data source allows us to (i) evaluate
models more fairly, even if some models can rapidly update their knowledge, thereby demonstrating
their power, and (ii) better track the model development. Besides evolving data, we also consider the
known data of LLMs, which means the data sources that all models have learned. Evaluations on
known data enable us to (i) fairly compare the learning efficiency of LLMs by comparing the different
knowledge they acquire from the same training data and (ii) assess the generalization ability by

2
comparing LLMs’ performance on known data and evolving data. We chose Wikipedia as our known
data source due to its common use. Considering the limitations of Wikipedia and our annotation
capabilities on the evolving data, we are unable to cover a very wide range of tasks.
For evaluation criteria, we design a contrastive evaluation system, including an overall standard
score system and a self-contrast knowledge creating metric. Conventional benchmarks report absolute
metrics for different tasks separately. The incomparability of scores across tasks makes it difficult for
audiences to intuitively compare the proficiency levels across different abilities. Additionally, the
sensitivity of different metrics varies, which may lead less experienced audiences to misinterpret the
ability differences represented by numerical differences. In the KoLA main leaderboard, we report
standard scores across different tasks, determined by the relative level compared to other evaluated
LLMs. This makes KoLA applicable to a broader range of audiences. Experienced audiences
can still refer to absolute metrics if desired. Furthermore, evaluating the knowledge creating is
particularly challenging as it involves distinguishing the correctly created knowledge and knowledge
hallucinations [13]. We design a self-contrast metric for evaluating knowledge hallucination by
contrasting the freely created completion and the knowledge-grounded completion of an LLM given
the same beginning text. This metric eliminates the influence of the stylistic difference between the
LLM and the human authors and focuses on whether the generated completions are consistent with
the actually presented knowledge.
In the first season of KoLA, we evaluate 21 widely-used LLMs, including 8 API-access commer-
cial LLMs, such as GPT-4 [1] and Cohere-command, and 13 open-source LLMs including GLM-
130B [17], LLaMa [18], etc. From the experimental results, we obtain some intriguing observations,
such as larger base models tend to memorize more knowledge, alignment unleashes the potential of
larger models in higher-level abilities but may harm memorization, and open-source models exhibit
overall inferiority compared to commercial models.
We welcome the participation of more LLMs in KoLA evaluation and encourage contributions to
the new seasons of KoLA. The data, leaderboard, participation information, and supporting tools
are publicly available at [Link] We hope KoLA can serve as a diagnostic tool
to facilitate the development of increasingly knowledgeable LLMs, and also help developers of
knowledge-related applications select appropriate LLMs.

2 KoLA Benchmark

2.1 Ability Modeling

Within the context of Artificial Intelligence (AI), Knowledge has long been employed to signify
information encompassing facts, events, and skills [19], serving as an indicator for the intelligence
level of AI. Hence various knowledge-intensive tasks [20, 21] are proposed to examine language
models’ knowledge-related abilities. Recently, the impressive performance of LLMs has encouraged
the development of more comprehensive benchmarks [8, 9] with broad human-subject exams [5, 6, 7].
Cognitive Ability Taxonomy. Confronted with such a vast array of evaluation datasets, we advocate
for considering the stratification and connection of abilities, rather than organizing these evaluations
discretely [4, 5, 8, 9] or straightforwardly based on disciplines [6] or difficulties [7]. Such viewpoints
have also been upheld by cognitive scientists for several decades, giving rise to a series of classic
cognitive learning theories [22]. Considering the ongoing debates surrounding high-order thinking [23,
24], we simplify and select four widely accepted cognitive processes in Bloom’s learning theory [16]
for organizing the tasks in KoLA benchmark.

(1) Knowledge Memorization (KM) aims to gauge the model’s ability in faithfully recalling
known facts, exemplified by the previous knowledge probing task [20].
(2) Knowledge Understanding (KU) focuses on evaluating the model’s ability in understand-
ing the underlying knowledge within texts, instantiated by the conventional information
extraction tasks [25, 26, 27, 28].
(3) Knowledge Applying (KA) reflects the ability of agents in employing knowledge to accom-
plish reasoning and problem-solving tasks. Consequently, this level is evaluated by various
knowledge reasoning tasks [29, 30, 31].

3
 (4) Knowledge Creating (KC) denotes the model’s ability to create novel and reasonable
knowledge given known facts. This is evaluated by the knowledge coherence and correct-
ness [32, 33] of contents generated by the model. It is worth noting that the evaluation goes
beyond merely assessing the generation quality (fluency, etc.).

2.2 Data Source and Selected Tasks

Known & Evolving Data: A common concern in evaluating LLMs is the fairness issue brought
by variations in training data and the potential test data leakage risk. To minimize these biases, we
propose the design of the following distinctive data sources:
(1) Known Data Source. Wikipedia1 is an acknowledged high-quality knowledge-rich corpus contain-
ing over 6.6 million English articles, which has been used in pre-training by numerous pre-trained
language models since BERT [34, 35, 36] and is widely included in open pre-training corpora [37].
Hence we believe assuming every LLM has been trained on Wikipedia is reasonable and adopt it
as our known data source. Considering that many LLMs state they can only provide answers based
on “Content before 2021” 2 , we select Wikidata5M [38], a high-quality subset of Wikidata, as the
basis, which allows linking to the 2019 version of Wikipedia dump, thus enabling the selection or
reconstruction of downstream tasks’ datasets.
(2) Evolving Data Source. Considering the time required for model training [17], it is less unlikely
for newly emerged data to be timely trained by LLMs. Therefore, we have devised an evolving
evaluation mechanism that continuously retrieves the web content published in recent 90 days as the
data source and constructs new datasets on them. This approach ensures fair assessment of LLMs’
performance on unseen content and whether they “secretly” involve knowledge updating modules
like the external search. Each update (we call it a Season of KoLA) requires crawling a minimum of
500 articles to support building test sets. For the first season reported in this paper, we adopt two
kinds of data: factual news 3 and fictional novels 4 .
Built upon these two data sources, we finally select and construct 19 tasks in KoLA, as shown in
Table 1. To ensure both the quality and efficiency of annotations for each season, we randomly select
one task at each level to annotate the new evolving evaluation dataset. For the existing datasets, we try
to ensure most of the test sets are not public, and this rigorous setting ensures a high level of fairness.
The data collection and task construction details are shown in Appendix C. We briefly introduce the
tasks of the four levels below.
Knowledge Memorization Tasks: We follow LAMA [20] to evaluate knowledge memorization
by probing facts from LLMs but re-construct the datasets on our data sources. Given a triplet in
Wikidata5M [38], we transform it into a sentence with a relation-specific template and let LLMs
complete its tail entity. Additionally, we want to explore whether the knowledge memorization of
LLMs correlates with training frequency. We sort the entities in Wikidata5M according to their
frequency of occurrence in Wikipedia [39], resulting in the creation of two test sets: (1-1) High-
Frequency Knowledge. Randomly selecting 100 entities from the top 2, 000 entities with the highest
frequency and construct data with triplets of them; (1-2) Low-Frequency Knowledge. Similarly, we
randomly select 100 entities from the lowest-frequency entities and construct a more challenging
evaluation set; (1-3) Evolving Test of Memorization (ETM). From the articles in evolving data sources,
we annotate the knowledge triplets shown in them and only preserve 100 triplets that cannot be
inferred from previously available corpora.
Knowledge Understanding Tasks: Knowledge understanding is evaluated by whether LLMs can
understand various genres of knowledge from texts, including concepts, entities, entity relations,
events, and event relations. (2-1/2-2/2-3) Concept Probing employs the three probing tasks (CSJ,
CPJ, CiC) of COPEN [28] to evaluate the models’ understanding of conceptual knowledge. (2-4)
Named Entity Recognition utilizes the FewNERD dataset [27], from which we randomly select
300 examples in our evaluation. (2-5) Relation Extraction selects the undisclosed test set from
the challenging document-level relation extraction dataset, DocRED [25]. (2-6) Event Detection
adopts the undisclosed test set of the finely annotated MAVEN [26] dataset. (2-7) Event Relation
1
[Link]
2
[Link]
3
An open source news API at Github. URL: [Link]
4
A well-known open license novel creating community. URL: [Link]

4
Extraction involves the undisclosed test set from MAVEN-ERE [40], which consists of 113k examples
of coreference, temporal, causal, and subevent relations between events. (2-8) Evolving Test of
Understanding (ETU). For the articles in evolving data, we conduct the entity recognition and follow
the same relation schema of DocRED to annotate a brand new test set containing 100 relation
instances from 50 articles. It is worth noting that apart from the evolving test, the other datasets are
all based on Wikipedia texts.
Knowledge Applying Tasks: Knowledge applying ability is evaluated by LLMs’ multi-hop reasoning
capabilities, specifically over world knowledge. This differs from several recent studies [41, 42],
which cover more general reasoning, such as mathematical reasoning. Therefore, the following
progressive Wikipedia-based datasets are included in KoLA: (3-1) HotpotQA [29] is a question-
answering dataset that involves a substantial number of natural language questions written by native
speakers, examining machine’s abilities in comparison, multi-hop reasoning, and more. However,
a limitation of HotpotQA is that some questions can be answered through shortcuts. To address
this, (3-2) 2WikiMultihopQA [43] ensures that questions cannot be solved through shortcuts by
manually-designed templates, but their questions lack naturalness in language. Furthermore, the (3-3)
MuSiQue [30] dataset tackles the challenges of shortcuts and naturalness simultaneously. Its questions
are composed of simple questions from existing datasets, with up to four-hop complex reasoning.
(3-4) KQA Pro [31] is a large-scale dataset, whose questions are relatively complex, allowing for
more fine-grained evaluation of LLMs’ multi-hop reasoning with logical operations and modifiers.
(3-5) KoRC [44] is a dataset that requires joint reasoning between the text and knowledge base. It
differs from the aforementioned four datasets as it requires implicit rather than explicit reasoning.
(3-6) Evolving Test of Applying (ETA) takes the same construction approach as KoRC, producing 49
questions upon 350 annotated knowledge triplets and 40 articles in the evolving data.
Knowledge Creating Tasks: As the highest level of Bloom’s Cognitive Taxonomy [16], how to
evaluate knowledge creation ability is an open and challenging question. We present a feasible
proposal based on knowledge-grounded text generation tasks [32, 45]. In the generation of narrative
texts such as history, news, and fiction, the core of creativity lies in describing subsequent events.
Consequently, we seek to evaluate the model’s creation ability by assessing their event knowledge
hallucination in generated texts. To establish a standard reference, we build an annotation platform
and conduct fine-grained event annotations on both Wikipedia texts and articles in the evolving data,
which constructs two evaluation datasets: (4-1) Encyclopedic Knowledge Creation, which is based on
Wikipedia articles and (4-2) Open Knowledge Creation, which is based on unseen news and novels,
serving as the Evolving Test of Creating (ETC).
Table 1 presents the features and statistics of each selected task. Further details regarding annotation
processes and task demonstrations are correspondingly presented in Appendix D.1.

2.3 Contrastive Evaluation System

Our contrastive evaluation system includes standardized overall scores based on relative model com-
parisons and a unique self-contrast metric, which can automatically evaluate knowledge hallucination
and enhance generation evaluation.
Standardized Overall Scoring. Since the metrics of different KoLA tasks are incomparable and
differently sensitive, less experienced audiences cannot easily compare and interpret results, which
is also prevalent in recent LLM benchmarks like Big-Bench-Hard [9] and MMLU [5]. Therefore,
we propose to introduce standardized scores [46] to enhance the applicability of KoLA results.
|D| |M |
Specifically, given a task set D = {di }i=1 and the evaluated model set M = {mj }j=1 , we first
select the most representative metric for each task, allowing us to compute the performance score xij
of model mj on task di . Then the standardized score z can be calculated as:


xij − µ xi1 , ..., xi|M |
zij =
, (1)
σ xi1 , ..., xi|M |

where µ (·) and σ (·) denote the mean and standard deviation. Subsequently, we apply Min-Max
scaling [47] to adjust all the results to the range of [0, 100], further enhancing the correlation and
readability of scores across tasks. The final scores are presented as:
zij − min (z)
sij = 100 , (2)
max (z) − min (z)

5
Table 1: The tasks in KoLA (Season 1st). Metrics in bold are selected for calculating standardized
scores. Exclusive task means their test sets are newly developed or sponsored by the original authors
and were not publicly disclosed. |Test Set| and |Pool| correspond to the testing instances used in this
season and the overall available instances. The evolving pool will be continuously updated.
Level ID Dataset Metrics Exclusive Context Type |Test Set| |Pool| Source
1-1 High-Freq. EM, F1 " Triple 100 20.6M
Known
KM 1-2 Low-Freq. EM, F1 " Triple 100 20.6M
1-3 RTM EM, F1 " Triple 100 2.7k Evolving
2-1 COPEN-CSJ Acc. " Entity, Concept 100 3.9k
2-2 COPEN-CPJ Acc. " Concept 100 4.7k
2-3 COPEN-CiC Acc. " Concept 100 2.3k
Known
KU 2-4 FewNERD F1 % Sentence 300 188.2k
2-5 DocRED F1 " Document, Entity 100 12k
2-6 MAVEN F1 " Document 100 20.4k
2-7 MAVEN-ERE F1 " Document(s), Event 199 1.3M
2-8 ETU F1 " Document, Entity 100 1.6k Evolving
3-1 HotpotQA F1 % Document(s) 100 7.4k
3-2 2WikiMulti. F1 " Document(s) 100 12.6k
3-3 MuSiQue F1 " Document(s) 100 2.5k Known
KA
3-4 KQA Pro F1 " KG 100 1.2k
3-5 KoRC F1 " Document(s), KG 100 5.2k
3-6 ETA F1 " Document(s), KG 49 1.6k Evolving
4-1 Encyclopedic BLEU, Rouge " Document, Event 95 4.5k Known
KC
4-2 ETC BLEU, Rouge " Document, Event 95 100 Evolving

where the functions max (z) and min (z) correspond to the maximum and minimum of all zij scores.
Self-contrast Metric. Evaluating knowledge creating is not only about evaluating generation
quality [10], but more about assessing whether the generated knowledge is faithful and reasonable,
i.e., knowledge hallucination [13]. We develop a unique self-contrast metric, which is defined by
contrasting two completions generated by the same model.
Let C denote the preceding context, R denote the human-written succeeding text, and K refer to the
knowledge in R. Each model is required to generate two completions: (a) Given context C, freely
generate a version of completion T ; (b) Given both the context C and the foreknowledge K, generate
another completion Tk . If T and Tk demonstrate a strong resemblance, it implies that the model
has the ability to spontaneously generate highly reasonable knowledge content (even resembling
foresight). It is worth noting that this approach does not rely on human evaluation [48, 49] and does
not prioritize the model’s ability to replicate a specific writing style. Meanwhile, to prevent evaluation
collapse caused by the model’s disregard for the knowledge K in the prompt of the process (b), we
define the final knowledge creating score as:
x = avg (∂ (T, Tk ) , ∂ (T, R) , ∂ (Tk , R)) , (3)
where avg (·) denotes the average. Function ∂ (·) is to calculate the similarity of the two texts, which
we employ the widely-used Rouge-L (F1) [50] in this work.

2.4 Availability

Platform. We develop an online platform at [Link] to offer a range of services

to the community, such as competition news updates, visualizations of evaluation results, and
convenient access to submit new models or modify previous submissions. Due to the dynamic nature
of the KoLA evolving data source, new results and rankings will be generated in each season, and a
selection of results from previous seasons will also be publicly available.
Participate in KoLA. Researchers can participate in KoLA evaluations in two roles. (1) Competitor:
KoLA welcomes open participation for each season by providing the model’s APIs or parameters,
and provides 5 high-quality examples for each task to help participants debug. It is worth noting that
KoLA does not allow the local evaluation to prevent test set leakage and unfairly overfitting datasets.
(2) Contributor: We maintain a special interest group (invitation link shown at the “About” page of

6
the platform) where volunteers who have ideas for result analysis, model refinement, and benchmark
improvement can discuss, propose suggestions, and participate in task construction.
Supporting Tools. We release a toolkit to support KoLA-related functions at [Link]
com/THU-KEG/KoLA, including: (1) Easy-to-submit. Competitors can employ this function to
independently maintain the in-context prompts for each task while providing a single model API,
making the submission and modification convenient. (2) Result Reproduction. We provide the code
and developed tools that used in our data visualization and standardization, which support result
reproduction and other analyses. (3) Data Acquisition. We also provide a data access API, which
assists authorized users in getting the evolving data and results of previous seasons.

3 Experiment
Evaluated Models. In the first season of KoLA, we evaluate LLMs of two categories: (1) Open-source
Model, including GPT-J (6B) [51], GPT-JT (6B) [52], GPT-NeoX (20B) [53], BLOOM (7B) [54],
T0++ (11B) [33], LLaMa (65B) [18], GLM (130B) [17], UL2 (20B) [55], FLAN-T5 (11B) [56],
FLAN-UL2 (20B) [57], Alpaca (7B) [58], ChatGLM (6B) [59]; (2) API service: GPT-3 curie v1
(6.7B)5 and davinci v1 (175B) [35], InstructGPT curie v1 (6.7B*)5 and davinci v2 (175B*) [60],
ChatGLM (130B) [17], Cohere-command (52.4B)6 , J2-Jumbo-Instruct (178B*) [61], GPT3.5-turbo5
and GPT-4 [1], where (*) represents that the size has not been officially confirmed.
Overall Performance. We report the standardized scores of all models in Table 2 and 3, where
“—” indicates that the result is unavailable due to the input is longer than the model context length.
As shown by the average standard scores (Avg), GPT-4 (2.06) and GPT-3.5-turbo (1.32) maintain
a substantial edge. Despite the overall consistency in rankings across different levels, we can still
obtain some intriguing findings from the results:
(1) For models without alignment or instruction tuning (e.g., GPT-J and BLOOM), there is a strong
correlation (Spearman’s coefficient of 0.79) between the ranking of the Knowledge Memory (KM)
level and the model size. This suggests that model size has an obvious positive impact on memorizing
seen knowledge, which corroborates some of the viewpoints expressed in previous evaluations [62].
(2) For models after alignment or instruction tuning, there is a significant increase in the correlations
between higher-level abilities and model size (exemplified by KA, whose Spearman’s coefficient
0.02->0.53). This suggests that alignment unleashes the greater potential of larger models in higher-
level capabilities. However, the correlation between model size and low-level KM performance
exhibits a decline (0.34), potentially demonstrating the widely discussed “alignment tax” [60].
(3) Compared to the commercial closed-source models like GPT4, GPT-3.5-turbo, and J2-Jumbo,
there is still a noticeable gap in the performance of open-source models. Open-source models
obtain an average z-score of −0.29, which is below the overall average. Among the top 8 models,
only FLAN-T5 and FLAN-UL2 can maintain a competitive edge in most of the tasks. The open-
source community should advocate for stronger collaboration and make efforts to support larger and
higher-quality models that are crucial for future research purposes.
Design Analysis. We further discuss several new observations brought by KoLA design factors.
First, there is a high correlation among tasks within each level, indicating that the abilities of LLM
indeed possess some inherent hierarchical structure. The knowledge memorization (KM) level shows
notable correlations with other levels, especially with the concept tasks in the understanding level
(2-1, 2-2, 2-3), as well as with the reasoning tasks (from 3-1 to 3-5) in the applying level, which
indicates that these high-level tasks rely heavily on knowledge memory. Moreover, in order to obtain
a more dissociated assessment of the LLMs’ competence in higher-order cognitive tasks, it is still
recommended to design tasks that exhibit substantial disparities from the pre-training corpus to
alleviate the potential biases stemming from data.
Second, the results of the models on evolving and non-evolving tasks show an obvious linear
correlation, indicating the reliability of our construction of evolving datasets. The performance
gap between known and evolving data is more prominent for shallower levels (KM, KU), whereas
it is less pronounced in higher-level tasks (KA, KC). The convergence of performance between
5
[Link]
6
[Link]

7
 Table 2: Standardized performance of Knowledge Memorization and Understanding level.
Level 1: KM Level 2: KU
Model
1-1 1-2 1-3 Rank 2-1 2-2 2-3 2-4 2-5 2-6 2-7 2-8 Rank
GPT-4 51.4 55.5 54.6 1st 63.5 42.9 46.0 62.3 100.0 72.3 72.8 59.5 1st
GPT-3.5-turbo 41.7 47.6 42.0 4th 37.5 43.8 44.8 49.2 47.2 44.1 50.5 25.5 2nd
InstructGPT davinci v2 (175B*) 30.8 37.2 32.4 7th 26.6 42.5 36.5 36.8 53.1 56.7 34.6 31.2 3rd
Cohere-command (52.4B) 46.6 42.6 56.8 2nd 33.1 41.2 40.6 21.4 33.5 13.2 40.9 18.6 4th
FLAN-UL2 (20B) 41.3 31.9 53.0 5th 52.7 41.2 47.8 10.7 18.6 13.2 16.3 18.6 6th
FLAN-T5 (11B) 44.1 39.9 49.6 3rd 57.0 42.1 43.6 13.4 — — — — 5th
J2-Jumbo-Instruct (178B*) 23.0 24.0 17.6 11th 20.1 15.8 24.5 32.1 26.3 25.7 45.2 22.0 7th
ChatGLM (130B) 27.8 44.5 36.1 6th 23.3 42.1 46.6 10.7 18.6 15.9 24.4 18.6 8th
InstructGPT curie v1 (6.7B*) 19.0 33.1 33.1 8th 22.3 34.9 35.9 17.1 19.1 14.6 19.9 18.6 9th
LLaMa (65B) 15.5 16.7 9.9 13th 14.6 10.3 10.7 50.8 25.7 23.0 19.6 18.6 11th
T0++ (11B) 31.3 28.2 25.3 9th 23.3 32.7 20.9 10.7 — — — — 13th
Alpaca (7B) 13.0 16.4 11.0 14th 14.6 10.3 11.3 20.2 20.2 24.3 16.3 18.6 16th
GLM (130B) 13.4 16.3 9.4 15th 14.6 10.3 10.7 44.7 27.6 22.6 16.3 27.6 10th
UL2 (20B) 17.6 19.0 10.9 12th 14.6 10.3 10.7 12.8 — — — — 21th
ChatGLM (6B) 22.3 23.3 19.5 10th 15.7 39.5 26.3 10.7 18.6 14.6 16.9 18.6 12th
GPT-J (6B) 12.4 10.6 8.9 18th 14.6 10.3 10.7 17.0 18.6 24.0 — 18.6 19th
GPT-3 davinci v1 (175B) 10.0 9.8 8.5 20th 14.6 10.7 10.7 25.1 18.6 22.5 16.4 18.6 15th
GPT-JT (6B) 11.5 10.7 9.2 19th 14.6 10.3 10.7 14.0 18.6 29.9 — 18.6 18th
GPT-NeoX (20B) 11.6 12.3 9.0 17th 14.6 10.3 10.7 20.6 18.6 25.1 — 18.6 17th
BLOOM (7B) 12.6 13.4 11.2 16th 14.6 10.3 10.7 25.0 21.9 22.1 16.3 18.6 14th
GPT-3 curie v1 (6.7B) 9.2 9.6 8.5 21th 14.6 10.3 10.7 16.3 18.6 18.4 18.0 18.6 20th

Table 3: Standardized performance of Knowledge Applying, Creating level and all the 4 levels.
Level 3: KA Level 4: KC Overall (1,2,3,4)
Model
3-1 3-2 3-3 3-4 3-5 3-6 Rank 4-1 4-2 Rank Avg Rank
GPT-4 56.2 58.3 72.4 26.9 56.5 55.6 1st 47.0 52.5 3rd 2.06 1st
GPT-3.5-turbo 54.7 37.2 48.5 42.5 24.7 24.3 4th 51.1 54.6 2nd 1.32 2nd
InstructGPT davinci v2 (175B*) 23.9 33.8 38.4 15.7 45.3 43.9 6th 53.6 53.3 1st 1.02 3rd
Cohere-command (52.4B) 30.1 36.1 39.5 47.0 49.9 53.8 3rd 11.4 35.4 7th 0.77 4th
FLAN-UL2 (20B) 44.9 43.0 33.3 49.3 38.1 51.5 2nd 24.1 15.2 12th 0.55 5th
FLAN-T5 (11B) 39.8 44.6 26.5 49.3 34.1 — 5th 15.0 17.0 16th 0.38 6th
J2-Jumbo-Instruct (178B*) 40.1 24.4 25.2 33.6 22.0 14.3 7th 41.5 42.6 4th 0.29 7th
ChatGLM (130B) 30.3 27.2 21.5 31.3 30.8 9.0 9th 19.7 17.8 13th 0.09 8th
InstructGPT curie v1 (6.7B*) 25.0 30.8 17.5 22.4 25.0 25.9 10th 23.5 22.6 10th -0.01 9th
LLaMa (65B) 8.2 28.8 35.7 17.9 15.0 18.1 12th 42.6 32.3 5th -0.09 10th
T0++ (11B) 14.9 14.3 17.1 4.5 34.1 — 16th 12.6 24.7 14th -0.29 11th
Alpaca (7B) 6.7 9.9 14.0 8.9 40.2 25.6 15th 32.5 23.7 6th -0.39 12th
GLM (130B) 16.1 2.5 11.4 13.4 39.6 39.6 13th 25.2 15.7 11th -0.40 13th
UL2 (20B) 17.9 20.1 19.3 33.6 9.0 — 14th 22.5 23.7 9th -0.47 14th
ChatGLM (6B) 13.5 19.6 15.3 11.2 12.6 18.3 17th 12.1 13.7 18th -0.49 15th
GPT-J (6B) 33.0 33.4 20.2 47.0 10.5 12.0 8th 26.6 1.4 17th -0.54 16th
GPT-3 davinci v1 (175B) 10.3 3.2 16.0 11.2 14.8 10.7 19th 28.6 17.6 8th -0.65 17th
GPT-JT (6B) 24.8 32.0 16.2 26.9 11.5 12.3 11th 16.1 0.0 20th -0.73 18th
GPT-NeoX (20B) 5.5 3.7 10.1 8.9 16.0 12.4 21th 28.5 4.0 15th -0.77 19th
BLOOM (7B) 10.7 13.8 10.1 11.2 21.0 16.9 18th 12.4 3.4 21th -0.80 20th
GPT-3 curie v1 (6.7B) 15.0 5.1 13.6 8.9 12.2 9.2 20th 18.9 6.5 19th -0.86 21th

Independent-Identical-Distribution and Out-of-Distribution evolving settings suggests a potential

enhancement in the model’s generalization capability and may support the opinion about the model’s
acquisition of divergent and reasoning abilities that go beyond simple data fitting [2, 6].
Third, we conduct manual annotation (Appendix F.1 for more details) on the results in the knowledge
creating tasks, where each annotator is required to read the contexts C and foreknowledge K, and
then evaluate the model’s outputs T in two aspects: overall quality and faithfulness. Ratings are
assigned on a scale of 1 (the worst rating) to 5 (the best rating). We calculate Spearman’s correlation
between the manually annotated results and the metrics introduced in § 2.3. We find that there is a
notable correlation (0.61) observed between the self-contrast metric ∂ (T, Tk ) and the faithfulness of
created content, while removing the self-contrast metric from the overall metric x in Eq. (3) brings a
significant 32% decrease in the correlation with the human-judged overall quality. We believe this
metric can contribute to future explorations on the assessment of generation abilities [10, 63, 64].

4 Conclusion and Future Work

This paper presents KoLA, a carefully designed Knowledge-oriented LLM assessment benchmark.
We design a cognitive ability taxonomy for more helpful diagnostic results, adopt both known and
evolving data sources for better fairness, and employ contrastive metrics for high applicability. In the

8
 1.00 level-1: KM level-2: KU
1-1 1 0.95 0.95 0.93 0.84 0.86 0.03 0.43 -0.17 0.47 0.39 0.65 0.72 0.77 0.52 0.68 0.37 0.03 0.66 Line of Best Fit, R² = 0.91 Line of Best Fit, R² = 0.92
1-2 0.95 1 0.94 0.9 0.88 0.9 0.07 0.47 -0.07 0.58 0.41 0.64 0.7 0.76 0.48 0.66 0.36 0.11 0.7 60 95% Confidence Interval 50 95% Confidence Interval
1-3 0.95 0.94 1 0.92 0.84 0.92 -0.04 0.4 -0.22 0.45 0.26 0.62 0.72 0.69 0.51 0.7 0.44 -0.08 0.55 0.75
2-1 0.93 0.9 0.92 1 0.94 0.94 -0.03 0.32 -0.12 0.49 0.37 0.73 0.77 0.75 0.51 0.65 0.34 0.04 0.54 40 40
2-2 0.84 0.88 0.84 0.94 1 0.92 0.04 0.34 0 0.63 0.43 0.66 0.68 0.72 0.42 0.57 0.31 0.14 0.58 0.50 30
2-3 0.86 0.9 0.92 0.94 0.92 1 -0.12 0.26 -0.12 0.51 0.29 0.69 0.7 0.68 0.49 0.65 0.37 0.06 0.5
2-4 0.03 0.07 -0.04-0.03 0.04 -0.12 1 0.83 0.69 0.49 0.67 0.02 0.04 0.26 -0.05 0.27 0.52 0.68 0.48 20 20
2-5 0.43 0.47 0.4 0.32 0.34 0.26 0.83 1 0.52 0.63 0.77 0.25 0.28 0.47 0.1 0.6 0.73 0.5 0.69 0.25
2-6 -0.17-0.07-0.22-0.12 0 -0.12 0.69 0.52 1 0.35 0.64 0.06 0.06 0.12 -0.14 0.05 0.31 0.74 0.21 0 10
2-7 0.47 0.58 0.45 0.49 0.63 0.51 0.49 0.63 0.35 1 0.53 0.44 0.39 0.64 0.2 0.31 0.39 0.38 0.7 0.00 20 40 10 20 30 40 50
2-8 0.39 0.41 0.26 0.37 0.43 0.29 0.67 0.77 0.64 0.53 1 0.43 0.24 0.4 0.1 0.47 0.5 0.63 0.57 level-3: KA level-4: KC
3-1 0.65 0.64 0.62 0.73 0.66 0.69 0.02 0.25 0.06 0.44 0.43 1 0.79 0.72 0.81 0.28 0.21 0.13 0.3 0.25 50
Line of Best Fit, R² = 0.63
50
Line of Best Fit, R² = 0.55
3-2 0.72 0.7 0.72 0.77 0.68 0.7 0.04 0.28 0.06 0.39 0.24 0.79 1 0.86 0.78 0.32 0.32 0.13 0.35 95% Confidence Interval 95% Confidence Interval
3-3 0.77 0.76 0.69 0.75 0.72 0.68 0.26 0.47 0.12 0.64 0.4 0.72 0.86 1 0.7 0.4 0.33 0.36 0.7 0.50 40 40
3-4 0.52 0.48 0.51 0.51 0.42 0.49 -0.05 0.1 -0.14 0.2 0.1 0.81 0.78 0.7 1 0.07 0.12 0.03 0.15
3-5 0.68 0.66 0.7 0.65 0.57 0.65 0.27 0.6 0.05 0.31 0.47 0.28 0.32 0.4 0.07 1 0.62 0.18 0.56 30 30
3-6 0.37 0.36 0.44 0.34 0.31 0.37 0.52 0.73 0.31 0.39 0.5 0.21 0.32 0.33 0.12 0.62 1 0.32 0.35 0.75 20 20
4-1 0.03 0.11 -0.08 0.04 0.14 0.06 0.68 0.5 0.74 0.38 0.63 0.13 0.13 0.36 0.03 0.18 0.32 1 0.51
4-2 0.66 0.7 0.55 0.54 0.58 0.5 0.48 0.69 0.21 0.7 0.57 0.3 0.35 0.7 0.15 0.56 0.35 0.51 1 1.00 10 10
1-1 1-2 1-3 2-1 2-2 2-3 2-4 2-5 2-6 2-7 2-8 3-1 3-2 3-3 3-4 3-5 3-6 4-1 4-2 10 20 30 40 50 10 20 30 40 50
(a) Inter-task performance ranking correlation (b) Evolving vs. Non-Evolving tasks
Figure 2: Analyses on the design factors of KoLA. Each cell of Figure (a) presents the Spearman’s
correlation coefficient between two tasks (name colored by levels). Figure (b) shows the scatter plots
of model performance on evolving tasks and its non-evolving counterparts (e.g., 3-5 v.s. 3-6). The
x-axis and y-axis of each subplot represent the standard scores (z value) correspondingly.

first season of KoLA, we evaluate 21 open and commercial LLMs and get some intriguing findings,
such as larger models tend to memorize more knowledge, and alignment unleashes the potential of
higher-level abilities but may harm the low-level knowledge memorization, etc. In the future, we will
continually host more seasons of KoLA to facilitate knowledgeable LLMs, help select backbones
for developing knowledge-related applications, and track the development of LLMs with evolving
evaluations. KoLA will always welcome open participation and contributions.

References
[1] OpenAI, “Gpt-4 technical report,” arXiv preprint arxiv:2303.08774, 2023.
[2] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee,
Y. Li, S. Lundberg, et al., “Sparks of artificial general intelligence: Early experiments with
GPT-4,” arXiv preprint arXiv:2303.12712, 2023.
[3] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task
benchmark and analysis platform for natural language understanding,” in Proceedings of
EMNLP Workshop BlackboxNLP, pp. 353–355, 2018.
[4] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R.
Bowman, “SuperGLUE: A stickier benchmark for general-purpose language understanding
systems,” in Proceedings of NIPS, pp. 3261–3275, 2019.
[5] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring
massive multitask language understanding,” in Proceedings of ICLR, 2021.
[6] W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan,
“AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models,” arXiv preprint
arxiv:2304.06364, 2023.
[7] Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al.,
“C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models,” arXiv
preprint arxiv:2305.08322, 2023.
[8] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro,
A. Gupta, A. Garriga-Alonso, et al., “Beyond the imitation game: Quantifying and extrapolating
the capabilities of language models,” arXiv preprint arxiv:2206.04615, 2022.
[9] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le,
E. H. Chi, D. Zhou, et al., “Challenging BIG-Bench tasks and whether chain-of-thought can
solve them,” arXiv preprint arxiv:2210.09261, 2022.
[10] L. Theis, A. van den Oord, and M. Bethge, “A note on the evaluation of generative models,” in
Proceedings of ICLR, 2016.

9
[11] M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly, “Assessing generative
models via precision and recall,” in Proceedings of NIPS, pp. 5234–5243, 2018.
[12] K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmentation reduces
hallucination in conversation,” in Findings of EMNLP, pp. 3784–3803, 2021.
[13] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung,
“Survey of hallucination in natural language generation,” ACM Computing Surveys, vol. 55,
no. 12, pp. 1–38, 2023.
[14] A. Belz, M. Popovic, and S. Mille, “Quantified reproducibility assessment of NLP results,” in
Proceedings of ACL, pp. 16–28, 2022.
[15] A. Belz, C. Thomson, and E. Reiter, “Missing information, unresponsive authors, experimental
flaws: The impossibility of assessing the reproducibility of previous human evaluations in NLP,”
in The Fourth Workshop on Insights from Negative Results in NLP, pp. 1–10, 2023.
[16] D. R. Krathwohl, “A revision of bloom’s taxonomy: An overview,” Theory into practice, vol. 41,
no. 4, pp. 212–218, 2002.
[17] A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia, et al.,
“Glm-130b: An open bilingual pre-trained model,” arXiv preprint arxiv:2210.02414, 2022.
[18] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal,
E. Hambro, F. Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv
preprint arxiv:2302.13971, 2023.
[19] E. A. Feigenbaum, “The art of artificial intelligence: Themes and case studies of knowledge
engineering,” in Proceedings of IJCAI, vol. 2, Boston, 1977.
[20] F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, and A. Miller, “Language
Models as Knowledge Bases?,” in Proceedings of EMNLP, pp. 2463–2473, 2019.
[21] F. Petroni, A. Piktus, A. Fan, P. Lewis, M. Yazdani, N. De Cao, J. Thorne, Y. Jernite,
V. Karpukhin, J. Maillard, V. Plachouras, T. Rocktäschel, and S. Riedel, “KILT: a Benchmark
for Knowledge Intensive Language Tasks,” in Proceedings of NAACL-HLT, pp. 2523–2544,
2021.
[22] A. Lewis and D. Smith, “Defining higher order thinking,” Theory into practice, vol. 32, no. 3,
pp. 131–137, 1993.
[23] B. Miri, B.-C. David, and Z. Uri, “Purposely teaching for the promotion of higher-order thinking
skills: A case of critical thinking,” Research in science education, vol. 37, pp. 353–369, 2007.
[24] R. Collins, “Skills for the 21st century: teaching higher-order thinking,” Curriculum & Leader-
ship Journal, vol. 12, no. 14, 2014.
[25] Y. Yao, D. Ye, P. Li, X. Han, Y. Lin, Z. Liu, Z. Liu, L. Huang, J. Zhou, and M. Sun, “DocRED:
A large-scale document-level relation extraction dataset,” in Proceedings of ACL, pp. 764–777,
2019.
[26] X. Wang, Z. Wang, X. Han, W. Jiang, R. Han, Z. Liu, J. Li, P. Li, Y. Lin, and J. Zhou,
“MAVEN: A Massive General Domain Event Detection Dataset,” in Proceedings of EMNLP,
pp. 1652–1671, 2020.
[27] N. Ding, G. Xu, Y. Chen, X. Wang, X. Han, P. Xie, H. Zheng, and Z. Liu, “Few-NERD: A
few-shot named entity recognition dataset,” in Proceedings of ACL, pp. 3198–3213, 2021.
[28] H. Peng, X. Wang, S. Hu, H. Jin, L. Hou, J. Li, Z. Liu, and Q. Liu, “COPEN: Probing conceptual
knowledge in pre-trained language models,” in Proceedings of EMNLP, pp. 5015–5035, 2022.
[29] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning, “Hot-
potQA: A dataset for diverse, explainable multi-hop question answering,” in Proceedings of
EMNLP, pp. 2369–2380, 2018.
[30] H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal, “MuSiQue: Multihop questions
via single-hop question composition,” Transactions of the Association for Computational
Linguistics, vol. 10, pp. 539–554, 2022.
[31] S. Cao, J. Shi, L. Pan, L. Nie, Y. Xiang, L. Hou, J. Li, B. He, and H. Zhang, “KQA Pro: A Dataset
with Explicit Compositional Programs for Complex Question Answering over Knowledge Base,”
in Proceedings of ACL, pp. 6101–6119, 2022.

10
[32] W. Chen, Y. Su, X. Yan, and W. Y. Wang, “KGPT: Knowledge-grounded pre-training for
data-to-text generation,” in Proceedings of EMNLP, pp. 8635–8648, 2020.
[33] Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung,
et al., “A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination,
and interactivity,” arXiv preprint arxiv:2302.04023, 2023.
[34] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional
transformers for language understanding,” in Proceedings of NAACL-HLT, pp. 4171–4186,
2019.
[35] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan,
P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child,
A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray,
B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei,
“Language models are few-shot learners,” in Proceedings of NIPS, 2020.
[36] K. Shuster, J. Xu, M. Komeili, D. Ju, E. M. Smith, S. Roller, M. Ung, M. Chen, K. Arora, J. Lane,
et al., “Blenderbot 3: a deployed conversational agent that continually learns to responsibly
engage,” arXiv preprint arxiv:2208.03188, 2022.
[37] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite,
N. Nabeshima, et al., “The pile: An 800gb dataset of diverse text for language modeling,” arXiv
preprint arxiv:2101.00027, 2021.
[38] X. Wang, T. Gao, Z. Zhu, Z. Zhang, Z. Liu, J. Li, and J. Tang, “KEPLER: A unified model for
knowledge embedding and pre-trained language representation,” Transactions of the Association
for Computational Linguistics, vol. 9, pp. 176–194, 2021.
[39] H. Jin, C. Li, J. Zhang, L. Hou, J. Li, and P. Zhang, “XLORE2: large-scale cross-lingual
knowledge graph construction and application,” Data Intelligence, vol. 1, no. 1, pp. 77–98,
2019.
[40] X. Wang, Y. Chen, N. Ding, H. Peng, Z. Wang, Y. Lin, X. Han, L. Hou, J. Li, Z. Liu, P. Li, and
J. Zhou, “MAVEN-ERE: A unified large-scale dataset for event coreference, temporal, causal,
and subevent relation extraction,” in Proceedings of EMNLP, pp. 926–941, 2022.
[41] P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao,
“Chameleon: Plug-and-play compositional reasoning with large language models,” arXiv
preprint arXiv:2304.09842, 2023.
[42] G. Mialon, R. Dessì, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Rozière, T. Schick,
J. Dwivedi-Yu, A. Celikyilmaz, et al., “Augmented language models: a survey,” arXiv preprint
arXiv:2302.07842, 2023.
[43] X. Ho, A.-K. Duong Nguyen, S. Sugawara, and A. Aizawa, “Constructing a multi-hop QA
dataset for comprehensive evaluation of reasoning steps,” in Proceedings of COLING, pp. 6609–
6625, 2020.
[44] Z. Yao, Y. Liu, X. Lv, S. Cao, J. Yu, J. Li, and L. Hou, “KoRC: Knowledge Oriented Reading
Comprehension Benchmark for Deep Text Understanding,” in Findings of ACL, 2023.
[45] J. Yu, X. Zhang, Y. Xu, X. Lei, X. Guan, J. Zhang, L. Hou, J. Li, and J. Tang, “XDAI: A
tuning-free framework for exploiting pre-trained language models in knowledge grounded
dialogue generation,” in Proceedings of KDD, pp. 4422–4432, 2022.
[46] P. J. Dyck, C. J. Boes, D. Mulder, C. Millikan, A. J. Windebank, P. J. B. Dyck, and R. Espinosa,
“History of standard scoring, notation, and summation of neuromuscular signs. a current survey
and recommendation,” Journal of the Peripheral Nervous System, vol. 10, no. 2, pp. 158–173,
2005.
[47] S. Patro and K. K. Sahu, “Normalization: A preprocessing stage,” arXiv preprint
arXiv:1503.06462, 2015.
[48] M. L. Maher, “Evaluating creativity in humans, computers, and collectively intelligent systems,”
in Proceedings of the 1st DESIRE Network Conference on Creativity and Innovation in Design,
pp. 22–28, 2010.
[49] C. Lamb, D. G. Brown, and C. L. Clarke, “Evaluating computational creativity: An interdisci-
plinary tutorial,” ACM Computing Surveys (CSUR), vol. 51, no. 2, pp. 1–34, 2018.

11
[50] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization
Branches Out, pp. 74–81, 2004.
[51] B. Wang and A. Komatsuzaki, “GPT-J-6B: A 6 Billion Parameter Autoregressive Language
Model.” [Link] 2021.
[52] T. Computer, “Releasing GPT-JT powered by open-source AI.” [Link]
xyz/blog/releasing-v1-of-gpt-jt-powered-by-open-source-ai, 2022.
[53] S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. Mc-
Donell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and
S. Weinbach, “GPT-NeoX-20B: An open-source autoregressive language model,” in Proceed-
ings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large
Language Models, pp. 95–136, 2022.
[54] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni,
F. Yvon, M. Gallé, et al., “Bloom: A 176b-parameter open-access multilingual language model,”
arXiv preprint arXiv:2211.05100, 2022.
[55] Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, D. Bahri, T. Schuster, H. S. Zheng, N. Houlsby, and
D. Metzler, “Unifying language learning paradigms,” arXiv preprint arXiv:2205.05131, 2022.
[56] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani,
S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, S. Narang,
G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin,
A. Roberts, D. Zhou, Q. V. Le, and J. Wei, “Scaling instruction-finetuned language models,”
arXiv preprint arxiv:2210.11416, 2022.
[57] G. Research, “A new open source flan 20b with ul2,” 2022.
[58] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto,
“Alpaca: A strong, replicable instruction-following model,” Stanford Center for Research on
Foundation Models., 2023.
[59] Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “Glm: General language model
pretraining with autoregressive blank infilling,” in Proceedings of the 60th Annual Meeting of
the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320–335, 2022.
[60] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal,
K. Slama, A. Ray, et al., “Training language models to follow instructions with human feedback,”
in Proceedings of NIPS, pp. 27730–27744, 2022.
[61] A. Studio, “Announcing jurassic-2 and task-specific apis,” 2023.
[62] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan,
Y. Wu, A. Kumar, et al., “Holistic evaluation of language models,” arXiv preprint
arxiv:2211.09110, 2022.
[63] K. Pillutla, S. Swayamdipta, R. Zellers, J. Thickstun, S. Welleck, Y. Choi, and Z. Harchaoui,
“MAUVE: measuring the gap between neural text and human text using divergence frontiers,” in
Proceedings of NIPS, pp. 4816–4828, 2021.
[64] J. Fu, S.-K. Ng, Z. Jiang, and P. Liu, “Gptscore: Evaluate as you desire,” arXiv preprint
arxiv:2302.04166, 2023.
[65] Y. T. Cao and H. Daumé, “Toward gender-inclusive coreference resolution,” in Proceedings of
ACL, 2020.
[66] M. Yoder, S. Khosla, Q. Shen, A. Naik, H. Jin, H. Muralidharan, and C. Rosé, “FanfictionNLP:
A text processing pipeline for fanfiction,” in Proceedings of the Third Workshop on Narrative
Understanding, pp. 13–23, 2021.
[67] S. Sun, K. Thai, and M. Iyyer, “ChapterBreak: A challenge dataset for long-range language
models,” in Proceedings of NAACL-HLT, pp. 3704–3714, 2022.
[68] K. Krishna, Y. Chang, J. Wieting, and M. Iyyer, “RankGen: Improving text generation with
large ranking models,” in Proceedings of EMNLP, pp. 199–232, 2022.
[69] W. Zhou, K. Huang, T. Ma, and J. Huang, “Document-level relation extraction with adaptive
thresholding and localized context pooling,” in Proceedings of AAAI, 2021.
[70] Z. Wang, X. Wang, X. Han, Y. Lin, L. Hou, Z. Liu, P. Li, J. Li, and J. Zhou, “Cleve: Contrastive
pre-training for event extraction,” in Proceedings of ACL, 2021.

12
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes] See Section 1.
(b) Did you describe the limitations of your work? [Yes] See Appendix A.1.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix A.2.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes] See Appendix A.3.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)? [Yes] See § 2.4.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [Yes] See Appendix E.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)? [No] Because of the high cost of running numerous LLMs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix E.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See § 2.2.
(b) Did you mention the license of the assets? [Yes] See Appendix C.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
See § 2.4.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating? [Yes] See Appendix B and § 2.4.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [Yes] See Appendix A.3 and Appendix C.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [No] Our evolving test tasks involve crowdsourcing annotation. But we do
not plan to publish the instructions because: (1) Our annotations are reproducing other
well-known tasks on the evolving data source. One can refer to the original papers
for details. (2) As a benchmark tested in zero-shot and few-shot ways, we need to
minimize the risk that one can achieve unfairly high scores by annotating training data.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation? [Yes] See Appendix A.3 and Appendix C.

13
A Broader Discussion
A.1 Limitation

The major limitation of KoLA is that our coverage is not as extensive as some other recent works [5,
6, 7]. KoLA evaluates LLMs’ world knowledge about concepts, entities, and events and only covers
19 English datasets now. While there is no doubt that expanding the evaluation "breadth" to test the
boundaries of LLM abilities is valuable, our emphasis lies more on the "depth" of evaluation. Due
to careful design considerations regarding data sources and our annotation capacity to host a new
competition season every 90 days, it is not easy to significantly broaden our evaluation coverage. The
first season results reported in this paper involve 21 LLMs. Although we strive to cover diverse and
representative LLMs, it is challenging to cover the ever-emerging and evolving LLMs with solely our
own efforts. We sincerely welcome community contributions and participations to introduce new
tasks or LLMs. The regarding information is shown at [Link]

A.2 Potential Impact

As a benchmark, the intended use of KoLA is not to construct applications or train LLMs, but rather
to evaluate the foundational abilities about world knowledge of LLMs. Our evaluation tasks do not
involve speculating personal sensitive information, making judgments on social issues, or interacting
with the real world. Therefore, we believe the likelihood of our benchmark directly leading to negative
impacts on safety, security, discrimination, surveillance, deception & harassment, human rights, bias
and fairness is very low. However, effective benchmarks will facilitate the development of powerful
LLMs, which poses a wide and serious risk of misuse. Although beyond the scope of this paper, we
earnestly call for strengthened cooperation from various sectors of society in enhancing the regulation
and safety control of LLMs.
Our evaluation needs to do inference with many LLMs on various datasets, which naturally results in
carbon emissions and potential environmental issues. The total carbon emissions can be estimated
based on the data provided in Appendix E.1. As our evaluation does not involve the pre-training and
fine-tuning of LLMs, we believe the impact caused is relatively marginal and controllable. In the
participation guidelines on the platform, we also state that we discourage improving evaluation scores
through repeated submissions or training specifically on benchmark-related data. This ensures the
reliability of the evaluation results while minimizing carbon emissions as much as possible.

A.3 Ethical Consideration

Appendix A.2 discusses the potential ethical concerns about our benchmark. In this section, we
discuss the ethical considerations regarding the data construction. (1) Data Risk Control. Regarding
the collected evolving data source, we have filtered out content that is inappropriate for presentation
to a general audience, and the relevant details are outlined in Appendix C.1. Seven of the authors
manual check all the newly constructed evolving test datasets as well as random samples of all the
previously released datasets included in KoLA. No instances of personally identifiable information,
discriminatory content, explicit, violence, or offensive content were found. (2) Annotator Treatment
and Consent. We hire crowdsourced annotators in the annotation of evolving test data and the human
evaluation for knowledge creating. The details are introduced in Appendix C.2. We have signed work
contracts with all the annotators and provided compensation according to mutually agreed-upon wage
standards and working hours. All employment arrangements are in compliance with local regulations.
(3) Copyright. Our known data source is Wikipedia, which is licensed under CC BY-SA 3.07 and
allows for free research use. For all the previously released datasets included in KoLA, we obtained
consent from the respective authors, and a detailed acknowledgment list can be found in Appendix B.
Our evolving data source contains public news and fictions. The news data is from The Guardian8
and we access it strictly following the terms and conditions9 . The fiction data is from Archive of
Our Own (AO3)10 , a fan-fiction archive site. Although AO3 data has been used in some previous
works [65, 66, 67, 68], there remains some ambiguity regarding its copyright status. We believe our
7
[Link]
8
[Link]
9
[Link]
10
[Link]

14
use of AO3 is appropriate because: (i) AO3 exhibits an open attitude towards data crawling11 . (ii) We
pledge that KoLA will always remain non-commercial and non-profit, and we do not redistribute the
crawled data (only samples are provided in our platform). According to the description12 provided by
the Organization for Transformative Works, the operator of AO3, such usage falls under fair use in the
context of the U.S. copyright law. (iii) If one believe that their copyrighted work has been included in
our datasets, one may contact the KoLA admin team (kola-benchmark@[Link]) at
any time to request its removal from KoLA.

11
[Link]
12
[Link]

15
B Acknowledgement
B.1 Author Contribution

Data Collection. Xiaozhi Wang, Shulin Cao, Xin Lv, Hao Peng, Zijun Yao collected the Known
Data Source from the open-source projects, private academic datasets and Wikidata. Jifan Yu, Daniel
Zhang-Li, Nianyi Lin, Linlu Gong and Kaifeng Yun collected and pre-processed the first season’s
Evovling Data Source. Hailong Jin supported the knowledgeable data from Xlore [39]. Yuan Yao
provided the test set of DocRED [25] and Ning Ding helped the construction of FewNERD [27] test
set.
Data Annotation. Xiaozhi Wang organized the crowdsourcing annotation of fine-grained event
arguments for knowledge creating (KC) tasks. Kaisheng Zeng and Yong Guan assisted in the quality
control process. Jifan Yu and Daniel Zhang-Li organized the annotation of knowledge triples from
the evolving articles, as well as the human evaluation of creating results.
Task Construction. Xin Lv designed the dataset and instruction of the knowledge memorization (KM)
tasks, (1-1) and (1-2). Hao Peng organized the knowledge understanding (KU) tasks and constructed
instructions for (2-1/2-2/2-3). Zhili Wu, Yunjia Qi, Jianhui Chen and Weikai Li correspondingly
constructed instructions for task (2-4) to (2-7). Shulin Cao and Zijun Yao organized the knowledge
applying (KA) tasks and constructed instructions for (3-1), (3-3), (3-4) and (3-5). Yantao Liu and
Amy Xin constructed task (3-2) and (3-6). Nianyi Lin and Daniel Zhang-Li organized the knowledge
creating (KC) tasks (4-1), (4-2) and most of the evolving tasks. Kaifeng Yun and Linlu Gong
constructed instructions for task (1-3) and (2-8).
Model Evaluation. Shangqing Tu organized the whole process of model evaluation. Hanming
Li deployed and conduct experiments of GPT-J (6B), GPT-JT (6B), GPT-NeoX (20B), BLOOM
(7B), T0++ (11B), LLaMa (65B), GLM (130B), UL2 (20B), FLAN-T5 (11B), Alpaca (7B), FLAN-
UL2 (20B), ChatGLM (6B). Chunyang Li evaluated GPT-3 curie v1 (6.7B) and davinci v1 (175B),
InstructGPT curie v1 (6.7B*) and davinci v2 (175B*), ChatGLM (130B), and GPT3.5-turbo. Zheyuan
Zhang evaluated Cohere-command (52.4B) and J2-Jumbo-Instrcut (178B*), while Yushi Bai evaluated
GPT-4. Ji Qi, Daniel Zhang-Li and Jinxin Liu assisted the data analysis and presentation on the
inference results. Yu Gu supported some of the candidate APIs.
Platform Development. Xiaohan Zhang’s team organized the overall platform development. Jifan Yu
and Daniel Zhang-Li respectively contributed to the visualization design and backend development.
Paper Writing. Jifan Yu and Xiaozhi Wang hosted the paper writting. Shangqing Tu, Ji Qi,
Daniel Zhang-Li supported the part of experimental analysis. All the authors contributed to their
corresponding working details for completing this paper (including Appendix).
Advising. Lei Hou, Zhiyuan Liu, Bin Xu, Jie Tang and Juanzi Li take advisor roles in this project.
Juanzi Li is the main advisor, initializing, supporting and organizing this project.

B.2 Acknowledgement

This research project is supported by a grant from the Institute for Guo Qiang, Tsinghua University
(2019GQB0003). Jie Tang is supported by NSFC distinguised young scholars (61825602).
This project would like to express its appreciation for the contributions and assistance of many other
participants. We appreciate the providing of the test set of MusiQue [30] by Harsh Trivedi, Stony
Brook University and the test set of 2Multiwikihop [43] by Xanh Ho, National Institute of Informatics,
Tokyo, Japan. Special thanks go to Gang Wang for his outstanding product prototype design. The
artistic design, including the beautiful illustrations for the paper, was skillfully provided by Shanshan
Wang. The platform development and feature implementation were carried out by Zhenfang Lu,
Shuai Xie, Shuaiming Wang, Liangliang Cui and Dingxiao Liu. The coordination of crowdsourcing
affairs was effectively managed by Jupeng Zhang and Yue Yang. Finally, the long-term assistance
and support from Yini Chen to the AIGC research team are greatly acknowledged.

16
C Details of Data Collection
There are two data sources used in KoLA: Known and Evolving, which are correspondingly from
Wikipedia and newly crawled corpus. In this section, erview of the collection and maintenance of
them, as well as the annotation process involved, including essential statistical information.

C.1 Raw Data Collection

Known. We collect the corresponding Wikipedia articles for the entities in Wikidata5M [38] from
Xlore2 [39], a cross-lingual knowledge base in Chinese and English, using its 2019 version, and align
them accordingly. This process generates a dataset of 5 million articles. Given that language model
training is reliant on textual data, we depart from the conventional graph-based methods typically
employed in knowledge graphs to calculate entity frequencies. Instead, we conduct statistical analysis
to determine and rank the occurrence frequencies of these entities and their aliases within the text
corpus. Subsequently, we establish two sets of high-frequency and low-frequency entities, each
containing 2, 000 entities, to fulfill the requirements of the knowledge memorization level.
Evolving. The data collection process for the Evolving dataset in the first season of KoLA concludes
on April 15, 2023. Therefore, we are collecting data from the preceding 90 days (January 15, 2023,
to April 14, 2023). In terms of news data, we are experimenting with multiple open-source news
scraping interfaces. Our primary focus is on gathering articles that have rich event elements, such
as factual news and entertainment news. We have collected a total of 1, 000 such articles, aiming
to avoid sensitive news categories like politics. As for the novel data, we randomly selected 1, 000
works from a renowned creative platform13 , striving to achieve a balanced representation of various
writing types (fan fiction and original works). Afterward, we employ an open-source tool14 to filter
out chapters containing explicit, violent, or other inappropriate content, sorting them based on the
number of views. Following this filtering process, we ultimately retain 250 articles each from the
news and novel categories as candidates (500 in total) for high-quality texts.

C.2 Data Annotation

During the construction of the KoLA dataset, we have two key annotation tasks: (1) Fact Triple
Annotation and (2) Event Argument Annotation. We elaborate on the composition of our annotation
team and outline the primary procedures employed for these two significant annotation tasks.
Annotation Team. We hire a 21-member annotation team
(based on market rates) comprising experienced annotators. Table 4: Statistics of the annotation team
With their permission, we gather some basic information for dataset construction in KoLA.
about the annotation team and present it in Table 4. In
general, the annotation team consists mostly of individuals Gender Rate
with graduate-level qualifications. The validators are three
Ph.D. holders from the KoLA team. With the collabo- Female 85.7%
ration of this high-quality team, we strive to ensure the Male 14.3%
efficiency and quality of data annotation. Before initiat- Education Rate
ing the annotation work, we enter into a legally binding
contract with the team to protect the rights of the annota- Bachelor 47.6%
tors. We also develop a dedicated platform specifically for Master 52.4%
the annotation process, which facilitates efficient review,
publishing, and exporting of the annotation results. The
platform allows annotators to have flexibility in choosing their working hours, including the ability to
save intermediate results, retrieve relevant resources, and log in or log out at any time.
Annotation Process. For the two aforementioned annotation tasks, we employ different methods for
data preprocessing to facilitate the annotation process for the annotators.
For Task (1), the text to be annotated includes only the Evolving data. Prior to annotation, we first
utilize a named entity recognition tool15 to extract entities from the articles. Based on these entities,
13
[Link]
14
[Link]
15
[Link]

17
 Figure 3: The distribution of the top-20 annotated triples in term of relation type.

Figure 4: The distribution of the events and text length in the articles from Wikipedia (MAVEN).

we employ a document-Level relation extraction method [69] to obtain candidate document-level

fact triples. Regarding these triples, annotators categorize them into three classes given the provided
document and the ability to search web content: a) incorrect triples, b) correct triples that can be
known prior to January 15th, and c) correct triples that can only be known after January 15th. These
triples will be used for constructing datasets for different tasks.
For Task (2), the text to be annotated includes both Known and Evolving data. The Known data
portion utilizes articles from the MAVEN dataset [26], where the event trigger words and event
types have already been annotated, requiring no additional preprocessing. As for the Evolving data,
we employ the Omni-Event toolkit16 and replace the backbone model with CLEVE [70] for event
extraction preprocessing. Given the document and event trigger words, annotators are required to
perform the following tasks: a) annotate candidate attributes for each event, and b) correct or delete
events if the event trigger words are incorrect. The evaluation system provides examples for 159
event types (with an average of 7 attributes per type) and constructs a convenient retrieval system to
facilitate annotation and quality control.

C.3 Statistics and Analysis

For Task (1), we annotate all 500 articles and retained 2.7K correct triples, out of which only 459
triples cannot be found in earlier corpora. Figure 3 illustrates the distribution of relations for all
correct triplets, where blue bars represent the category b) and red bars are the number of category c).
It can be observed that even in the Evolving data, the triplets still exhibit a long-tail distribution. Most
of the triplets, such as "country of citizen" and "country," do not effectively convey the main content
of the articles. This further reinforces our goal of annotating fine-grained event-level knowledge.

16
[Link]

18
 Figure 5: The distribution of the events and text length in the articles from Evolving Data.

For Task (2), we specifically select 100 articles from the MAVEN and Evolving datasets, ensuring
they possess extensive knowledge (i.e., a substantial number of entities and event triggers that are not
within the lowest 20% frequency range). After completing the annotation process, we examine the
number of valid events (events containing at least one argument) and the article lengths, as illustrated
in Figure 4 and Figure 5.
In general, the distribution of event knowledge between the two datasets is quite similar. The articles
in the Evolving dataset are generally longer compared to the Wikipedia articles in MAVEN, resulting
in a higher number of valid events. Due to this difference, models may encounter more challenges
when performing tasks on the Evolving dataset, but they may also benefit from a greater exposure
to knowledge. Therefore, the varying performance of models on the tasks upon these two data
sources may be influenced by factors such as the model’s parameter size or training adequacy. This
phenomenon warrants further analysis and discussion.
Aligned Tasks. Due to the subsequent use of standard scores for evaluation, we align certain abilities
related to the tasks before initiating the task design. This alignment ensures the comparability of the
scores when calculating the standard scores. Specifically, the tasks in the Evolving data for each level
are aligned with the corresponding non-Evolving tasks. An exception is the KM layer, where all three
tasks are highly correlated, and thus, their results are calculated together.

19
D Details of Task Instruction
After completing the data collection, we proceed to construct separate test sets for tasks at different
levels, thereby transforming knowledge-related tasks into language tasks driven by instructions, which
facilitates the execution by large-scale models. In this section, we first present the design principles
for each level of tasks, followed by specific approaches to constructing detailed instructions, and
provide corresponding task examples.

D.1 Converting to Task Format

There are 7 tasks’ test sets that need to be constructed from scratch ((1-1) High-Freq., (1-2) Low-
Freq., (4-1) Encyclopedic Knowledge Creating and 4 Evolving Test tasks). These tasks require the
reconstruction and quality control of the data based on our annotations. As for the other 12 tasks,
we only focus on designing how they can be transformed into sequence tasks that can be solved
by language models, considering the dataset construction methods provided in the original text as
references. Overall, we follow two principles during the process of constructing instructions: a)
Simplicity: We aim to describe the task objectives using the least amount of text, thus saving the
model’s in-context length; b) Standardization: We use special markers to identify all structured
knowledge, assisting the model in quickly capturing the knowledge objectives.

D.2 Knowledge Memorization Tasks

Knowledge Memorization (KM) level primarily assesses the model’s ability to retain knowledge
triples. However, this format is not inherently suitable for large models. Therefore, we transform the
triple prediction task into a question-answering task that considers 1-to-N relationships. For each
type of relationship, we design specific templates to facilitate this transformation.

I NSTRUCTION : Please give answers to the following questions about knowledge. Note: If there are more than
one answer, print them all and separate them with a semicolon (;). Please do not give anything other than the
answers.

Q UESTION : What is the occupation of Wang Guozhen?

A NSWER : poet

Table 5: The instruction and an example of Task 1-1 High-Freq. KM.

I NSTRUCTION : Please give answers to the following questions about knowledge. Note: If there is more than
one answer, print them all and separate them with a semicolon (;). Please do not give anything other than the
answers.

Q UESTION : Which country does White Hall Township belong to?

A NSWER : United States of America

Table 6: The instruction and an example of Task 1-2 Low-Freq. KM.

I NSTRUCTION : Please give answers to the following questions about knowledge. Note: If there is more than
one answer, print them all and separate them with a semicolon (;). Please do not give anything other than the
answers.

Q UESTION : Will Messi still serve in Paris Saint-Germain?

A NSWER : No

Table 7: The instruction and an example of Task 1-3 RTM KM.

20
D.3 Knowledge Understanding Tasks

Knowledge Memorization (KM) level involves various levels of structured information, such as
concepts, entities, relationships, and events. It also incorporates multiple documents, even at the
multi-document level, which can easily overwhelm the model and distract from the main task.
Therefore, we have placed a strong emphasis on standardizing the input and output of this layer to
facilitate the model’s comprehension of the task objectives. Additionally, we strive to balance the
simplicity of the input while ensuring comprehensive understanding.
Specifically, the instructions for each task are as follows:

I NSTRUCTION : Conceptual similarity judgment

Q UERY: Among Tutu Chengcui, The Pierre, Waddesdon Manor, Astro Orbitor, Heian period, 2019 Canadian
federal election, Paradiski, Tenughat Dam, Gros Michel banana, Reedy Glacier, Gangotri Glacier, Pinatubo,
Interwar period, djon djon, Qiu Shiliang, Caciotta, Firth of Forth, 2011 Rugby World Cup, Cheng Yuanzhen,
Pliocene, Sri Maha Bodhi, which one is the most conceptually similar with Botryosphaeria stevensii? Please
answer the entity name only.

A NSWER : djon djon

Table 8: The instruction and an example of Task 2-1 COPEN-CSJ, KU.

I NSTRUCTION : Conceptual property judgment

Q UERY: Is the statement “Specieses have a cellulose wall and other polysaccharides.” true or false? Please
answer true or false.

A NSWER : False

Table 9: The instruction and an example of Task 2-2 COPEN-CPJ, KU.

I NSTRUCTION : Conceptualization in contexts

Q UERY: Given the context “The next year, he made his stock car racing debut in the American Speed Association,
where he won a pole at Winchester Speedway and had four top-tens.”, neglect your knowledge about Winchester
Speedway and select the most contextually related concept for it from the concept set: Racecourse, Place,
ArchitecturalStructure, Infrastructure, RaceTrack, Venue, Road, SportFacility, RouteOfTransportation, Building.
Please answer the concept name only.

A NSWER : Racecourse

Table 10: The instruction and an example of Task 2-3 COPEN-CiC, KU.

21
I NSTRUCTION : Please recognize entities for the given text and classify them into a suitable type. The collection
of types is as follows: <set of types>

Q UERY: Agrippa succeeded in blocking the more manoeuvrable ships of Sextus and, after a long and bloody
fight, to defeat his enemy.

A NSWER : Agrippa: person-politician; Sextus: person-politician;

Table 11: The instruction and an example of Task 2-4 FewNERD, KU.

I NSTRUCTION : Please follow the above demonstration, and extract relations from the [Question text]. Note
the relation needs to be in the predefined set of relations. The output format required to is the same as the
demonstration, format:(<entity_ID>, relation, <entity_ID>). The predefined set of relations: <set of types>

Q UERY: <entity_0> Rickon Stark </entity_0> is a fictional character in the <entity_1> A Song of Ice and
Fire </entity_1> series of fantasy novels by <entity_2> American </entity_2> author <entity_3> George R.
R. Martin </entity_3>, and its television adaptation <entity_4> Game of Thrones </entity_4> . Introduced in
<entity_5> 1996 </entity_5> ’s <entity_6> A Game of Thrones </entity_6>, <entity_0> Rickon </entity_0>
is the youngest child of <entity_7> Eddard Stark </entity_7>, the honorable lord of <entity_8> Winterfell
</entity_8>, an ancient fortress in the <entity_9> North </entity_9> of the fictional kingdom of <entity_10>
Westeros </entity_10>. He subsequently appeared in <entity_3> Martin </entity_3> ’s <entity_11> A Clash
of Kings </entity_11> (<entity_12> 1998 </entity_12>) . The <entity_13> Publishers Weekly </entity_13>
review of <entity_6> A Game of Thrones </entity_6> noted, Ït is fascinating to watch <entity_3> Martin
</entity_3> ’s characters mature and grow, particularly <entity_14> Stark </entity_14> ’s children, who stand at
¨
the center of the book. <entity_0> Rickon </entity_0> is played by <entity_15> Art Parkinson </entity_15> in
the <entity_16> HBO </entity_16> television adaptation.

A NSWER : (<entity_0>, father, <entity_7>); (<entity_0>, present in work, <entity_4>); (<entity_0>, creator,
<entity_3>); (<entity_7>, child, <entity_0>); (<entity_7>, present in work, <entity_6>); (<entity_7>, present
in work, <entity_4>); (<entity_6>, publication date, <entity_5>); (<entity_6>, characters, <entity_7>); (<en-
tity_6>, author, <entity_3>); (<entity_4>, publication date, <entity_5>); (<entity_4>, characters, <entity_7>);
(<entity_4>, has part, <entity_6>); (<entity_4>, author, <entity_3>); (<entity_4>, screenwriter, <entity_3>);
(<entity_4>, original network, <entity_16>); (<entity_3>, notable work, <entity_6>); (<entity_10>, present in
work, <entity_4>); (<entity_11>, publication date, <entity_12>); (<entity_11>, series, <entity_6>); (<entity_11>,
follows, <entity_6>); (<entity_11>, series, <entity_4>); (<entity_11>, author, <entity_3>); (<entity_14>, present
in work, <entity_4>); (<entity_14>, creator, <entity_3>);

Table 12: The instruction and an example of Task 2-5 DocRED, KU.

I NSTRUCTION : Please identify the events in the text and classify them into appropriate categories; The
collection of categories is <set of types>

Q UERY: The ruling National Command of the Arab Socialist Ba’ath Party were removed from power by a union
of the party’s Military Committee and the Regional Command, under the leadership of Salah Jadid.

A NSWER : removed:Removing

Table 13: The instruction and an example of Task 2-6 MAVEN, KU.

22
I NSTRUCTION : Please classify the relation between two events/“Time” in a given document. There are 10 types
of relations: [“before”, “overlap”, “contains”, “simultaneous”, “begins-on”, “ends-on”, “cause”, “precondition”,
“subeven”, and “coreference”]. In each document, 2 events/“Timex” are marked as “<Event> event name
</Event>” or “<Timex> Timex name </Timex>”. If there is a relation type or multiple relation types, the answer
form is “Answer: [relation type 1, relation type 2, ...]”.

Q UERY: Document: The Central Park jogger case was a criminal case in the United States based on the assault
and <Event> rape </Event> of Trisha Meili, a 28-year-old white woman who was jogging in the park, and attacks
on eight other persons, in areas ranging from the North Woods of Manhattan’s Central Park to the Reservoir,
on the night of April 19, 1989. Three of the victims were black or Latino. Meili was so injured that she was
in a coma for 12 days. “The New York Times” in 1990 described the attack on her as “one of the most widely
publicized crimes of the 1980s”. Attacks in Central Park that night were allegedly committed by a loose group of
302̆01332 teenagers, and police attempted to apprehend suspects after crimes began to be reported between 9 and
10 p.m. The brutally beaten Meili was not found until 1:30 a.m., after which the police hunt greatly intensified.
They took into custody 14 or more other suspects over the next few days, and arrested a total of ten suspects who
were ultimately tried for the attacks. Among them were four African American and two Hispanic American
teenagers who were indicted on May 10 on charges of assault, robbery, riot, rape, sexual abuse, and attempted
murder of Meili and an unrelated man, John Loughlin. The prosecutor planned to try the defendants in two
groups, then scheduled the sixth defendant to be tried last. The latter pleaded guilty in January 1991 on lesser
charges and received a reduced sentence. Prosecution of the five remaining defendants in the rape and assault
case was based primarily on confessions which they had made after police interrogations. None had counsel
during this questioning. Within weeks, they each withdrew these confessions, pleaded not guilty, and refused
plea deals on the rape and assault charges. None of the suspects ’ DNA matched the DNA collected from the
crime scene: two semen samples that both belonged to one unidentified man. No substantive physical evidence
connected any of the five teenagers to the rape scene, but each was convicted in 1990 of related assault and other
charges. Subsequently known as the Central Park Five, they received stiff sentences ranging from 5 to 15 years.
Four of the defendants appealed their convictions, but these were affirmed by appellate courts. The four juvenile
defendants served 62̆0137 years each; the 16-year-old, tried and sentenced as an adult, served 13 years in adult
prison. The five other defendants, <Event> indicted </Event> for assaults of other victims, pleaded guilty to
reduced charges and received less severe sentences. In 2001, Matias Reyes, a convicted murderer and serial rapist
serving life in prison, confessed to officials that he had raped the female jogger. His DNA matched that found
at the scene, and he provided other confirmatory evidence. He said he committed the rape alone. Reyes could
not be prosecuted for raping Meili, because the statute of limitations had passed. In 2002 Robert Morgenthau,
District Attorney for New York County, had his office conduct an investigation and recommended to the state
court that the convictions of the five men on all charges be vacated. The court vacated their convictions in
2002, and the state withdrew all charges against the men. In 2003, the five men sued the City of New York for
malicious prosecution, racial discrimination, and emotional distress. The city refused to settle the suits for a
decade, because its lawyers believed that the city could win a court case. After a change in administration, the
city settled in 2014 with the five plaintiffs for $41 million. The five men also filed suit against the State of New
York for additional damages; this case was settled in 2016 for a total of $3.9 million. The first event/“Timex”:
<Event> rape </Event>. The second event/“Timex”: <Event> indicted </Event>.

A NSWER : before

Table 14: The instruction and an example of Task 2-7 MAVEN-ERE, KU.

I NSTRUCTION : Please follow the above demonstration, extract relations from the [Question text]. Note
the relation need to be in the predefined set of relations. The output format required to is the same as the
demonstration, format:(<entity_ID>, relation, <entity_ID>). The predefined set of relations: <set of types>

Q UERY: Text: Less than four months removed from the entity0 World Cup entity0’s bright lights , the U.S.
men’s national soccer team visited an 8,000 - seat bayside stadium on a tiny entity1 Caribbean entity1 island
Friday to face an opponent with players from regional leagues and some of entity2 England entity2’s lowest
divisions entity3. <text continued> Relations in the predefined set of relations in the above text:?

A NSWER : (entity4,part of,entity13); (entity6,part of,entity13); (entity13,has part,entity4); (entity15,participant

of,entity0); (entity15,country of citizenship,entity9); (entity16,country of citizenship,entity9); (entity17,country
of citizenship,entity9); (entity18,country of citizenship,entity9); (entity19,country of citizenship,entity9); (en-
tity20,country of citizenship,entity9);

Table 15: The instruction and an example of Task 2-8 ETU, KU.

23
D.4 Knowledge Applying Tasks

Knowledge Applying (KA) level naturally involves multi-hop reasoning in the form of question
answering, which is suitable for large-scale models to perform inference. One major challenge is that
the contexts required for these reasoning steps are not consistent. One particular aspect is the KoRC
task and ETA task, which assume that the model has access to a corresponding knowledge base. The
original dataset for KoRC confirms Wikidata as the knowledge base. However, in the Evolving data,
many pieces of knowledge cannot be directly found. Therefore, we construct a virtual knowledge
base using the annotated triples and generate questions based on this knowledge base.
Finally, we adopt the following task instructions:

I NSTRUCTION : Please answer the following question.

Q UERY: Jeremy Theobald and Christopher Nolan share what profession?

A NSWER : Jeremy Theobald is an actor and producer. Christopher Nolan is a director, producer, and screenwriter.
Therefore, they both share the profession of being a producer. So the answer is: producer.

Table 16: The instruction and an example of Task 3-1 HotpotQA, KA.

I NSTRUCTION : Please answer the following question.

Q UERY: Which film came out first, Blind Shaft or The Mask Of Fu Manchu?

A NSWER : Blind Shaft is a 2003 film. The Mask Of Fu Manchu is a 1932 film. So the answer is: The Mask Of
Fu Manchu.

Table 17: The instruction and an example of Task 3-2 2WikiMultihopQA, KA.

I NSTRUCTION : Please answer the following question.

Q UERY: When did the first large winter carnival take place in the city where CIMI_FM is licensed to broadcast?

A NSWER : CIMI_FM is licensed to broadcast in Quebec City. The first large winter carnival in Quebec City
took place in 1894. So the answer is: 1894.

Table 18: The instruction and an example of Task 3-3 MuSiQue, KA.

24
I NSTRUCTION : Please answer the following question.

Q UERY: When was Neville A. Stanton’s employer founded?

A NSWER : The employer of Neville A. Stanton is University of Southampton. The University of Southampton
was founded in 1862. So the answer is: 1862.

Table 19: The instruction and an example of Task 3-4 KQA Pro, KA.

I NSTRUCTION : You are given one document and one anonymized real-world entity with one or more mentions
in the passage. Then we will ask your a question about this anonymized entity. The questions cannot be answered
solely within the document or the background knowledge. Your task is to leverage world knowledge you have
like Wikipedia or Wikidata as background knowledge combined with the given document to answer the question
related to the anonymized entity. You must output all answers in the end.

Q UERY: Allen is a county in the U.S. state of Ohio. As of the 2010 census, the population was 106,331. The
county seat is Lima. The county was created in 1820 and organized in 1831. The county is named for Colonel
[a human being], who was killed leading his men at the Battle of Frenchtown, during the War of 1812. It has
also been claimed the county was named for Revolutionary War soldier Ethan Allen, but the weight of the
evidence in favor of [the human being] led the General Assembly to declare in 1976 that the county was named
for him. Allen comprises the Lima, OH Metropolitan Statistical Area, which is also part of the Lima - Van Wert -
Wapakoneta, OH Combined Statistical Area.
Q UESTION :: Which place was this human being born?

A NSWER : Rockbridge County.

Table 20: The instruction and an example of Task 3-5 KoRC, KA.

I NSTRUCTION : You are given one document and one anonymized real-world entity with one or more mentions
in the passage. Then we will ask your a question about this anonymized entity. The questions cannot be answered
solely within the document or the background knowledge. Your task is to leverage world knowledge you have
like Wikipedia or Wikidata as background knowledge combined with the given document to answer the question
related to the anonymized entity. You must output all answers in the end.

Q UERY: Six months after its New Shepard rocket suffered a failure during flight, Blue Origin said Friday its
review of the incident pinpointed a problem with its engine nozzle and that it is expecting to return to flight
“soon.” In September, the rocket lifted off and flew for just over a minute before bright flames flashed from the
booster and the capsule’s emergency abort system kicked in, propelling it away from the rocket. The mission
carried only science experiments; no one was on board, and no one was injured on the ground. In a statement
Friday, Blue Origin, the space venture founded by Amazon executive chairman Jeff Bezos, said that it would
refly the mission, again carrying scientific payloads. (Bezos owns [daily newspaper].) A flight with people could
come later. The vehicle is designed to carry as many as six people to the edge of space and back on suborbital
tourist trips that allow passengers to experience weightlessness and view the earth from above. In the statement,
Blue Origin said its investigation, which was overseen by the Federal Aviation Administration and included
members of the National Transportation Safety Board, concluded that the problem was caused by a failure of the
engine nozzle, which experienced “temperatures that exceeded the expected and analyzed values of the nozzle
material.” Engineers are “implementing corrective actions, including design changes to the combustion chamber
and operating parameters,” the statement said. “Additional design changes to the nozzle have improved structural
performance under thermal and dynamic loads.” The FAA said in a statement that it is reviewing Blue Origin’s
mishap report but that the investigation remains open. “FAA approval is required to close the investigation and
for the New Shepard system to return to flight.” It was unclear how long that could take. While the booster was
lost, the capsule and the 36 payloads it was carrying landed safely under parachutes and can fly again, Blue
Origin said. The booster, which under normal circumstances falls back to Earth and touches down softly on
a landing pad so that it can be reused, was a total loss. The company was able to recover all the debris from
the rocket within the designated hazard area, it said. Bezos flew on the first flight with people in 2021. It had
since flown five other missions with people on board, including one with Star Trek actor William Shatner and
television commentator Michael Strahan. It has not flown since the September incident.
Q UESTION :: What is the home country of [daily newspaper]?

A NSWER : United States

Table 21: The instruction and an example of Task 3-6 ETA, KA.

25
D.5 Knowledge Creating Tasks

The Knowledge Creation (KC) level is particularly unique as each task involves two generation
processes. Here, we present the process that considers generating subsequent knowledge, which is
the most informative. However, for direct generation, the instruction can be replaced with "Complete
the following generate" without specifying the "TRIPLETS" item.
The example instructions of creating tasks are shown below:

I NSTRUCTION : Complete the following generated texts and make sure to contain all the events provided.

T RIPLETS : ### Death of Freddie Gray;#### Known Events;##### relation: charges; Agent: Marilyn Mosby;
Patient: six police officers; Reason: the medical examiner’s report ruled Gray’s death a homicide;#####
relation: culminating; Agent: Larry Hogan; Patient: civil disorder; Symbol event: a state of emergency
declaration; Location: Baltimore;##### relation: establishment; Creator: Larry Hogan; Patient: a curfew;
Location: Baltimore; <Triplets Continued>

R EFERENCE C ONTEXT: #### Generated Context; On April 12, 2015, Freddie Carlos Gray, Jr., a 25-year-old
black man, was arrested by the Baltimore Police Department for possessing what the police alleged was an
illegal knife under Baltimore law. While being transported in a police van, Gray fell into a coma and was taken
to a trauma center. Gray died on April 19, 2015; <Text Continued>

G ROUND T RUTH : On May 1, 2015, the Baltimore City State’s Attorney, Marilyn Mosby, announced her office
had filed charges against six police officers after the medical examiner’s report ruled Gray’s death a homicide.
The prosecutors stated that they had probable cause to file criminal charges against the six police officers who
were believed to be involved in his death. <Text Continued>

Table 22: The instruction and an example of Task 4-1 Encyclopedic, KC.

I NSTRUCTION : Complete the following generated texts and make sure to contain all the events provided.

T RIPLETS : ### <Title if Contained>;#### Known Events;##### relation: uncovered; Speaker: Alston & Bird;
Message: no facts to show that U.S. Soccer knew of the 1992 Incident when it hired Mr. Berhalter; Receiver:
Gregg Berhalter;##### relation: harm; Agent: Gregg Berhalter; Cause: others; Location: United States;#####
relation: injured; Agent: Gregg Berhalter; Cause: Rosalind; Location: the University of North Carolina ;#####
relation: preparing; Agent: The next coach; Content: the U.S. team for the 2026 World Cup; Upcoming activity:
World Cup; Location: United States; <Triplets Continued>

R EFERENCE C ONTEXT: #### Generated Context; Details of a sordid rift between two prominent U.S. soccer
families — one that included allegations of domestic abuse against men’s national team coach Gregg Berhalter
and parental complaints about Gio Reyna’s playing time at the 2022 World Cup — continued to spill out Monday
when the findings of an independent investigation were released. <Text Continued>

G ROUND T RUTH : Earnie Stewart left the job last month. Anthony Hudson, a World Cup assistant, is the
interim coach. The next coach will begin preparing the U.S. team for the 2026 World Cup, which will take place
in the United States, Mexico and Canada. Berhalter guided the United States for four years, leading a young
squad to two regional championships and a place in the World Cup, where it finished second in group play and
lost to the Netherlands in the round of 16. <Text Continued>

Table 23: The instruction and an example of Task 4-2 ETC, KC.

26
E Details of Result Inference
Given the instructions and test sets for each task, we evalute a total of 21 models in the first season of
KoLA. Here, we present some of the deployment environments of the models that participated in our
first season, as well as some specific solutions implemented during the evaluations.

E.1 Deployment Environment and Model Information

The participating models in the evaluation include two types: closed-source models that return
answers through API calls, and open-source models that are deployed directly for inference (with a
temperature set to 0). Here, we primarily introduce the software and hardware environment used for
deploying the models. We utilize the widely-used PyTorch and transformers library to load open-
source models. The evaluation experiments are conducted on an Ubuntu 20.04.4 server equipped
with 112 Intel Xeon(R) Platinum 8336C CPU cores, and graphic cards that contained 8 NVIDIA
A100 SXM 80GB GPUs. Besides, The CUDA version is 11.4, the Python version is 3.10.0, the
PyTorch version is 2.0.0 and the transformers version is 4.28.1.
Table 24 presents the features of the selected LLMs in the first season. For the open-source models,
we deploy them using their official versions, with particular emphasis on the HuggingFace versions.
As for the closed-source models, we utilize the various model APIs available as of May 15, 2023. We
also conduct thorough checks and re-inferencing in case of any network-related errors.

Table 24: Selected LLMs. Instruct and |Context| correspond to whether the model is with instruction
tuning and the input context’s length limitation. * indicates the size has not been officially confirmed.
Model Size Type Instruct |Context| Website Url
GPT-J 6B Open w/o 2, 048 [Link]
GPT-JT 6B Open w/ 2, 048 [Link]
GPT-NeoX 20B Open w/o 2, 048 [Link]
BLOOM 7B Open w/o 2, 048 [Link]
T0++ 11B Open w/ 512 [Link]
LLaMa 65B Open w/o 2, 048 [Link]
Alpaca 7B Open w/ 2, 048 [Link]
UL2 20B Open w/o 512 [Link]
FLAN-T5 11B Open w/ 512 [Link]
FLAN-UL2 20B Open w/ 2, 048 [Link]
GLM 130B Open w/o 2, 048 [Link]
ChatGLM 6B Open w/ 2, 048 [Link]
ChatGLM 130B API w/ 2, 048 Not Publicly Availabile, Comming Soon
GPT-3 curie v1 6.7B API w/o 2, 048 [Link]
GPT-3 davinci v1 175B API w/o 2, 048 [Link]
InstructGPT curie v1 6.7B* API w/ 2, 048 [Link]
InstructGPT davinci v2 175B* API w/ 2, 048 [Link]
GPT3.5-turbo * API w/ 2, 048 [Link]
GPT-4 * API w/ 2, 048 [Link]
Cohere-command 52.4B API w/ 4, 096 [Link]
J2-Jumbo-Instruct 178B* API w/ 8, 192 [Link]

E.2 Solution for Run-time Exceptions

Apart from issues such as user permissions and network environment when invoking the model API,
the main challenges we encountered during the model evaluation process were limited input length
for some models and output inconsistencies with the required format. Therefore, we have devised the
following strategies to handle these exceptional cases during evaluation:
Over-length Issue: Due to the length limitations of certain models, performing 5-shot zero-shot
inference becomes challenging for tasks with lengthy instructions. Therefore, we have devised the
following strategies to enable the models to produce desired outputs: a) Reduce the number of
examples until the input-output length requirements are met; b) If reducing the number of examples
to one still fails to meet the requirements of all cases, skip the non-compliant cases and treat them as
0; c) If a model skips a substantial number of examples (over 90%) on a particular task, consider it a
failure on that task and record it as “–”.
Disregarding Instruction: Another factor that significantly affects the model’s performance is its
potential inability to comprehend the task instructions, resulting in failure to produce outputs in the

27
specified format. This poses challenges for evaluating the generation-based open-ended answers.
Therefore, we attempt to extract key information from the answers using techniques such as regular
expressions and perform fuzzy matching. Unfortunately, there are still many scenarios where certain
models fail to generate correct and valid answers. For models that fail to provide reasonable answers
on over 90% of the test cases for a particular task, we mark them as N/A.
Sensitive Result: The models may also trigger or bypass their safety mechanisms in certain tasks,
such as refusing to provide answers (due to mistakenly perceiving the given information as containing
unsafe content) or generating sensitive content. For cases where the model refuses to provide an
answer, we conduct manual checks to ensure that the data in the test set does not contain explicit,
violent, discriminatory, or other inappropriate content. In the secure test set, for situations where the
model refuses to answer or exhibits similar behavior, we handle them in the same way as mentioned
above. For models that cannot provide reasonable answers on over 90% of the test cases for a
particular task, we mark them as N/A.
In general, for all tasks, if a model’s performance is marked as “–” or “N/A” on a particular task, we
do not include that score in the calculation of the standard score. However, when calculating the
overall rankings of the models across different levels, we consider these cases as "missing" and assign
them a score of 0. There is still room for improvement in this handling approach. Nonetheless, we
firmly believe that the model’s ability to handle input length, adhere to guidelines, and handle sensitive
information is an important foundational skill for dealing with real-world knowledge problems.

28
F More Evaluation Result
Due to the length constraints of the paper, including the manual annotation process in the experimental
section, a series of specific results are not fully presented. In this section, we first introduce the
annotation process and results for the knowledge creation level. Then, we provide a detailed list of
all absolute performance values for each model in the first season and discuss some notable findings.

F.1 Annotation of Creating Tasks

Annotation Team for Evaluation. We recruit members

from the annotation team who are willing to participate in Table 25: Statistics of the annotation
result annotation and further confirm their compensation team for creating evaluation in KoLA.
and bonuses to protect their earnings and rights. Currently,
14 annotators participate in the annotation of knowledge Gender Rate
creation results. The composition of these individuals
closely resembles that of the overall annotation team, with Female 71.4%
all members having a bachelor’s degree or higher. The an- Male 28.6%
notation results undergo quality checks conducted by three Education Rate
Ph.D. students from KoLA teams, ensuring both efficiency
17 Bachelor 57.1%
and quality in the annotation process. We use Streamlit
to build an annotation platform where annotators can si- Master 42.9%
multaneously view the text to be annotated, the context,
and the historical annotations.
Annotation Process and Result. For each model, we provide the following information: a) preceding
context, b) continuation generated by the model, c) event knowledge content of the actual subsequent
text, and d) actual subsequent text. It is important to note that in item b), the model generates
the continuation without being informed of the knowledge in the subsequent text. Annotators are
required to evaluate two aspects: i) the overall quality of the text, which includes assessing the novelty,
coherence, and fluency of the model’s creative content; and ii) the plausibility of the generated content
in terms of knowledge hallucination, ensuring it does not conflict with real-world situations. The
latter aspect is a subtask of the former. The annotation is conducted on a 5-point scale, where 1
represents the worst and 5 represents the best. We collect 4.2K scoring results and calculate the
average score for each model. Table 26 presents the results and variances of the two ratings for
each model. It is based on these scores and the model’s rank that we calculate various correlation
coefficients.

Table 26: The human evaluation results of Knowledge Creating (KC).

Model Overall Knowledge Model Overall Knowledge Model Overall Knowledge
FLAN-T5 2.25 3.00 LLaMa 2.25 2.75 InstructGPT curie v1 1.75 2.75
UL2 2.00 2.25 Alpaca 1.25 2.00 InstructGPT davinci v2 2.50 2.75
FLAN-UL2 3.00 2.00 J2-Jumbo-Instruct 2.50 3.00 GLM 2.75 3.00
GPT-J 2.00 2.25 Cohere-command 2.75 3.00 GPT-4 2.75 3.75
GPT-NeoX 3.25 3.25 GPT-3.5-turbo 2.50 3.00 ChatGLM 2.50 2.00
BLOOM 3.25 2.00 GPT-3 curie v1 1.75 3.50 ChatGLM (130B) 2.50 3.00
T0++ 2.25 2.75 GPT-3 davinci v1 2.50 2.50 GPT-JT 2.50 2.25

17
[Link]

29
F.2 Detailed Results of Each Task

Knowledge Memorization (KM). Table 27 present the absolute performance on the tasks of knowl-
edge memorization. This level employs two main scoring methods, namely Exactly Match (EM) and
Token-level F1. A key observation is that due to the limited control over generation by many models,
the scores obtained using EM are often lower, resulting in numerous cases where a score cannot be
assigned. During the analysis of the results, we observe that some models, even without access to
external resources, can achieve good performance on evolving task. However, this often requires a
substantial scale and instruction tuning. This may be attributed to the fact that certain new knowledge
can be inferred from existing knowledge, which also relies on the model’s memorization.

Table 27: Absolute Performance of all metrics on task (1-1), (1-2), and (1-3), KM.
1-1 1-2 1-3
Model
EM F1 EM F1 EM F1
FLAN-T5 (11B) 13.6 20.1 12.5 17.7 21.8 23.2
UL2 (20B) N/A 5.1 N/A 5.9 N/A 1.4
FLAN-UL2 (20B) 14.0 18.5 9.5 13.2 23.7 25.1
GPT-JT (6B) N/A 1.8 N/A 1.3 N/A 0.4
GPT-J (6B) N/A 2.3 N/A 1.2 N/A 0.3
GPT-NeoX (20B) N/A 1.8 N/A 2.2 N/A 0.3
BLOOM (7B) N/A 2.3 N/A 2.8 N/A 1.5
T0++ (11B) 7.5 12.9 6.0 11.1 4.9 9.5
LLaMa (65B) 0.7 4.0 N/A 4.7 N/A 0.8
Alpaca (7B) N/A 2.6 1.0 4.5 N/A 1.5
J2-Jumbo-Instruct (178B*) 4.6 8.2 5.0 8.8 1.0 5.2
Cohere-command (52.4B) 17.0 21.5 12.5 19.3 24.7 27.3
GPT-3.5-turbo 9.8 18.7 18.0 22.1 13.8 18.9
GPT-3 curie v1 (6.7B) N/A 0.4 N/A 0.7 N/A N/A
GPT-3 davinci v1 (175B) N/A 0.9 N/A 0.8 N/A N/A
InstructGPT curie v1 (6.7B*) 1.3 5.9 10.5 13.9 7.9 13.9
InstructGPT davinci v2 (175B*) 5.6 12.6 13.0 16.2 9.6 13.5
GLM (130B) N/A 2.8 N/A 4.4 N/A 0.5
GPT-4 17.1 24.2 20.8 26.5 21.0 26.0
ChatGLM (130B) 7.4 10.9 16.5 20.3 13.8 15.6

30
Knowledge Understanding (KU). The results in this level are unexpected, as a significant amount
of knowledge understanding relies on longer texts or generating highly structured content. There-
fore, in complex tasks such as document-level relation extraction and event relation extraction, the
performance of many models is not satisfactory. This aspect deserves further exploration.

Table 28: Absolute Performance of accuracy on COPEN (2-1), (2-2), (2-3), KU.
Model 2-1 2-2 2-3
FLAN-T5 (11B) 39.0 75.0 55.0
UL2 (20B) N/A N/A N/A
FLAN-UL2 (20B) 35.0 73.0 62.0
GPT-JT (6B) N/A N/A N/A
GPT-J (6B) N/A N/A N/A
GPT-NeoX (20B) N/A N/A N/A
BLOOM (7B) N/A N/A N/A
T0++ (11B) 8.0 53.0 17.0
LLaMa (65B) N/A N/A N/A
Alpaca (7B) N/A N/A 1.0
J2-Jumbo-Instruct (178B*) 5.0 13.0 23.0
Cohere-command (52.4B) 17.0 73.0 50.0
GPT-3.5-turbo 21.0 79.0 57.0
GPT-3 curie v1 (6.7B) N/A N/A N/A
GPT-3 davinci v1 (175B) N/A 1.0 N/A
InstructGPT curie v1 (6.7B*) 7.0 58.0 42.0
InstructGPT davinci v2 (175B*) 11.0 76.0 43.0
GLM (130B) N/A N/A N/A
GPT-4 45.0 77.0 59.0
ChatGLM (130B) 8.0 75.0 60.0

Table 29: Absolute Performance on RE tasks, i.e., FewNERD (2-4), DocRED (2-5), ETU (2-8), KU.
2-4 2-5 2-8
Model
P R F1 P R F1 P R F1
FLAN-T5 (11B) 7.8 0.4 0.7 — — — — — —
UL2 (20B) 16.7 0.3 0.5 — — — — — —
FLAN-UL2 (20B) N/A N/A N/A N/A N/A N/A N/A N/A N/A
GPT-JT (6B) 20.0 0.4 0.8 N/A N/A N/A N/A N/A N/A
GPT-J (6B) 1.9 1.7 1.6 N/A N/A N/A N/A N/A N/A
GPT-NeoX (20B) 2.6 2.5 2.5 N/A N/A N/A N/A N/A N/A
BLOOM (7B) 3.3 4.6 3.7 1.0 1.8 1.3 N/A N/A N/A
T0++ (11B) N/A N/A N/A — — — — — —
LLaMa (65B) 10.0 11.1 10.4 2.2 3.8 2.8 N/A N/A N/A
Alpaca (7B) 2.1 3.2 2.4 0.6 0.7 0.6 N/A N/A N/A
J2-Jumbo-Instruct (178B*) 5.7 5.5 5.5 3.0 3.0 3.0 1.5 1.2 1.3
Cohere-command (52.4B) 5.2 1.9 2.8 5.2 6.6 5.8 N/A N/A N/A
GPT-3.5-turbo 10.4 9.5 10.0 11.9 10.6 11.2 4.1 2.0 2.7
GPT-3 curie v1 (6.7B) 1.3 1.7 1.4 N/A N/A N/A N/A N/A N/A
GPT-3 davinci v1 (175B) 3.6 3.9 3.7 N/A N/A N/A N/A N/A N/A
InstructGPT curie v1 (6.7B*) 2.4 1.3 1.6 0.3 0.2 0.2 N/A N/A N/A
InstructGPT davinci v2 (175B*) 6.0 7.9 6.8 13.2 13.9 13.6 5.3 4.6 5.0
GLM (130B) 7.9 10.5 8.8 2.8 4.6 3.5 2.8 4.6 3.5
GPT-4 12.4 14.5 13.4 35.5 29.2 32.0 19.8 13.6 16.1
ChatGLM (130B) N/A N/A N/A N/A N/A N/A N/A N/A N/A

31
 Table 30: Absolute Performance of all metrics on two sub-tasks of MAVEN (2-6), KU.
2-6
Model
Identification Classification
P R F1 P R F1
FLAN-T5 (11B) — — — — — —
UL2 (20B) — — — — — —
FLAN-UL2 (20B) N/A N/A N/A N/A N/A N/A
GPT-JT (6B) 21.8 12.1 15.6 15.1 8.4 10.8
GPT-J (6B) 25.0 8.4 12.5 13.9 4.7 7.0
GPT-NeoX (20B) 11.8 12.6 12.2 7.5 7.9 7.7
BLOOM (7B) 30.6 8.8 13.7 12.9 3.7 5.8
T0++ (11B) — — — — — —
LLaMa (65B) 52.9 16.7 25.4 13.2 4.2 6.4
Alpaca (7B) 48.6 7.9 13.6 25.7 4.2 7.2
J2-Jumbo-Instruct (178B*) 28.0 10.7 15.5 14.6 5.6 8.1
Cohere-command (52.4B) N/A N/A N/A N/A N/A N/A
GPT-3.5-turbo 54.7 34.9 42.6 25.5 16.3 19.9
GPT-3 curie v1 (6.7B) 52.2 5.6 10.1 17.4 1.9 3.4
GPT-3 davinci v1 (175B) 44.2 10.7 17.2 15.4 3.7 6.0
InstructGPT curie v1 (6.7B*) 100.0 0.5 0.9 100.0 0.5 0.9
InstructGPT davinci v2 (175B*) 52.4 45.1 48.5 30.3 26.0 28.0
GLM (130B) 52.5 19.5 28.5 11.2 4.2 6.1
GPT-4 66.3 58.6 62.2 40.5 35.8 38.0
ChatGLM (130B) 50.0 2.3 4.4 20.0 0.9 1.8

Table 31: Absolute Performance of all metrics on four sub-tasks of MAVEN-ERE (2-7), KU.
2-7
Model
Temporal Causal Subevent Coreference
P R F1 P R F1 P R F1 P R F1
FLAN-T5 (11B) — — — — — — — — — — — —
UL2 (20B) — — — — — — — — — — — —
FLAN-UL2 (20B) N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
GPT-JT (6B) — — — — — — — — — — — —
GPT-J (6B) — — — — — — — — — — — —
GPT-NeoX (20B) — — — — — — — — — — — —
BLOOM (7B) N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
T0++ (11B) — — — — — — — — — — — —
LLaMa (65B) N/A N/A N/A 33.3 5.0 8.7 N/A N/A N/A N/A N/A N/A
Alpaca (7B) N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
J2-Jumbo-Instruct (178B*) 10.7 3.4 5.2 25.0 30.0 27.3 22.2 20.0 21.1 25.0 20.0 22.2
Cohere-command (52.4B) 7.5 6.9 7.2 15.0 15.0 15.0 N/A N/A N/A 44.4 40.0 42.1
GPT-3.5-turbo 18.6 14.9 16.6 15.7 55.0 24.4 4.0 10.0 5.7 33.3 60.0 42.9
GPT-3 curie v1 (6.7B) 4.7 4.6 4.7 N/A N/A N/A N/A N/A N/A N/A N/A N/A
GPT-3 davinci v1 (175B) 0.3 1.1 0.5 N/A N/A N/A N/A N/A N/A N/A N/A N/A
InstructGPT curie v1 (6.7B*) 6.9 14.9 9.4 N/A N/A N/A N/A N/A N/A N/A N/A N/A
InstructGPT davinci v2 (175B*) 22.0 14.9 17.8 11.8 30.0 16.9 20.0 10.0 13.3 N/A N/A N/A
GLM (130B) N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
GPT-4 19.0 26.4 22.1 23.8 25.0 24.4 21.4 30.0 25.0 72.7 80.0 76.2
ChatGLM (130B) 18.7 3.4 5.8 N/A N/A N/A N/A N/A N/A 33.3 10.0 15.4

32
Knowledge Applying (KA). In the evaluation of the KA level, a notable phenomenon is that
knowledge graph (KG)-based reasoning question answering tasks are almost impossible to complete
without the use of KG. This phenomenon is evident in the three tasks (3-4)-(3-6). Furthermore, due
to the clear quality progression exhibited by multiple tasks in this layer, the performance of models
generally follows a decreasing trend.

Table 32: Absolute Performance of F1-score on task (3-1), (3-2) and (3-3), KA.
Model 3-1 3-2 3-3
FLAN-T5 (11B) 23.7 35.3 7.5
UL2 (20B) 9.2 16.3 4.2
FLAN-UL2 (20B) 27.1 34.0 10.6
GPT-JT (6B) 13.8 25.5 2.8
GPT-J (6B) 19.2 26.6 4.6
GPT-NeoX (20B) 1.0 3.6 N/A
BLOOM (7B) 4.4 11.4 N/A
T0++ (11B) 7.2 11.8 3.2
LLaMa (65B) 2.8 23.0 11.7
Alpaca (7B) 1.8 8.4 1.8
J2-Jumbo-Instruct (178B*) 23.9 19.6 6.9
Cohere-command (52.4B) 17.3 28.7 13.4
GPT-3.5-turbo 33.6 29.5 17.5
GPT-3 curie v1 (6.7B) 7.3 4.7 1.6
GPT-3 davinci v1 (175B) 4.2 3.2 2.7
InstructGPT curie v1 (6.7B*) 13.9 24.6 3.4
InstructGPT davinci v2 (175B*) 13.2 26.9 12.9
GLM (130B) 8.0 2.7 0.6
GPT-4 34.6 45.9 28.4
ChatGLM (130B) 17.4 21.8 5.2

Table 33: Absolute Performance of accuracy of different types of questions on KQA Pro (3-4), KA.
3-4
Model
All Multi. Quali. Comp. Logi. Count. Veri.
FLAN-T5 (11B) 20.0 21.3 19.4 29.4 15.4 N/A 64.3
UL2 (20B) 13.0 13.3 22.6 17.6 15.4 N/A 14.3
FLAN-UL2 (20B) 20.0 21.3 19.4 41.2 19.2 N/A 42.9
GPT-JT (6B) 10.0 12.0 19.4 N/A 15.4 N/A 28.6
GPT-J (6B) 19.0 17.3 22.6 35.3 19.2 N/A 42.9
GPT-NeoX (20B) 2.0 2.7 N/A N/A N/A N/A 14.3
BLOOM (7B) 3.0 4.0 6.5 N/A 3.8 N/A 7.1
T0++ (11B) N/A N/A N/A N/A N/A N/A N/A
LLaMa (65B) 6.0 8.0 12.9 5.9 11.5 N/A 7.1
Alpaca (7B) 2.0 2.7 N/A 5.9 N/A N/A N/A
J2-Jumbo-Instruct (178B*) 13.0 12.0 12.9 17.6 3.8 11.1 28.6
Cohere-command (52.4B) 19.0 21.3 22.6 23.5 23.1 11.1 50.0
GPT-3.5-turbo 17.0 16.0 19.4 41.2 19.2 N/A 21.4
GPT-3 curie v1 (6.7B) 2.0 1.3 3.2 5.9 3.8 N/A N/A
GPT-3 davinci v1 (175B) 3.0 2.7 3.2 N/A N/A N/A 14.3
InstructGPT curie v1 (6.7B*) 8.0 8.0 3.2 29.4 7.7 N/A 14.3
InstructGPT davinci v2 (175B*) 5.0 4.0 6.5 11.8 7.7 N/A N/A
GLM (130B) 4.0 5.3 6.5 N/A 7.7 N/A 7.1
GPT-4 10.0 12.0 16.1 5.9 7.7 N/A 42.9
ChatGLM (6B) 3.0 2.7 6.5 5.9 3.8 N/A N/A
ChatGLM (130B) 12.0 10.7 9.7 35.3 7.7 N/A 21.4

33
Table 34: Absolute Performance of all metrics on KoRC (3-5) and ETA (3-6), KA.
3-5 3-6
Model
EM F1 EM F1
FLAN-T5 (11B) 20.0 23.3 — —
UL2 (20B) N/A N/A — —
FLAN-UL2 (20B) 21.0 27.1 32.6 39.6
GPT-JT (6B) N/A 2.3 N/A 3.0
GPT-J (6B) N/A 1.4 N/A 2.7
GPT-NeoX (20B) 2.0 6.5 N/A 3.1
BLOOM (7B) 7.0 11.2 N/A 7.3
T0++ (11B) 18.0 23.4 — —
LLaMa (65B) 2.0 5.6 N/A 8.4
Alpaca (7B) 23.0 29.1 4.1 15.4
J2-Jumbo-Instruct (178B*) 6.0 12.1 2.0 4.9
Cohere-command (52.4B) 28.0 38.1 36.7 41.8
GPT-3.5-turbo 10.0 14.6 10.2 14.2
GPT-3 curie v1 (6.7B) 2.0 3.0 N/A 0.2
GPT-3 davinci v1 (175B) 3.0 5.3 N/A 1.5
InstructGPT curie v1 (6.7B*) 9.0 14.9 8.2 15.7
InstructGPT davinci v2 (175B*) 25.0 33.8 22.4 32.6
GLM (130B) 22.0 28.5 22.0 28.5
GPT-4 33.0 44.3 36.7 43.5
ChatGLM (6B) 2.0 3.3 N/A 8.7
ChatGLM (130B) 17.0 20.3 N/A N/A

34
Knowledge Creating (KC). Here, we present the scores of three sub-criteria used to calculate
the overall score for each model. If only these scores are considered, it is observed that some
well-regarded models such as GPT4 and GPT-3.5-turbo do not necessarily demonstrate superiority.

Table 35: Absolute Performance of the key metrics on Encyclopedia Creating task (4-1), KC.
Model ∂ (Tk , R) ∂ (T, R) ∂ (T, Tk )
FLAN-T5 (11B) 15.6 9.7 22.5
UL2 (20B) 23.3 18.3 31.6
FLAN-UL2 (20B) 17.3 8.7 14.5
GPT-JT (6B) 22.1 17.8 36.0
GPT-J (6B) 27.5 19.4 32.9
GPT-NeoX (20B) 28.2 19.2 31.6
BLOOM (7B) 25.4 19.5 44.6
T0++ (11B) 16.3 10.0 25.7
LLaMa (65B) 30.9 24.0 25.5
Alpaca (7B) 26.5 20.0 26.9
J2-Jumbo-Instruct (178B*) 27.5 17.8 17.0
Cohere-command (52.4B) 27.6 19.3 47.5
GPT-3.5-turbo 45.6 21.3 29.2
GPT-3 curie v1 (6.7B) 26.5 22.1 42.0
GPT-3 davinci v1 (175B) 28.0 21.7 33.7
InstructGPT curie v1 (6.7B*) 20.2 18.8 28.0
InstructGPT davinci v2 (175B*) 43.7 22.4 26.0
GLM (130B) 30.1 20.3 37.7
GPT-4 37.9 14.8 19.0
ChatGLM (6B) 20.0 15.4 35.3
ChatGLM (130B) 17.5 16.0 26.1

Table 36: Absolute Performance of the key metrics on ETC (4-2), KC.
Model ∂ (Tk , R) ∂ (T, R) ∂ (T, Tk )
FLAN-T5 (11B) 1.0 7.3 3.6
UL2 (20B) 1.2 15.6 5.6
FLAN-UL2 (20B) 9.1 6.1 12.1
GPT-JT (6B) 17.4 14.9 43.9
GPT-J (6B) 17.7 16.0 44.1
GPT-NeoX (20B) 18.0 16.5 42.3
BLOOM (7B) 19.5 16.4 44.3
T0++ (11B) 0.9 15.8 4.5
LLaMa (65B) 26.0 20.0 26.6
Alpaca (7B) 14.9 20.7 24.5
J2-Jumbo-Instruct (178B*) 27.0 17.8 15.4
Cohere-command (52.4B) 26.1 15.0 18.6
GPT-3.5-turbo 45.7 19.1 23.9
GPT-3 curie v1 (6.7B) 18.9 17.4 41.7
GPT-3 davinci v1 (175B) 24.0 20.3 39.0
InstructGPT curie v1 (6.7B*) 20.8 21.1 31.8
InstructGPT davinci v2 (175B*) 43.1 21.3 24.7
GLM (130B) 25.8 17.1 39.4
GPT-4 43.8 17.5 22.3
ChatGLM (6B) 17.7 14.9 31.1
ChatGLM (130B) 10.0 8.5 13.0

35