Fact Extraction Judgment Guidelines
1 Task Overview
The goal of this task is to evaluate if the fact which is extracted from the Input passage is accurate or not.
A fact contains Triple of Subject, Predicate and Object value:
Subject is some entity which holds the object value;
Predicate is the property which indicates the relationship between the subject and the object value.
Once you log in to UHRS and have launched the HitApp, you will see a screen similar to this example below:
The HitApp shows two columns side-by-side:
• On the top of left is a short Input passage. Bellow the Input passage is the fact extraction result. The extraction
result contains Subject, Predicate and Object:
• On the right is a pane. It contains an information card for the Subject entity (People, Place, Organization, Event,
etc.)
You need to judge How the Object value matches the Predicate and Subject entity card according to the Input passage.
2 Judgment Instructions
2.1 Process Steps
1. Read the Input passage.
2. Read the Object Value and find the object value in the Input passage. Determine if the object value is valid and
complete. A valid object value could be any string with meaningful information.
a. It could be but not limited to: numbers, dates, locations, names. Good examples are: “Oct 12 th, 2019”,
“Seattle, WA”, “1.75 meter”, “2 kilo grams”.
b. Bad value could be some generics words like: “is”, “a”, “his”, “a piece of”.
c. The value should also cover the complete information. Examples are:
i. The Input passage contains string “1.25 inch”, then the object value “1.25” should be considered an
incomplete value.
ii. If “Tom cruise” is in the Input passage, then the object value “Tom” is incomplete.
iii. If “January 2019” is in the Input passage, then the object value “2019” is incomplete.
d. The value should not contain too much noise. For example, “Tom’s height is 6 feet” is not a good object value.
e. Small value string boundary issue should be ignored. For example, object value like “April, 23 1982)” or “8 feet.”
should be considered as good Object value.
3. Read the Predicate Description: Determine if the predicate description is accurate to indicate the property of the object
value. Bad descriptions could be but not limited to:
a. Type error. For example, the value is “1.75 meter” but the description is “weight of a person”.
b. The unit such as “kilometers”, “kilograms” in the predicate description should be ignored. For example, it should
not be considered as inaccurate predicate when the value is “150 pounds” but the predicate description is “weight
of the person in Kilograms”.
c. It is not the property of the object value. For example, if the value is “Oct 12 th, 2016” and in the Input passage it is
a release date of a music album but the description is “person’s birthday”, the object value should be considered
to not match the predicate even if the type matches.
4. Read the Subject entity card. Determine if the Object Value belonged to this entity according to the Input passage.
a. The Subject entity name or part of the subject entity name should appear in the input passage.
b. The Subject entity card should match the input passage. For example, If the entity card is about “Melania Trump”
but the Input passage is only talk about “President Trump”, it should not be considered as a match.
c. The Object value should relate to the Subject entity card.
5. Choose the first option below which applies:
a. Object value is invalid or incomplete: The object value is invalid or incomplete.
b. Object Value does not match the predicate:
i. The object value is valid and complete.
ii. The predicate Description is not accurate for the object value.
c. Object Value does not match the subject:
i. The object value is valid and complete.
ii. The predicate description is accurate for describing the property of the object value.
iii. The subject entity card does not match the object value.
d. All subject, predicate and object value are consistent with the input passage:
i. The Object value is valid and complete.
ii. All subject, predicate and object value are consistent with the input passage.
iii. The subject entity card matches the object value.
6. Click “Submit”.
3 Rating Examples
Input Passage Subject Predicate Object Label Comment
About Mark Salling’s weight: Mark Salling Property relating an 175 All subject, The predicate
Mark weighs in at around 175 mso:[Link] to its weight pounds predicate and should be
pounds which is in kilograms object value are consider as
average for his height. consistent with accurate even
the input though the
passage. weight unit of
the
Object and
Predicate
description do
not match
"I'll Be" is a song recorded by Ill Na Na A property linking an album to the Foxy All subject,
American rapper Foxy Brown for primary recording artist of the Brown predicate and
her debut studio album Ill Na Na album. Note that this property object value are
(1996), featuring American should be single-valued, except in consistent with
rapper Jay-Z . the case of split albums. Note too the input
that this property should not be passage.
used in cases of compilation albums
featuring various artists.
Nightmare Beach (also released Nightmare A property that relates a film to its Umberto All subject,
as Welcome to Spring Break) is a Beach director. Lenzi and predicate and
1988 American-Italian slasher Harry object value are
film directed by Umberto Lenzi Kirkpatrick consistent with
and Harry Kirkpatrick, and the input
starring Nicolas de Toth, Sarah passage.
Buxton, John Saxon , and
Michael Parks .
The Orobians are a ska band The A property that relates a dated 1997 Object Value "1997 " is found
founded during 1997 in Bergamo Orobians location to the date that the does not match date for a band
, Italy and still based in location came into existence, e.g. the predicate but not a
Bergamo . They play ska and the date that Czechoslovakia split location.
rocksteady standards (mostly into the Czech Republic and
instrumental) with a modern jazz Slovakia.
twist.
In 2001, Saharneanu founded Moldova A property that relates a dated 2001 Object Value "2001" is the
Gazeta Romaneasca [The location to the date that the does not match found date of
Romanian Gazette], a weekly location came into existence, e.g. the predicate the magazine
magazine sold in Moldova that the date that Czechoslovakia split but not the
compiled articles from major into the Czech Republic and country.
Romanian media outlets. The Slovakia.
project failed primarily because
of financial reasons.
Still on My Mind is the fifth Girl Who Got A property that relates an album to 8 March Object value 8 March 2019 is
studio album by English singer Away the date that the album was first 2019 does not match the release date
Dido , released on 8 March 2019 released. the subject of album "Still
through BMG . It is her first on My Mind"
studio album since 2013's Girl
Who Got Away .
Among the most prominent Shanghai Height of structure. For Skyscrapers 492 Object value "492 metres
examples are the Jin Mao Tower Tower this should not include antennae or metres does not match (1,614 ft)" is the
and the taller Shanghai World spires. See theskyscraper type (1,614 ft) the subject height of
Financial Center , which at 492 Shanghai World
metres (1,614 ft) tall is the third Financial Center
tallest skyscraper in mainland but not
China and ranks tenth in the Shanghai Tower
world. The Shanghai Tower ,
completed in 2015, is the tallest
building in China, as well as the
second tallest in the world. With
a height of 632 metres (2,073 ft),
the building has 128 floors and a
total floor area of 380,000
square metres
(4,100,000 sq ft) above
ground.
The first ITTF International A property that relates a dated 1926 Object Value is “1926” is an
international tournament was Table Tennis location to the date that the invalid or
held in January 1926 in Berlin Federation location came into existence, e.g. incomplete
while the first World Table the date that Czechoslovakia split incomplete
Tennis Championships was held into the Czech Republic and object value.
in December 1926 in London. Slovakia. The complete
object value
should be
“January 1926”
or “December
1926”
--You are ready judge now---