Direct Answer Blocks Task Guide

Assessing the Helpfulness of Direct Answer Blocks

Task Instructions

In this task, you will be given

● a query,
● one or more Direct Answer Blocks, which are results that provide answers to that query,
and
● other results that may be used as reference.

Your job is to

1. Consider whether the query likely implies a direct request for information that can be
satisfied by a Direct Answer Block.
2. Also consider whether the query has multiple interpretations.
3. Assign any flags, defined below, as appropriate.
4. Using the Answer Satisfaction scale defined below, rate how well each Direct Answer Block
responds to the query. For this rating scale, please:
● Base your rating on the answers in the Direct Answer Blocks, using the site title and
URL for additional context.
● Use the other results for reference, if you are unsure whether an answer is correct.
● Consider whether the query is seeking an answer or information (if not, a Direct Answer
Block would likely be rated no higher than Slighty Answers)

If the SCRB alone should receive a low Answer Satisfaction rating, do not raise the rating to
reflect the landing page quality.

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 Answer Satisfaction Scale

Rating Explanation

The block's answer correctly answers the query,

immediately and fully, for all or almost all users. If the
Fully Answers (FullyA) query has multiple interpretations, all interpretations
are covered by the block. Very few users would wish
to see additional results to answer their question.

The block's answer correctly answers the query in a

way that would be very helpful for many or most
Highly Answers (HA) users. If the query has multiple interpretations, all
interpretations are covered by the block. Some users
may wish to see additional results.

The block's answer would be helpful for many users

OR very helpful for some users. It might give a partial
or clearly incomplete answer to the query. It may also
Moderately Answers (MA) lack context that connects the block's answer to the
query. Some or many users may wish to see
additional results.

The block content would be helpful for few users

issuing this query either because the

● query is not directly requesting an answer or

information, or
Slightly Answers (SA) ● the block’s answer addresses related topics but
does not directly answer the query.

Many or most users would wish to see additional

results.

The block content would be helpful for very few or no

users. This can be because:
Fails to Answer (FailsA) ● the block content is incorrect or misleading for
the query
● the block content addresses a different query

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 Flags

Please assign any result or query flags when appropriate. If a query flag is applicable, please
assign the flag to all results. Make sure to read the examples to better understand when to
assign these flags.

● Result flags
○ Foreign Language, Upsetting-Offensive: Please assign these flags as defined in the
General Guidelines.
● Query flags
○ False Premise: The user's query assumes something untrue, such as asking when
something happened that did not actually happen.
○ User Error: The query contains an obvious typo or similar small mistake. This can
include a substitution that leaves the intent clear, such as "Greg Washington" in
place of "George Washington."
○ No Clear Question: The query does not seem to suggest a question. It might be a
Website query, such as a company or website name.
○ Can't Understand Intent: It is difficult or not possible to understand the intent of the
query.

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 Examples
Query: Older Living President
Flag: User Error

Note: For a query like this, assume minor human error. It is more likely that the user meant [oldest
living president], so treat the query as [oldest living president]

Direct Answer Block Rating

The oldest living U.S. president is Jimmy The text in this block directly and fully answers
Carter, born October 1, 1924. On March 22, the user's question so it is Fully Answers.
2019, he also became the nation's longest-lived
president, surpassing the lifespan of George H.
W. Bush, who died at the age of 94 years, 171
days.
List of Presidents of the United States by age - Wikipedia, the free
...
[Link]/wiki/List_of_Presidents_of_the_United_States_by
_age

The oldest living U.S. president is Jimmy The text in this block directly answers the user's
Carter. question so it is Highly Answers. Some users
might want to learn more context, putting this
List of Presidents of the United States by age - Wikipedia, the free
... beneath FullyA.
[Link]/wiki/List_of_Presidents_of_the_United_States_by
_age

On March 22, 2019, Jimmy Carter also became The text in this block seems to directly answer
the nation's longest-lived president, surpassing the user’s question, but lacks context and could
the lifespan of George H. W. Bush, who died at be from a passage that actually answers
something different. The user would have to
the age of 94 years, 171 days.
view additional results to be certain.
List of Presidents of the United States by age - Wikipedia, the free
...
[Link]/wiki/List_of_Presidents_of_the_United_States_by
_age

Jimmy Carter was born October 1, 1924. This answers a different but related question,
that of Jimmy Carter’s age. It does not directly
List of Presidents of the United States by age - Wikipedia, the free
...
address the user's query about who the oldest
[Link]/wiki/List_of_Presidents_of_the_United_States_by living president is.
_age

Joseph R. Biden, elected President in 2020 This answer looks plausible, but is wrong.
(born November 20, 1942), is 78 years old. Incorrect information is rated Fails to Answer,
even though the landing page includes the
correct answer. It might be necessary to
examine other results to determine whether this
List of Presidents of the United States by age - Wikipedia, the free
...
answer is incorrect.
[Link]/wiki/List_of_Presidents_of_the_United_States_by
_age

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 Examples
Query: Symptoms of Cancer
Flag: <None>

Direct Answer Block Rating

It would be very difficult for an answer block to FULLY

ANSWER this query, because the answer is broad and
is not easily summarized.

Cancer can cause many symptoms, but these This is a broad request for medical information. Different
symptoms are most often caused by… other problems. users will have different needs. The passage explains
If you have symptoms that do not get better after a few generally that there are a lot of variability and
weeks, see your doctor so that problems can be complexity, and also lists some specific examples of
diagnosed and treated… Some symptoms of cancer symptoms, making it very useful for most users.
include:

● Breast changes
● Bladder changes
● Bleeding or bruising

Symptoms of Cancer - National Cancer Institute

[Link]/about-cancer/diagnosis-staging

Cancer can cause many symptoms, but these This passage explains generally that there is a lot of
symptoms are most often caused by… other problems. variability and complexity to the answer, but does not
If you have symptoms that do not get better after a few provide any specific examples. While some users might
be satisfied, those who wanted to see specific examples
weeks, see your doctor so that problems can be
of symptoms would need to view other results.
diagnosed and treated as early as possible. Some
symptoms of cancer include: …

Symptoms of Cancer - National Cancer Institute

[Link]/about-cancer/diagnosis-staging

Common symptoms of lung cancer include:

● Coughing
● Shortness of Breath (Dyspnea)
● Wheezing The answer is for a specific, overly-narrow interpretation
● Hoarseness or Change in Voice of the query, and is not likely to be useful to most users.
● Chronic Fatigue
● Fever

Lung Cancer Symptoms - Coughing Up Blood, Wheezing

[Link]

Common symptoms of cancer include: The answer describes the symptoms for lung cancer but
does not mention they are specifically for lung cancer.
● Coughing The user could get confused and think they are
● Shortness of Breath (Dyspnea) symptoms for all cancer, so it should be rated as Fails
● Wheezing to Answer.
● Hoarseness or Change in Voice
● Chronic Fatigue
● Fever

Lung Cancer Symptoms - Coughing Up Blood, Wheezing

[Link]
TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.
 Examples
Query: facebook login
Flag: No Clear Question

Note: This is almost certainly a query looking for the page to log in to Facebook, rather than asking
a question.

Direct Answer Block Rating

No Block could answer this query since it is not

a question.

Log into Facebook to start sharing and It is unlikely that the query is seeking
connecting with your friends, family, and people information, but even if it were, the answer
you know… Log In. Forgot Account? Sign up for provided would be helpful for few users. This is
Facebook. English (US) Slightly Answers even though the landing
page is likely correct, because the rating applies
to the answer itself.

Log into Facebook

[Link]/login

With the new login, people can select the data It is extremely unlikely that the query is seeking
an app gets by unchecking categories of this (or any) answer, so it is Fails to Answer
information, one by one. For example, if
someone wants to share their email address
with an app, but not their birthday, they can
make a choice with a couple of taps.

Introducing Anonymous Login and an Updated Facebook Login

[Link]
and-an-updated-facebook-login

TELUS International Digital Solutions This material contains information that is proprietary and confidential to TELUS International. It cannot be
shared with third parties without TELUS International written consent. DO NOT COPY. DO NOT DISTRIBUTE.