Maps Search Evaluation Guidelines

June 2024
1. Introduc on ..............................................................................................................................................5
[Link] Rating Tool ....................................................................................................................................................................................................5
[Link] Work low...............................................................................................................................................................................................11
[Link] Interface ................................................................................................................................................................................................13
[Link]-Level Navigational Result Question.................................................................................................................................................14
[Link]-Level Rating Checkboxes ...................................................................................................................................................................14
[Link] Relevance Rating ..................................................................................................................................................................................15
[Link] Relevance Demotion Checkboxes ...................................................................................................................................................17
[Link] Accuracy ...................................................................................................................................................................................................17
[Link] ..........................................................................................................................................................................................................17
2. Understanding User Intent .......................................................................................................................18
[Link] Types .......................................................................................................................................................................................................18
[Link] Expectation .......................................................................................................................................................................................22
[Link] Types .......................................................................................................................................................................................................22
[Link] Intent ..................................................................................................................................................................................................23
[Link] Location ................................................................................................................................................................................................24
[Link] Location ..............................................................................................................................................................................................24
3. Ra ng the Query-Level Naviga onal Result Ques on ...............................................................................25
4. Ra ng Result-Level Issues ........................................................................................................................25
[Link] Name/Title in Unexpected Language or Script.....................................................................................................................25
[Link] of Business/POI Results ........................................................................................................................................................................27
[Link] of Address Results ....................................................................................................................................................................................30
[Link]/POI is Closed or Does not Exist .............................................................................................................................................30
[Link]/Does Not Exist vs. Inaccurate Name and Address......................................................................................................................31
[Link] Relevance of Closed/Non-Existing Business/POI .......................................................................................................................31
[Link]/POI Status is PERMANENT_CLOSURE ................................................................................................................................31
[Link] Status Shown ..............................................................................................................................................................................................32
[Link] Shown as PERMANENT_CLOSURE .................................................................................................................................................32
5. Relevance ................................................................................................................................................32
[Link]-Result Connection..............................................................................................................................................................................33
5.1.1. General Connection .........................................................................................................................................................................................34
5.1.2. Abbreviation/Alternate Name Connection ..............................................................................................................................................34
5.1.3. Category Connection .....................................................................................................................................................................................34
5.1.4. Spell Correction Connection .........................................................................................................................................................................35
5.1.5. Transit Intent Connection ..............................................................................................................................................................................35
5.1.6. Special Character Connection ......................................................................................................................................................................35
5.1.7. Address-Result Connection ............................................................................................................................................................................36
5.1.8. Lack of Connection .........................................................................................................................................................................................36
5.2. Satisfying User Intent ....................................................................................................................................................................................38
[Link].........................................................................................................................................................................................................39
[Link] ................................................................................................................................................................................................................40
[Link] vs. Prominence ...............................................................................................................................................................................42
[Link] Possible Results ....................................................................................................................................................................................42
[Link] Possible Results.......................................................................................................................................................................................45
[Link] Possible Results and Greater Distance ..........................................................................................................................................46
[Link] Areas .........................................................................................................................................................................................................47
[Link] – User Intent Deviation ............................................................................................................................................................48
[Link] for User Inside and Outside Viewport ...................................................................................................................................49
[Link] Address Does Not Exist ................................................................................................................................................................51
[Link] vs. Municipality .....................................................................................................................................................................................52
[Link] Results ......................................................................................................................................................................................52
5.15. Multiple Transit POIs with the Same Name ......................................................................................................................................54
ti
ti
ti
f
ti
ti
[Link] and Transit Intent ...............................................................................................................................................................................55
[Link] Queries .................................................................................................................................................................................................56
[Link] vs. Stations ............................................................................................................................................................................................59
[Link] Intent .................................................................................................................................................................................................61
[Link]: Service-Level Mismatch ......................................................................................................................................................61
[Link] Relevance When the Result Status is PERMANENT_CLOSURE ...................................................................................64
[Link] PERMANENT_CLOSURE Status ..................................................................................................................................................64
[Link].Expected PERMANENT_CLOSURE Status: Navigational Query......................................................................................................66
[Link].Expected PERMANENT_CLOSURE Status: All chains closed in area ............................................................................................68
[Link] PERMANENT_CLOSURE Status ............................................................................................................................................69
[Link].Unexpected PERMANENT_CLOSURE Status: Open Options are Nearby ....................................................................................70
6. Name and Category Accuracy ...................................................................................................................71
[Link] Not Applicable (n/a) .........................................................................................................................................................................71
[Link] Names .....................................................................................................................................................................................................72
6.2.1. Correct Name ....................................................................................................................................................................................................72
6.2.2. Partially Correct Name ..................................................................................................................................................................................76
[Link]. Minor and Moderate Misspellings and Missing or Extra Words .....................................................................................................76
[Link]. Service-Level Mismatch ....................................................................................................................................................................................79
[Link]. Holding Name and Corporate Structure ....................................................................................................................................................79
6.2.3. Incorrect Name................................................................................................................................................................................................80
6.3 Result Category .................................................................................................................................................................................................82
6.3.1. Correct Category ..............................................................................................................................................................................................83
6.3.2. Incorrect Category..........................................................................................................................................................................................84
6.4. Can’t Verify .........................................................................................................................................................................................................86
6.5 Transit Names ...................................................................................................................................................................................................86
6.6. Parking Names .................................................................................................................................................................................................87
6.7. Final Name and Category Accuracy Rating (Summary Table).....................................................................................................88
7. Address Accuracy: Components and Issues ..............................................................................................89
[Link] Components .....................................................................................................................................................................................89
[Link] Number .....................................................................................................................................................................................................89
[Link]/Apt................................................................................................................................................................................................................91
[Link] Name ........................................................................................................................................................................................................92
[Link]-Locality and Administrative Subdivisions ..........................................................................................................................................93
[Link] .................................................................................................................................................................................................................94
[Link] or State...................................................................................................................................................................................................94
[Link] Code .........................................................................................................................................................................................................95
[Link] .................................................................................................................................................................................................................95
[Link] Does Not Exist .................................................................................................................................................................................96
[Link]/Script Issue in Address ............................................................................................................................................................96
[Link]-Speci ic Issue ...................................................................................................................................................................................97
[Link]-Speci c Issue vs. Correct with Formatting Issue .......................................................................................................................97
[Link] Issue ..........................................................................................................................................................................................................97
[Link] with Formatting Issue ...................................................................................................................................................................98
[Link]’t Verify ..........................................................................................................................................................................................................98
[Link]’t Verify for Street Number ........................................................................................................................................................................99
[Link]’t Verify for Street Name ...........................................................................................................................................................................99
8. Address Accuracy: Result Expecta ons ...................................................................................................100
[Link]/POI Result ....................................................................................................................................................................................100
[Link]/POI Address Research and Resources .......................................................................................................................................100
[Link] O cial Addresses ......................................................................................................................................................................101
[Link].Department Addresses ....................................................................................................................................................................................101
[Link].Result Missing the Street Address ..............................................................................................................................................................103
[Link].P.O. Box, Mailing Addresses, and Management Of ices ......................................................................................................................104
[Link]/POI with Moving Locations ....................................................................................................................................................105
[Link] .................................................................................................................................................................................................................105
[Link] Type Results ..................................................................................................................................................................................106

2
ffi
fi
f
ti
f
 [Link] Result..................................................................................................................................................................................................106
[Link] Formats ............................................................................................................................................................................................106
[Link] Result ....................................................................................................................................................................................................107
[Link] Result.................................................................................................................................................................................................107
[Link] Code Result .........................................................................................................................................................................................108
[Link]/Region/Territory Result .....................................................................................................................................................................108
[Link] Result.................................................................................................................................................................................................108
[Link] Without an Expected Address ..............................................................................................................................................108
[Link] Features .............................................................................................................................................................................................108
[Link] Without an Expected Address ...........................................................................................................................................................110
[Link].Minimum Address Component ....................................................................................................................................................................110
[Link].Of icial Address Present ..................................................................................................................................................................................111
[Link].No Of icial Address ............................................................................................................................................................................................113
9. Pin Accuracy ..........................................................................................................................................115
[Link] Considerations ..............................................................................................................................................................................116
[Link] Resources .....................................................................................................................................................................................................116
[Link] Imagery vs. Vector Map .................................................................................................................................................................116
[Link] of the Feature .............................................................................................................................................................................117
[Link].Half ’n Half and Tennis Rules.........................................................................................................................................................................118
[Link].Boundaries in Shared Spaces ........................................................................................................................................................................120
[Link] Door ..........................................................................................................................................................................................................122
[Link]’t Verify (Smallest Identi ed Area) .......................................................................................................................................................123
[Link] Rooftop .................................................................................................................................................................................................124
[Link] Buildings ...........................................................................................................................................................................................139
[Link] Property with Multiple Buildings .........................................................................................................................................141
[Link] Rooftops ..........................................................................................................................................................................................142
[Link]-Rooftop Rating ...............................................................................................................................................................................143
[Link] Rating...............................................................................................................................................................................................149
[Link].Single-Complex Campus ..................................................................................................................................................................................151
[Link].Dispersed Campus .............................................................................................................................................................................................154
[Link] Without a Rooftop ......................................................................................................................................................................157
[Link] Features ..............................................................................................................................................................................................157
[Link] Divisions ..............................................................................................................................................................................160
[Link] ................................................................................................................................................................................................................161
[Link] Lots and Structures .....................................................................................................................................................................163
[Link] POIs .....................................................................................................................................................................................................164
[Link], Tram, and Streetcar Stops ....................................................................................................................................................................166
[Link] POI with Multiple Rooftops/Platforms .......................................................................................................................................167
[Link] POIs with a Single Rooftop............................................................................................................................................................170
[Link] Transit Stations ....................................................................................................................................................................170
[Link].How to Establish an Entrance Polygon .....................................................................................................................................................171
[Link].Examples ................................................................................................................................................................................................................172
[Link] to Rate Results ..............................................................................................................................176
[Link] ic Address ..........................................................................................................................................................................................176
[Link]-Speci ic Address ................................................................................................................................................................................191
[Link] Address Does Not Exist ..............................................................................................................................................................199
[Link] of Interest ...........................................................................................................................................................................................201
10.5. Understanding Multiple Query Interpretations: Beyond Viewport and User Location ..............................................205
10.6. Business Queries........................................................................................................................................................................................208
10.6.1. Non-Chain Businesses ................................................................................................................................................................................209
10.6.2. Chain Businesses .........................................................................................................................................................................................212
10.6.3. Chain Business With Location Modi er ................................................................................................................................................216
[Link]. General Location Modi ier ..........................................................................................................................................................................216
[Link]. Speci ic Location Modi ier ..........................................................................................................................................................................224
10.6.4. Back O ce and Businesses with No Physical Location....................................................................................................................228
10.6.5. Business/POI Does Not Exist ...................................................................................................................................................................230
[Link] Queries ........................................................................................................................................................................................231
3
f
f
f
f
ffi
f
fi
f
f
fi
 [Link] for Distance with User Inside Viewport ...............................................................................................................................232
[Link] for Distance with User Outside Viewport ..........................................................................................................................234
[Link] for Distance with Fewer Possible Results...........................................................................................................................236
[Link] Queries and User Intent ...........................................................................................................................................................238
[Link] Queries and Prominence .........................................................................................................................................................239
[Link] Query with Location Modi er ................................................................................................................................................239
[Link] Results for Category Queries ............................................................................................................................................241
[Link] Categories ...........................................................................................................................................................................................245
[Link] Categories..............................................................................................................................................................................................246
[Link] Query: Parking ...........................................................................................................................................................................251
[Link] Results with PERMANENT_CLOSURE Status ...................................................................................................................255
[Link] PERMANENT_CLOSURE Status ...............................................................................................................................................255
[Link] PERMANENT_CLOSURE Status ........................................................................................................................................256
[Link] Query Types .....................................................................................................................................................................................258
[Link] Queries ............................................................................................................................................................................................258
[Link] and “My Location” Queries..................................................................................................................................................260
[Link] Results .........................................................................................................................................................................................262
[Link] Ra ng Tips ......................................................................................................................................269
[Link]’s the relevance when query is business/POI name and address and the result is only the address? .......269
[Link]’s the relevance when the query is a street name and the result is a single business on that street? ........269
[Link] do I know what the location intent is when user is outside/inside a fresh viewport?......................................270
[Link] do I know what the location intent is when user is outside/inside a stale viewport? ......................................271
[Link] full address result does not exist. How do I rate it? ...........................................................................................................271
Appendix: Release Survey ..........................................................................................................................273

4
ti
fi
1. Introduction
Welcome to Maps Search Evaluation. These guidelines will teach you how to use the evaluation tool
and rate Maps Search results.

For each task, you will be presented with a search query and the results that are provided to Maps
users. The goals of this task are:

• Search Relevance and Search 2.0: To determine how well the results satisfy the user’s
query (result relevance)
• Search 2.0 only: To check the data accuracy (name, address, and pin location) of the results

Note: Examples shown in this document reflected real locations at the time of publication. The
examples continue to demonstrate the rating principles described even if a business or point of
interest has changed its name.

1.1. The Rating Tool

This is the web-based rating tool used to rate Search 2.0 and Search Relevance queries.

5
Anatomy of the Tool

Name Screenshot Description

The task bar shows:

• Task Type: The name of the task.

• Task ID: A number that identifies the
specific task item. Use this number
whenever you communicate about a
task or rating.
Task Bar • Request ID: A reference number for
internal use only. Do not use this
number to communicate about a
task or rating.
• Estimated Rating Time: Time to
rate all results

Rating The Rating Guidelines shortcut button

Guidelines opens the guidelines for this task.

Allows you to skip a survey in rare cases

Release where a technical or other issue prevents
Survey you from rating. See Appendix for
complete instructions.
Viewport shortcut buttons provide
convenient controls to pan or zoom to
common map viewport settings. Show
Show User
User moves the result viewport to the
user’s location without changing the zoom
level (map scale).

Show Viewport moves the result

Show viewport to the center of the user
Viewport viewport and zooms in on the viewport
area.

Use Show All to return from any zoom

Show All level to the initial display showing the
user, the viewport, and all the results.

6
 Name Screenshot Description
The query header shows:
Query: The term the user entered into the
search bar.
Viewport Age: How long ago the user
moved the viewport.
• A fresh viewport is indicated by the
word FRESH displayed in green
text.
Query • A stale viewport is indicated by the
Header
word STALE displayed in red text.
Locale: The country and language
combination this query is for.
Country: Should match the country in the
Locale field. If it does not, defer to the
country found in the Locale field.
User Lat,Lng: The user’s location.

Top-Level A top-level question that must be

Question answered once for each query.

The result returned for a user query.

The color of the result heading

corresponds to the color of the pin on the
map that marks that result’s location. (See
Result Pin on page 9.)

The result address, category and status (if

included) will be shown beneath the result
Result
name/title.

You will evaluate the relevance of each

individual result. If you are rating Search
2.0, you will also rate the name/address/
pin accuracy for each result.

Note: A query can generate more than

one result.

7
 Name Screenshot Description

A result may also display its status. If the

status appears and is set to
Status PERMANENT_CLOSURE, the result
requires special treatment when rating
relevance.

` You may occasionally see one or more

normal looking or grayed-out results with
Result with
no rating fields beneath them. No rating
no Rating
is required for these results. This is
Choices
expected and is not a technical issue or
a reason to release the survey.

User viewport: A transparent purple box

centered over a device icon shows the
user’s viewport at the time the query was
issued (if known). The user viewport
represents the map area the user was
looking at when making the query.

The shape of the viewport may vary

depending on the size and orientation of
User
the device being used.
Viewport
Note: In some cases, the user viewport
may be so small that it cannot be seen,
but (if the viewport information is
available) the device icon will always be
visible, regardless of the map scale.

Note: The viewports you see in the

examples in these guidelines may be grey.

8
 Name Screenshot Description
This device icon shows the center of the
user viewport at the time of the user
query. The purpose of the device icon is
to make it easier to find the user viewport.
It does not indicate the user's location.

This icon should not be used for

Device Icon
measuring distance. To measure
distance, start from the outer edge of the
user viewport.

This icon has no rating significance. Its

only purpose is to help you find the
viewport on the map.
A blue and white icon indicates the user
location at the time of the query (if
known).
User
Location
Note: Clicking on the icon reveals the
coordinates (latitude/longitude) of the
user’s location.
A pin indicates a result’s location. The
head of the pin is only an indicator. The
tip marks the actual location.

Each pin on the map has a unique

number and color, which matches the
Result Pin number and color of its corresponding
result heading.(See Result, above)

Note: The pins you see in the examples

in these guidelines may look different
from the pin shown in this chart and the
ones in the tool.

9
 Name Screenshot Description
Clicking any individual result or pin causes
a pop-up box to appear over the pin. The
text within the box may show additional
information and should only be used for
research purposes to help understand the
result:
Pop-up
Information • Phone number (for research only,
Box never call a business)
• URL
• Coordinates (latitude/longitude)

Note: Do not rate the information in the

pop-up box. Rate only the information in
the result.

Map Layer Allows you to select which map layer is

Selector displayed.

Allows zooming in and out on the map.

Zoom
Change the zoom level by clicking the +
Controller
(zoom in) or – (zoom out) buttons.

Copy to
Click to copy information to a clipboard
Clipboard

Click the pin icon at the top right of the

map to place a pin anywhere on the map.

Drop
Location Pin

A purple pin and a pop-up window

showing the pin’s address and latitude/
longitude will appear.

10
 Name Screenshot Description

To measure the distance, click the ruler

icon at the top right of the map.
Measure
Distance
You can now drop a start pin (green) and
an end pin (red). The tool will display the
distance between them.

Draw Tool Click to draw custom polygons.

Click Submit to save your ratings and

Submit
display the next available task.

[Link] Workflow

Complete each step of the following workflow before you assign a rating.

Evaluate and rate each rating drop-down independently of the others. Rate result Relevance based
on whether the result fits the query intent, even if the result is closed or does not exist or has
incorrect Name, Address, or Pin data. If you are rating Search Relevance you will stop after you’ve
rated result relevance. If you are rating Search 2.0, you will continue by rating Name and Category
Accuracy, Address Accuracy, and Pin Accuracy individually.

Note: For any result in a Search 2.0 project, you may be asked to make all of the possible ratings, some of
them, just one, or none. This is expected and not a reason to escalate or release a task.

Rate each result on its own merits. Do not demote duplicate or very similar results.

11
12
1.3. Rating Interface

Your ratings and comments will be collected in the rating interface shown below. You will first answer
one top-level question about the query itself, then rate each individual result using the other
checkboxes and drop-downs.

13
1.3.1. Query-Level Navigational Result Question
You will answer a top-level question for each query to indicate whether there are any results in the
real world that would completely satisfy the user intent.

Query-level question Explanation Notes

Is there a navigational result Are there any results in the real You will only need to answer
for this query? world that can completely this question once for each
satisfy the user intent? query.

Answer to Query-Level
Query Explanation
Question
[Eiffel Tower] Yes There is only one result in the
real world that can completely
satisfy the user intent.
[Starbucks] No There is not only one result in
the real world that can satisfy
the intent of this query.

Note: If you are presented with a query that has no results, you must still answer this question and
submit your answer.

1.3.2. Result-Level Rating Checkboxes

For each result, you will be presented with two high-level questions. These will allow you to highlight
any problems with the result.

High-level
Explanation Notes
question
Result is not in the language of the If you check this box, your rating for this
Result name/
query, the test locale, or the result result will be complete (all other questions
title is in
region (the expected languages) or is will be unavailable).
unexpected
in a mix of expected and unexpected
language or
languages (unless it is an official
script
company, chain, or brand name).

14
 High-level
Explanation Notes
question
If you check this box, only the relevance
needs to be rated.

Pay attention to the status of the POI. If

the result’s status is
PERMANENT_CLOSURE, refer to 5.19
Rating Relevance when Status is set to
PERMANENT_CLOSURE and 10.8
Permanently Closed Results for rating
Business/POI Research shows that the business is
instructions.
is closed or closed or that no such business exists
does not exist at the address listed in the result.
If the status is blank or displays any other
text, rate the relevance of the result as if it
did exist.

Use thorough research and your best

judgement to determine whether the
result is closed, has never existed, or the
result component(s) are [Link]
Unclear Results for further guidance.

1.3.3. Result Relevance Rating

The relevance rating will take into consideration user intent as well as distance/prominence.
Relevance is always rated independently of any data (name/category, address, or pin) inaccuracies.
This means that when rating relevance we always assume that the result exists (even if research
reveals the location is closed) and that the data presented is correct.

Always rate against the real world: If there is a better result available but it is not shown, demote the
existing result(s) while considering the missing one(s).

15
 Relevance
Explanation Notes
Rating

The most likely result implied by the user’s

query, location, and/or viewport that
completely satisfies the user’s distinct
intent.
A distinct user intent is defined by a A rating of Navigational for one
combination of the following result does not prevent other
Navigational characteristics: results from satisfying the query to
• Extreme prominence (e.g. Eiffel Tower) a lesser degree.
• Uniqueness (e.g. complete address)
• Proximity (e.g. incomplete address that
points to an unambiguous option that is
is extremely close)

Highest possible initial rating for

A high-quality result that clearly satisfies the
ambiguous queries as well as for
Excellent user’s intent. Multiple results can be
queries that are not eligible for a
considered high quality.
Navigational rating.
Only partially satisfies the user’s intent due
Good
to relevance, prominence, or distance.
Any relevance rating of Good or
Technically satisfies the user’s intent but below requires an indication of the
Acceptable
does so poorly due to relevance or distance. reason(s) for the demotion:
• User Intent Issue
Does not satisfy the user’s intent due to lack • Distance/Prominence Issue
Bad of relevance or great distance (when closer
satisfying results are available).

16
1.3.4. Result Relevance Demotion Checkboxes

If you give a result a Relevance rating of Good or below, you will be presented with two checkboxes.
Use them to select the reason(s) for your rating:

• User Intent issue

• Distance/Prominence issue

You must also leave a comment explaining your rating.

Checkbox Explanation Notes

User Intent Select when the demotion is due to a result only
issue partially fulfilling the query intent.
Select when the demotion is due to a distance
issue based on the user or viewport location or If both issues were reasons for
Distance/ when you demoted for a result that is less your relevance rating, select both
Prominence prominent based on the initial query intent. You checkboxes.
issue can also select this option when the query asks
for something at a specific location and the
result is not at or near the requested location.

1.3.5. Data Accuracy

If you are rating Search 2.0, once you've rated relevance, you’ll evaluate three result components for
data accuracy: Name, Address, and Pin.

1.3.6. Comments

Comments are mandatory for any result relevance rating of Good or below and for any data issues
not rated Correct/Perfect and should explain the reasons for all rating demotions. Be sure to:

• Include user intent: Indicate what you think the user intent is.
• Refer to the guidelines: If the demotion is due to a guideline instruction, note the guideline
section you used to make your decision.
• Include sources: If you find incorrect information in a result, your comment should include
the correct information and its source. Be sure that any links you include lead directly to the
information you provide. If they don’t, include a link to the source you used and explain how to
get to the information from there.
• Use URL shorteners when inserting long links into your comments
• Be concise: Don’t write elaborate comments. Be short and specific.
• Comment in English: Your comments will sometimes be reviewed or used by people who are
not familiar with your test locale or its language. Comments in English can be understood by
the widest possible audience.

17
 Issue Expectations Example Rating and Comment
The comment should explain the Relevance -> Good (Distance/
reasoning behind your rating. A demotion Prominence)
Relevance may be due to distance from the user’s Comment: Demoted -1 for a few closer
Demotions location compared to similar results/ real world locations in the Middletown/
potential results, secondary intent, or Anchorage area
other reasons.
Address Accuracy -> Incorrect — Street
Number
Include a comment when you rate a
Comment: Correct street number is: 332
Name, Address, or Pin something other
W El Camino Real, Sunnyvale, CA 94087
than Correct or Perfect or you give a
Data Issues as found on the official site: https://
rating of Closed/Does not Exist or
[Link]/locations/sunnyvale
Unexpected Language.
Pin Accuracy -> Wrong
Comment: Wrong pin: correct location is:
52.368067, 4.895389

2. Understanding User Intent

Before evaluating any results, you’ll need to understand what the user wanted to find when they
typed in their query. To determine user intent, use a combination of your own research and local
knowledge plus the information found in the rating interface, which includes:

• User location
• User viewport
• Viewport age

User intent refers to the interaction between query, user location, user viewport, and viewport age.

2.1. Query Types

The queries you are evaluating are real queries submitted by users. Because they are real, they can:

• Be misspelled
• Contain mistakes
• Be inconsistent
• Be ambiguous
• Be badly formatted
• Be, or contain, emojis
• Have no maps intent

The first step in the rating process is to make sure you understand the query.

If you see a foreign-language query, do not release the survey. Research the query or use an online
translation tool to find out what it means, then rate it as usual.

18