Project Overview

Welcome to the Egocentric Annotation Program

Page 1 of 9

What is This Project?

This project facilitates a human egocentric video annotation workflow. You will be annotating
videos showing humans completing physical tasks from a first-person (ego) perspective.

Understanding Segments

Each video is divided into multiple events (segments): distinct actions performed by egos (human
operators with first-person cameras) as they complete specific tasks (e.g., sewing, cleaning gym
equipment, etc.) within an environment.

Your Role

As a reviewer, you are responsible for reviewing text annotations (labels) segment-by-segment and
correcting labels when necessary.

The goal is to ensure the ego's main actions and the objects they are interacting with are accurate,
as well as the timestamps for each segment.

Do NOT label the ego's movement through a location (walking, navigating, etc.).

Important: The guidelines in this document take precedence over any other labeling instructions. All
project communication is conducted via our Discord server.

Key Focus Areas

Focus On

Main Actions

Focus on the primary task being performed

Hand Dexterity

Hands and their meaningful object interactions

Don't Focus On

Movement Through Space

Walking/navigation is not the focus

Idle Hand Gestures

Gestures unrelated to the work environment

Step-by-Step Instructions

How to review and edit annotations

Page 2 of 9

1. Reserve a batch of episodes from the Task page

Click 'Reserve Episodes' to get a batch of episodes assigned to you

2. Click on the Review button to start reviewing

Click 'Review' to start reviewing a video (episode)
3. Review and edit segments one at a time

Click on each segment entry box to review and edit the label.
The main review interface showing video player and segment list

Click on a segment to select it for editing

Editing Controls

Click on the icons on the right side or use hotkeys to edit, split, delete, or play a segment.

Action icons: Edit (pencil), Split (scissors), Delete (trash), Play

Use the - and + buttons to edit the duration of a segment (timestamps).

Adjust segment duration in 0.2 second increments

Keyboard Shortcuts

EEdit label

SSplit segment

DDelete segment

PPlay segment

LLoop video

SpacePlay/Pause

EnterSave edits

EscCancel

What to Verify

Label accurately reflects the main actions involving hand dexterity

Label accurately describes the main objects being interacted with

Label is not missing any important actions or objects

Timestamps are correct and align with the described actions

Unrelated movements are labeled as "No Action"

Submit Your Review

After reviewing all segments and checking for typos, click the blue Complete button to submit your
review.

Click 'Complete' to submit your review

Core Mental Model

Definitions and what requires a label
Page 3 of 9
Definitions
Episode
A full video task.
Segment
A continuous time span paired with one label.
Core Mental Model
A segment represents one continuous interaction with a primary object toward a single goal.
A segment typically begins when the hands engage the primary object and ends when that interaction is
complete, when the hands disengage, or when the interaction focus or goal changes.
A change in primary object or interaction focus may justify a split, even if the hands remain in contact with
an object.
No hand contact → No Action (default)
✅ What Requires a Label
Goal-oriented hand–object actions that matter to the task
❌ Do NOT Label (as actions)
Walking / navigating through space
Idle gestures
Looking / visually examining (no "inspect", "check")
Unrelated actions (e.g., adjusting camera, checking phone when not important to task)
"Reach" (see Action Verb Rules section)
❌ Do not use numerical characters (e.g., 1, 2, 5th, 10th).
Use words ("three") or omit quantities when not required.
Label Format Rules
How to write consistent, natural labels
Page 4 of 9

2.1 Imperative Voice

 Write labels as commands:
pick up spoon
place box on table
2.2 Consistency within an episode
Use consistent verbs and nouns where possible:
If you choose "wash," don't alternate with "clean" without reason.
2.3 Action separators
When multiple actions are in one label, separate actions with comma or and:
pick up cup, place cup on table
pick up cup and place cup on table
pick up cup place cup on table
(Last example fails because it has no separator)
2.4 No numerals
pick up 3 knives
pick up three knives
pick up knives
2.5 No intent-only language
Don't add mental-state intent that isn't a physical action.
 • Prefer the physical verb that occurred.
 • Example: instead of "prepare to cut tape," label the actual observed action ("pick up scissors", "cut
tape", etc.).
Dense vs. Coarse Labels
Understanding label granularity
Page 5 of 9

3.1 Segment Rule

A segment is either Dense OR Coarse – do not mix within a single segment.

3.2 Video Rule

A video may contain both dense and coarse segments.
3.3 When to use Coarse vs. Dense
Use coarse when:
 • A clear goal exists, AND
 • Listing atomic steps risks errors/hallucination, or
 • The atomic steps are too many to list safely
Use dense when:
 • Multiple distinct hand actions are required to be accurate (no single goal verb fits).
Dense is not "better" than coarse. Coarse is often preferred for accuracy.

3.4 Length as a Granularity Signal

As a general guideline, a single segment label should usually contain no more than ~20 words or ~4
atomic actions.
 • This is not a strict limit.
 • Long labels increase the risk of inaccuracies or hallucinated steps.
 • If a label is becoming long, consider whether a coarse label would be more accurate.
 • Accuracy and completeness always take priority over label length.
 Action Verb Rules
 Verb definitions and usage guidelines
 Page 6 of 9
 4.1 Verbs you must NOT use
 inspect / check
  reach (except only if the action is truncated/cut off at the end of the episode and no better verb is
possible)
 If "reach" feels necessary, timestamps are usually wrong. Fix timestamps instead.
 4.2 Verb definitions
Verb Meaning Audit Notes

pick
Object leaves a surface/container resting position Required when usin
up

place Object contacts a surface and is released/positioned Required when usin

Coarse relocation describing pick up + place as one goal, OR repositioning without ✅ Allowed coarse s
move
detailing steps the goal

adjust Small corrective change in position/orientation Use instead of insp

hold Maintain grip without relocating Only if task-relevan

grab Grip itself is meaningful Rare; use sparingly

 4.3 "Move" clarification (important)

 Yes — "move __" can represent "pick up and place __" as a coarse label when the goal is
relocation.
 move mat to table (coarse)
 move box onto shelf (coarse)
 If you choose dense, you must be explicit:
 pick up mat, place mat on table (dense)
 4.4 Attach verbs to objects
 Every verb should clearly apply to an object.
 pick up, place on table
 pick up cup, place cup on table
 No Action & Object Rules
 When to use No Action and how to describe objects
 Page 7 of 9
 5.1 When to use "No Action"
 Use No Action only when:
 Hands touch nothing, OR
 Ego is idle / doing irrelevant behavior unrelated to the task
 5.2 No Action Rules
 Do not split solely to isolate "No Action" pauses.
 Do not combine "No Action" with real actions in a single label.
 Do not use "No Action" if the ego is holding an object and that hold is task-relevant.


 6.1 Identify only what you can defend

 • If unsure, use a general noun ("tool", "container", "cloth").
 • Incorrect object naming must be fixed (spoon vs fork, blue vs orange).
 6.2 Consistency
 Stay consistent in object naming through the episode where possible.
 6.3 Adjectives
 Use adjectives only to disambiguate (two similar items):
 blue cloth vs white cloth
 pick up cloth
 6.4 "Place" requires a location (can be general)
 place cup on table
 place cup in bin
 Overly specific location is not required.
 6.5 Left/right
 Allowed if accurate from ego view, but not required.
 6.6 Body parts
 Avoid referencing body parts unless unavoidable.
 wash spoon
 Acceptable if needed: "apply glue to shoe with finger" (if it's the only clear description)
 Segment Editing Rules
 Timestamps, Merge, Split, and Delete
 Page 8 of 9
 7.1 Timestamps
 Start:when the action begins (hands begin engaging toward contact to cover the full interaction)
 End:when hands disengage and the interaction ends
 Minor idle time inside the segment is acceptable if the segment still represents one continuous
interaction.
 7.2 Extend / Shorten
 Use to align boundaries to the true action.
 Don't extend into a new action.
 Don't cut off completion of the action.
 7.3 Merge (when allowed)
 Merge adjacent segments only if:
 Same action/goal, AND
 Hands never disengage between them.
 7.4 Do NOT merge
 Do not merge when:
 There are repeated pick up → place cycles with clear disengagement
 Different objects or different goals
 7.5 Split (when required)
 Split when:
 Hands disengage and a new interaction begins, OR
 A new goal/action begins that must be labeled separately
 Final Rules & Checklist
 Repeated actions, audit fail conditions, and ideal segment
 Page 9 of 9
 8.1 Repeated actions
 If the ego disengages and repeats, they are separate segments (unless "move"/coarse goal safely
covers a continuous relocation objective without changing interaction boundaries).
 If the ego never disengages, it is one segment (often coarse).
 8.2 Simultaneous actions
 If multiple task-relevant actions happen in the same segment, include them either:
 • As a coarse goal label, OR
 • As dense enumerated actions
 …but do not invent steps.

  9. Audit Fail Conditions
 A segment fails audit if any of the following are true:
 Missed major task-relevant hand action
 Hallucinated (non-occurring) action/object
 Timestamps cut off the action or include a different action
 Forbidden verbs used ("inspect/check", "reach" except truncated-end exception)
 Dense/coarse mixed in one label
 "No Action" combined with action


 10. Ideal Segment Checklist

 One goal
 Full action coverage
 Accurate verbs
 No hallucinated steps
 Dense OR coarse (not mixed)
 Remember: Quality over quantity. A well-labeled segment accurately captures the main hand-
object interaction from start to finish, using clear and consistent language.

References

Reference & Edge Case Library

Resolve disputes and rare scenarios
Reference 1 of 6

Purpose: This section helps resolve disputes and handle rare scenarios.
Not required for memorization — use as reference.

1. Canonical Mistakes → Correct Interpretation

❌ "Don't annotate minute actions"
Wrong. Missed actions are a major mistake.
 • If an object is picked up and placed → it must be captured
 • (dense or coarse via move)
❌ "Dense is always better"
Wrong.
 • Coarse is preferred when it reduces hallucinations
 • Dense is required only when no single goal verb is accurate
❌ "Always split out No Action"
Wrong.
 • No Action inside a segment is acceptable
 • Splitting just to isolate pauses is unnecessary
Move vs Pick Up + Place
Edge clarification for relocation actions
Reference 2 of 6

2. Move vs Pick Up + Place

move is valid when:
 • The goal is relocation
 • Intermediate steps add no value
 • Accuracy improves by abstraction
 Examples:
move mat to table
move eggs in crate
Use pick up + place when:
 • Order matters
 • You are labeling densely
 • Multiple discrete interactions must be explicit
Merge vs Split Decision Flow
When to merge or split segments
Reference 3 of 6

3. Merge vs Split — Decision Flow

Decision Tree:
1

Hands disengage?
• Yes → split
• No → continue
2

Same goal?
• Yes → merge / keep
• No → split
3

Different object?
• Yes → split
Never merge:

 • Repeated pick up → place cycles with disengagement

 • Different goals "just to reduce count"
Repeated & Simultaneous Actions
Handling multiple or repeated actions
Reference 4 of 6

4. Repeated & Simultaneous Actions

Repeated Actions
Disengage + repeat → multiple segments

Never disengage → one segment

Simultaneous Actions
Capture all task-relevant actions

Ignore irrelevant side actions (phone, camera touch, etc.)

Objects, Ambiguity & Body Parts

Handling unclear objects and body part references
Reference 5 of 6
5. Objects & Ambiguity
Use general nouns if unsure (tool, container)
Adjectives only to disambiguate
Color not required unless needed
Left/right acceptable from ego view
"Place" always needs some location (general is fine)

6. Body Parts as Tools (Edge Case)

Avoid mentioning body parts unless unavoidable.
Preferred:
apply glue to shoe
Acceptable if needed:
dip finger in glue and apply to shoe
Formatting & When to Escalate
Common formatting mistakes and escalation guidelines
Reference 6 of 6
7. Common Formatting Mistakes (Minor)
Do not use numerical numbers:
pick up 3 knives from table
pick up three knives from table
pick up knives from table

8. When to Escalate
Escalate via Discord if:
Object cannot be identified after reasonable effort
Action cannot be labeled without guessing
Segment cannot be made accurate via coarse abstraction
You've completed the Reference & Edge Case Library!
Use this section as a reference when you encounter edge cases during review.