E-commerce Interaction Understanding -
Intent Classification
Step 1 - Input
Create example text to be classified or processed. This text should come
from the perspective of a customer as they attempt to interact with a
customer service agent.
Step 2 - Context
Previous conversation before the input, user and agent’s turns.
Step 3 - Reasoning Trace
This is a step-by-step explanation of how the agent should arrive at the
answer. All reasoning traces must:
• Contain 3-7 logical steps.
• Show clear progression from input analysis to conclusion.
• Highlight key information and its relevance.
• Explain any ambiguities and how they are resolved.
• Demonstrate the logical decision-making process.
Step 4 - Ground Truth
1. Product_Information_Request - General questions about product
details, specifications, or materials
2. Price_Inquiry - Questions specifically about the current price of
products
3. Availability_Check - Inquiries about whether items are in stock or
available for purchase
4. Comparison_Request - Requests to compare features or benefits
between two or more products
5. Return_Policy_Question - Questions about how to return items or return
policy details
6. Warranty_Inquiry - Questions about warranty coverage, duration, or
claims process
7. Order_Status_Check - Inquiries about tracking or the current status of
an order
8. Product_Complaint - Expressing dissatisfaction with product quality or
performance
9. Feature_Inquiry - Specific questions about particular product features
or capabilities
10. Compatibility_Question - Questions about whether products work with
specific systems or other products
11. Size_Guidance_Request - Questions about sizing charts, fit, or
appropriate sizes
12. Installation_Help - Requests for assistance with installing or setting up
products
13. Troubleshooting_Request - Help with diagnosing or fixing issues with
products already owned
14. Payment_Issue - Problems related to payment processing, charges, or
payment methods
15. Shipping_Question - Inquiries about shipping methods, costs,
timeframes, or policies
16. Account_Issue - Problems accessing or managing customer accounts
17. Recommendation_Request - Asking for product suggestions based on
needs or preferences
18. Discount_Inquiry - Questions about sales, promotions, coupons, or
special offers
19. Cancellation_Request - Requests to cancel an order that hasn't shipped
yet
20. Exchange_Request - Inquiries about exchanging items for different
sizes, colors, or models
21. Gift_Option_Inquiry - Questions about gift wrapping, gift receipts, or gift
cards
22. International_Shipping_Question - Specific inquiries about shipping to
countries outside domestic service
23. Bulk_Order_Inquiry - Questions about ordering in large quantities or
wholesale
24. Customization_Question - Inquiries about personalizing or customizing
products
25. Subscription_Management - Questions about managing recurring
orders or subscriptions
26. Product_Restock_Inquiry - Questions about when out-of-stock items will
be available again
27. Store_Location_Inquiry - Questions about physical store locations or
hours
28. Loyalty_Program_Question - Inquiries about rewards programs, points,
or member benefits
29. Technical_Specification_Question - Detailed questions about product
specifications or technical aspects
30. Authenticity_Verification - Inquiries about product authenticity or how
to verify genuine items
31. Other - In case none of the above intents matches the conversation
Step 5 - Metadata
For each example you create, include metadata such as:
• Categories under Diversity Requirements where applicable (such as
product categories, customer issues, conversation length, linguistic
complexity, and so on) where applicable.
• Complexity level (see below).
• Difficulty level (see below).
• Any other relevant information about the data creation.
Best Practices
To ensure the success of this project, follow these best practices.
Create Challenging Examples
To create challenging examples:
• Use product descriptions with subtle differences between options.
• Include ambiguous customer statements that could map to multiple
intents.
• Create scenarios where context from previous turns is essential.
• Introduce rare edge cases that test classification boundaries.
• Use domain-specific terminology that requires specialized knowledge.
• Include examples where the appropriate next action changes based on
conversation context.
• Ensure some examples feature sensitive scenarios that require careful
handling.
• Include examples where "Unknown_At_Current_State" is the correct
category, especially for initial customer contacts.
• Create examples showing how different customer tones (frustrated,
confused, pleased) might influence the next action.
• Include scenarios that demonstrate when automated handling is
sufficient versus when human intervention is necessary.
What to Avoid
Samples will be rejected if they:
• Contain inappropriate content, hateful language, or illegal activities.
• Have reasoning traces that do not logically support the ground truth.
• Contain factually incorrect product information.
Metadata
Product Categories
Create samples across each of the following product categories:
• Electronics: Smartphones, laptops, headphones, cameras, gaming
consoles, smart home devices, wearables, TVs, audio equipment
• Fashion: Clothing, shoes, accessories, jewelry, active wear, formal
attire, seasonal items, undergarments
• Home Goods: Furniture, kitchen appliances, decor, bedding, bathroom
fixtures, cleaning supplies, storage solutions
• Groceries: Fresh produce, pantry staples, frozen foods, specialty
ingredients, beverages, snacks, dietary-specific foods
• Beauty & Personal Care: Skincare, makeup, hair-care, fragrances, oral
care, personal hygiene, men's grooming
• Health & Wellness: Vitamins, supplements, fitness equipment, medical
devices, first aid supplies, mobility aids, health monitors
• Toys & Games: Board games, action figures, educational toys, puzzles,
outdoor play equipment, collectibles, hobby kits
• Books & Media: Books, e-books, magazines, music albums, movies,
video games, digital downloads, audiobooks
• Automotive: Car parts, accessories, maintenance products, tools, car
electronics, motorcycle gear, vehicle care products
• Pet Supplies: Pet food, toys, beds, grooming products, carriers, training
aids, aquarium supplies, pet medications
• Sports & Outdoors: Sporting goods, camping gear, exercise equipment,
outdoor recreation, bikes, water sports, team sports equipment
• Baby & Kids: Baby gear, children's clothing, toys for specific age
groups, nursery furniture, feeding supplies, diapers
• Office & School Supplies: Stationery, desk organization, office furniture,
art supplies, educational materials, printing supplies
Customer Issues
Create content across various customer issues including:
• Product defects
• Sizing issues
• Compatibility problems
• Delivery delays
• Missing items
• Account issues
• Website navigation problems
• Product comparisons
• Feature questions
Language Variations
Create content with language variations including:
• Formal language
• Informal language
• Technical jargon
• Slang
• Grammatical variations
• Abbreviated text
• Emoji usage
Cultural Contexts
Create content with various cultural contexts including:
• Region-specific product references
• Holiday-related purchases
• Cultural celebrations
• International shopping terminology
Conversation Length
Distribute the conversation lengths of your examples as follows (applicable
to multi-turn scenarios):
• Short (1-3 turns): 40%
• Medium (4-9 turns): 40%
• Long (10+ turns): 20%
Linguistic Complexity
Include a range of linguistic styles in your examples including:
• Simple, direct requests.
• Complex queries with multiple parameters.
• Ambiguous requests requiring clarification.
• Technical specifications and jargon-heavy text.
• Casual, conversational language with colloquialisms.
Examples
Below are both good and bad examples related to this project:
Good Example 1
Input: "The left earbud of the wireless headphones I bought last week isn't
charging properly."
Context: [Previous conversation about wireless headphones purchase]
Reasoning Trace:
1. The customer mentions a specific product: wireless headphones
2. They're describing an issue with the product's functionality
3. Specifically, one earbud isn't charging properly
4. This is a technical issue with a purchased product
5. The appropriate classification is Troubleshooting_Request
Ground Truth: Troubleshooting_Request
Metadata: Electronics, Moderate complexity, Level 1 difficulty
Note: For context “[Previous conversation about wireless headphones
purchase]”, this is expected to be replaced with made-up customer
conversations, like: “Customer: Hi. I have a problem with product xyz.
Assistant: Understood, please tell me ... Customer: ...” If Input is the first
interaction, then Context can be None.
Bad Example 2
Input: "Do these headphones work with iPhone?"
Context: None
Reasoning Trace: It's about headphones
Ground Truth: Product_Question
Metadata: Electronics, Simple, Level 1
[FEEDBACK: This example is inadequate because it uses a vague intent class
"Product_Question" instead of the more specific "Compatibility_Question".
The reasoning trace is too brief and doesn't show the logical steps. The
context is missing previous conversation turns that would be valuable.]
Self-Review Checklist
IMPORTANT: Each task must fully meet all the requirements specified for each
item in the Quality Metric. If any requirement is not met, the task will be marked
as Failed.
Quality Metric
Fulfille
Item Requirement Explanation d?
Contex The context must clearly
t belong to the e-commerce
domain. It should not be a
E-commerce Interaction tech support scenario or a
generic set of questions
one might search on
Google or ChatGPT.
The conversation should
reflect a genuine e-
commerce interaction.
Think like a real customer.
Realistic Conversation
What would you ask an e-
commerce agent, and how
would the agent
realistically respond?
Natural Conversational Flow The dialogue must be
logical, coherent, and easy
to follow.
The agent’s inputs must be
professional,
Agent Inputs are free of Grammar
grammatically correct, and
Errors, Typos, Colloquial Language
free from informal
expressions.
The agent must provide
factual, accurate and
Agent is Truthful
consistent information with
no contradictions.
The agent’s responses
should directly address the
Agent is Helpful customer’s request,
providing complete and
useful information.
Context must be about real
Real Product product, no fictional
products are allowed
Conversation must match
metadata perfectly. For
example, if metadata calls
Match Metadata
for Informal language, User
inputs must be clearly
informal.
The input should represent
a natural next line in the
conversation coming from
Logical and Realistic Continuation
User, right after the
of Context
context. It must be
consistent with a real-life
e-commerce scenario.
Input Input must match
metadata perfectly. For
example, if metadata calls
for Simple, direct requests,
Match Metadata
User prompt must be
clearly simple and direct
with no complex multi-
question inputs.
Ground GT must be logically tied to
Truth all other elements of the
Consistent with Context & Input
task -
context/input/metadata
Justifiable by the Reasoning Trace GT must be logically
explained in RT
Most Appropriate Classification GT must be undoubtfully
(not just an acceptable one) correct
Each reasoning trace must
include between 3 and 7
3-7 Logical Steps
distinct, logical steps. No
more, no less.
The reasoning should
demonstrate a logical
analytical sequence of
Clear Progression from Input
input and context (with the
Analysis to Conclusion.
main focus on input)
leading to the final
decision - ground truth.
Emphasize all the crucial
Highlight Key Information details that inform the
decision.
Reason The analysis must be in-
In Depth Analysis of Input
ing depth, no superficial points
Trace All points must serve to
Free of Unrelated or Repetitive proving the ground truth,
Points avoid repetitive and
irrelevant information.
Clearly address any
uncertainties in the
Explain Any Ambiguities
input/context and explain
how they are resolved.
Do not rely on a single
format or repeated
Avoid Patterns
phrasing when writing
reasoning traces.
Reasoning traces should
Avoid "We" & "I" Pronouns remain objective and
impersonal.
Product Category match
Customer Issues match
Must directly and
Metada Cultural Contexts match
undoubtfully match
ta Language Style match context/input
Linguistic Complexity match
Context Turns match
All elements of the task align clearly and consistently with one another.
Each component - context, input, GT, RT, and metadata form a
cohesive, logically connected, and realistic representation of an e-
commerce interaction.