Chapter : The Testing Lifecycle: Unit Testing vs.

System
Testing
Objective: Understand the testing lifecycle for AI agent applications, distinguish between unit
and system testing in Copilot Studio, and learn how to validate agentic workflows, custom tools,
and context-passing mechanisms.

Core Concepts & Theory

As AI agents become more autonomous and integrated into complex enterprise systems, quality
assurance (QA) becomes essential. Ensuring that your multi-agent ecosystems, prompt
evaluations, and external data integrations (such as MCP servers) behave reliably requires a
structured testing strategy.

1. The AI Agent Testing Lifecycle

 Development & Evaluation: Local testing of individual components such as system

prompts, custom tools, and API schemas.
 Unit Testing: Validating individual components in isolation to ensure they function
correctly before being orchestrated by the master agent.
 System Testing: Testing the end-to-end interactions across the entire ecosystem,
including master-to-subordinate agent handoffs, context handoffs, and multi-turn
conversations.

2. Unit Testing in Copilot Studio

Unit testing focuses on the smallest testable parts of your copilot application:

 Topic-level Testing: Verifying that a specific topic triggers under the correct conditions
and produces the expected response.
 Schema Validation: Testing custom tool inputs and outputs (e.g., verifying that an MCP
server returns the correct JSON schema to prevent parsing errors).
 Prompt Evaluation: Testing the prompt templates used by the AI model to ensure
consistency and prevent hallucinations.

3. System Testing in Copilot Studio

System testing evaluates the agent ecosystem as an integrated whole:

[Link]
  Generative Orchestration Testing: Confirming that the master agent correctly selects
the right topic or tool from a set of available sub-agents.
 Context Preservation: Checking that variables (such as [Link] or
[Link]) are passed correctly between different topics and agents.
 Performance Testing: Measuring the turn-around latency and token consumption during
multi-turn interactions.

Practical Case Study

Case Study: Northwind Electric Utility Assistant

Scenario: Northwind Electric provides an automated customer support copilot designed to

handle billing issues, submit outage reports, and check account status.

The Challenge: The team performed unit tests on the billing topic and the outage topic
individually, and both passed. However, during system testing, when a customer asked about an
outage and then immediately asked about a bill, the context variable was overwritten, which
caused the bot to route the user to the wrong sub-topic.

The Solution: The development team implemented an automated system-level test using
conversation transcripts.

 They isolated the conversation variables and added test assertions.

 The tests confirmed that context is properly cleared and reset when the user switches
topics.
 This update prevented data leakage across turns, ensuring that the assistant resolves both
queries correctly in sequence.

Step-by-Step Example
Unit Testing an MCP Tool Output with Python

This example demonstrates how to write a unit test to validate the schema and output of a custom
tool or MCP server response before deploying it to Copilot Studio.

Step 1: Initialize the Testing Environment

Ensure you have the necessary testing packages installed:

[Link]
Bash
pip install pytest

Step 2: Create the Test Module

Create a file named test_agent_tools.py containing the unit test for an inventory checking
tool.

Python
import pytest

# Mock function simulating the response from an MCP tool or API action
def check_inventory(item_id: str) -> dict:
"""
Mock function to simulate inventory validation.
"""
inventory_db = {
"SKU-100": {"name": "Solar Panel", "stock": 45, "status":
"Available"},
"SKU-200": {"name": "Inverter", "stock": 0, "status": "Out of Stock"}
}

if item_id not in inventory_db:

return {"error": "Item not found."}

return inventory_db[item_id]

# Test Case 1: Validate item existence and stock levels

def test_valid_item_stock():
result = check_inventory("SKU-100")
assert result["status"] == "Available"
assert result["stock"] > 0

# Test Case 2: Validate edge case for out-of-stock items

def test_out_of_stock_item():
result = check_inventory("SKU-200")
assert result["status"] == "Out of Stock"
assert result["stock"] == 0

# Test Case 3: Validate handling of non-existent items

def test_invalid_item():
result = check_inventory("SKU-999")
assert "error" in result

Step 3: Run the Unit Tests

Execute the tests using the pytest runner in your development pipeline:

Bash
pytest test_agent_tools.py

[Link]
Multiple-Choice Questions (MCQs)
Question 1

Which type of testing is most appropriate for validating the schema and output of a single custom
tool or MCP server action before integrating it into a master agent? A. System Testing

B. Unit Testing

C. User Acceptance Testing (UAT)

D. Penetration Testing

 Correct Answer: B
 Explanation: Unit testing focuses on the individual components of the system in
isolation to ensure they function correctly on their own.

Question 2

During system testing of a multi-agent ecosystem in Copilot Studio, what is the primary risk of
not validating context handoffs between agents? A. The LLM consumes fewer tokens.

B. The application scale-to-zero feature is triggered prematurely.

C. State variables can be overwritten, which can cause the agent to route the user to the incorrect
sub-topic.

D. The API endpoint returns an unencrypted connection error.

 Correct Answer: C
 Explanation: Without proper variable tracking and state management, values can be
overwritten between topics, leading to incorrect routing and poor user experience.

Question 3

Which testing practice helps identify issues with generative orchestration, such as when a master
agent incorrectly routes a user's request to a sub-agent? A. Static Code Analysis

[Link]
B. System-level integration testing

C. Load testing the network interface

D. Compiling the local database schema

 Correct Answer: B
 Explanation: System-level integration testing evaluates the interactions between
components, ensuring that orchestration and context-passing work correctly when
combined.

[Link]
Chapter: Creating Representative Test Sets for AI
Evaluation
Objective: Understand the methodology for designing, curating, and executing representative
test sets to evaluate AI agent behavior, prevent hallucinations, and ensure robust performance in
Microsoft Copilot Studio.

Core Concepts & Theory

In generative AI and multi-agent ecosystems, evaluating performance requires more than just
testing standard happy paths. Creating a Representative Test Set ensures that your system is
evaluated across the variety of inputs it will encounter in production.

1. Components of an Evaluation Set

 Golden Dataset: A curated collection of real-world user queries paired with the expected
or ideal responses (ground truth).
 Edge Cases: Inputs containing ambiguity, compound intents, out-of-domain requests, or
unusual vocabulary.
 Adversarial Prompts: Tests designed to provoke hallucinations, prompt injections, or
inappropriate behavior.
 Negative Test Cases: Scenarios where the agent should decline to answer rather than
providing incorrect information.

2. Key Evaluation Metrics

 Groundedness: Measures if the response is derived exclusively from the agent's

knowledge base.
 Relevance: Evaluates whether the generated response directly answers the user's prompt.
 Completeness: Determines if all parts of the user's multi-part question were addressed.

3. Continuous Evaluation in the Enterprise

[Link]
Using continuous evaluation pipelines lets developers run regression tests when upgrading
system prompts, ensuring that improvements do not negatively impact the performance of
existing scenarios.

Practical Case Study

Case Study: Contoso Retail Support Bot

Scenario: Contoso Retail uses a generative assistant in Copilot Studio to handle customer
service queries regarding product returns and shipping policies.

The Challenge: The team noticed that while the bot performed well on standard inquiries, it
frequently hallucinated return windows for international orders, leading to customer complaints
and increased support ticket volumes.

The Solution: The QA team built a representative test set using historical transcripts and
designed adversarial prompts.

 They added a test suite that included inquiries about international shipping, bulk returns,
and holiday exceptions.
 They ran the test suite across several system prompt iterations, identifying and
eliminating the hallucination points.
 This decreased the agent's error rate by 42% on edge cases.

Step-by-Step Example
Creating a Representative Test Set Evaluation using Python

This example demonstrates how to write a Python script that evaluates a model's responses
against a Golden Dataset using a simple validation metric.

Step 1: Define the Golden Dataset

Python
golden_dataset = [
{
"id": "1",
"input": "What is the return policy for electronics?",
"expected_response_keyword": "30 days"
},

[Link]
 {
"id": "2",
"input": "Can I return an item without the receipt?",
"expected_response_keyword": "original receipt"
}
]

Step 2: Define the Evaluation Function

Python
def evaluate_agent(input_text, agent_response, expected_keyword):
"""
Evaluates the agent response against the expected keyword.
"""
passed = expected_keyword.lower() in agent_response.lower()
return {
"input": input_text,
"response": agent_response,
"passed": passed
}

Step 3: Run Evaluation and Analyze Results

Python
def run_tests():
# Mock agent responses
test_runs = [
{
"input": "What is the return policy for electronics?",
"response": "The electronics return policy allows returns within
30 days of purchase.",
"expected": "30 days"
},
{
"input": "Can I return an item without the receipt?",
"response": "You can return items but you must bring your
original receipt.",
"expected": "original receipt"
}
]

results = []
for run in test_runs:
res = evaluate_agent(run["input"], run["response"], run["expected"])
[Link](res)

success_rate = sum(1 for r in results if r["passed"]) / len(results) *

100
print(f"Evaluation Completed. Success Rate: {success_rate}%")
return results

if __name__ == "__main__":

[Link]
 run_tests()

Multiple-Choice Questions (MCQs)

Question 1

What is the primary purpose of a "Golden Dataset" in the AI evaluation lifecycle? A. To test the
processing power of local hardware.

B. To provide a baseline of user queries paired with the expected or ideal responses.

C. To generate random user queries without any expected outcome.

D. To replace the master agent in Copilot Studio.

 Correct Answer: B
 Explanation: A golden dataset establishes the ground truth by providing known inputs
and expected outputs, which allows you to measure consistency and accuracy.

Question 2

Which metric measures whether the agent's response is derived exclusively from the provided
data sources or knowledge base? A. Groundedness

B. Token Consumption

C. Turn-Around Latency

D. Handoff Success Rate

 Correct Answer: A
 Explanation: Groundedness ensures that the generated text does not hallucinate facts and
stays true to the provided context.

Question 3

Why are adversarial prompts included in a representative test set? A. To slow down the server
response time.

[Link]
B. To test the system's ability to decline answers or resist prompt injections.

C. To increase token consumption.

D. To simulate normal, repetitive, and uninteresting user patterns.

 Correct Answer: B
 Explanation: Adversarial prompts push the limits of the system to ensure it is secure
against prompt injections and can gracefully handle harmful or invalid inputs.

Chapter : Choosing Evaluation Methods: Manual vs. AI-

Assisted
Objective: Understand the differences between manual spot-checking and AI-assisted evaluation
methods in Copilot Studio, and learn when to apply each approach to ensure robust AI agent
validation.

Core Concepts & Theory

As AI agents take on more responsibilities across enterprise applications, verifying their
performance becomes a crucial quality assurance step. Copilot Studio allows developers to test
these systems using both manual validation and built-in AI-assisted evaluations. Choosing the
right method depends on the complexity of the task, processing scale, and the need for
explainability.

1. Manual Evaluation Methods

Manual testing involves subject matter experts (SMEs), developers, or QA analysts directly
interacting with the agent to verify its outputs.

 Pros: Highly intuitive, allows for fine-grained nuance checking, and is excellent for
evaluating user experience (UX) and overall tone.

[Link]
  Cons: Time-consuming, subjective, and struggles to keep pace with large knowledge
bases. Manual evaluation cannot easily scale to hundreds of test cases.

2. AI-Assisted Evaluation Methods

AI-assisted evaluation uses automated graders built into Copilot Studio (or via the Microsoft
Foundry tools API) to measure specific performance metrics against a golden dataset.

 Pros: Scalable, repeatable, and fast. It allows teams to test up to 100 scenarios at once
without requiring human intervention for every turn.
 Cons: Relies on the consistency of the judge model. Edge cases with complex irony or
highly domain-specific sarcasm might still require human review.

3. Comparing Manual and AI-Assisted Evaluation

Feature Manual Evaluation AI-Assisted Evaluation

Throughput &
Low (few scenarios per hour) High (hundreds of test cases in minutes)
Scale

Variable (subject to human High (uses objective evaluation

Consistency
fatigue/mood) methods)

Evaluation Tone, empathy, UX flow, and Groundedness, relevance, completeness,

Dimensions layout and similarity

Cost & Effort High human resource cost Low operational cost per test suite

Practical Case Study

Case Study: Northwind Health Insurance

Scenario: Northwind Health uses a multi-turn Copilot Studio assistant to help employees
navigate benefits policies, out-of-network claims, and regional coverage limitations.

[Link]
The Challenge: When the company updated its benefits documents, the QA team needed to
verify that the agent was not hallucinating old policies. Manual testing required reading through
hundreds of transcripts, which slowed the deployment of critical updates.

The Solution: The QA team implemented a dual-method evaluation process:

1. AI-Assisted Evaluation: They set up a golden dataset with 100 common and edge-case
questions. The built-in General Quality and Groundedness Graders were used to evaluate
the responses against the knowledge base automatically.
2. Manual Evaluation: SMEs reviewed a subset of 10 conversations that failed the AI-
assisted checks to ensure the tone and clarity were appropriate.

This combination reduced testing time by 80% while ensuring high accuracy.

Step-by-Step Example
Implementing an AI-Assisted Evaluation Request

This example shows how to validate an agent's response using an API call to evaluate its
groundedness and relevance against a golden dataset.

Step 1: Initialize the Evaluation Payload

Define the payload used to send the agent's output to the evaluation service.

Python
import json
import requests

def run_ai_evaluation(agent_response, reference_context, input_query):

"""
Simulates a call to an AI-assisted evaluation endpoint (like Microsoft
Foundry)
to assess groundedness and relevance.
"""
url = "[Link]
headers = {
"Content-Type": "application/json",
"Authorization": "Bearer YOUR_EVAL_API_KEY"
}

payload = {
"query": input_query,
"response": agent_response,
"context": reference_context,

[Link]
 "metrics": ["groundedness", "relevance"]
}

# In production, this would be a real POST request:

# response = [Link](url, data=[Link](payload),
headers=headers)
# return [Link]()

# Mocking the response for educational demonstration

return {
"evaluation_run": "test_run_019",
"results": {
"groundedness": {"score": 0.95, "reasoning": "Response is fully
supported by context."},
"relevance": {"score": 0.88, "reasoning": "Response directly
addresses the user query."}
}
}

if __name__ == "__main__":
query = "What is the deductible for the standard plan?"
context = "Standard Plan Details: The annual deductible is $500 for
individual coverage."
response = "For the standard plan, your annual individual deductible is
$500."

result = run_ai_evaluation(response, context, query)

print([Link](result, indent=2))

Step 2: Review the Evaluation

1. Extract the scores to determine if the agent response passed the minimum quality gates.
2. Use the reasoning output to understand which components of the response need
improvement.

Multiple-Choice Questions (MCQs)

Question 1

What is the primary benefit of using AI-assisted evaluations over manual evaluation for testing
large agent sets in Copilot Studio? A. AI evaluations can fully replace the user acceptance testing
(UAT) phase without human oversight.

B. AI evaluations can process large test sets quickly and provide objective scoring for
dimensions like groundedness.

[Link]
C. AI evaluations require more processing power from the client's local computer.

D. AI evaluations work without an internet connection.

 Correct Answer: B
 Explanation: Automated evaluation allows developers to scale testing to dozens or
hundreds of test cases without manual effort.

Question 2

Which metric is best suited to determine whether the model hallucinated or included information
not present in the reference documents? A. Turn-around Latency

B. Groundedness

C. Exact Match Rate

D. Intent Recognition

 Correct Answer: B
 Explanation: Groundedness ensures that the agent's response is supported by the
provided data sources and knowledge base.

Question 3

In what scenario would a development team choose to use Manual Evaluation over AI-assisted
evaluation? A. When testing 10,000 variations of user queries at the same time.

B. When evaluating if the response tone aligns with company culture and user experience
guidelines.

C. When checking JSON schemas returned by an MCP server.

D. When calculating the average turn latency across all users.

 Correct Answer: B
 Explanation: Human judgment is better suited for assessing subjective factors like tone
and user experience, which are harder for an AI evaluator to quantify.

[Link]
Chapter : Debugging Topic Flows and Generative Answers
Objective: Understand how to identify, trace, and resolve issues within topic triggers, variable
handoffs, and Generative Answers in Microsoft Copilot Studio, ensuring accurate and compliant
AI responses.

Core Concepts & Theory

Building conversational AI agents requires a systematic approach to quality assurance and
debugging. In Microsoft Copilot Studio, developers must frequently troubleshoot why a bot fails
to trigger the correct topic, how it processes variables, and why it might hallucinate or pull
incorrect data during Generative Answers.

1. Topic Triggering and Intent Recognition

[Link]
When an end-user inputs a prompt, the agent's natural language understanding (NLU) engine
evaluates the user's intent against trigger phrases.

 NLU Matching Issues: If trigger phrases are too narrow or overlap with other topics, the
agent may fire the wrong topic or default to the conversational system topic.
 Debugging Strategy: Use the Test bot panel's Track between topics feature to see which
trigger phase node was matched.

2. Generative Answers Node Troubleshooting

The Create Generative Answers node allows the copilot to extract answers from specified data
sources (such as SharePoint or custom websites). Common errors include:

 Data Source Mismatch: The model searches the wrong index or ignores the uploaded
documents.
 Low Groundedness Scores: The response relies on the public foundational model rather
than the custom data source.
 Authentication Failures: The data source connection uses improper credentials,
resulting in a fallback message.

3. Copilot Studio Diagnostics Toolkit

 Variable Watch Window: Inspects the current values of session and global variables as
the user moves between nodes.
 Error Handling Nodes: Captures system exceptions and routes users to a human agent
or a recovery topic.

Practical Case Study

Case Study: Cloud 9 Retail Assistant

Scenario: Cloud 9 Retail operates a customer service agent built in Copilot Studio. The bot is
designed to handle returns and check shipping statuses.

The Challenge: Customers asking about the "holiday return policy" were provided with the
wrong information. Instead of using the company's internal return documents, the bot generated
an answer using standard web data, which resulted in a compliance error.

The Solution: The development team used Copilot Studio's conversation diagnostics and the
variable watch window to trace the error:

[Link]
  They isolated the Generative Answers node and discovered that the SharePoint connector
had lost its indexing authorization.
 They reconfigured the authentication scope and added explicit data source filtering to the
Generative Answers node.
 This ensured that the agent only searches the approved enterprise knowledge base.

Step-by-Step Example
Validating and Logging Generative Answer Payloads with Python

This example demonstrates how to write a script that validates the sources and confidence levels
of a Generative Answer response before surfacing it in Copilot Studio.

Step 1: Initialize the Environment

Ensure you have the requests library installed:

Bash
pip install requests

Step 2: Write the Verification Script

Create a Python module to evaluate the payload and sources returned by the Generative Answers
node:

Python
import json

def validate_generative_response(payload: dict) -> dict:

"""
Validates the Generative Answer payload to ensure sources are
grounded and no hallucinations are present.
"""
confidence_score = [Link]("confidence_score", 0.0)
sources = [Link]("sources", [])

if confidence_score < 0.75:

return {
"status": "Needs_Review",
"reason": "Low confidence score from the generative model.",
"score": confidence_score
}

if not sources:
return {

[Link]
 "status": "Failed",
"reason": "No valid knowledge sources found. Potential
hallucination risk.",
"score": confidence_score
}

return {
"status": "Passed",
"reason": "Generative Answer is well-grounded.",
"sources_count": len(sources)
}

if __name__ == "__main__":
# Simulate a payload received from Copilot Studio's Generative Answers
Node
mock_payload = {
"user_query": "What is the warranty on the Cloud 9 Solar Panel?",
"confidence_score": 0.88,
"sources": [
{"title": "Warranty Policy 2026", "url":
"[Link]
]
}

result = validate_generative_response(mock_payload)
print([Link](result, indent=2))

Step 3: Integrate with Copilot Studio

1. Export your Generative Answers output payload to an external workflow using a Power
Automate flow.
2. Call this Python-based validation logic using an Azure Function or HTTP action.
3. If the status returns Failed, route the conversation flow to a human customer support
topic.

Multiple-Choice Questions (MCQs)

Question 1

Which built-in tool in Copilot Studio allows developers to inspect the value of a session variable
during a multi-turn conversation? A. The System Performance Dashboard

B. The Variable Watch Window in the Test Bot panel

C. The Power Platform Admin Center

[Link]
D. The Azure Container Registry

 Correct Answer: B
 Explanation: The variable watch window allows you to view the live values of your
variables as the user steps through the topic nodes, making it easy to identify data handoff
issues.

Question 2

When an agent's Generative Answers node pulls information from unverified internet sources
rather than enterprise documents, which setting should be reviewed first? A. The system topic
trigger phrases.

B. The data source scope and authentication settings.

C. The conversational node's scale-to-zero setting.

D. The API Gateway rate limits.

 Correct Answer: B
 Explanation: To prevent the model from using external web sources, you must configure
the Generative Answers node to restrict searches to your specific data sources (such as
SharePoint or custom URLs).

Question 3

What happens when the confidence score of a Generative Answers node falls below the
configured threshold in Copilot Studio? A. The bot closes the browser window automatically.

B. The conversation automatically terminates.

C. The bot defaults to its fallback action (e.g., rephrasing or escalating to a human agent).

D. The model re-runs the previous SQL database query.

 Correct Answer: C
 Explanation: When the copilot cannot find a clear answer in the provided sources with
sufficient confidence, it triggers the fallback path to prevent hallucinations.

[Link]
Chapter : Performance Tuning: Speed vs. Accuracy
Objective: Understand the trade-offs between model latency and output accuracy, and learn how
to optimize agent parameters in Microsoft Copilot Studio and external agent frameworks to meet
enterprise service-level agreements (SLAs).

[Link]
Core Concepts & Theory
As you build multi-turn, multi-agent architectures in Copilot Studio and the Microsoft
ecosystem, you will encounter the fundamental trade-off between response speed and output
accuracy. Balancing these factors is critical to providing a responsive, high-quality user
experience.

1. The Speed vs. Accuracy Dilemma

 Latency (Speed): The time it takes for an agent to process input, orchestrate tools,
generate tokens, and return a response. High latency can degrade the user experience and
cause conversational timeouts.
 Accuracy: The precision, groundedness, and relevance of the agent's response.
Achieving higher accuracy often requires deeper context retrieval, more complex chains
of thought, or larger, more capable models, all of which increase latency.

2. Factors Affecting Latency

 Orchestration Depth: Multi-agent architectures that pass context through several sub-
agents require multiple LLM calls, increasing round-trip time.
 Knowledge Retrieval Overhead: Searching large enterprise knowledge bases (e.g.,
SharePoint, vector databases, or local MCP databases) takes longer than generating a
generic response.
 Payload Size & Token Limits: Processing large input prompts and generating long
responses requires more compute time.

3. Strategies for Performance Tuning

 Caching: Store the outputs of frequent or static queries in a key-value store to bypass the
LLM entirely for repeated intents.
 Model Routing: Route simple, routine intents to smaller, faster models (e.g., smaller tier
models in the Azure OpenAI Service) and complex queries to larger, more accurate
models.
 Streamed Responses: Implement token streaming using WebSockets or Server-Sent
Events (SSE) to reduce perceived latency while the model completes its generation.

Practical Case Study

Case Study: Global Logistics Inc.

[Link]
Scenario: Global Logistics operates an AI-powered assistant in Copilot Studio to help customs
agents verify international shipping codes.

The Challenge: During peak operational hours, the assistant took up to 8 seconds per response
because it ran complex validation steps and searched a large database of shipping regulations.
This delay caused users to abandon the chat, despite the high accuracy of the responses.

The Solution: The development team tuned the assistant's performance:

 They split the orchestrator into a dual-model architecture: simple queries were handled by
a smaller, faster model, while complex verifications used the standard model.
 They implemented an intent-caching mechanism for common shipping codes.
 They added response streaming so that the user interface displayed intermediate steps
while the backend processed the data.

These changes reduced the average response time from 8 seconds to 2.5 seconds without
decreasing accuracy.

Step-by-Step Example
Measuring and Optimizing Latency with Timeout Controls in Python

This example shows how to implement a timeout mechanism to prevent the system from hanging
when waiting for a slow API action or knowledge retrieval step.

Step 1: Install Dependencies

Bash
pip install requests

Step 2: Implement the Timeout Mechanism

Create a Python script that sets a maximum execution time for the tool call.

Python
import requests
import time

def call_slow_agent_tool(payload: dict, timeout_seconds: float = 3.0) ->

dict:
"""
Simulates an agent tool call with a strict timeout limit to manage
latency.

[Link]
 """
start_time = [Link]()
url = "[Link]

try:
# Simulate a request with a timeout
# response = [Link](url, json=payload,
timeout=timeout_seconds)

# Simulating the response time for demonstration:

[Link](2.5) # Simulating execution well within the 3.0s limit

elapsed = [Link]() - start_time

return {
"status": "success",
"latency_seconds": round(elapsed, 4),
"output": "Action completed successfully"
}

except [Link]:
elapsed = [Link]() - start_time
return {
"status": "timeout",
"latency_seconds": round(elapsed, 4),
"output": "Action timed out. Fallback triggered."
}

except Exception as e:
return {
"status": "error",
"message": str(e)
}

if __name__ == "__main__":
sample_payload = {"intent": "query_shipping_database", "entity": "SHP-
001"}
result = call_slow_agent_tool(sample_payload, timeout_seconds=2.0)
print(f"Execution Result: {result['status']} | Latency:
{result['latency_seconds']}s")

Step 3: Integrate into Copilot Studio

1. Export your API actions or tool code to an Azure Function with strict execution limits.
2. If the tool exceeds the latency limit, configure the error branch in Copilot Studio to use a
cached or generic response.
3. Turn on streaming in the copilot properties to improve the user experience while the
backend executes.

[Link]
Multiple-Choice Questions (MCQs)
Question 1

Which of the following actions helps reduce the latency of an agent without decreasing its
response accuracy? A. Increasing the number of orchestration steps between sub-agents.

B. Using a large foundational model for simple, routine intents.

C. Caching the outputs of frequent or static queries.

D. Removing authentication layers from your data sources.

 Correct Answer: C
 Explanation: Caching frequent queries reduces turnaround time by returning known
answers without needing to run the model again.

Question 2

When an agent takes too long to respond during peak hours, which configuration should you
review to manage user experience and prevent connection drops? A. The variable watch window.

B. Token streaming and execution timeouts.

C. The scale-to-zero settings of your web browser.

D. The language trigger phrases.

 Correct Answer: B
 Explanation: Implementing timeouts and token streaming helps manage long-running
operations and provides feedback to the user while the query is processing.

Question 3

Why is model routing considered an effective performance tuning strategy for enterprise agents?
A. It routes all complex calculations to the local browser, freeing up cloud resources.

B. It directs simple queries to smaller, faster models, and complex queries to more powerful
models.

[Link]
C. It stops token consumption once the user's intent is identified.

D. It deletes session and global variables to free up storage space.

 Correct Answer: B
 Explanation: Model routing optimizes resource usage by matching the complexity of the
query to the capabilities of the model, which saves both time and computing costs.

[Link]
Chapter: Using Application Insights for Deep Tracing
Objective: Understand how to configure Azure Application Insights, track agent turns, trace
multi-agent handoffs, and analyze telemetry data to optimize performance in Microsoft Copilot
Studio.

Core Concepts & Theory

As enterprise AI agents scale to handle complex, multi-turn interactions and integrate with
external APIs and micro-services, standard conversational logging is no longer sufficient. Deep
tracing gives developers clear visibility into the agent's internal reasoning steps, API calls, and
context handoffs.

1. The Role of Observability in AI Agents

Observability in conversational AI goes beyond checking whether an application is running. It

involves monitoring the internal state of the agent as it processes a prompt:

 Token Usage Tracking: Measures the resources consumed during prompt evaluation
and response generation.
 Orchestration Tracing: Tracks interactions as context moves from a master agent to
subordinate sub-agents.
 Dependency Monitoring: Identifies performance bottlenecks in external data sources,
such as Enterprise Knowledge Bases or custom MCP (Model Context Protocol) servers.

2. Azure Application Insights Integration

By linking your Copilot Studio environment to Azure Application Insights, you can stream
telemetry and conversation data to a central location. This setup uses several standard telemetry
tables:

 Traces: Custom diagnostic messages and execution logs.

 Dependencies: External API calls made during the conversation (e.g., retrieving
inventory data).
 Page Views & Events: User interactions and custom business metrics.

[Link]
3. Kusto Query Language (KQL) for Agent Diagnostics

KQL is the primary tool used to analyze telemetry in Azure Monitor. Using KQL, you can
inspect the performance of individual agent turns and troubleshoot system errors.

Practical Case Study

Case Study: Adventure Works Cycles

Scenario: Adventure Works uses a multi-agent system in Copilot Studio to help customers
customize and order bicycle parts.

The Challenge: During peak shopping seasons, the system occasionally slowed down, causing
conversational timeouts. Because the development team lacked end-to-end tracing, they could
not determine whether the latency was caused by the language model, the internal knowledge
base, or an external inventory lookup API.

The Solution: The team enabled deep tracing with Azure Application Insights.

 They traced API calls and LLM evaluation steps in real time.
 The logs revealed that an external inventory API was causing the delays during complex
searches.
 The development team optimized the API and cached the responses, which reduced the
average response time by 60%.

Step-by-Step Example
Instrumenting an Agent Application with Azure Monitor

This step-by-step example shows how to configure a Python-based custom tool or agent
application to send custom traces and dependency logs to Azure Application Insights.

Step 1: Install Required Libraries

Install the Azure Monitor OpenTelemetry exporter in your local environment:

Bash
pip install azure-monitor-opentelemetry

[Link]
Step 2: Initialize Tracing and Metrics

Create a file named [Link] to configure the OpenTelemetry exporter and send data to Azure
Application Insights:

Python
import logging
import time
from [Link] import configure_azure_monitor

# Configure the logger and the Azure Monitor connection

def initialize_telemetry():
# In a production environment, connection string is read from the
APPLICATIONINSIGHTS_CONNECTION_STRING environment variable
# [Link]["APPLICATIONINSIGHTS_CONNECTION_STRING"] =
"InstrumentationKey=..."
try:
configure_azure_monitor()
logger = [Link]("AgentLogger")
[Link]([Link])
[Link]("Azure Monitor telemetry pipeline initialized
successfully.")
return True
except Exception as e:
print(f"Failed to configure telemetry: {e}")
return False

def trace_agent_action(action_name: str, payload: dict):

logger = [Link]("AgentLogger")
start_time = [Link]()

# Simulate processing logic

[Link](f"Starting agent action: {action_name}")

# Simulate action time

[Link](0.4)

duration = [Link]() - start_time

[Link](
f"Completed action {action_name}",
extra={"custom_dimensions": {"action": action_name,
"duration_seconds": duration, "status": "Success"}}
)
return True

if __name__ == "__main__":
if initialize_telemetry():
trace_agent_action("CheckInventory", {"item_id": "SKU-5001"})

Step 3: Query Telemetry with KQL

[Link]
You can analyze your logs in Azure Application Insights using this KQL query:

Code snippet
traces
| where message startswith "Completed action"
| extend ActionName = custom_dimensions.action
| extend Duration = todouble(custom_dimensions.duration_seconds)
| project timestamp, ActionName, Duration, message
| order by timestamp desc

Multiple-Choice Questions (MCQs)

Question 1

What is the primary benefit of streaming agent telemetry to Azure Application Insights instead of
using default local log files? A. It completely removes the need for a connection string or
authentication.

B. It provides a scalable, centralized platform for monitoring dependencies, tracking tokens, and
running diagnostic queries across multiple turns.

C. It limits the agent's ability to use generative answers.

D. It automatically generates web content without LLM evaluation.

 Correct Answer: B
 Explanation: Application Insights offers centralized observability, letting you analyze
performance, debug errors, and track dependencies in distributed, multi-agent systems.

Question 2

Which Kusto Query Language (KQL) operator is used to extract custom properties from the
custom_dimensions dictionary? A. extend

B. summarize

C. join

D. render

 Correct Answer: A

[Link]
  Explanation: The extend operator adds new calculated columns to your result set,
allowing you to parse and analyze custom dimensions.

Question 3

Why should you include custom dimensions when logging agent actions? A. To prevent users
from editing or viewing the conversation transcript.

B. To allow the bot to bypass the Authentication Manager.

C. To provide context (such as run duration or user ID) for downstream troubleshooting and
analysis.

D. To increase token consumption.

 Correct Answer: C
 Explanation: Custom dimensions enrich log data by adding structured properties,
making it easier to filter, search, and analyze performance issues.

[Link]
Chapter : Interpreting Analytics: Satisfaction vs. Deflection
Rates
Objective: Learn how to measure, analyze, and balance deflection and customer satisfaction
(CSAT) metrics in Microsoft Copilot Studio to optimize AI agent performance and return on
investment (ROI).

Core Concepts & Theory

Analyzing the performance of your AI agents requires balancing operational efficiency with user
experience. Two of the most important performance indicators are the Deflection Rate and
Customer Satisfaction (CSAT).

1. Understanding Deflection and CSAT

 Deflection Rate: The percentage of user queries or requests handled automatically by the
agent, avoiding the need for escalation to a human representative.

Deflection Rate=(1−Total Engaged SessionsEscalated Sessions+Abandoned Sessions

)×100

[Link]
  Customer Satisfaction Score (CSAT): The average score obtained from user feedback
collected at the end of a conversation or topic when the user confirms their issue is
resolved.
 Cost Savings vs. Quality: While high deflection rates reduce operational costs, if they
come at the expense of user satisfaction, they can lead to frustration and trust issues.

2. Key Copilot Studio Analytics Metrics

To properly evaluate performance, you should track these additional metrics on the Copilot
Studio Analytics dashboard:

 Engagement Rate: The percentage of total sessions where a custom topic or key system
topic (like Conversational Boosting, Escalate, or Fallback) is triggered.
 Resolution Rate: The percentage of engaged sessions where the user receives an End of
Conversation question and indicates their issue was resolved.
 Escalation Rate: The percentage of engaged sessions that are passed to a human
representative.

3. Interpreting the Analytics Matrix

Assessment
Finding Impact & Meaning
Method
Effective Self- High Deflection + The agent operates effectively, providing accurate and
Service High CSAT helpful self-service.
High Deflection + Users are successfully diverted from human agents, but
Frustrated Users
Low CSAT are frustrated by unhelpful or rigid flows.
Low Deflection + The agent works well but has low scenario coverage or
Niche Coverage
High CSAT misses common user intents.
Configuration Low Deflection + Indicates gaps in your knowledge base, unclear
Gap Low CSAT responses, or broken topic flows.
Export to Sheets

Practical Case Study

Case Study: Contoso Energy Support Bot

Scenario: Contoso Energy uses a Copilot Studio agent to help customers check their energy
usage, pay bills, and report outages.

The Challenge: The dashboard indicated a deflection rate of 75%. However, the CSAT scores
for the self-service channel dropped by 20% over a single month.

[Link]
The Solution: The analytics team reviewed the conversation transcripts and discovered that the
bot was "deflecting" calls by providing generic links to PDF documents, instead of retrieving
specific billing account information through Power Automate.

 They updated the topics to include more conversational follow-ups.

 They added an integration to check the user's account balance directly rather than asking
users to check external documents.
 Deflection held steady at 70%, while the CSAT score increased by 25%.

Step-by-Step Example
Analyzing Deflection and Satisfaction Metrics via Python

This example demonstrates how to process session logs from Copilot Studio to calculate the
Deflection Rate and classify the performance of your topics.

Step 1: Initialize the Session Data

Python
def calculate_deflection_metrics(total_sessions: int, escalated_sessions:
int, abandoned_sessions: int) -> dict:
"""
Calculates the deflection rate based on session outcomes.
"""
engaged_sessions = total_sessions # Assume all sessions are engaged for
simplicity
if engaged_sessions == 0:
return {"deflection_rate": 0.0, "status": "No data"}

deflected_sessions = engaged_sessions - (escalated_sessions +

abandoned_sessions)
deflection_rate = (deflected_sessions / engaged_sessions) * 100

return {
"engaged_sessions": engaged_sessions,
"deflected_sessions": deflected_sessions,
"deflection_rate": round(deflection_rate, 2)
}

if __name__ == "__main__":
# Simulate a dataset of session logs for the past month
session_data = {
"total_sessions": 1250,
"escalated_sessions": 250,
"abandoned_sessions": 100
}

[Link]
 result = calculate_deflection_metrics(
session_data["total_sessions"],
session_data["escalated_sessions"],
session_data["abandoned_sessions"]
)

print("Analytics Output:")
print(f"Total Engaged Sessions: {result['engaged_sessions']}")
print(f"Deflected Sessions: {result['deflected_sessions']}")
print(f"Deflection Rate: {result['deflection_rate']}%")

Step 2: Implement Topic Escalation Review

1. Navigate to the Analytics tab in Copilot Studio.

2. Review the Escalation Rate Drivers to identify topics with high escalation rates (like
"Returns & Exchanges").
3. Update those specific topics using the analytics guidelines to improve deflection rates and
user satisfaction.

Multiple-Choice Questions (MCQs)

Question 1

What is the primary indicator that an AI agent is providing a poor experience, despite having a
high deflection rate?

A. A high conversation engagement rate and long response times.

B. Low customer satisfaction scores (CSAT) alongside a high escalation rate to other sub-agents.

C. High deflection with a sharp drop in CSAT scores, indicating users are frustrated by unhelpful
or incomplete answers.

D. An increase in the total number of sessions logged in the analytics overview dashboard.

 Correct Answer: C
 Explanation: A high deflection rate means the bot is handling inquiries on its own, but
low CSAT scores suggest that the automated assistance does not meet the user's
expectations.

[Link]
Question 2

Which metric represents the percentage of sessions that end with the user indicating their issue
was resolved successfully?

A. Abandonment Rate

B. Resolution Rate

C. Engagement Rate

D. Direct Escalation Rate

 Correct Answer: B
 Explanation: Resolution rate tracks sessions where the End of Conversation topic
triggers and the user confirms their issue is resolved.

Question 3

If your copilot telemetry indicates High Usage but a Low Success Rate, what is the
recommended diagnostic approach?

A. Increase the number of available sub-agents without testing.

B. Check for topic coverage gaps, missing enterprise data connections, or ambiguous system
prompts.

C. Delete older versions of conversational system topics.

D. Remove the feedback surveys to reduce response latency.

 Correct Answer: B
 Explanation: High usage paired with low success rates points to gaps in knowledge
sources or missing configuration, rather than a lack of interest from the users.

[Link]
Chapter : Automated Testing for Multi-Agent Scenarios
Objective: Understand how to design, execute, and monitor automated testing strategies for
multi-agent ecosystems in Microsoft Copilot Studio, including context-passing, dynamic
orchestration, and custom tool verification.

Core Concepts & Theory

As enterprise workloads shift from single conversational bots to advanced multi-agent
ecosystems, testing requires more than evaluating static topic triggers. In a multi-agent scenario,

[Link]
a "Master Agent" coordinates with subordinate agents (e.g., a Database Agent, an Inventory
Agent, or an ERP Integration Agent) to resolve a query.

1. Challenges in Testing Multi-Agent Architectures

 Non-Determinism: Generative responses from different LLMs can vary, making exact-
string matching difficult.
 Context Handoffs: Ensuring that session variables, user profiles, and conversation
context are passed accurately between the master agent and subordinate agents.
 Orchestration Failure: When the master agent routes a query to the incorrect
subordinate agent, leading to incorrect tool execution.

2. Automated Testing Strategies

 Traceability Tests: Verifying the execution path by tracking session IDs and topic
histories.
 Orchestration Validation: Using synthetic inputs to ensure the master agent chooses the
correct sub-agent based on the user's intent.
 Continuous Integration (CI) Pipelines: Running regression test sets every time you
update the system prompt or agent logic.

3. Key Metrics for Automation

 Routing Accuracy: The percentage of intents routed to the correct subordinate agent.
 Tool Calling Success Rate: The reliability of custom API actions or Model Context
Protocol (MCP) servers.

Practical Case Study

Case Study: Contoso Supply Chain Logistics

Scenario: Contoso Supply Chain uses a multi-agent ecosystem in Copilot Studio. A master agent
handles initial requests, then routes complex tasks to either a Shipping Agent or a Supplier
Management Agent.

The Challenge: After an update to the master agent's system prompt, the master agent began
incorrectly routing warehouse inquiries to the Supplier Management Agent. This created
processing delays and errors in the inventory database.

The Solution: The development team built an automated test suite.

[Link]
  They generated synthetic conversation logs covering common routing scenarios.
 The tests verified the chosen sub-agents in the orchestration tree.
 The team added the tests to their deployment pipeline. This caught similar routing issues
in future builds before they affected the production environment.

Step-by-Step Example
Automating Multi-Agent Routing Tests with Python

This example shows how to write an automated test script to verify that a master agent routes
intents to the correct subordinate agent.

Step 1: Define the Master Agent's Routing Logic

Python
def route_intent(user_query: str) -> str:
"""
Simulates the master agent's intent recognition and routing logic.
"""
query = user_query.lower()

if "shipment" in query or "tracking" in query:

return "ShippingAgent"
elif "inventory" in query or "stock" in query:
return "InventoryAgent"
elif "supplier" in query or "vendor" in query:
return "SupplierAgent"
else:
return "MasterAgentFallback"

Step 2: Create the Test Cases

Python
def test_routing_scenarios():
test_cases = [
{"id": "TC-001", "query": "Where is my shipment SHP-4992?",
"expected": "ShippingAgent"},
{"id": "TC-002", "query": "Check stock levels for the solar
inverter", "expected": "InventoryAgent"},
{"id": "TC-003", "query": "Contact the raw materials vendor",
"expected": "SupplierAgent"}
]

results = []
for case in test_cases:
actual_agent = route_intent(case["query"])

[Link]
 passed = actual_agent == case["expected"]
[Link]({
"id": case["id"],
"passed": passed,
"expected": case["expected"],
"actual": actual_agent
})

return results

Step 3: Run and Review the Results

Python
if __name__ == "__main__":
test_results = test_routing_scenarios()

for run in test_results:

print(f"Test ID: {run['id']} | Expected: {run['expected']} | Actual:
{run['actual']} | Status: {'PASS' if run['passed'] else 'FAIL'}")

total_passed = sum(1 for r in test_results if r["passed"])

success_rate = (total_passed / len(test_results)) * 100
print(f"\nMulti-Agent Routing Success Rate: {success_rate}%")

Multiple-Choice Questions (MCQs)

Question 1

What is the primary risk when changing the system prompt of a master agent in Copilot Studio
without running automated routing tests?

A. The agent will run out of available tokens during the conversation.

B. The master agent may route intents to the wrong subordinate agent, causing failures in
downstream tools.

C. The web application will automatically scale to zero compute resources.

D. User session variables will be exposed to public internet websites.

 Correct Answer: B
 Explanation: Changing the master agent's system prompt can alter its intent recognition,
causing it to select the wrong sub-agent.

[Link]
Question 2

When testing multi-agent handoffs, which session data should you inspect to ensure context was
preserved?

A. The total number of users logged into the system.

B. The duration of the user's web browser session.

C. Session and global variables containing conversation history and user credentials.

D. The API timeout limits set in the master dashboard.

 Correct Answer: C
 Explanation: Context variables store user data and settings as the conversation moves
between agents. Preserving this data is essential for a smooth user experience.

Question 3

Why are automated, non-deterministic test suites important for validating multi-agent
ecosystems?

A. They prevent the chat interface from rendering intermediate steps.

B. They allow you to test how the agent handles variations in natural language queries.

C. They remove the need for manual customer service teams.

D. They force the subordinate agents to use the same system prompt.

 Correct Answer: B
 Explanation: Real users phrase questions in many different ways. Non-deterministic test
suites verify that the agent understands intent across different phrasing styles.

Chapter : Analyzing User Feedback Loops

[Link]
Objective: Understand how to design, monitor, and analyze user feedback loops in Microsoft
Copilot Studio, and learn how to use both explicit and implicit signals to improve AI agent
performance.

Core Concepts & Theory

In Copilot Studio, building and maintaining high-performing AI agents relies on continuous
improvement. A well-designed feedback loop helps you identify areas where your topics,
prompts, and data sources fall short of user expectations.

1. Types of Feedback Mechanisms

 Explicit Feedback: Direct ratings and responses provided by the user. Examples include
end-of-conversation surveys, thumbs-up or thumbs-down ratings, and open-ended text
comments.
 Implicit Feedback: Behavioral signals that indicate a user's experience without requiring
direct input. Examples include session abandonment, rapid re-prompting, and topic
escalation.

2. Copilot Studio Analytics and Dataverse Integration

To capture and analyze feedback, Copilot Studio stores interaction data within Microsoft
Dataverse. This pipeline tracks:

 Conversational Metrics: Total engagement time, node traversal path, and fallback
triggers.
 Topic Feedback: Aggregated sentiment and survey ratings for individual dialogue
nodes.
 Escalation Data: Points in the conversation where users choose to transfer to a human
agent.

3. Analyzing Feedback Trends

When analyzing feedback, you should categorize the signals to isolate the root causes of issues:

 High Engagement + Negative Feedback: The user spent a long time interacting with
the agent but was unhappy with the resolution, suggesting the topic flows are too
complex or unhelpful.
 High Deflection + Low CSAT: The agent successfully closed the session, but the user
felt the interaction was poor, which often points to rigid, generic, or unhelpful responses.

[Link]
Practical Case Study
Case Study: Contoso Retail Support Bot

Scenario: Contoso Retail uses a Copilot Studio assistant to process customer service returns and
delivery inquiries.

The Challenge: The team noticed an increase in thumbs-down ratings for the order tracking
topic. The logs showed that users were getting stuck in a loop and leaving the chat without
resolving their issue.

The Solution: The quality assurance team implemented an improved feedback loop:

 They added an explicit, short-form survey that triggered whenever the user abandoned a
conversation.
 They found that the bot provided tracking numbers without links to the courier's website.
 They updated the topic to include actionable adaptive cards with direct links. As a result,
the CSAT score for the order tracking topic increased by 35%.

Step-by-Step Example
Analyzing Feedback Ratings with Python

This example demonstrates how to process session logs and feedback data to identify low-
performing topics that need review.

Step 1: Initialize the Feedback Dataset

Python
def analyze_feedback_data(session_feedback: list) -> dict:
"""
Analyzes feedback scores to identify topics that require improvement.
"""
total_ratings = len(session_feedback)
if total_ratings == 0:
return {"status": "No feedback available"}

positive_ratings = sum(1 for item in session_feedback if item["rating"]

>= 4)
negative_ratings = sum(1 for item in session_feedback if item["rating"]
<= 2)

[Link]
 avg_score = sum(item["rating"] for item in session_feedback) /
total_ratings

return {
"total_responses": total_ratings,
"average_score": round(avg_score, 2),
"positive_percentage": round((positive_ratings / total_ratings) *
100, 2),
"negative_percentage": round((negative_ratings / total_ratings) *
100, 2)
}

if __name__ == "__main__":
# Simulate a set of feedback ratings from Copilot Studio conversations
mock_feedback = [
{"topic": "Returns", "rating": 5, "comment": "Clear and fast."},
{"topic": "Shipping", "rating": 1, "comment": "Could not find my
tracking link."},
{"topic": "Shipping", "rating": 2, "comment": "Kept repeating the
same answer."},
{"topic": "Account", "rating": 4, "comment": "Helpful support."}
]

results = analyze_feedback_data(mock_feedback)
print("Feedback Analysis Results:")
print(f"Total Responses: {results['total_responses']}")
print(f"Average Score: {results['average_score']} / 5")
print(f"Negative Feedback: {results['negative_percentage']}%")

Step 2: Review and Take Action

1. Extract topics with a negative feedback score greater than 30%.

2. Open those topics in the Copilot Studio authoring canvas to inspect the dialogue nodes.
3. Add escalation or fallback paths to route users to a human representative when the AI
cannot resolve the issue.

Multiple-Choice Questions (MCQs)

Question 1

Which type of feedback is captured by tracking user behavior like session abandonment or topic
re-triggering, without requiring the user to fill out a survey?

A. Explicit Feedback

B. Implicit Feedback

[Link]
C. Direct System Evaluation

D. Knowledge Base Feedback

 Correct Answer: B
 Explanation: Implicit feedback consists of behavioral signals that indicate user
frustration or success without requiring direct input from the user.

Question 2

What does a high deflection rate paired with low customer satisfaction (CSAT) scores usually
indicate?

A. The bot cannot handle complex intents and lacks data sources.

B. The user is satisfied with the conversation but prefers speaking with a human agent.

C. The bot successfully deflects queries, but the answers provided are unhelpful or frustrating.

D. The bot's response time is too fast for the user to read.

 Correct Answer: C
 Explanation: A high deflection rate means the bot is handling inquiries on its own, but
low CSAT scores suggest that the automated assistance does not meet the user's
expectations.

Question 3

When a topic generates negative feedback, what is the best practice for improving its
performance in Copilot Studio?

A. Increase the number of models used for knowledge extraction.

B. Review the conversation transcripts to find the root cause, then update the topic with clearer
steps or a human handoff.

C. Delete the topic from the Copilot Studio authoring canvas.

D. Decrease the node connection timeout limit.

[Link]
 Correct Answer: B
 Explanation: Analyzing conversation transcripts reveals the specific conversational steps
where users get stuck, allowing you to optimize that part of the topic flow.

[Link]