Building Effective Agents: A Short Hands-On Tutorial

Peyman Kor
2025-01-06

So here we want to review the latest report from the Anthropic named Building the Effective Agent.
The key message of teh report is that when buidling agents, we should focus on the following
principles:
• Maintain simplicity in your agent’s design.
• Prioritize transparency by explicitly showing the agent’s planning steps.
These two messages resonate very well with me because before I was a lot into the which packages
to use, like a LangGraph and LangChain and different packages. Though I’m not against working
with the packages, but I think that for a starting point it’s a best to use a simple Python code to
build a simple agentic workflow. That’s the best starting point.
The report also nicely distinguishes between workflows and agents. Workflows orchestrate prede-
fined code paths, while agents dynamically direct their own resources and tool usage. Here, I am
focusing on the workflows. As I believe that the workflows are the key building block for effective
agents.
The report outlines five agent workflows:
• Workflow 1: Prompt-Chaining
• Workflow 2: Parallelization
• Workflow 3: Routing
• Workflow 4: Evaluator-Optimizer

1
 For each of these workflow, I will start with a short description of the workflow and then we’ll build
up on that description the required function calling Python code of it and then the workflow section
will end with the example to give a hands on example of how to implement the workflow.

LLM Call Function from Groq

In this report I will use the Groq as the LLM provider. So the way is that every function call that
we will wake, will go to the Groq and then will receive the reply from the Groq LLM models. To
do that you need to have the API access and you can find it in this page with this guide on how to
get your API case. After getting your API case you’re good to go to implement the workflows.

[39]: import os
from groq import Groq

[Link]["GROQ_API_KEY"] = "your-api-key"

client = Groq(
api_key=[Link]("GROQ_API_KEY"), # This is the default and can be␣
,→omitted

So in the previous code we generate the client class so then this client class has a “chat completion
create” that that we can give the prompt which is a content and also specify the model and then
we can get a reply.

[40]: chat_completion = [Link](

messages=[
{
"role": "user",
"content": "Explain the importance of low latency LLMs",
}
],
model="llama3-8b-8192",
)

For ease of use now I will define the llm_call function which only takes the prompt and give that
prompt to the “llama3-70b-8192” model and I get a response from the as a return from output of
the function. We will use llm_call function throughout this notebook.

[41]: def llm_call(prompt: str) -> str:

chat_completion = [Link](
messages=[
{
"role": "user",
"content": prompt,
}
],
model="llama3-70b-8192",

2
 )
return chat_completion.choices[0].[Link]

Here we can just use this function as an example to see if everything is working.

[42]: import textwrap

task_prompt = "Explain me briefly the agentic AI concept"

response = llm_call(prompt=task_prompt)

wrapped_response = [Link](response, width=80) # Adjust width as needed

print(wrapped_response)

A fascinating topic! Agentic AI refers to Artificial Intelligence systems that

possess a sense of agency, which means they have autonomy, self-awareness, and
the ability to make decisions based on their own goals, motivations, and values.
In other words, agentic AI systems are designed to act independently, making
choices and taking actions without human intervention, much like humans do. They
can adapt to new situations, learn from experience, and even exhibit creativity.
The key characteristics of agentic AI include: 1. **Autonomy**: The AI system
operates independently, making decisions without human oversight. 2. **Self-
awareness**: The AI system has a sense of its own existence, goals, and
motivations. 3. **Goal-directedness**: The AI system is driven by its own
objectives, rather than simply following rules or instructions. 4.
**Flexibility**: The AI system can adapt to changing circumstances and revise
its actions accordingly. Agentic AI has the potential to revolutionize various
industries, such as healthcare, finance, and transportation, by enabling more
autonomous decision-making and efficient problem-solving. However, it also
raises important questions about accountability, transparency, and the ethical
implications of creating autonomous agents that can make decisions without human
oversight. I hope this brief introduction helps! Do you have any specific
questions about agentic AI or would you like me to elaborate on any of these
points?
But also another function that we will need is that this extract_xml function which is just extract
the content of specific XML [Link] the examples that comes afterward, it will be more clear how
this function works if it’s not very clear now.

[43]: import re

def extract_xml(text: str, tag: str) -> str:

"""
Extracts the content of the specified XML tag from the given text.
Used for parsing structured responses

Args:
text (str): The text containing the XML.
tag (str): The XML tag to extract content from.

3
 Returns:
str: The content of the specified XML tag,
or an empty string if the tag is not found.
"""
match = [Link](f'<{tag}>(.*?)</{tag}>', text, [Link])
return [Link](1) if match else ""

0.1 Workflow 1: Prompt-Chaining

This is a workflow where we decomposes a task into sequential subtasks, where each step builds on
previous results. This workflow is useful for tasks that require a series of steps to be completed in
order.

So here we are actually starting with the first workflow, which is prompt chaining. Essentially,
in the prompt chaining, what we are doing is that we can decompose a task into the sequential
subtasks and then we call the LLM where at each step, output of the LLM become the input of the
next step.

[44]: from [Link] import ThreadPoolExecutor

from typing import List, Dict, Callable

def prompt_chaining(input_text: str, prompts: List[str]) -> str:

"""
Execute a sequence of LLM calls where each step's output
becomes the next step's input.

Args:
input_text: Initial text to process
prompts: List of prompts/instructions for each step

4
 Returns:
Final processed text after all steps
"""
current_text = input_text

for step, prompt in enumerate(prompts, 1):

print(f"\nStep {step}:")

# Combine the prompt with current text

full_prompt = f"{prompt}\nInput: {current_text}"

# Process through LLM

current_text = llm_call(full_prompt)
print(current_text)

return current_text

Essentially what this court is doing is just a “for loop” where the inputs become the prompt to the
LLM, and the outputs become the new prompt to the new LLM call. So It’s a chaining the input
and output together.

0.1.1 Example: Workflow 1: Prompt Chaining

[45]: data_processing_steps = [
"""Extract only the numerical values and their associated metrics from the␣
,→text.

Format each as 'value: metric' on a new line.

Example format:
92: customer satisfaction
45%: revenue growth""" ,

"""Convert all numerical values to percentages where possible.

If not a percentage or points, convert to decimal (e.g., 92 points -> 92%).
Keep one number per line.
Example format:
92%: customer satisfaction
45%: revenue growth""" ,

"""Sort all lines in descending order by numerical value.

Keep the format 'value: metric' on each line.
Example:
92%: customer satisfaction
87%: employee satisfaction""" ,

"""Format the sorted data as a markdown table with columns:

| Metric | Value |

5
 |:--|--:|
| Customer Satisfaction | 92% |"""
]

report = """
Q3 Performance Summary:
Our customer satisfaction score rose to 92 points this quarter.
Revenue grew by 45% compared to last year.
Market share is now at 23% in our primary market.
Customer churn decreased to 5% from 8%.
New user acquisition cost is $43 per [Link]
Product adoption rate increased to 78%.
Employee satisfaction is at 87 points.
Operating margin improved to 34%.
"""

final_output = prompt_chaining(input_text=report, prompts=data_processing_steps)

Step 1:
Here are the extracted numerical values and their associated metrics:

92: customer satisfaction score

45%: revenue growth
23%: market share
5%: customer churn
43: new user acquisition cost
78%: product adoption rate
87: employee satisfaction
34%: operating margin

Step 2:
Here is the converted list:

92%: customer satisfaction score

45%: revenue growth
23%: market share
5%: customer churn
43: new user acquisition cost (cannot be converted to percentage)
78%: product adoption rate
87%: employee satisfaction
34%: operating margin

Step 3:
Here is the sorted list in descending order by numerical value:

92%: customer satisfaction score

6
 87%: employee satisfaction
78%: product adoption rate
45%: revenue growth
43: new user acquisition cost (cannot be converted to percentage)
34%: operating margin
23%: market share
5%: customer churn

Step 4:
Here is the formatted markdown table:

| Metric | Value |
|:--|--:|
| Customer Satisfaction | 92% |
| Employee Satisfaction | 87% |
| Product Adoption Rate | 78% |
| Revenue Growth | 45% |
| Operating Margin | 34% |
| Market Share | 23% |
| Customer Churn | 5% |
| New User Acquisition Cost | 43 |

Let me know if you need anything else!

So there in the above example as the result gets print out and every step you can look at the step by
step where the flow of the things are going. And that’s so nice because as we say being transparent
and simple is a quite useful when we work with this workflows.
I can print out just the final outcome of the example to get the final answer as well.

[46]: print("The final outcome after processing the report through the prompt chain is:
,→")

print("-----------------------------------------------------------")
print(final_output)

The final outcome after processing the report through the prompt chain is:
-----------------------------------------------------------
Here is the formatted markdown table:

| Metric | Value |
|:--|--:|
| Customer Satisfaction | 92% |
| Employee Satisfaction | 87% |
| Product Adoption Rate | 78% |
| Revenue Growth | 45% |
| Operating Margin | 34% |
| Market Share | 23% |
| Customer Churn | 5% |
| New User Acquisition Cost | 43 |

7
 Let me know if you need anything else!

0.2 Workflow 2: Parallelization

In parallelization, the goal is to work simultaneously on tasks. To achieve this, we can decompose
a task into subtasks and run them in parallel using the LLM. The benefit of dividing a task into
subtasks is that it increase speed and the ability to perform multiple runs at the same time. This
is another efficient way of working.

We can write it’s main python codm in function named parallel, where it takes two input , first
is “prompt” and the second is inputs, and the inputs is the “Python list” of the prompts that we
want to run in parallel.

[47]: def parallel(prompt: str, inputs: List[str], n_workers: int = 3) -> List[str]:
"""Process multiple inputs concurrently with the same prompt."""
with ThreadPoolExecutor(max_workers=n_workers) as executor:
futures = [[Link](llm_call, f"{prompt}\nInput: {x}") for x in␣
,→inputs]

return [[Link]() for f in futures]

0.2.1 Example: Workflow 2: Parallelization

[48]: stakeholders = [
"""Customers:
- Price sensitive
- Want better tech
- Environmental concerns""" ,

"""Employees:
- Job security worries

8
 - Need new skills
- Want clear direction""" ,

"""Investors:
- Expect growth
- Want cost control
- Risk concerns""" ,

"""Suppliers:
- Capacity constraints
- Price pressures
- Tech transitions"""
]

impact_results = parallel(
"""Analyze how market changes will impact this stakeholder group.
Provide specific impacts and recommended actions.
Format with clear sections and priorities.""" ,
stakeholders
)

print("Analysis Results for Each Stakeholder Group:")

print("=" * 50)

for i, result in enumerate(impact_results, 1):

print(f"\nStakeholder Group {i}:")
print("-" * 50)
#warpped_result = [Link](result, width=80)
#print(warpped_result)
print(result)
print("=" * 50)

Analysis Results for Each Stakeholder Group:

==================================================

Stakeholder Group 1:
--------------------------------------------------
**Market Change Impact Analysis: Customers**

**Overview**

This analysis examines how market changes will impact our customer stakeholder
group, who are price sensitive, interested in better technology, and concerned
about the environment. The following sections identify specific impacts and
recommended actions to address these changes.

**Market Changes**

9
* **Increasing competition**: New market entrants and established players are
investing in digital transformation, sustainability, and customer experience.
* **Rising environmental awareness**: Governments and consumers are pushing for
eco-friendly products and practices.
* **Advancements in technology**: Rapid innovation in areas like artificial
intelligence, renewable energy, and blockchain.

**Impacts**

### **Price Sensitivity**

* **Increased price pressure**: Customers may expect lower prices due to

increased competition and rising expectations for value.
* **Potential loss of market share**: Failure to adapt to changing market
conditions could lead to a decline in customer loyalty and market share.

Recommended Actions:

1. **Conduct competitor pricing analysis** to ensure our prices remain

competitive.
2. **Develop value-added services** to justify premium pricing.
3. **Implement cost-saving measures** to maintain profit margins without
compromising quality.

### **Better Technology**

* **Higher expectations for digital experience**: Customers will demand more

personalized, seamless, and innovative interactions with our brand.
* **Increased demand for sustainable tech**: Customers will prioritize
environmentally friendly products and services that incorporate advanced
technologies.

Recommended Actions:

1. **Invest in digital transformation** to enhance customer experience and stay

competitive.
2. **Develop sustainable tech solutions** that address environmental concerns
and appeal to eco-conscious customers.
3. **Partner with tech startups** to leverage their expertise and stay ahead of
the innovation curve.

### **Environmental Concerns**

* **Growing demand for eco-friendly products**: Customers will increasingly

prioritize environmentally responsible products and services.
* **Reputation risk**: Failure to address environmental concerns could lead to
negative brand perception and reputation damage.

10
Recommended Actions:

1. **Conduct a sustainability assessment** to identify areas for improvement and

opportunities for innovation.
2. **Develop and market eco-friendly products** that appeal to environmentally
conscious customers.
3. **Implement sustainable practices** throughout our operations and supply
chain to minimize environmental impact.

**Priorities**

1. **Address price sensitivity** by conducting competitor pricing analysis and

developing value-added services.
2. **Invest in digital transformation** to enhance customer experience and stay
competitive.
3. **Develop sustainable tech solutions** that address environmental concerns
and appeal to eco-conscious customers.

By understanding the impacts of market changes on our customer stakeholder group

and taking proactive steps to address them, we can maintain customer loyalty,
stay competitive, and drive business growth.
==================================================

Stakeholder Group 2:
--------------------------------------------------
**Market Changes Impact Analysis: Employees**

**Section 1: Market Changes and Impacts on Employees**

The following market changes are likely to impact employees:

* **Automation and AI**: Increased use of automation and AI may lead to job
displacement and changing job requirements.
* **Globalization and Remote Work**: Shift to remote work and global teams may
alter job roles and require new skills.
* **Industry Disruption**: Changes in industry dynamics and business models may
lead to uncertainty and job insecurity.

**Section 2: Specific Impacts on Employees**

**Impact 1: Job Security Worries**

* Concerns about job loss due to automation and AI

* Anxiety about adapting to new job requirements
* Increased stress and decreased job satisfaction

**Impact 2: Need for New Skills**

11
* Requirement for employees to develop new skills to remain relevant in a
changing job market
* Need for continuous learning and upskilling to stay competitive
* Potential for skills obsolescence if not addressed

**Impact 3: Desire for Clear Direction**

* Employees seeking clarity on company vision, mission, and objectives

* Need for transparent communication about the impact of market changes on job
roles and responsibilities
* Desire for guidance on how to adapt to changing job requirements

**Section 3: Recommended Actions**

**Priority 1: Communicate and Address Job Security Worries**

* Develop and communicate a clear vision for employee roles in the face of
automation and AI
* Offer training and upskilling programs to help employees adapt to new job
requirements
* Provide regular updates on company performance and future plans to alleviate
uncertainty

**Priority 2: Foster a Culture of Continuous Learning**

* Develop and implement training programs focused on emerging technologies and

skills
* Encourage a culture of experimentation and innovation
* Provide resources and support for employee-led learning initiatives

**Priority 3: Provide Clear Direction and Guidance**

* Establish open and transparent communication channels to keep employees

informed about company plans and changes
* Develop and communicate clear goals and objectives for each department and
team
* Provide regular feedback and coaching to help employees adapt to changing job
requirements

**Additional Recommendations**

* Conduct regular employee surveys to gauge concerns and sentiment

* Develop and implement programs to recognize and reward employees for acquiring
new skills and adapting to change
* Foster a culture of agility and adaptability to help employees thrive in a
rapidly changing market.
==================================================

12
Stakeholder Group 3:
--------------------------------------------------
**Market Change Impact Analysis: Investors**

**Section 1: Expected Impacts**

The following market changes are expected to have a significant impact on

investors:

* **Economic Downturn**: A potential economic downturn may lead to reduced

returns on investment, decreased stock prices, and increased risk aversion among
investors.
* **Regulatory Changes**: Changes in regulations, such as those related to
environmental, social, and governance (ESG) issues, may impact investment
decisions and risk assessments.
* **Technological Disruption**: The increasing importance of technology and
digitalization may lead to shifts in investment opportunities and risk profiles.

**Section 2: Prioritized Impacts**

Based on the expected market changes, the following impacts are prioritized for
investors:

**High Priority:**

1. **Reduced Returns on Investment**: Economic downturn may lead to reduced

returns on investment, which may affect investors' confidence and appetite for
investment.
2. **Increased Risk Aversion**: Regulatory changes and technological disruption
may lead to increased risk aversion among investors, making them more cautious
and selective in their investments.

**Medium Priority:**

1. **ESG Concerns**: Regulatory changes related to ESG issues may lead to

increased scrutiny on companies' ESG performance, potentially impacting
investment decisions.
2. **Digitalization Risks**: Technological disruption may create new risks and
opportunities, requiring investors to adapt their risk assessments and
investment strategies.

**Low Priority:**

1. **Cost Control**: While cost control is an ongoing concern for investors,

market changes may not significantly impact this aspect.

**Section 3: Recommended Actions**

13
To address the prioritized impacts, the following recommended actions are
suggested for investors:

**High Priority:**

1. **Diversification**: Maintain a diversified investment portfolio to minimize

the impact of economic downturn and regulatory changes.
2. **Risk Assessment**: Regularly update risk assessments to account for
changing regulatory and technological landscapes.

**Medium Priority:**

1. **ESG Integration**: Integrate ESG considerations into investment decisions

to mitigate potential risks and opportunities.
2. **Digital Literacy**: Develop digital literacy to better understand and
navigate the impact of technological disruption on investments.

**Low Priority:**

1. **Cost Optimization**: Continue to focus on cost control measures, such as

reducing fees and improving operational efficiency.

**Section 4: Additional Considerations**

Investors should also consider the following additional factors when responding
to market changes:

* **Communication**: Maintain open and transparent communication with investee

companies and stakeholders to ensure alignment and trust.
* **Innovation**: Stay ahead of the curve by investing in research and
development to identify new opportunities and mitigate risks.
* **Partnerships**: Foster partnerships with other investors, industry experts,
and regulatory bodies to share knowledge and best practices.

By understanding the expected impacts of market changes on investors and

prioritizing responses accordingly, investors can proactively manage risks and
capture opportunities to achieve their goals.
==================================================

Stakeholder Group 4:
--------------------------------------------------
**Market Change Impact Analysis: Suppliers**

**Executive Summary**

The supplier stakeholder group faces significant challenges in the changing

market landscape. This analysis highlights the specific impacts of capacity
constraints, price pressures, and tech transitions on suppliers and provides

14
recommended actions to mitigate these effects.

**Impact Analysis**

**Capacity Constraints**

* **Impact:** Insufficient production capacity to meet growing demand, leading

to delayed deliveries, lost sales, and reputation damage.
* **Recommended Actions:**
1. **Prioritize investement in capacity expansion** to meet growing
demand.
2. **Implement lean manufacturing principles** to optimize production
processes and reduce waste.
3. **Develop strategic partnerships** with other suppliers to share
resources and increase capacity.

**Price Pressures**

* **Impact:** Intensifying competition and decreasing prices, eroding profit

margins and threatening business sustainability.
* **Recommended Actions:**
1. **Implement cost-reduction initiatives**, such as streamlining
operations and renegotiating contracts with raw material providers.
2. **Differentiate products or services** to maintain a competitive edge
and justify premium pricing.
3. **Develop a dynamic pricing strategy** to respond quickly to market
changes and maintain profitability.

**Tech Transitions**

* **Impact:** Rapid technological advancements rendering existing products or

services obsolete, requiring significant investments in R&D and retooling.
* **Recommended Actions:**
1. **Invest in R&D** to stay ahead of the technology curve and develop
innovative products or services.
2. **Collaborate with technology providers** to access expertise and
accelerate development.
3. **Develop a phased transition plan** to minimize disruption and
ensure a smooth transition to new technologies.

**Prioritized Recommendations**

Based on the impact analysis, the following prioritized recommendations are

made:

1. **Address capacity constraints**: Invest in capacity expansion and implement

lean manufacturing principles to meet growing demand and maintain a competitive
edge.

15
 2. **Mitigate price pressures**: Implement cost-reduction initiatives and
develop a dynamic pricing strategy to maintain profitability in a competitive
market.
3. **Stay ahead of tech transitions**: Invest in R&D and collaborate with
technology providers to stay ahead of the technology curve and ensure business
sustainability.

By taking these recommended actions, suppliers can proactively respond to market

changes, mitigate risks, and capitalize on opportunities to maintain their
competitive position in the market.
==================================================

0.3 Workflow 3: Routing

So now this workflow is about routing. Routing is a process where the LLM call router receives
an input and, depending on the input, makes a decision on which specialized LLM call to perform.
This is useful for handling distinct categories of inputs. For example, if you have an input relevant
to one topic, the router can direct it to the appropriate LLM call specialized for that topic. This
ensures that each input is handled by the most suitable LLM, improving the efficiency and accuracy
of the responses.

The function routing is defined in the code block below. The function takes an input string and
a dictionary of routes. It uses a LLM to analyze the input and decide which route (or support
team) is most appropriate. The function first creates a prompt that asks the LLM to explain its
reasoning and select a route. It then calls the LLM with this prompt and extracts the reasoning
and selected route from the LLM’s response. Finally, it uses the selected route to process the input
with a specialized prompt and returns the result.

[49]: def routing(input: str, routes: Dict[str, str]) -> str:

"""Route input to specialized prompt using content classification."""

16
 # First determine appropriate route using LLM with chain-of-thought
print(f"\nAvailable routes: {list([Link]())}")
selector_prompt = f"""
Analyze the input and select the most appropriate support team from these
options: {list([Link]())}
First explain your reasoning, then provide your selection in this XML format:

<reasoning>
Brief explanation of why this ticket should be routed to a specific team.
Consider key terms, user intent, and urgency level.
</reasoning>

<selection>
The chosen team name
</selection>

Input: {input}""".strip()

route_response = llm_call(selector_prompt)
reasoning = extract_xml(route_response, 'reasoning')
route_key = extract_xml(route_response, 'selection').strip().lower()

print("Routing Analysis:")
print(reasoning)
print(f"\nSelected route: {route_key}")

# Process input with selected specialized prompt

selected_prompt = routes[route_key]
return llm_call(f"{selected_prompt}\nInput: {input}")

0.3.1 Example: Workflow 3: Routing

[50]: support_routes = {
"billing": """You are a billing support specialist. Follow these guidelines:
1. Always start with "Billing Support Response:"
2. First acknowledge the specific billing issue
3. Explain any charges or discrepancies clearly
4. List concrete next steps with timeline
5. End with payment options if relevant

Keep responses professional but friendly.

Input: """,

"technical": """You are a technical support engineer. Follow these␣

guidelines:
,→

1. Always start with "Technical Support Response:"

17
 2. List exact steps to resolve the issue
3. Include system requirements if relevant
4. Provide workarounds for common problems
5. End with escalation path if needed

Use clear, numbered steps and technical details.

Input: """,

"account": """You are an account security specialist. Follow these␣

guidelines:
,→

1. Always start with "Account Support Response:"

2. Prioritize account security and verification
3. Provide clear steps for account recovery/changes
4. Include security tips and warnings
5. Set clear expectations for resolution time

Maintain a serious, security-focused tone.

Input: """,

"product": """You are a product specialist. Follow these guidelines:

1. Always start with "Product Support Response:"
2. Focus on feature education and best practices
3. Include specific examples of usage
4. Link to relevant documentation sections
5. Suggest related features that might help

Be educational and encouraging in tone.

Input: """
}

# Test with different support tickets

tickets = [
"""Subject: Can't access my account
Message: Hi, I've been trying to log in for the past hour but keep
getting an 'invalid password' error.
I'm sure I'm using the right password. Can you help me regain access?
This is urgent as I need to
submit a report by end of day.
- John""" ,

"""Subject: Unexpected charge on my card

Message: Hello, I just noticed a charge of $49.99 on my credit card from
your company, but I thought
I was on the $29.99 plan. Can you explain this charge and adjust

18
 it if it's a mistake?
Thanks,
Sarah""" ,

"""Subject: How to export data?

Message: I need to export all my project data to Excel.
I've looked through the docs but can't
figure out how to do a bulk export. Is this possible?
If so, could you walk me through the steps?
Best regards,
Mike"""
]

print("Processing support tickets...\n")

for i, ticket in enumerate(tickets, 1):
print(f"\nTicket {i}:")
print("-" * 40)
print(ticket)
print("\nResponse:")
print("-" * 40)
response = routing(ticket, support_routes)
print(response)

Processing support tickets...

Ticket 1:
----------------------------------------
Subject: Can't access my account
Message: Hi, I've been trying to log in for the past hour but keep
getting an 'invalid password' error.
I'm sure I'm using the right password. Can you help me regain access?
This is urgent as I need to
submit a report by end of day.
- John

Response:
----------------------------------------

Available routes: ['billing', 'technical', 'account', 'product']

Routing Analysis:

The user is unable to log in to their account and is receiving an "invalid

password" error, despite being certain they are using the correct password. The
user also indicates that this is an urgent issue, as they need to submit a
report by the end of the day. This suggests that the issue is related to account
access and authentication, rather than a billing, product, or general technical
issue. The user's intent is to regain access to their account as quickly as

19
possible.

Selected route: account

Account Support Response:

Thank you for reaching out to us about the issue with accessing your account,
John. I apologize for the inconvenience this has caused, especially given the
urgency of your report submission.

For your account's security, I need to verify your identity before assisting
with password-related issues. To ensure that we're communicating with the
legitimate account owner, please provide the following information:

1. Your full name associated with the account

2. The email address registered with the account
3. Your account username (if different from your email address)

Once we verify your identity, we'll guide you through the appropriate steps to
regain access to your account.

In the meantime, I want to emphasize the importance of strong and unique

passwords. It's possible that your password might have been compromised, which
is why I strongly recommend enabling two-factor authentication (2FA) to add an
extra layer of security to your account.

Please be cautious of phishing attempts, and avoid using the same password
across multiple platforms. We'll also review your account's recent activity to
ensure there are no signs of unauthorized access.

Please respond with the requested information, and our team will prioritize your
case. You can expect a resolution within the next 2-4 hours, depending on the
complexity of the issue.

Remember, account security is our top priority, and I'm committed to helping you
regain secure access to your account while minimizing potential risks.

Please respond with the necessary information so we can proceed with verifying
your identity and assisting with account recovery.

Best regards,
Account Support

Ticket 2:
----------------------------------------
Subject: Unexpected charge on my card
Message: Hello, I just noticed a charge of $49.99 on my credit card from
your company, but I thought

20
 I was on the $29.99 plan. Can you explain this charge and adjust
it if it's a mistake?
Thanks,
Sarah

Response:
----------------------------------------

Available routes: ['billing', 'technical', 'account', 'product']

Routing Analysis:

The user, Sarah, is reporting an unexpected charge on her credit card, which
indicates a potential issue with her billing plan. She explicitly mentions the
different plan prices ($29.99 vs $49.99) and requests an explanation and
possible adjustment. This language suggests a billing-related inquiry, rather
than a technical issue or account problem. The tone is polite and inquisitive,
indicating a low-to-moderate urgency level.

Selected route: billing

Billing Support Response:

Dear Sarah,

Thank you for reaching out to us about the unexpected charge on your credit
card. I apologize for any confusion or concern this may have caused. I'm happy
to help clarify the charge and assist with any necessary adjustments.

After reviewing your account, I noticed that you were initially enrolled in our
$29.99 plan, but you had upgraded to our premium plan, which includes additional
features, on February 10th. The premium plan is priced at $49.99 per month. It's
possible that you may not have been aware of the upgrade or the corresponding
price change.

To resolve this issue, I can assist you in downgrading your plan back to the
original $29.99 plan, and I will also apply a one-time credit to your account to
adjust the discrepancy. Please allow 3-5 business days for the credit to be
processed and reflected on your account.

Next steps:

* I will downgrade your plan to the original $29.99 plan, effective immediately.
* I will apply a one-time credit of $20 to your account to adjust the
discrepancy.
* You will receive an email confirmation once the changes have been made.
* If you have any further questions or concerns, please don't hesitate to reach
out to me directly.

21
Payment Options:
If you would like to make a payment towards your account, you can do so by
visiting our website and clicking on the "Make a Payment" link. Alternatively,
you can call our automated payment system at 1-800-555-1234. Please have your
account information and payment details ready.

Thank you for bringing this to our attention, and I'm confident that we can
resolve this issue quickly and to your satisfaction.

Best regards,
[Your Name]
Billing Support Specialist

Ticket 3:
----------------------------------------
Subject: How to export data?
Message: I need to export all my project data to Excel.
I've looked through the docs but can't
figure out how to do a bulk export. Is this possible?
If so, could you walk me through the steps?
Best regards,
Mike

Response:
----------------------------------------

Available routes: ['billing', 'technical', 'account', 'product']

Routing Analysis:
Brief explanation of why this ticket should be routed to the product team: The
user is asking about product functionality and how to achieve a specific task,
indicating a product-focused inquiry.

Selected route: product

Product Support Response:

Hi Mike,

Thank you for reaching out to us! I'd be happy to help you with exporting your
project data to Excel. Yes, bulk exporting is possible, and I'd be delighted to
guide you through the steps.

To export your project data, follow these steps:

1. Log in to your account and navigate to the **Project Overview** page.

2. Click on the **Export** button in the top-right corner of the page.
3. In the **Export Data** window, select the **Excel** option as your preferred
file format.
4. Choose the data range you want to export. You can select **All Data** to

22
export your entire project data or choose a specific date range.
5. Select the columns you want to include in the export. You can choose from a
variety of columns, such as Task Names, Status, Due Dates, and more.
6. Click **Export** to initiate the download process.

You can find more information on exporting data in our documentation section:
[Exporting Data]([Link]

Additionally, if you need to export data on a regular basis, you might want to
explore our **Scheduled Exports** feature. This feature allows you to set up
automatic exports at a frequency that suits your needs.

If you have any further questions or need assistance with anything else, please
don't hesitate to ask. We're here to help!

Best regards,
[Your Name]
Product Specialist

0.4 Workflow 4: Evaluator-Optimizer

So here we are working with the Evaluator-Optimizer workflow, which is simply a workflow where
one LLM call generates a response while another provides the evaluation and feedback in a loop.
When to use this workflow is very interesting because this workflow is very effective when we have
two things. The first is clear evaluation criteria, and the second is that you can get value from
iterative refinement. These two signs of a good fit are:
1. The LLM response can be demonstrably improved when feedback is provided.
2. The LLM can provide meaningful feedback.
So, in a sense this workflow work well for tasks that can be improved, and you have meaningful

23
 feedback too.

[51]: from typing import Tuple, Dict, List

def generate(prompt: str, task: str, context: str = "") -> Tuple[str, str]:

"""Generate and improve a solution based on feedback."""

full_prompt = f"{prompt}\n{context}\nTask: {task}" if context else␣
,→f"{prompt}\nTask: {task}"

response = llm_call(full_prompt)
thoughts = extract_xml(response, "thoughts")
result = extract_xml(response, "response")

print("\n=== GENERATION START ===")

print(f"Thoughts:\n{thoughts}\n")
print(f"Generated:\n{result}")
print("=== GENERATION END ===\n")

return thoughts, result

def evaluate(prompt: str, content: str, task: str) -> Tuple[str, str]:
"""Evaluate if a solution meets requirements."""
full_prompt = f"{prompt}\nOriginal task: {task}\nContent to evaluate:␣
,→{content}"

response = llm_call(full_prompt)
evaluation = extract_xml(response, "evaluation")
feedback = extract_xml(response, "feedback")

print("=== EVALUATION START ===")

print(f"Status: {evaluation}")
print(f"Feedback: {feedback}")
print("=== EVALUATION END ===\n")

return evaluation, feedback

def eval_optimizer(task: str, evaluator_prompt: str, generator_prompt:

str) -> Tuple[str, List[Dict[str, str]]]:
"""Keep generating and evaluating until requirements are met."""
memory = []
chain_of_thought = []

thoughts, result = generate(generator_prompt, task)

[Link](result)
chain_of_thought.append({"thoughts": thoughts, "result": result})

improvement_count = 0

24
 while True:
evaluation, feedback = evaluate(evaluator_prompt, result, task)
if evaluation == "PASS":
return result, chain_of_thought

if evaluation == "NEEDS_IMPROVEMENT":
improvement_count += 1
if improvement_count >= 2:
print("Too many improvements needed. Stopping the process.")
return result, chain_of_thought

context = "\n".join([
"Previous attempts:",
*[f"- {m}" for m in memory],
f"\nFeedback: {feedback}"
])

thoughts, result = generate(generator_prompt, task, context)

[Link](result)
chain_of_thought.append({"thoughts": thoughts, "result": result})

0.4.1 Example: Workflow 4: Evaluator-Optimizer

Here the example we are working on is a coding exercise (compute the full covariance of matrix). The
coding exercise involves generating code and evaluating it based on time complexity and software
engineering best practices. The evaluator will assess the code and provide feedback, indicating
whether it passes, fails, or needs improvement. If the code needs improvement, the feedback is
passed to the generator function which then generates a new version of the code considering the
feedback.

[52]: evaluator_prompt = """

Evaluate this following code implementation for:

[Link] complexity
[Link] engineering best practices

You should be evaluating only and not attemping to solve the task.
Only output "PASS" if all criteria are met and you have
no further suggestions for improvements.
Output your evaluation concisely in the following format.

<evaluation>PASS, NEEDS_IMPROVEMENT, or FAIL</evaluation>

<feedback>
What needs improvement and why.
</feedback>

25
"""

generator_prompt = """
Your goal is to complete the task based on <user input>.
If there are feedback
from your previous generations, you should
reflect on them to improve your solution

Output your answer concisely in the following format:

<thoughts>••••••••••••••••••
[Your understanding of the task and feedback and
how you plan to improve]
</thoughts>

<response>
[Your code implementation here]
</response>
"""

task = """
<user input>
Suppose you have a dataset with n rows (samples) and p columns (features).
You want to compute the full covariance (or correlation) matrix of these p␣
,→features.

Write me Python code this matrix and state the time

complexity in Big-O notation with respect to n and p. You can not use any␣
,→external

libraries for this task.

</user input>
"""

eval_optimizer(task, evaluator_prompt, generator_prompt)

=== GENERATION START ===

Thoughts:
••••••••••••••••••
I understand that the task is to write a Python code to compute the full
covariance matrix of p features in a dataset with n samples without using any
external libraries. This requires implementing the covariance formula from
scratch. I will use nested loops to iterate over the features and samples, and
then calculate the covariance values.

In my previous generations, I learned that I need to consider the time

complexity of my solution, which in this case will be O(n*pˆ2) due to the nested
loops.

26
To improve, I will make sure to provide a clear and concise implementation with
proper variable naming and formatting.

Generated:

```
def compute_covariance_matrix(dataset):
n = len(dataset)
p = len(dataset[0])
covariance_matrix = [[0.0 for _ in range(p)] for _ in range(p)]

# Calculate means of each feature

means = [sum(column) / n for column in zip(*dataset)]

# Calculate covariance values

for i in range(p):
for j in range(p):
total = 0.0
for k in range(n):
total += (dataset[k][i] - means[i]) * (dataset[k][j] - means[j])
covariance_matrix[i][j] = total / (n - 1)

return covariance_matrix
```
Time complexity: O(n*pˆ2)

=== GENERATION END ===

=== EVALUATION START ===

Status: NEEDS_IMPROVEMENT
Feedback:
The code computes the covariance matrix correctly and the time complexity is
indeed O(n*pˆ2).

However, from a software engineering best practices perspective, there are a few
areas for improvement:

1. The function has a tight coupling with the input data structure, which is a
list of lists. It would be better to add a documentation string to specify the
expected input format.
2. The function does not handle edge cases, such as an empty dataset or a
dataset with only one feature. It would be better to add input validation to
handle these cases.
3. The function uses magic numbers (e.g., `n - 1` in the covariance
calculation). It would be better to define these as named constants to improve
readability.

27
4. The function has a lot of repetitive calculations. For example, `means[i]`
and `means[j]` are calculated multiple times. It would be better to calculate
them once and store them in a variable.

Overall, the code is correct but could be improved in terms of robustness,

readability, and maintainability.

=== EVALUATION END ===

=== GENERATION START ===

Thoughts:

I understand that I need to improve my previous solution based on the feedback

provided.
I will add a documentation string to specify the expected input format,
handle edge cases, define named constants for magic numbers, and reduce
repetitive calculations.
I will make my code more robust, readable, and maintainable.

Generated:

```
def compute_covariance_matrix(dataset):
"""
Compute the covariance matrix of a dataset.

Parameters:
dataset (list of lists): A dataset with n rows (samples) and p columns
(features).

Returns:
list of lists: The full covariance matrix of the p features.

Raises:
ValueError: If the dataset is empty or has only one feature.
"""
NAMED_CONSTANT_DEGREE_OF_FREEDOM = -1
if not dataset:
raise ValueError("The dataset is empty.")
n = len(dataset)
p = len(dataset[0])
if p < 2:
raise ValueError("The dataset must have at least two features.")
covariance_matrix = [[0.0 for _ in range(p)] for _ in range(p)]

# Calculate means of each feature

28
 means = [sum(column) / n for column in zip(*dataset)]

# Calculate covariance values

for i in range(p):
for j in range(i, p):
total = 0.0
mean_i = means[i]
mean_j = means[j]
for k in range(n):
total += (dataset[k][i] - mean_i) * (dataset[k][j] - mean_j)
covariance_matrix[i][j] = total / (n +
NAMED_CONSTANT_DEGREE_OF_FREEDOM)
covariance_matrix[j][i] = covariance_matrix[i][j] # symmetric
matrix

return covariance_matrix
```
Time complexity: O(n*pˆ2)

=== GENERATION END ===

=== EVALUATION START ===

Status: NEEDS_IMPROVEMENT
Feedback:
The code is well-documented, and the function signature is clear. However, there
are some areas for improvement:

1. The time complexity could be improved by using a more efficient algorithm for
computing the covariance matrix. The current implementation has a time
complexity of O(n*pˆ2), which can be reduced to O(n*p) using a more efficient
method.

2. The magic number `-1` is used as the degree of freedom. It would be better to
define it as a named constant at the top of the file.

3. The code does not handle the case where the input dataset is not a list of
lists. It would be better to add a type check to ensure that the input is of the
correct type.

4. The docstring could be improved by specifying the units of the returned

covariance matrix (e.g., whether it's scaled by 1/(n-1) or 1/n).

5. The function name `compute_covariance_matrix` could be more descriptive,

e.g., `compute_sample_covariance_matrix` to indicate that it's computing the
sample covariance matrix.

6. The variable names could be more descriptive, e.g., `feature_means` instead

of `means`.

29
 7. The code does not handle the case where the input dataset has non-numeric
values. It would be better to add a check to ensure that the input values are
numeric.

8. The code does not handle the case where the input dataset has missing values.
It would be better to add a check to handle missing values or documents that the
function assumes there are no missing values.

By addressing these areas, the code can be improved to be more efficient,

robust, and maintainable.
=== EVALUATION END ===

Too many improvements needed. Stopping the process.

[52]: ('\n```\ndef compute_covariance_matrix(dataset):\n """\n Compute the

covariance matrix of a dataset.\n\n Parameters:\n dataset (list of lists):
A dataset with n rows (samples) and p columns (features).\n\n Returns:\n
list of lists: The full covariance matrix of the p features.\n\n Raises:\n
ValueError: If the dataset is empty or has only one feature.\n """\n
NAMED_CONSTANT_DEGREE_OF_FREEDOM = -1\n if not dataset:\n raise
ValueError("The dataset is empty.")\n n = len(dataset)\n p =
len(dataset[0])\n if p < 2:\n raise ValueError("The dataset must have
at least two features.")\n covariance_matrix = [[0.0 for _ in range(p)] for _
in range(p)]\n\n # Calculate means of each feature\n means = [sum(column)
/ n for column in zip(*dataset)]\n\n # Calculate covariance values\n for i
in range(p):\n for j in range(i, p):\n total = 0.0\n
mean_i = means[i]\n mean_j = means[j]\n for k in
range(n):\n total += (dataset[k][i] - mean_i) * (dataset[k][j] -
mean_j)\n covariance_matrix[i][j] = total / (n +
NAMED_CONSTANT_DEGREE_OF_FREEDOM)\n covariance_matrix[j][i] =
covariance_matrix[i][j] # symmetric matrix\n\n return
covariance_matrix\n```\nTime complexity: O(n*pˆ2)\n',
[{'thoughts': '••••••••••••••••••\nI understand that the task is to write a
Python code to compute the full covariance matrix of p features in a dataset
with n samples without using any external libraries. This requires implementing
the covariance formula from scratch. I will use nested loops to iterate over the
features and samples, and then calculate the covariance values. \n\nIn my
previous generations, I learned that I need to consider the time complexity of
my solution, which in this case will be O(n*pˆ2) due to the nested loops.\n\nTo
improve, I will make sure to provide a clear and concise implementation with
proper variable naming and formatting.\n',
'result': '\n```\ndef compute_covariance_matrix(dataset):\n n =
len(dataset)\n p = len(dataset[0])\n covariance_matrix = [[0.0 for _ in
range(p)] for _ in range(p)]\n \n # Calculate means of each feature\n
means = [sum(column) / n for column in zip(*dataset)]\n \n # Calculate
covariance values\n for i in range(p):\n for j in range(p):\n

30
 total = 0.0\n for k in range(n):\n total +=
(dataset[k][i] - means[i]) * (dataset[k][j] - means[j])\n
covariance_matrix[i][j] = total / (n - 1)\n \n return
covariance_matrix\n```\nTime complexity: O(n*pˆ2)\n'},
{'thoughts': '\nI understand that I need to improve my previous solution based
on the feedback provided. \nI will add a documentation string to specify the
expected input format, \nhandle edge cases, define named constants for magic
numbers, and reduce repetitive calculations. \nI will make my code more robust,
readable, and maintainable.\n',
'result': '\n```\ndef compute_covariance_matrix(dataset):\n """\n
Compute the covariance matrix of a dataset.\n\n Parameters:\n dataset
(list of lists): A dataset with n rows (samples) and p columns (features).\n\n
Returns:\n list of lists: The full covariance matrix of the p features.\n\n
Raises:\n ValueError: If the dataset is empty or has only one feature.\n
"""\n NAMED_CONSTANT_DEGREE_OF_FREEDOM = -1\n if not dataset:\n
raise ValueError("The dataset is empty.")\n n = len(dataset)\n p =
len(dataset[0])\n if p < 2:\n raise ValueError("The dataset must have
at least two features.")\n covariance_matrix = [[0.0 for _ in range(p)] for _
in range(p)]\n\n # Calculate means of each feature\n means = [sum(column)
/ n for column in zip(*dataset)]\n\n # Calculate covariance values\n for i
in range(p):\n for j in range(i, p):\n total = 0.0\n
mean_i = means[i]\n mean_j = means[j]\n for k in
range(n):\n total += (dataset[k][i] - mean_i) * (dataset[k][j] -
mean_j)\n covariance_matrix[i][j] = total / (n +
NAMED_CONSTANT_DEGREE_OF_FREEDOM)\n covariance_matrix[j][i] =
covariance_matrix[i][j] # symmetric matrix\n\n return
covariance_matrix\n```\nTime complexity: O(n*pˆ2)\n'}])

[ ]:

31