RAG DEEP DIVE 🤿

Extending RAG with Function Calling

[Link]/ragdeepdive/extending/slides

Anthony Shaw
Python Cloud Advocate
RAG DEEP DIVE 🤿
🚀 1/13: The RAG solution for Azure
🎨 1/15: Customizing the RAG solution
🔎 1/21: Optimal retrieval with Azure AI Search
1/22: Multimedia data ingestion
, 1/27: User login and data access control
📗 1/29: Storing chat history
🎤 2/3: Adding speech input and output
🔒 2/5: Private deployment
💯 2/10: Evaluating RAG answer quality
📈 2/12: Monitoring and tracing LLM calls
🦾 2/18: Extending RAG with function calling
Our RAG chat solution
Azure OpenAI +
Azure AI Search +
Azure Container Apps / App
Service

Features:
• Simple & Advanced RAG
• Conversations ("multi-turn")
• Optional vision integration
• Optional data access control
Code:
[Link]/ragchat

Demo (public):
[Link]/ragchat/demo
Today we'll extend that RAG!
• Function calling 101
• Query rewriting with function calling
• API calling with function calling
• Calling Bing API via Agents SDK
Function calling
Function calling with OpenAI API
Use JSON schema to describe the functions that the model can call:

tools = [{ response = [Link](

"type": "function", model=MODEL_NAME,
"function": { messages=[
"name": "lookup_weather", {"role": "system",
"description": "Lookup the weather "content": "You're a weather chatbot."},
for a given city name or zip code.", {"role": "user",
"parameters": { "content": "whats the weather in
"type": "object", berkeley?"},
"properties": { ],
"city_name": { tools=tools,
"type": "string", )
"description": "The city
name", tool_call =
}, [Link][0].message.tool_calls[0]
"zip_code": { func_name = tool_call.[Link]
"type": "string", [Link]
func_args = tool_call.[Link]
"description": "The zip code"
[Link]
}}}}}]
 Providing multiple functions
response = [Link](
Your LLM can choose model=MODEL_NAME,
from multiple function messages=[
definitions. {"role": "system",
"content": "You're a tourism chatbot."},
{"role": "user",
"content": "what film can I watch in
berkeley?"},
],
tool_choice can be: tools=[get_weather, get_movies],
tool_choice="auto"
• "auto": call 0, 1, or multiple )
functions
• "required": call at least one tool_call =
function [Link][0].message.tool_calls[0]
• specific function name func_name = tool_call.[Link]
• "none": don't call any func_args = tool_call.[Link]
* In our experience, it rarely/never calls more than one function.
Function calling in RAG
Simple RAG flow: No function
calling
This flow is used by the "Ask" tab in the RAG solution.

Yes, the Northwind Health Plus

Does the Northwind plan covers eye exams. 1
Health Plus plan cover
eye exams?

“[Link]:
Health Plus is a
comprehensive plan that
Question
Retrieval with offers more coverage than answering
User Northwind Standard.
AI Search Northwind Health Plus with
Question offers coverage for
emergency services,
OpenAI LLM
mental health and
substance abuse
coverage, and out-of-
network services, while
 Advanced RAG flow: Function
calling!
This flow is used by the "Chat" tab in the RAG solution.

Does the Northwind Health Plus

plan cover eye exams?

Yes, the Northwind Health Plus plan Yes, the Northwind Health Plus
covers eye exams. 1 plan also covers hearing tests. 1

Hearing too?

“Northwind “[Link]:
Health Plus plan Health Plus is a Question
Conversation coverage for eye comprehensive plan that
Retrieval with answering
Query exams and offers more coverage than
hearing” AI Search Northwind Standard. with
rewriting Northwind Health Plus
offers coverage for OpenAI LLM
with OpenAI emergency services...
Query rewriting function call
[{
"type": "function",
"function": {
"name": "search_sources",
"description": "Retrieve sources from the Azure AI Search index",
"parameters": {
"type": "object",
"properties": {
"search_query": {
"type": "string",
"description": "Query string to retrieve documents from Azure
Search eg: 'Health care plan'"
}
},
"required": ["search_query"]
}
}
}]
Query rewriting function call:
Processing
response_message = chat_completion.choices[0].message

if response_message.tool_calls:
for tool in response_message.tool_calls:
if [Link] != "function":
continue
function = [Link]
if [Link] == "search_sources":
arg = [Link]([Link])
search_query = [Link]("search_query",
self.NO_RESPONSE)
Why even use function calling?
1. In practice, we got better results from the LLM when
asking for the rewritten query as a function argument
2. It makes it easier for developers to extend the function
calling to cover more scenarios...
Function calling scenarios
Example: Searching a GitHub issue
tracker
{
"type": "function",
"function": {
"name": "github_search_issues",
"description": "Retrieve issues from the azure-search-openai-demo issue
tracker. Use this function for questions like 'what are the top errors with
deployment?'",
"parameters": {
"type": "object",
"properties": {
"search_query": {
"type": "string",
"description": "Query string to retrieve issues from github eg:
'Deployment failure' - should only contain the search terms, does not need
'issue' or 'issues' in the search query."
}
},
"required": ["search_query"]}}}
[Link]
ub
Example: Generating filters for AI
Search
{ "type": "function",
"function": {
"name": "search_by_filename",
"description": "Retrieve a specific filename from the Azure AI Search index",
"parameters": {
"type": "object",
"properties": {
"filename": {
"type": "string",
"description": "The filename, like '[Link]'"}},
"required": ["filename"]}}}

if [Link] == User can now ask

"search_by_filename":
arg = [Link]([Link]) "Summarize
filename = [Link]("filename", "") [Link]"
filename_filter = filename
and it will find the exact
[Link]
file.
Example: Escalate to a real
person
{"type": "function",
"function": {
"name": "human_escalation",
"description": "Check if user wants
to escalate to a human",
"parameters": {
"type": "object",
"properties": {
"requires_escalation": {
"type": "boolean",
"description": "If user is
showing signs of frustration or anger
in the query. Also if the user says
they want to talk to a real person and
not a chat bot."}},
[Link]
"required": -search-openai-demo/pull/1176
["requires_escalation"]}}}
Calling the Bing API
 Former way of searching Bing:
HTTP API
base_url = f"[Link]
headers = {
"Ocp-Apim-Subscription-Key": api_key
}
params = {
"q": query,
"mkt": lang,
"textDecorations": True,
"textFormat": "HTML",
"responseFilter": "Webpages",
"safeSearch": "Strict",
"setLang": lang,
}
async with [Link]() as client:
response = await [Link](self.base_url, headers=[Link], params=params)
response.raise_for_status()
return WebAnswer.model_validate([Link]()["webPages"])

[Link]
New way of searching Bing: AI
Agents SDK
project_client = AIProjectClient.from_connection_string(connection_string,
cred)
async with project_client:
thread = await project_client.agents.create_thread()
message = await project_client.agents.create_message(
thread_id=[Link], role="user", content=query)
run = await project_client.agents.create_and_process_run(
thread_id=[Link], assistant_id=agent_id)
messages = await project_client.agents.list_messages(thread_id=[Link])
first_message = [Link][0].content[0]
url = first_message.[Link][0].as_dict()['url_citation']['url']
title = first_message.[Link][0].as_dict()['url_citation']
['title']
snippet = first_message.[Link]
return WebAnswer(totalEstimatedMatches=1,
webSearchUrl="[Link]
value=[WebPage(id="1", name=title, url=url,
[Link]
displayUrl=url,
language="en", snippet=snippet)])
 RAG DEEP DIVE 🤿
Next steps
🚀 1/13: The RAG solution for Azure
• Watch past streams! → 🎨 1/15: Customizing the RAG solution
🔎 1/21: Optimal retrieval with Azure AI Sea
• Deploy the RAG chat app: 1/22: Multimedia data ingestion
[Link]/ragchat , 1/27: User login and data access contro
Post questions in the issue 📗 1/29: Storing chat history
🎤 2/3: Adding speech input and output
tracker or discussions
🔒 2/5: Private deployment
💯 2/10: Evaluating RAG answer quality
• Come to Pamela's Office 📈 2/12: Monitoring and tracing LLM calls
Hours on Thursdays in 🦾 2/18: Extending RAG with function callin
Discord:
[Link]/ragdeepdive/oh Watch all the sessions @
[Link]
ist
Sign up for more RAG
resources!

[Link]/thesource/RAG