Azure OpenAI Workshop
Introduction
Agenda
• Azure Open AI Overview
• LLM Adaptation with Prompt Engineering
• Lab 1 : Build your first Open AI app or Use Open AI Python SDK
• Lab 2 (optional ): Ask question on SQL data
• Lab 3 (optional): Ask question on unstructured data
• Lab 4 (optional): Build Open AI knowledge extraction pipeline
Azure OpenAI | Overview
Foundation Models Tasks
Question and
Answering
Data
Sentiment
Analysis
Text
Images Information
Extraction
Speech Training Foundation Adaptation
Model
Structured Image
Data Captioning
3d Signals Transformer Model
Object
Recognition
Instruction
Follow
Generative AI
GPT-3 Codex DALL·E ChatGPT (GPT-Turbo)
Prompt: Prompt: Prompt: A white Siamese cat What is the fastest animal on land?
Write a tagline for an ice Table customers, columns = Response:
cream shop. [CustomerId, FirstName, the cheetah (Acinonyx jubatus),
LastName, Company, Address, which can reach speeds of up to 60
miles (97 kilometers) per hour.
Response: City, State, Country,
PostalCode]
We serve up smiles with What makes them so fast?
every scoop! Create a SQL query for all
customers in Texas named
Jane
query = Cheetahs are built for speed and
have several adaptations that make
them the fastest land animal: lean
Response: body, long legs, flexible spine,
large nostrils and claws that don’t
retract.
SELECT *
FROM customers
WHERE State = 'TX' AND
FirstName = 'Jane'
Azure AI
Applicati Partner
Solutions
ons
Business
Users
Application
Platform
AI Builder
Power BI Power Apps Power Automate Power Virtual Agents
Scenario-Based
Services
Applied AI Services
Bot Service Cognitive Search Form Recognizer Video Indexer Metrics Advisor Immersive Reader
Customizable AI
Models
Cognitive Developers &
Services Vision Speech Language Decision OpenAI Service Data Scientists
ML Azure Machine Learning
Platform
Deployed within your Azure
subscription, secured by you, accessed
only by you, and tied to your datasets
and applications
Azure OpenAI Service Large, pretrained AI models to
unlock new scenarios
GPT-3 Codex Custom AI models fine-tuned with
your data and hyperparameters
DALL·E (preview)
Built-in responsible AI to detect and
mitigate harmful use
Enterprise-grade security with role-based
access control (RBAC) and private
networks
Azure OpenAI | Top 4 Capabilities & Use Cases
Content Code Semantic
Summarizatio generation
Generatio n Search
n
Call Center Analytics:
Call Center Analytics: Convert Natural Language to
Automatically generate Search reviews for a specific
Summary of customer SQL (or vice versa) for
responses to customer product / service
support conversation logs telemetry data
inquiries
Subject Matter Expert
Convert Natural Language to
Generate personalised UI for Document Summarization Information Discovery and
Query Proprietary Data
your website (e.g. financial reporting, Knowledge Mining
Models
analyst articles)
Social Media Trends
Code Documentation
Summarization
Examples of Multiple Model Use
Cases
End to End Call Center Analytics: Classification, Sentiment, Entity Extraction, Summarization and Email Generation
Customer 360: Hyper-personalisation using timely Summarization of customer queries & trends, Search, and Content Generation
Business Process Automation: Search through structured & unstructured documentation, Generate Code to query data models,
Content Generation
Lab Introduction
Lab Overview [Link]
Prompt
Engineering
Build your first
OpenAI Use OpenAI
application with Python SDK
Power App
Option 2: An AOAI Option 3: Make
Option 1: Asking
data pipeline to ChatGPT works on
natural language
extract insights your own
question on SQL
from unstructured unstructured
data
data dataset
Build your first Open AI App
2 min demo
Enable natural language question on structured
data
2 min demo
Enable natural language question on unstructured
data
Prompt Engineering
How to adapt GPT-3 model for your task
No Gradient Updates
Zero-Shot One-Shot Few-Shot
The model predicts the answer given only In addition to the task description, the In addition to the task description, the
a natural language description of the task. model sees a single example of the task model sees a few examples of the task.
Fine Tuning
The model is trained via repeated gradient updates using a large corpus of example
tasks.
Prepare and
Train a new fined Use your fine-
upload training
tuned model tuned model
data
1. 2. 3. 4.
Higher quality results Ability to train on more Token savings due Lower latency requests
than prompt design examples than can fit in a to shorter prompts
prompt
2 min demo
Prompt Engineering Basic
Prompt engineering is the process of designing and refining prompts or input prompts for
machine learning models that enable them to produce relevant and useful outputs.
Instruction - a
specific task or
Output Indicator - indicates
instruction you
the type or format of output.
want the model to
perform
Input Data - is the input or
question that we are
interested to find a response
for Context - can involve
external information or
additional context that can
steer the model to better
responses
2 min demo
LLM setting
• Temperature - The lower the temperature the more
deterministic the results in the sense that the highest probable
next token is always picked. Increasing temperature could lead
to more randomness encouraging more diverse or creative
outputs. We are essentially increasing the weights of the other
possible tokens. In terms of application, we might want to use
lower temperature for something like fact-based QA to
encourage more factual and concise responses. For poem
generation or other creative tasks it might be beneficial to
increase temperature.
• Top_p - Similarly, with top_p, a sampling technique with
temperature called nucleus sampling, you can control how
deterministic the model is at generating a response. If you are
looking for exact and factual answers keep this low. If you are
looking for more diverse responses, increase to a higher value.
• The general recommendation is to alter one not both.
Basic applications (use playground demo)
• Text Summarization
• Information Extraction
• Question Answering
• Text Classification
• Conversation
• Code Generation
• Data Generation
• Recommendations
2 min demo
Tips for designing prompt
• Start Simple
• The Instruction
• Specificity
• Avoid Impreciseness
• To do or not to do?
Advanced tips
• Zero shot, few-shot
• LLM may suffer from greedy problem. Requires step by step reasoning or
provide step by step instruction will provide better result in complex scenario
• Conversation: control style and behavior of chatbot
Mini-challenge for audience
• Use Open AI to generate a customer support transcript for internet service provider ABC
wireless corp. Customer called to complain about technical issue and tech rep handled
the call. The transcript has a timestamp at the beginning to indicate date and time of
the call
• Write a prompt to extract the information in the following structured format from the
generated customer support’s transcript in ISP company.
{ "issue": "product fault", "product": "Linksys router", "resolution": "resolved",
"sentiment": "positive“, “datetime:”Mar-12-2023 13:30:21” }
Fine tuning vs. prompt engineering
• Why fine tuning?
• You may have language patterns, context or communication or
special behavior that you need to hardwire into the LLM’s weights
to simplify prompt
• Prompt engineering offers flexibility but comes at longer prompt ->
higher cost
• When to use fine-tuning
- It’s a repeated pattern across multiple applications (i.e. why
attaching the same instruction/context again and again?)
- Users expect the model/chatbot to already have a domain
understanding (context) which can be too large to insert into every
prompt
- You have good quality data (prompt-completion format)
- You want your LLM to have special behavior based on input data
alone that prompt cannot be prepared in advance such as refusing
to answer certain question due to lack of information, asking
Envision your intelligent
Application
Thank
you
Evolution of NLP
Word2Vec FastText BlazingText GPT GPT-3
Jan 2013 Jan 2016 Nov 2017 Jul 2018 May 2020
Glove Transforme ELMo BERT
Jan 2014 rs Feb 2018 Oct 2018
Jun 2017
In-context Learning
Zero-shot
The model predicts the answer given only a natural
Zero-shot - Predicting with no sample
language description of the task. No gradient
updates are performed.
provided 1 Translate English to
French:
<------ task description
2 <------ prompt
Cheese =>
One-shot
In addition to the task description, the model sees a
single example of the task. No gradient updates are
One-shot - Predicting with one sample performed.
provided
Translate English to <------ task description
French:
2 <------ example
Sea otter => loutre de
3 mer <------ prompt
Cheese =>
Few-shot
In addition to the task description, the model sees a
few examples of the task. No gradient updates are
performed
1 Translate English to <------ task description
Few-shot – Predicting with a few samples 2
French:
Sea otter => loutre de mer
<------ examples
provided
3 <------
Peppermint => menthe
4 poivre <------
5 Plush giraffe => girafe <------ prompt
peluche
Cheese =>
Microsoft Confidential
Family of Models
A series of models that can understand and generate natural language.
GPT-3
A series of models that can understand and generate code, including
Codex translating natural language to code.
A set of models that can understand and use embeddings. An embedding is
a special format of data representation that can be easily utilized by
Embeddings machine learning models and algorithms. The embedding is an information
dense representation of the semantic meaning of a piece of text. Currently,
we offer three families of Embeddings models for different functionalities:
similarity, text search, and code search.
*[Link]
models
Azure OpenAI Service models | Family of Models
GPT-3 Codex
Davinci Davinci-codex
• 4000 Tokens
• Summarizing for
specific audience
• Generating creative content
Curie Cushman-
Inferencing
• 2048 Tokens
codex
time
• Answering questions
• Complex, nuanced
Babbage classification
• 2048 Tokens
• Semantic search ranking
• Moderately complex
Ada
classification
• 2048 Tokens
• Simple classification
• Parsing and formatting
text Capability
Capability
Regional Availability – Embeddings Models
Model Supports East US South Central West
Embeddings US Europe
text-similarity-ada-001 X X X X
text-similarity-babbage-001 X X X
text-similarity-curie-001 X X X X
text-similarity-davinci-001 X X X
text-search-ada-doc-001 X X X
text-search-ada-query-001 X X X
text-search-babbage-doc-001 X X X
text-search-babbage-query-001 X X X
text-search-curie-doc-001 X X X
text-search-curie-query-001 X X X
text-search-davinci-doc-001 X X X
text-search-davinci-query-001 X X X
code-search-ada-code-001 X X X
code-search-ada-text-001 X X X
code-search-babbage-code-001 X X X
code-search-babbage-text-001 X X X
Regional Availability – GPT-3
Model Supports Completions East US South Central US West Europe
Text-ada-001 X X X X
Text-babbage-001 X X X X
Text-curie-001 X X X X
Text-davinci-001 X X X
Text-davinci-002 X X X X
Text-davinci-003 X X
GPT-3
• 3rd generation of Generative Pre-Trained transformer
model
• Autoregressive language model, Generative in nature
i.e., continues generating text from previously
generated token
• 175B parameters!
• Trained with a massive amount of text from web
scraping (Common Crawl), Books, Wikipedia etc.
• Achieves strong performance on many NLP datasets
including:
• Summarization
• Translation
• Question-answering
• Cloze tasks
• Performing 3-digit arithmetic
Reference: [Link]
Azure OpenAI | GPT-3 Family of Models
Model Request Description, performance, costUse cases
Davinci 4,000 Most capable GPT-3 model. Can Complex intent, cause
tokens do any task the other models can and effect,
do, often with higher quality, summarization for
longer output and better audience
instruction-following.
Curie
2048 Very capable, but faster and Language translation,
tokens lower cost than Davinci. complex classification,
text sentiment,
Babbag summarization
e Capable of straightforward tasks,
2048 very fast, and lower cost. Moderate classification,
tokens semantic search
Ada Capable of very simple tasks, classification
usually the fastest model in the
2048 GPT-3 series, and lowest cost. Parsing text, simple
Azure OpenAI | Overview of GPT-3
Generative pre-trained transformer
3 (GPT-3)
Autoregressive language model that uses deep
learning to produce human-like text
Pre-trained on trillions of words
Predicts the most likely next word based on
input text
General text-in/text-out interface
Azure OpenAI | GPT-3 Prompt Design
Prompt – Text input that Completion – Output
some that GPT-3 generates
provides some context to
context based on the prompt.
the engine on what is
expecting.
Models and platform roadmap
March 2022 December 2022 Q1 2023
June 2020 Jan 2021 June 2021 Aug 2021
DALL-E and Codex Embeddings Azure OpenAI DALL*E 2
GPT-3 Customize
CLIP models & Instruction Service GA Private Preview
Models models
+ models (Fine-tuning) Following
OpenAI API Models
Sept 2021 Nov 2021 October Early 2023
April 2021
2022 Text-davinci-003
LoRA GitHub Azure OpenAI Azure OpenAI
Efficient Copilot Private preview Public preview
Fine-tuning on Azure Codex Fine
OpenAI Tuning
Unlock new use cases
Top Use Cases
Writing Government agency using Azure OpenAI Service to extract and
summarize key information from their extensive library of rural
assistance development reports
Financial services using Azure OpenAI Service to summarize
financial reporting for peer risk analysis and customer
conversation summarization
Code Aircraft company using to convert natural language to SQL for
generation aircraft telemetry data
Consulting service using Azure OpenAI Service to convert natural
language to query propriety data models
Reasoning Financial services firm using Azure OpenAI Service to improve
search capabilities and the conversational quality of a
over data customer’s Bot experience.
Insurance companies extract information from volumes of
unstructured data to automate claim handling processes
Summarizatio International insurance company using Azure OpenAI Service to
provide summaries of call center customer support conversation
n logs
Azure OpenAI Languag
Service Use e
Reason over structured and unstructured data:
Classification, Sentiment, Entity Extraction,
Summarization
• Call center call transcripts
Cases Search
• Refined text-based search with embeddings
• Subject Matter Expert
Documents
• Product feedback sentiment • Competitive analysis
• Customer and employee feedback • Peer Analysis
classification
• Technical reports
• Claims and risk analyses
• Product and service feedback
• Support emails and call transcripts
• Social media trends
Codex • Social media trends
• Natural Language to Code
Writing Assistance Conversational AI
• Natural Language to SQL
• Marketing copy / email taglines • Smart assists for call centers
• Code to Natural Language
• Long format text • Tech support chat bots
• Code documentation
•Use
Paragraphs from
Cases that bullets
use multiple model capabilities • Virtual assistants
• Refactoring
• Contact Centers • Artificial humans
• Classification – route mails to appropriate team
DALL·E 2
• Sentiment – prioritize angry customers
• Creative ideation
• Entity extraction and search – analyze liability and risk
• Podcast and music playlist images
• Mail and call transcript summarization
• Content syndication
• Customer response email generation
• Marketing campaign
personalization • Rapid response marketing campaigns: classification, sentiment,
summarization, content generation, image generation
• Hyper-personalization