Machine learning

What is AI?
Simply put, AI is software that imitates human behaviors and capabilities. Key workloads include:

Machine learning - This is often the foundation for an AI system, and is the way we "teach" a computer
model to make predictions and draw conclusions from data.

Computer vision - Capabilities within AI to interpret the world visually through cameras, video, and images.

Natural language processing - Capabilities within AI for a computer to interpret written or spoken language,
and respond in kind.

Document intelligence - Capabilities within AI that deal with managing, processing, and using high volumes
of data found in forms and documents.

Knowledge mining - Capabilities within AI to extract information from large volumes of often unstructured
data to create a searchable knowledge store.

Generative AI - Capabilities within AI that create original content in a variety of formats including natural
language, image, code, and more.

How do machines learn?

The answer is, from data. In today's world, we create huge volumes of data as we go about our everyday
lives. From the text messages, emails, and social media posts we send to the photographs and videos we
take on our phones, we generate massive amounts of information. More data still is created by millions of
sensors in our homes, cars, cities, public transport infrastructure, and factories.

Data scientists can use all of that data to train machine learning models that can make predictions and
inferences based on the relationships they find in the data.

Machine learning models try to capture the relationship between data. For example, suppose an
environmental conservation organization wants volunteers to identify and catalog different species of
wildflower using a phone app. The following animation shows how machine learning can be used to
enable this scenario.

1. A team of botanists and scientists collect data on wildflower samples.

2. The team labels the samples with the correct species.

3. The labeled data is processed using an algorithm that finds relationships between the features of the
samples and the labeled species.

4. The results of the algorithm are encapsulated in a model.

5. When new samples are found by volunteers, the model can identify the correct species label.
Approaches to AI have advanced to complete tasks of much greater complexity. These complex models
form the basis of AI capabilities.
Machine learning in Microsoft Azure
Microsoft Azure provides the Azure Machine Learning service - a cloud-based platform for creating,
managing, and publishing machine learning models. Azure Machine Learning Studio offers multiple
authoring experiences such as:

1. Automated machine learning: this feature enables non-experts to quickly create an effective machine
learning model from data.

2. Azure Machine Learning designer: a graphical interface enabling no-code development of machine
learning solutions.

3. Data metric visualization: analyze and optimize your experiments with visualization.

4. Notebooks: write and run your own code in managed Jupyter Notebook servers that are directly
integrated in the studio.

Computer Vision

Most computer vision solutions are based on machine learning models that can be applied to visual input
from cameras, videos, or images. The following table describes common computer vision tasks.

Image classification involves training a machine learning model to classify images based on their contents.
For example, in a traffic monitoring solution you might use an image classification model to classify
images based on the type of vehicle they contain, such as taxis, buses, cyclists, and so on.

Object detection machine learning models are trained to classify individual objects within an image, and
identify their location with a bounding box. For example, a traffic monitoring solution might use object
detection to identify the location of different classes of vehicle.

Semantic segmentation is an advanced machine learning technique in which individual pixels in the image
are classified according to the object to which they belong. For example, a traffic monitoring solution might
overlay traffic images with "mask" layers to highlight different vehicles using specific colors.

You can create solutions that combine machine learning models with advanced image analysis techniques
to extract information from images, including "tags" that could help catalog the image or even descriptive
captions that summarize the scene shown in the image.

Face detection is a specialized form of object detection that locates human faces in an image. This can be
combined with classification and facial geometry analysis techniques to recognize individuals based on
their facial features.

Optical character recognition is a technique used to detect and read text in images. You can use OCR to
read text in photographs (for example, road signs or store fronts) or to extract information from scanned
documents such as letters, invoices, or forms.

Computer vision services in Microsoft Azure

You can use Microsoft's Azure AI Vision to develop computer vision solutions. The service features are
available for use and testing in the Azure Vision Studio and other programming languages. Some features
of Azure AI Vision include:
Image Analysis: capabilities for analyzing images and video, and extracting descriptions, tags, objects, and
text.
Face: capabilities that enable you to build face detection and facial recognition solutions.
Optical Character Recognition (OCR): capabilities for extracting printed or handwritten text from images,
enabling access to a digital version of the scanned text.
NLP
Natural language processing (NLP) is the area of AI that deals with creating software that understands
written and spoken language.

NLP enables you to create software that can:

1. Analyze and interpret text in documents, email messages, and other sources.
2. Interpret spoken language, and synthesize speech responses.
3. Automatically translate spoken or written phrases between languages.
4. Interpret commands and determine appropriate actions.

Understand document intelligence and knowledge mining

Document Intelligence is the area of AI that deals with managing, processing, and using high volumes of a
variety of data found in forms and documents. Document intelligence enables you to create software that
can automate processing for contracts, health documents, financial forms and more

Document intelligence in Microsoft Azure

You can use Microsoft's Azure AI Document Intelligence to build solutions that manage and accelerate
data collection from scanned documents. Features of Azure AI Document Intelligence help automate
document processing in applications and workflows, enhance data-driven strategies, and enrich document
search capabilities. You can use prebuilt models to add intelligent document processing for invoices,
receipts, health insurance cards, tax forms, and more. You can also use Azure AI Document Intelligence
to create custom models with your own labeled datasets. The service features are available for use and
testing in the Document Intelligence Studio and other programming languages.

Knowledge Mining
Knowledge mining is the term used to describe solutions that involve extracting information from large
volumes of often unstructured data to create a searchable knowledge store.

Understand generative AI

Generative AI describes a category of capabilities within AI that create original content. People typically
interact with generative AI that has been built into chat applications. Generative AI applications take in
natural language input, and return appropriate responses in a variety of formats including natural language,
image, code, and audio.
Generative AI in Microsoft Azure
Azure OpenAI Service is Microsoft's cloud solution for deploying, customizing, and hosting generative AI
models. It brings together the best of OpenAI's cutting edge models and APIs with the security and
scalability of the Azure cloud platform.
Azure OpenAI Service supports many generative model choices that can serve different needs. You can
use Azure AI Studio to create generative AI solutions, such as custom copilot chat-based assistants that use
Azure OpenAI Service models

In this example, an Azure OpenAI Service model is used to power a copilot application that can be used to
generate original content in response to user prompts, such as a request to write a cover letter.
 Challenges and risks with AI

Artificial Intelligence is a powerful tool that can be used to greatly benefit the world. However, like any tool,
it must be used responsibly.

The following table shows some of the potential challenges and risks facing an AI application developer.

Challenge or Risk

Bias can affect results

A loan-approval model discriminates by gender due to bias in the data with which it was trained

Errors may cause harm

An autonomous vehicle experiences a system failure and causes a collision

Data could be exposed

A medical diagnostic bot is trained using sensitive patient data, which is stored insecurely

Solutions may not work for everyone

A home automation assistant provides no audio output for visually impaired users

Users must trust a complex system

An AI-based financial tool makes investment recommendations - what are they based on?

Who's liable for AI-driven decisions?

An innocent person is convicted of a crime based on evidence from facial recognition who's
responsible?

Understand Responsible AI
At Microsoft, AI software development is guided by a set of six principles, designed to ensure that AI
applications provide amazing solutions to difficult problems without any unintended negative consequences.

MACHINE LEARNING

Machine learning is in many ways the intersection of two disciplines - data science and software engineering
. The goal of machine learning is to use data to create a predictive model that can be incorporated into a
software application or service. To achieve this goal requires collaboration between data scientists who
explore and prepare the data before using it to train a machine learning model, and software developers
who integrate the models into applications where they're used to predict new data values (a process known
as inferencing).

In this module, you'll explore some of the core concepts on which machine learning is based, learn how to
identify different kinds of machine learning models, and examine the ways in which machine learning
models are trained and evaluated. Finally, you'll learn how to use Microsoft Azure Machine Learning to train
and deploy a machine learning model, without needing to write any code.

Note

Machine learning is based on mathematical and statistical techniques, some of which are described at a
high level in this module. Don't worry if you're not a mathematical expert though! The goal of the module is
to help you gain an intuition of how machine learning works - we'll keep the mathematics to the minimum
required to understand the core concepts.
Machine learning has its origins in statistics and mathematical modeling of data. The fundamental idea of
machine learning is to use data from past observations to predict unknown outcomes or values. For
example:

1. The proprietor of an ice cream store might use an app that combines historical sales and weather
records to predict how many ice creams they're likely to sell on a given day, based on the weather forecast
.2. A doctor might use clinical data from past patients to run automated tests that predict whether a new
patient is at risk from diabetes based on factors like weight, blood glucose level, and other measurements.
3. A researcher in the Antarctic might use past observations automate the identification of different
penguin species (such as Adelie, Gentoo, or Chinstrap) based on measurements of a bird's flippers, bill,
and other
physical attributes.

Machine learning as a function : Because machine learning is based on mathematics and statistics, it's
common to think about machine learning models in mathematical terms. Fundamentally, a machine
learning model is a software application that encapsulates a function to calculate an output value based on
one or more input values. The process of defining that function is known as training. After the function has
been defined, you can use it to predict new values in a process called inferencing.

Let's explore the steps involved in training and inferencing.

Diagram showing the training and inferencing phases in machine learning.

The training data consists of past observations. In most cases, the observations include the observed
attributes or features of the thing being observed, and the known value of the thing you want to train a
model to predict (known as the label).

In mathematical terms, you'll often see the features referred to using the shorthand variable name x, and
the label referred to as y. Usually, an observation consists of multiple feature values, so x is actually a
vector (an array with multiple values), like this: [x1,x2,x3,...].

To make this clearer, let's consider the examples described previously:

In the ice cream sales scenario, our goal is to train a model that can predict the number of ice cream sales
based on the weather. The weather measurements for the day (temperature, rainfall, windspeed, and so
on ) would be the features (x), and the number of ice creams sold on each day would be the label (y).
In the medical scenario, the goal is to predict whether or not a patient is at risk of diabetes based on their
clinical measurements. The patient's measurements (weight, blood glucose level, and so on) are the
features (x), and the likelihood of diabetes (for example, 1 for at risk, 0 for not at risk) is the label (y).
In the Antarctic research scenario, we want to predict the species of a penguin based on its physical
attributes. The key measurements of the penguin (length of its flippers, width of its bill, and so on) are the
features (x), and the species (for example, 0 for Adelie, 1 for Gentoo, or 2 for Chinstrap) is the label (y).
An algorithm is applied to the data to try to determine a relationship between the features and the label,
and generalize that relationship as a calculation that can be performed on x to calculate y. The specific
algorithm used depends on the kind of predictive problem you're trying to solve (more about this later), but
the basic principle is to try to fit a function to the data, in which the values of the features can be used to
calculate the label.

The result of the algorithm is a model that encapsulates the calculation derived by the algorithm as a
function - let's call it f. In mathematical notation:

y = f(x)

Now that the training phase is complete, the trained model can be used for inferencing. The model is
essentially a software program that encapsulates the function produced by the training process. You can
input a set of feature values, and receive as an output a prediction of the corresponding label. Because the
output from the model is a prediction that was calculated by the function, and not an observed value, you'll
often see the output from the function shown as (which is rather delightfully verbalized as "y-hat").
 Types of machine learning

Supervised machine learning

Supervised machine learning is a general term for machine learning algorithms in which the training data
includes both feature values and known label values. Supervised machine learning is used to train
models by determining a relationship between the features and labels in past observations, so that
unknown labels can be predicted for features in future cases.
Regression is a form of supervised machine learning in which the label predicted by the model is a numeric
value. For example:

The number of ice creams sold on a given day, based on the temperature, rainfall, and windspeed.
The selling price of a property based on its size in square feet, the number of bedrooms it contains, and
socio-economic metrics for its location.
The fuel efficiency (in miles-per-gallon) of a car based on its engine size, weight, width, height, and length.

Classification is a form of supervised machine learning in which the label represents a categorization, or
class. There are two common classification scenarios.

Binary classification
In binary classification, the label determines whether the observed item is (or isn't) an instance of a specific
class. Or put another way, binary classification models predict one of two mutually exclusive outcomes. For
example:

Whether a patient is at risk for diabetes based on clinical metrics like weight, age, blood glucose level, and
so on.
Whether a bank customer will default on a loan based on income, credit history, age, and other factors.
Whether a mailing list customer will respond positively to a marketing offer based on demographic
attributes and past purchases.
In all of these examples, the model predicts a binary true/false or positive/negative prediction for a single
possible class.

Multiclass classification extends binary classification to predict a label that represents one of multiple
possible classes. For example,

The species of a penguin (Adelie, Gentoo, or Chinstrap) based on its physical measurements.
The genre of a movie (comedy, horror, romance, adventure, or science fiction) based on its cast, director,
and budget.
In most scenarios that involve a known set of multiple classes, multiclass classification is used to predict
mutually exclusive labels. For example, a penguin can't be both a Gentoo and an Adelie. However, there
are also some algorithms that you can use to train multilabel classification models, in which there may be
more than one valid label for a single observation. For example, a movie could potentially be categorized
as both science fiction and comedy.

Unsupervised machine learning involves training models using data that consists only of feature values
without any known labels. Unsupervised machine learning algorithms determine relationships between the
features of the observations in the training data.

The most common form of unsupervised machine learning is clustering. A clustering algorithm identifies
similarities between observations based on their features, and groups them into discrete clusters. For
example:

Group similar flowers based on their size, number of leaves, and number of petals.
Identify groups of similar customers based on demographic attributes and purchasing behavior.
In some ways, clustering is similar to multiclass classification; in that it categorizes observations into
discrete groups. The difference is that when using classification, you already know the classes to which the
observations in the training data belong; so the algorithm works by determining the relationship between
the features and the known classification label. In clustering, there's no previously known cluster label and
the algorithm groups the data observations based purely on similarity of features.

In some cases, clustering is used to determine the set of classes that exist before training a classification
model. For example, you might use clustering to segment your customers into groups, and then analyze
those groups to identify and categorize different classes of customer (high value - low volume, frequent
small purchaser, and so on). You could then use your categorizations to label the observations in your
clustering results and use the labeled data to train a classification model that predicts to which customer
category a new customer might belong.
Regression

Regression models are trained to predict numeric label values based on training data that includes both
features and known labels. The process for training a regression model (or indeed, any supervised machine
learning model) involves multiple iterations in which you use an appropriate algorithm (usually with some
parameterized settings) to train a model, evaluate the model's predictive performance, and refine the model
by repeating the training process with different algorithms and parameters until you achieve an acceptable
level of predictive accuracy.

Regression evaluation metrics

Based on the differences between the predicted and actual values, you can calculate some common
metrics that are used to evaluate a regression model.

Mean Absolute Error (MAE)

The variance in this example indicates by how many ice creams each prediction was wrong. It doesn't
matter if the prediction was over or under the actual value (so for example, -3 and +3 both indicate a
variance of 3). This metric is known as the absolute error for each prediction, and can be summarized for
the whole validation set as the mean absolute error (MAE).

In the ice cream example, the mean (average) of the absolute errors (2, 3, 3, 1, 2, and 3) is 2.33.

Mean Squared Error (MSE)

The mean absolute error metric takes all discrepancies between predicted and actual labels into account
equally. However, it may be more desirable to have a model that is consistently wrong by a small amount
than one that makes fewer, but larger errors. One way to produce a metric that "amplifies" larger errors by
squaring the individual errors and calculating the mean of the squared values. This metric is known as the
mean squared error (MSE).

In our ice cream example, the mean of the squared absolute values (which are 4, 9, 9, 1, 4, and 9) is 6.

Root Mean Squared Error (RMSE)

The mean squared error helps take the magnitude of errors into account, but because it squares the error
values, the resulting metric no longer represents the quantity measured by the label. In other words, we can
say that the MSE of our model is 6, but that doesn't measure its accuracy in terms of the number of ice
creams that were mispredicted; 6 is just a numeric score that indicates the level of error in the validation
predictions.

If we want to measure the error in terms of the number of ice creams, we need to calculate the square root
of the MSE; which produces a metric called, unsurprisingly, Root Mean Squared Error. In this case √6,
which is 2.45 (ice creams).

Coefficient of determination (R2)

All of the metrics so far compare the discrepancy between the predicted and actual values in order to
evaluate the model. However, in reality, there's some natural random variance in the daily sales of ice
cream that the model takes into account. In a linear regression model, the training algorithm fits a straight
line that minimizes the mean variance between the function and the known label values. The coefficient of
determination (more commonly referred to as R2 or R-Squared) is a metric that measures the proportion of
variance in the validation results that can be explained by the model, as opposed to some anomalous
aspect of the validation data (for example, a day with a highly unusual number of ice creams sales because
of a local festival).

The calculation for R2 is more complex than for the previous metrics. It compares the sum of squared
differences between predicted and actual labels with the sum of squared differences between the actual
label values and the mean of actual label values, like this:

R2 = 1- ∑(y-)2 ÷ ∑(y-y)2
Iterative training
The metrics described above are commonly used to evaluate a regression model. In most real-world
scenarios, a data scientist will use an iterative process to repeatedly train and evaluate a model, varying:

Feature selection and preparation (choosing which features to include in the model, and calculations applied
to them to help ensure a better fit).
Algorithm selection (We explored linear regression in the previous example, but there are many other
regression algorithms)
Algorithm parameters (numeric settings to control algorithm behavior, more accurately called
hyperparameters to differentiate them from the x and y parameters).
After multiple iterations, the model that results in the best evaluation metric that's acceptable for the specific
scenario is selected.

Binary classification

Classification, like regression, is a supervised machine learning technique; and therefore follows the same
iterative process of training, validating, and evaluating models. Instead of calculating numeric values like a
regression model, the algorithms used to train classification models calculate probability values for class
assignment and the evaluation metrics used to assess model performance compare the predicted classes to
the actual classes.

Binary classification algorithms are used to train a model that predicts one of two possible labels for a single
class. Essentially, predicting true or false. In most real scenarios, the data observations used to train and
validate the model consist of multiple feature (x) values and a y value that is either 1 or 0.

Training a binary classification model

To train the model, we'll use an algorithm to fit the training data to a function that calculates the probability of
the class label being true (in other words, that the patient has diabetes). Probability is measured as a value
between 0.0 and 1.0, such that the total probability for all possible classes is 1.0. So for example, if the
probability of a patient having diabetes is 0.7, then there's a corresponding probability of 0.3 that the patient
isn't diabetic.

There are many algorithms that can be used for binary classification, such as logistic regression, which
derives a sigmoid (S-shaped) function with values between 0.0 and 1.0, like this:
Binary classification evaluation metrics
The first step in calculating evaluation metrics for a binary classification model is usually to create a matrix of
the number of correct and incorrect predictions for each possible class label:

Diagram of a confusion matrix.

This visualization is called a confusion matrix, and it shows the prediction totals where:

yhat =0 and y=0: True negatives (TN)

yhat =1 and y=0: False positives (FP)
yhat =0 and y=1: False negatives (FN)
yhat =1 and y=1: True positives (TP)

The arrangement of the confusion matrix is such that correct (true) predictions are shown in a diagonal
line from top-left to bottom-right. Often, color-intensity is used to indicate the number of predictions in each
cell, so a quick glance at a model that predicts well should reveal a deeply shaded diagonal trend.
Accuracy
The simplest metric you can calculate from the confusion matrix is accuracy - the proportion of predictions
that the model got right. Accuracy is calculated as:

(TN+TP) ÷ (TN+FN+FP+TP)

In the case of our diabetes example, the calculation is:

(2+3) ÷ (2+1+0+3) = 5 ÷ 6 = 0.83

So for our validation data, the diabetes classification model produced correct predictions 83% of the time.

Accuracy might initially seem like a good metric to evaluate a model, but consider this. Suppose 11% of the
population has diabetes. You could create a model that always predicts 0, and it would achieve an accuracy
of 89%, even though it makes no real attempt to differentiate between patients by evaluating their features.
What we really need is a deeper understanding of how the model performs at predicting 1 for positive cases
and 0 for negative cases.

Recall
Recall is a metric that measures the proportion of positive cases that the model identified correctly. In other
words, compared to the number of patients who have diabetes, how many did the model predict to have
diabetes?

The formula for recall is: TP ÷ (TP+FN)

For our diabetes example:

= 3 ÷ (3+1) = 3 ÷ 4 = 0.75

So our model correctly identified 75% of patients who have diabetes as having diabetes.

Precision
Precision is a similar metric to recall, but measures the proportion of predicted positive cases where the true
label is actually positive. In other words, what proportion of the patients predicted by the model to have
diabetes actually have diabetes?

The formula for precision is: TP ÷ (TP+FP)

For our diabetes example:

= 3 ÷ (3+0) = 3 ÷ 3 = 1.0

So 100% of the patients predicted by our model to have diabetes do in fact have diabetes.

F1-score

F1-score is an overall metric that combined recall and precision. The formula for F1-score is:

(2 x Precision x Recall) ÷ (Precision + Recall)

For our diabetes example:

= (2 x 1.0 x 0.75) ÷ (1.0 + 0.75) = 1.5 ÷ 1.75 = 0.86

Area Under the Curve (AUC)
Another name for recall is the true positive rate (TPR), and there's an equivalent metric called the false
positive rate (FPR) that is calculated as FP÷(FP+TN). We already know that the TPR for our model when
using a threshold of 0.5 is 0.75, and we can use the formula for FPR to calculate a value of 0÷2 = 0.

Of course, if we were to change the threshold above which the model predicts true (1), it would affect the
number of positive and negative predictions; and therefore change the TPR and FPR metrics. These metrics
are often used to evaluate a model by plotting a received operator characteristic (ROC) curve that
compares the TPR and FPR for every possible threshold value between 0.0 and 1.0:

Diagram of a ROC curve.

The ROC curve for a perfect model would go straight up the TPR axis on the left and then across the FPR
axis at the top. Since the plot area for the curve measures 1x1, the area under this perfect curve would be 1.
0 (meaning that the model is correct 100% of the time). In contrast, a diagonal line from the bottom-left to
the top-right represents the results that would be achieved by randomly guessing a binary label; producing
an area under the curve of 0.5. In other words, given two possible class labels, you could reasonably expect
to guess correctly 50% of the time.

In the case of our diabetes model, the curve above is produced, and the area under the curve (AUC) metric
is 0.875. Since the AUC is higher than 0.5, we can conclude the model performs better at predicting whether
or not a patient has diabetes than randomly guessing.

Multiclass classification

Multiclass classification is used to predict to which of multiple possible classes an observation belongs. As a
supervised machine learning technique, it follows the same iterative train, validate, and evaluate process as
regression and binary classification in which a subset of the training data is held back to validate the trained
model.

Example - multiclass classification

Multiclass classification algorithms are used to calculate probability values for multiple class labels, enabling
a model to predict the most probable class for a given observation.

Let's explore an example in which we have some observations of penguins, in which the flipper length (x) of
each penguin is recorded. For each observation, the data includes the penguin species (y), which is
encoded as follows:

0: Adelie
1: Gentoo
2: Chinstrap
Training a multiclass classification model
To train a multiclass classification model, we need to use an algorithm to fit the training data to a function
that calculates a probability value for each possible class. There are two kinds of algorithm you can use to do
this:

One-vs-Rest (OvR) algorithms

Multinomial algorithms
One-vs-Rest (OvR) algorithms
One-vs-Rest algorithms train a binary classification function for each class, each calculating the probability
that the observation is an example of the target class. Each function calculates the probability of the
observation being a specific class compared to any other class. For our penguin species classification model
, the algorithm would essentially create three binary classification functions:

f0(x) = P(y=0 | x)
f1(x) = P(y=1 | x)
f2(x) = P(y=2 | x)
Each algorithm produces a sigmoid function that calculates a probability value between 0.0 and 1.0. A model
trained using this kind of algorithm predicts the class for the function that produces the highest probability
output.

Multinomial algorithms
As an alternative approach is to use a multinomial algorithm, which creates a single function that returns a
multi-valued output. The output is a vector (an array of values) that contains the probability distribution for all
possible classes - with a probability score for each class which when totaled add up to 1.0:

f(x) =[P(y=0|x), P(y=1|x), P(y=2|x)]

An example of this kind of function is a softmax function, which could produce an output like the following
example:

[0.2, 0.3, 0.5]

The elements in the vector represent the probabilities for classes 0, 1, and 2 respectively; so in this case, the
class with the highest probability is 2.

Regardless of which type of algorithm is used, the model uses the resulting function to determine the most
probable class for a given set of features (x) and predicts the corresponding class label (y).

Evaluating a multiclass classification model

To calculate the overall accuracy, recall, and precision metrics, you use the total of the TP, TN, FP, and
FN metrics:

Overall accuracy = (13+6)÷(13+6+1+1) = 0.90

Overall recall = 6÷(6+1) = 0.86
Overall precision = 6÷(6+1) = 0.86
The overall F1-score is calculated using the overall recall and precision metrics:

Overall F1-score = (2x0.86x0.86)÷(0.86+0.86) = 0.86

Clustering

Clustering is a form of unsupervised machine learning in which observations are grouped into clusters
based on similarities in their data values, or features. This kind of machine learning is considered
unsupervised because it doesn't make use of previously known label values to train a model. In a
clustering model, the label is the cluster to which the

Training a clustering model

There are multiple algorithms you can use for clustering. One of the most commonly used algorithms
is K-Means clustering, which consists of the following steps:

The feature (x) values are vectorized to define n-dimensional coordinates (where n is the number of
features). In the flower example, we have two features: number of leaves (x1) and number of petals
(x2). So, the feature vector has two coordinates that we can use to conceptually plot the data points in
two-dimensional space ([x1,x2])

You decide how many clusters you want to use to group the flowers - call this value k. For example, to
create three clusters, you would use a k value of 3. Then k points are plotted at random coordinates.
These points become the center points for each cluster, so they're called centroids.

Each data point (in this case a flower) is assigned to its nearest centroid.

Each centroid is moved to the center of the data points assigned to it based on the mean distance
between the points.

After the centroid is moved, the data points may now be closer to a different centroid, so the data
points are reassigned to clusters based on the new closest centroid.

The centroid movement and cluster reallocation steps are repeated until the clusters become stable or
a predetermined maximum number of iterations is reached.
Evaluating a clustering model
Since there's no known label with which to compare the predicted cluster assignments, evaluation of a
clustering model is based on how well the resulting clusters are separated from one another.

There are multiple metrics that you can use to evaluate cluster separation, including:

Average distance to cluster center: How close, on average, each point in the cluster is to the centroid of the
cluster.
Average distance to other center: How close, on average, each point in the cluster is to the centroid of all
other clusters.
Maximum distance to cluster center: The furthest distance between a point in the cluster and its centroid.
Silhouette: A value between -1 and 1 that summarizes the ratio of distance between points in the same
cluster and points in different clusters (The closer to 1, the better the cluster separation).

Deep learning

Deep learning is an advanced form of machine learning that tries to emulate the way the human brain
learns. The key to deep learning is the creation of an artificial neural network that simulates
electrochemical activity in biological neurons by using mathematical functions, as shown here.

Artificial neural networks are made up of multiple layers of neurons - essentially defining a deeply
nested function. This architecture is the reason the technique is referred to as deep learning and the
models produced by it are often referred to as deep neural networks (DNNs). You can use deep
neural networks for many kinds of machine learning problem, including regression and classification,
as well as more specialized models for natural language processing and computer vision.

Just like other machine learning techniques discussed in this module, deep learning involves fitting
training data to a function that can predict a label (y) based on the value of one or more features (x).
The function (f(x)) is the outer layer of a nested function in which each layer of the neural network
encapsulates functions that operate on x and the weight (w) values associated with them. The
algorithm used to train the model involves iteratively feeding the feature values (x) in the training data
forward through the layers to calculate output values for , validating the model to evaluate how far off
the calculated values are from the known y values (which quantifies the level of error, or loss, in the
model), and then modifying the weights (w) to reduce the loss. The trained model includes the final
weight values that result in the most accurate predictions.

Deep Learning For Classification

The feature data (x) consists of some measurements of a penguin. Specifically, the measurements are:

The length of the penguin's bill.

The depth of the penguin's bill.
The length of the penguin's flippers.
The penguin's weight.
In this case, x is a vector of four values, or mathematically, x=[x1,x2,x3,x4].

The label we're trying to predict (y) is the species of the penguin, and that there are three possible species
it could be:

Adelie
Gentoo
Chinstrap

This is an example of a classification problem, in which the machine learning model must predict the most
probable class to which an observation belongs. A classification model accomplishes this by predicting a
label that consists of the probability for each class. In other words, y is a vector of three probability values;
one for each of the possible classes: [P(y=0|x), P(y=1|x), P(y=2|x)].

The process for inferencing a predicted penguin class using this network is:

The feature vector for a penguin observation is fed into the input layer of the neural network, which consists
of a neuron for each x value. In this example, the following x vector is used as the input: [37.3, 16.8, 19.2,
30.0]
The functions for the first layer of neurons each calculate a weighted sum by combining the x value and w
weight, and pass it to an activation function that determines if it meets the threshold to be passed on to the
next layer.
Each neuron in a layer is connected to all of the neurons in the next layer (an architecture sometimes
called a fully connected network) so the results of each layer are fed forward through the network until they
reach the output layer.
The output layer produces a vector of values; in this case, using a softmax or similar function to calculate
the probability distribution for the three possible classes of penguin. In this example, the output vector is: [0
.2, 0.7, 0.1]
The elements of the vector represent the probabilities for classes 0, 1, and 2. The second value is the
highest, so the model predicts that the species of the penguin is 1 (Gentoo).
How does a neural network learn?
The weights in a neural network are central to how it calculates predicted values for labels. During the
training process, the model learns the weights that will result in the most accurate predictions. Let's explore
the training process in a little more detail to understand how this learning takes place.

Diagram of a neural network being trained, evaluated, and optimized.

The training and validation datasets are defined, and the training features are fed into the input layer.
The neurons in each layer of the network apply their weights (which are initially assigned randomly) and feed
the data through the network.
The output layer produces a vector containing the calculated values for . For example, an output for a
penguin class prediction might be [0.3. 0.1. 0.6].
A loss function is used to compare the predicted values to the known y values and aggregate the difference
(which is known as the loss). For example, if the known class for the case that returned the output in the
previous step is Chinstrap, then the y value should be [0.0, 0.0, 1.0]. The absolute difference between this
and the vector is [0.3, 0.1, 0.4]. In reality, the loss function calculates the aggregate variance for multiple
cases and summarizes it as a single loss value.
Since the entire network is essentially one large nested function, an optimization function can use differential
calculus to evaluate the influence of each weight in the network on the loss, and determine how they could
be adjusted (up or down) to reduce the amount of overall loss. The specific optimization technique can vary,
but usually involves a gradient descent approach in which each weight is increased or decreased to
minimize the loss.
The changes to the weights are backpropagated to the layers in the network, replacing the previously used
values.
The process is repeated over multiple iterations (known as epochs) until the loss is minimized and the model
predicts acceptably accurately.

While it's easier to think of each case in the training data being passed through the network one at a time, in
reality the data is batched into matrices and processed using linear algebraic calculations. For this reason,
neural network training is best performed on computers with graphical processing units (GPUs) that are
optimized for vector and matrix manipulation.
Machine learning is the foundation on which artificial intelligence is built.

Azure AI services are easy to use AI capabilities made available as resources on the Azure platform. Azure
AI service capabilities include Language, Speech, Vision, Decision, Search, and Azure OpenAI. In this
module we've used several different terms relating to AI services. Here's a recap:

1. API application programming interfaces (APIs) enable software components to communicate, so one
side can be updated without stopping the other from working.
2. Artificial Intelligence (AI) computer programs that respond in ways that are normally associated with
human reasoning, learning, and thought.
3. Azure AI services a portfolio of AI services that can be incorporated into applications quickly and easily
without specialist knowledge. Azure AI services is also the name for the multi-service resource created in the
Azure portal that provides access to several different Azure AI services with a single key and endpoint.
4. Endpoint the location of a resource, such as an Azure AI service.
5. Key a private string that is used to authenticate a request.
6. Machine learning the ability for computer programs to learn from large amounts of data, in a process
known as "training".
7. Multi-service resource the AI service resource created in the Azure portal that provides access to a
bundle of AI services.
8. Single-service resource a resource created in the Azure portal that provides access to a single Azure
AI service, such as Speech, Vision, Language, etc. Each Azure AI service has a unique key and endpoint.
RESTful API a scalable web application programming interface used to access Azure AI services.
 MICROSOFT - Challenge
Module 1

Explore types of generative AI models

To begin building with Azure OpenAI, you need to choose a base model and deploy it. Microsoft provides
base models and the option to create customized base models. This module covers the currently
available base models.

Azure OpenAI includes several types of model:

1. GPT-4 models are the latest generation of generative pretrained (GPT) models that can generate
natural language and code completions based on natural language prompts.
2. GPT 3.5 models can generate natural language and code completions based on natural language
prompts. In particular, GPT-35-turbo models are optimized for chat-based interactions and work well in
most generative AI scenarios.
3. Embeddings models convert text into numeric vectors, and are useful in language analytics scenarios
such as comparing text sources for similarities.
4. DALL-E models are used to generate images based on natural language prompts. Currently, DALL-E
models are in preview. DALL-E models aren't listed in the Azure OpenAI Studio interface and don't need
to be explicitly deployed.

A prompt is the text portion of a request that is sent to the deployed model's completions endpoint.
Responses are referred to as completions, which can come in form of text, code, or other formats.

Completions Playground parameters

There are many parameters that you can adjust to change the performance of your model:

Temperature: Controls randomness. Lowering the temperature means that the model produces more
repetitive and deterministic responses. Increasing the temperature results in more unexpected or creative
responses. Try adjusting temperature or Top P but not both.

Max length (tokens): Set a limit on the number of tokens per model response. The API supports a
maximum of 4000 tokens shared between the prompt (including system message, examples, message
history, and user query) and the model's response. One token is roughly four characters for typical
English text.

Stop sequences: Make responses stop at a desired point, such as the end of a sentence or list. Specify
up to four sequences where the model will stop generating further tokens in a response. The returned text
won't contain the stop sequence.

Top probabilities (Top P): Similar to temperature, this controls randomness but uses a different method.
Lowering Top P narrows the model s token selection to likelier tokens. Increasing Top P lets the model
choose from tokens with both high and low likelihood. Try adjusting temperature or Top P but not both.

Frequency penalty: Reduce the chance of repeating a token proportionally based on how often it has
appeared in the text so far. This decreases the likelihood of repeating the exact same text in a response.

Presence penalty: Reduce the chance of repeating any token that has appeared in the text at all so far.
This increases the likelihood of introducing new topics in a response.
Pre-response text: Insert text after the user s input and before the model s response. This can help
prepare the model for a response.
Post-response text: Insert text after the model s generated response to encourage further user input, as
when modeling a conversation.
Analyse Image -

Available visual features are contained in the VisualFeatures enum:

[Link]: Identifies tags about the image, including objects, scenery, setting, and actions
[Link]: Returns the bounding box for each detected object
[Link]: Generates a caption of the image in natural language
VisualFeatures.DENSE_CAPTIONS: Generates more detailed captions for the objects detected
[Link]: Returns the bounding box for detected people
VisualFeatures.SMART_CROPS: Returns the bounding box of the specified aspect ratio for the area of
interest
[Link]: Extracts readable text

Code:

from [Link] import ImageAnalysisClient

from [Link] import VisualFeatures
from [Link] import AzureKeyCredential

client = ImageAnalysisClient(
endpoint=[Link]["ENDPOINT"],
credential=AzureKeyCredential([Link]["KEY"])
)

result = [Link](
image_url="<url>",
visual_features=[[Link], [Link]],
gender_neutral_caption=True,
language="en",
)

Understand the development lifecycle of a large language model (LLM) app

1. Initialization: Define the use case and

design the solution.

2. Experimentation: Develop a flow and test

with a small dataset.

3. Evaluation and refinement: Assess the

flow with a larger dataset.

4. Production: Deploy and monitor the flow

and application.
Initialization
Imagine you want to design and develop an LLM application to classify news articles. Before you start
creating anything, you need to define what categories you want as output. You need to understand what a
typical news article looks like, how you present the article as input to your application, and how the
application generates the desired output.

In other words, during initialization you:

Diagram of the four steps during initialization.

1. Define the objective

2. Collect a sample dataset
3. Build a basic prompt
4. Design the flow

To design, develop, and test an LLM application, you need a sample dataset that serves as the input. A
sample dataset is a small representative subset of the data you eventually expect to parse as input to your
LLM application.

When collecting or creating the sample dataset, you should ensure diversity in the data to cover various
scenarios and edge cases. You should also remove any privacy sensitive information from the dataset to
avoid any vulnerabilities.

Experimentation
You collected a sample dataset of news articles, and decided on which categories you want the articles to
be classified into. You designed a flow that takes a news article as input, and uses an LLM to classify the
article. To test whether your flow generates the expected output, you run it against your sample dataset.

Diagram of the four steps during experimentation.

The experimentation phase is an iterative process during which you (1) run the flow against a sample
dataset. You then (2) evaluate the prompt's performance. If you're (3) satisfied with the result, you can
move on to evaluation and refinement. If you think there's room for improvement, you can (4) modify the
flow by changing the prompt or flow itself.
Evaluation and refinement
When you're satisfied with the output of the flow that classifies news articles, based on the sample
dataset, you can assess the flow's performance against a larger dataset.

By testing the flow on a larger dataset, you can evaluate how well the LLM application generalizes to new
data. During evaluation, you can identify potential bottlenecks or areas for optimization or refinement.

When you edit your flow, you should first run it against a smaller dataset before running it again against a
larger dataset. Testing your flow with a smaller dataset allows you to more quickly respond to any issues.

Once your LLM application appears to be robust and reliable in handling various scenarios, you can
decide to move the LLM application to production.

Production
Finally, your news article classification application is ready for production.

Diagram of the three steps during production.

During production, you:

Optimize the flow that classifies
incoming articles for efficiency
and effectiveness.
Deploy your flow to an endpoint.
When you call the endpoint, the
flow is triggered to run and the
desired
output is generated.
Monitor the performance of your
solution by collecting usage
data and end-user feedback. By

understanding how the

application performs, you can
improve the flow whenever
necessary.

Overview of the lifecycle -->

Understand the types of flows
There are three different types of flows you can create with prompt flow:

Standard flow: Ideal for general LLM-based application development, offering a range of versatile tools.
Chat flow: Designed for conversational applications, with enhanced support for chat-related functionalities.
Evaluation flow: Focused on performance evaluation, allowing the analysis and improvement of models or
applications through feedback on previous runs.

Understanding RAG

RAG is a technique that you can use to ground a language model. In other words, it's a process for
retrieving information that is relevant to the user's initial prompt. In general terms, the RAG pattern
incorporates the following steps:

Diagram of the retrieval augmented generation pattern.

Retrieve grounding data based on the initial user-entered prompt.

Augment the prompt with grounding data.
Use a language model to generate a grounded response.
By retrieving context from a specified data source, you ensure that the language model uses relevant
information when responding, instead of relying on its training data.

Using RAG is a powerful and easy-to-use technique for many cases in which you want to ground your
language model and improve the factual accuracy of your copilot's responses.

What are language models?

Generative AI applications are powered by language models, which are a specialized type of machine
learning model that you can use to perform natural language processing (NLP) tasks, including:

Determining sentiment or otherwise classifying natural language text.

Summarizing text.
Comparing multiple text sources for semantic similarity.
Generating new natural language.
While the mathematical principles behind these language models can be complex, a basic understanding
of the architecture used to implement them can help you gain a conceptual understanding of how they
work.

Transformer models
Machine learning models for natural language processing have evolved over many years. Today's cutting
-edge large language models are based on the transformer architecture, which builds on and extends
some techniques that have been proven successful in modeling vocabularies to support NLP tasks - and
in particular in generating language. Transformer models are trained with large volumes of text, enabling
them to represent the semantic relationships between words and use those relationships to determine
probable sequences of text that make sense. Transformer models with a large enough vocabulary are
capable of generating language responses that are tough to distinguish from human responses.

Transformer model architecture consists of two components, or blocks:

The model is trained with a large volume of natural language text, often sourced from the internet or other
public sources of text.

The sequences of text are broken down into tokens (for example, indovidual words) and the encoder block
processes these token sequences using a technique called attention to determine relationships between
tokens (for example, which tokens influence the presence of other tokens in a sequence, different tokens
that are commonly used in the same context, and so on.)

The output from the encoder is a collection of vectors (multi-valued numeric arrays) in which each element of
the vector represents a semantic attribute of the tokens. These vectors are referred to as embeddings.
The decoder block works on a new sequence of text tokens and uses the embeddings generated by the
encoder to generate an appropriate natural language output.
For example, given an inut sequence like "When my dog was", the model can use the attention techique to
analyze the input tokens and the semantic attributes encoded in the embeddings to predict an appropriate
completion of the sentence, such as "a puppy."
In practice, the specific implementations of the architecture vary for example, the Bidirectional Encoder
Representations from Transformers (BERT) model developed by Google to support their search engine uses
only the encoder block, while the Generative Pretrained Transformer (GPT) model developed by OpenAI
uses only the decoder block.

While a complete explanation of every aspect of transformer models is beyond the scope of this module, an
explanation of some of the key elements in a transformer can help you get a sense for how they support
generative AI

Tokenization
The first step in training a transformer model is to decompose the training text into tokens - in other words,
identify each unique text value. For the sake of simplicity, you can think of each distinct word in the training
text as a token (though in reality, tokens can be generated for partial words, or combinations of words and
punctuation).
For example, consider the following sentence:
I heard a dog bark loudly at a cat
To tokenize this text, you can identify each discrete word and assign token IDs to them. For example:

- I (1)
- heard (2)
- a (3)
- dog (4)
- bark (5)
- loudly (6)
- at (7)
- *("a" is already tokenized as 3)*
- cat (8)
 Embeddings
While it may be convenient to represent tokens as simple IDs - essentially creating an index for all the
words in the vocabulary, they don't tell us anything about the meaning of the words, or the relationships
between them. To create a vocabulary that encapsulates semantic relationships between the tokens, we
define contextual vectors, known as embeddings, for them. Vectors are multi-valued numeric
representations of information, for example [10, 3, 1] in which each numeric element represents a
particular attribute of the information. For language tokens, each element of a token's vector represents
some semantic attribute of the token. The specific categories for the elements of the vectors in a language
model are determined during training based on how commonly words are used together or in similar
contexts.
Vectors represent lines in multidimensional space, describing direction
and distance along multiple axes
(you can impress your mathematician friends by calling these amplitude
and magnitude). It can be useful
to think of the elements in an embedding vector for a token as
representing steps along a path in
multidimensional space. For example, a vector with three elements
represents a path in 3-dimensional
space in which the element values indicate the units traveled forward/
back, left/right, and up/down.
Overall, the vector describes the direction and distance of the path from
origin to end.
The elements of the tokens in the embeddings space each represent some semantic attribute of the token ,
so that semantically similar tokens should result in vectors that have a similar orientation in other words
they point in the same direction. A technique called cosine similarity is used to determine if two
vectors have similar directions (regardless of distance), and therefore represent semantically linked words .
As a simple example, suppose the embeddings for our tokens consist of vectors with three elements, for
example:

There are multiple ways you can calculate appropriate embeddings for a given set of tokens, including
language modeling algorithms like Word2Vec or the encoder block in a transformer model.

Attention
The encoder and decoder blocks in a transformer model include multiple layers that form the neural network
for the model. We don't need to go into the details of all these layers, but it's useful to consider one of the
types of layers that is used in both blocks: attention layers. Attention is a technique used to examine a
sequence of text tokens and try to quantify the strength of the relationships between them. In particular, self-
attention involves considering how other tokens around one particular token influence that token's meaning.

In an encoder block, each token is carefully examined in context, and an appropriate encoding is determined
for its vector embedding. The vector values are based on the relationship between the token and other
tokens with which it frequently appears. This contextualized approach means that the same word might have
multiple embeddings depending on the context in which it's used - for example "the bark of a tree" means
something different to "I heard a dog bark."

In a decoder block, attention layers are used to predict the next token in a sequence. For each token
generated, the model has an attention layer that takes into account the sequence of tokens up to that point.
The model considers which of the tokens are the most influential when considering what the next token
should be. For example, given the sequence I heard a dog, the attention layer might assign greater
weight to the tokens heard and dog when considering the next word in the sequence:

I heard a dog [bark]

Remember that the attention layer is working with numeric vector representations of the tokens, not the
actual text. In a decoder, the process starts with a sequence of token embeddings representing the text to be
completed. The first thing that happens is that another positional encoding layer adds a value to each
embedding to indicate its position in the sequence:
[1,5,6,2] (I)
[2,9,3,1] (heard)
[3,1,1,2] (a)
[4,10,3,2] (dog)

During training, the goal is to predict the vector for the final token in the sequence based on the preceding
tokens. The attention layer assigns a numeric weight to each token in the sequence so far. It uses that value
to perform a calculation on the weighted vectors that produces an attention score that can be used to
calculate a possible vector for the next token. In practice, a technique called multi-head attention uses
different elements of the embeddings to calculate multiple attention scores. A neural network is then used to
evaluate all possible tokens to determine the most probable token with which to continue the sequence. The
process continues iteratively for each token in the sequence, with the output sequence so far being used
regressively as the input for the next iteration essentially building the output one token at a time.

The following animation shows a simplified representation of how this works in reality, the calculations
performed by the attention layer are more complex; but the principles can be simplified as shown:

A sequence of token embeddings is fed into the attention layer. Each token is represented as a vector of
numeric values.
The goal in a decoder is to predict the next token in the sequence, which will also be a vector that aligns to
an embedding in the model s vocabulary.
The attention layer evaluates the sequence so far and assigns weights to each token to represent their
relative influence on the next token.
The weights can be used to compute a new vector for the next token with an attention score. Multi-head
attention uses different elements in the embeddings to calculate multiple alternative tokens.
A fully connected neural network uses the scores in the calculated vectors to predict the most probable
token from the entire vocabulary.
The predicted output is appended to the sequence so far, which is used as the input for the next iteration.
During training, the actual sequence of tokens is known we just mask the ones that come later in the
sequence than the token position currently being considered. As in any neural network, the predicted value
for the token vector is compared to the actual value of the next vector in the sequence, and the loss is
calculated. The weights are then incrementally adjusted to reduce the loss and improve the model. When
used for inferencing (predicting a new sequence of tokens), the trained attention layer applies weights that
predict the most probable token in the model s vocabulary that is semantically aligned to the sequence so
far.

What all of this means, is that a transformer model such as GPT-4 (the model behind ChatGPT and Bing) is
designed to take in a text input (called a prompt) and generate a syntactically correct output (called a
completion). In effect, the magic of the model is that it has the ability to string a coherent sentence
together. This ability doesn't imply any knowledge or intelligence on the part of the model; just a
large vocabulary and the ability to generate meaningful sequences of words. What makes a large language
model like GPT-4 so powerful however, is the sheer volume of data with which it has been trained (public
and licensed data from the Internet) and the complexity of the network. This enables the model to generate
completions that are based on the relationships between words in the vocabulary on which the model was
trained; often generating output that is indistinguishable from a human response to the same prompt.

In addition to the Azure OpenAI models, the model catalog includes the latest open-source models from
Microsoft and multiple partners, including:

1. OpenAI
2. HuggingFace
3. Mistral
4. Meta and others.