Module 5
Reinforcement Learning
Reinforcement Learning (RL) is a branch of machine learning that
focuses on how agents can learn to make decisions through trial and
error to maximize cumulative rewards. RL allows machines to learn by
interacting with an environment and receiving feedback based on their
actions. This feedback comes in the form of rewards or penalties.
Reinforcement Learning revolves around the idea that an agent
(the learner or decision-maker) interacts with an environment to
achieve a goal. The agent performs actions and receives feedback
to optimize its decision-making over time.
• Agent: The decision-maker that performs actions.
• Environment: The world or system in which the agent
operates.
• State: The situation or condition the agent is currently in.
• Action: The possible moves or decisions the agent can make.
• Reward: The feedback or result from the environment based
on the agent’s action.
Reinforcement Learning is a feedback-based
Machine learning technique in which an agent
learns to behave in an environment by
performing the actions and seeing the results of
actions. For each good action, the agent gets
positive feedback, and for each bad action, the
agent gets negative feedback or penalty.
Working of Reinforcement Learning
The RL process involves an agent performing actions in an environment,
receiving rewards or penalties based on those actions, and adjusting its
behavior accordingly. This loop helps the agent improve its decision-
making over time to maximize the cumulative reward.
• Here’s a breakdown of RL components:
• Policy: A strategy that the agent uses to determine the next action
based on the current state.
• Reward Function: A function that provides feedback on the actions
taken, guiding the agent towards its goal.
• Value Function: Estimates the future cumulative rewards the agent
will receive from a given state.
• Model of the Environment: A representation of the environment
that predicts future states and rewards, aiding in planning.
Robot in a room
actions: UP, DOWN,
+1 LEFT, RIGHT
-1 UP
80% move UP
10% move LEFT
START
10% move RIGHT
• reward +1 at [4,3], -1 at [4,2]
• reward -0.04 for each step
Reinforcement Learning Working
To understand the working process of the RL, we
need to consider two main things:
• Environment: It can be anything such as a
room, maze, football ground, etc.
• Agent: An intelligent agent such as AI robot.
Now, the agent
has successfully
stored the
previous steps
assigning the 1
value to each
previous block.
But what will the agent do if he starts moving
from the block, which has 1 value block on
both sides? Consider the below diagram:
1) Policy: A policy can be defined as a way how an
agent behaves at a given time. It maps the perceived
states of the environment to the actions taken on
those states. A policy is the core element of the RL as
it alone can define the behavior of the agent.
2) Reward Signal: The goal of reinforcement learning is
defined by the reward signal. At each state, the
environment sends an immediate signal to the learning
agent, and this signal is known as a reward signal. These
rewards are given according to the good and bad actions
taken by the agent. The agent's main objective is to
maximize the total number of rewards for good actions.
3) Value Function: The value function gives information
about how good the situation and action are and how
much reward an agent can expect. A reward indicates
the immediate signal for each good and bad action,
whereas a value function specifies the good state and
action for the future. The value function depends on the
reward as, without reward, there could be no value. The
goal of estimating values is to achieve more rewards.
4) Model: The last element of reinforcement learning is
the model, which mimics the behavior of the
environment. With the help of the model, one can make
inferences about how the environment will behave. Such
as, if a state and an action are given, then a model can
predict the next state and reward.
Transfer Learning
Transfer learning is a machine learning
technique where a model trained on one task is
repurposed as the foundation for a second task.
This approach is beneficial when the second task
is related to the first or when data for the
second task is limited.
Leveraging learned features from the initial task, the model can adapt more efficiently
to the new task, accelerating learning and improving performance. Transfer learning
also reduces the risk of overfitting, as the model already incorporates generalizable
features useful for the second task.
Transfer learning offer solutions to key challenges like:
• Limited Data: Acquiring extensive labeled data is often challenging and
costly. Transfer learning enables us to use pre-trained models, reducing the
dependency on large datasets.
• Enhanced Performance: Starting with a pre-trained model, which has already
learned from substantial data, allows for faster and more accurate results on
new tasks—ideal for applications needing high accuracy and efficiency.
• Time and Cost Efficiency: Transfer learning shortens training time and
conserves resources by utilizing existing models, eliminating the need for
training from scratch.
• Adaptability: Models trained on one task can be fine-tuned for related tasks,
making transfer learning versatile for various applications, from image
recognition to natural language processing.
Working of Transfer Learning
Transfer learning involves a structured process to leverage existing knowledge from a
pre-trained model for new tasks:
• Pre-trained Model: Start with a model already trained on a large dataset for a
specific task. This pre-trained model has learned general features and patterns that
are relevant across related tasks.
• Base Model: This pre-trained model, known as the base model, includes layers that
have processed data to learn hierarchical representations, capturing low-level to
complex features.
• Transfer Layers: Identify layers within the base model that hold generic information
applicable to both the original and new tasks. These layers, often near the top of
the network, capture broad, reusable features.
• Fine-tuning: Fine-tune these selected layers with data from the new task. This
process helps retain the pre-trained knowledge while adjusting parameters to meet
the specific requirements of the new task, improving accuracy and adaptability.
Low-level features learned for task A should be beneficial for learning of model for task
B.
Frozen vs. Trainable Layers in Transfer Learning
In Transfer Learning, two main components help in adapting models effectively:
Frozen Layers: These layers from a pre-trained model remain unchanged during fine-
tuning. They retain general features learned from the original task, extracting universal
patterns from input data.
Trainable Layers: These layers are adjusted during fine-tuning to learn task-specific
features from the new dataset, allowing the model to meet the new task’s unique
requirements.
Deciding Which Layers to Freeze or Train:
The extent to which you freeze or fine-tune layers depends on the similarity and size of
your target dataset:
• Small, Similar Dataset: For smaller datasets that resemble the original dataset, you
freeze most layers and only fine-tune the last one or two layers to prevent
overfitting.
• Large, Similar Dataset: With large, similar datasets, you can unfreeze more layers,
allowing the model to adapt while retaining learned features from the base model.
• Small, Different Dataset: For smaller, dissimilar datasets, fine-tuning layers closer to
the input layer helps the model learn task-specific features from scratch.
• Large, Different Dataset: In this case, fine-tuning the entire model helps the model
adapt to the new task while leveraging the broad knowledge from the pre-trained
model.
Applications of Transfer Learning
Transfer learning is widely used across multiple domains, including:
• Computer Vision: Transfer learning is prevalent in image recognition tasks, where
models pre-trained on large image datasets are adapted to specific tasks such as
medical imaging, facial recognition, and object detection.
• Natural Language Processing (NLP): In NLP, models like BERT, GPT, and ELMo are
pre-trained on vast text corpora and later fine-tuned for specific tasks such as
sentiment analysis, machine translation, and question-answering.
• Healthcare: Transfer learning helps develop medical diagnostic tools, leveraging
knowledge from general image recognition models to analyze medical images like
X-rays or MRIs.
• Finance: Transfer learning in finance assists in fraud detection, risk assessment,
and credit scoring by transferring patterns learned from related financial datasets.
Advantages of Transfer Learning
• Speed up the training process: By using a pre-trained model, the model can learn more
quickly and effectively on the second task, as it already has a good understanding of the
features and patterns in the data.
• Better performance: Transfer learning can lead to better performance on the second task, as
the model can leverage the knowledge it has gained from the first task.
• Handling small datasets: When there is limited data available for the second task, transfer
learning can help to prevent overfitting, as the model will have already learned general
features that are likely to be useful in the second task.
Disadvantages of Transfer Learning
• Domain mismatch: The pre-trained model may not be well-suited to the second task if the
two tasks are vastly different or the data distribution between the two tasks is very different.
• Overfitting: Transfer learning can lead to overfitting if the model is fine-tuned too much on
the second task, as it may learn task-specific features that do not generalize well to new
data.
• Complexity: The pre-trained model and the fine-tuning process can be computationally
expensive and may require specialized hardware.
Transfer learning enhances model performance by leveraging knowledge from previously
trained models. By starting with pre-existing models and fine-tuning them for specific tasks,
transfer learning saves time, improves accuracy, and enables effective learning even with limited
data.
Pretrained models
Pretrained models are machine learning models
that have already been trained on a large
dataset for a general task and can be reused or
fine-tuned for specific tasks. These models save
time, computation, and data because the initial
learning phase is already done.
A pre-trained model is a machine learning (ML) model that has been trained on a large dataset
and can be fine-tuned for a specific task. Pre-trained models are often used as a starting point
for developing ML models, as they provide a set of initial weights and biases that can be fine-
tuned for a specific task.
• There are several advantages to using pre-trained models, including the ability to leverage the
knowledge and experience of others, the ability to save time and resources, and the ability to
improve model performance. Pre-trained models are often trained on large, diverse datasets
and have been trained to recognize a wide range of patterns and features. As a result, they
can provide a strong foundation for fine-tuning and can significantly improve the performance
of the model.
• Pre-trained models come in a variety of forms, such as language models, object detection
models, and picture classification models. Convolutional neural networks are frequently used
as the foundation for image classification models, which are trained to categorize images into
predetermined categories (CNNs).
• CNNs or region-based convolutional neural networks are frequently used as the foundation
for object recognition models, which are taught to recognise and categorize items in photos
or videos (R-CNNs). Recurrent neural networks (RNNs) or transformers are frequently used as
the foundation for language models, which are trained to predict the next word in a
sequence.
• Overall, pre-trained models are a useful tool in ML, and can be used as a starting point for
developing ML models. They provide a set of initial weights and biases that can be fine-tuned
for a specific task, and can significantly improve the performance of the model.
• A pre-trained model, having been trained on extensive data,
serves as a foundational model for various tasks, leveraging its
learned patterns and features. In natural language processing
(NLP), these models are commonly employed as a starting point
for tasks like language translation, sentiment analysis, and text
summarization. Utilizing pre-trained models allows NLP
practitioners to economize on time and resources, bypassing
the need to train a model from scratch on a large dataset.
• Some popular pre-trained models for NLP include BERT, GPT-2,
ELMo, and RoBERTa. These models are trained on large datasets
of text and can be fine-tuned for specific tasks.
Pretrained models are effective and efficient solutions for
developers, researchers and businesses as they eliminate the
need to write and train the code from scratch, saving time.
Other advantages of using pre-trained models for projects are:
• Reduces the computational burden required for initial model
training hence, making development more accessible.
• The learned knowledge can be used for various applications.
• Models can be fine-tuned according to the task and can result
in superior performance to training from the initial point.
• Less labelled data is required for fine-tuning specific tasks.
Applications of pre-trained Models in NLP
• Language translation: NLP algorithms facilitate the automatic translation of text or speech, promoting
communication across language barriers.
• Sentiment analysis: NLP algorithms enable automated analysis of text or speech to discern sentiment
(positive, negative, or neutral), beneficial for tasks like customer feedback assessment and social
media surveillance.
• Chatbot development: NLP algorithms can be used to develop chatbots that can understand and
respond to human language. This is useful for applications such as customer service, where chatbots
can handle routine inquiries and free up human agents to handle more complex tasks.
• Text summarization: NLP algorithms can be used to automatically summarize long documents or
articles by extracting the most important information and presenting it in a condensed form. This is
useful for applications such as news aggregation and information extraction.
• Sentence completion: NLP algorithms can automatically generate sentence or paragraph
continuations by considering the context and content of the entered text.
The application of pretrained models is not limited to NLP, it is also used for image classification, image
segmentation and other computer vision applications.
Pretrained models in NLP
Here are a few excellent pretrained models for natural language processing (NLP):
1. BERT (Bidirectional Encoder Representations from Transformers)
BERT (Bidirectional Encoder Representations from Transformers) is a state-of-the-art
language representation model developed by Google. It is trained on a large dataset of
unannotated text and can be fine-tuned for a wide range of natural language processing
(NLP) tasks. BERT has achieved state-of-the-art performance on a variety of NLP tasks,
such as language translation, sentiment analysis, and text summarization.
Architecture and Working
• The processes the textual sequence bidirectionally, considering both left and right
context simultaneously for each word. This allows BERT to better capture the
meaning and context of words in a sentence, leading to improved performance on a
variety of NLP tasks.
• BERT is a transformer-based model, which means it uses self-attention mechanisms
to process input text.
• The model is trained on extensive amount of dataset with diverse range of text. The
pretraining phase allows the model to learn and understand the linguistic patterns.
• The BERT uses attention mechanism to assign different weights to different parts of
input sequence.
3. ELMo (Embeddings from Language Models)
ELMo (Embeddings from Language Models) is a deep contextualized word representation model
developed by researchers at the Allen Institute for Artificial Intelligence. It is trained on a large
dataset of unannotated text and can be fine-tuned for a wide range of natural language
processing (NLP) tasks. ELMo word vectors are generated through a two-layer bidirectional
language model (biLM), featuring both forward and backward passes in each layer. Diverging
from approaches like Glove and Word2Vec, ELMo takes a holistic perspective by representing
word embeddings based on the entire sentence that encompasses the word. This unique
characteristic enables ELMo embeddings to effectively capture the contextual nuances of a word
within a given sentence. Consequently, ELMo has the capability to produce distinct embeddings
for the same word deployed in diverse contexts across different sentences, setting it apart in its
ability to grasp the intricacies of language use.
4. Transformer-XL
Transformer-XL is a state-of-the-art language representation model developed by researchers at
Carnegie Mellon University and Google Brain. Transformer -XL is a variant of transformer model,
which includes relative positional encoding and a recurrence mechanism. Transformers XL tackles
the challenge of long-term dependency by retaining the previously learned segment in a hidden
state. This means that instead of recalculating each segment's hidden state from scratch, the
model utilizes the existing knowledge from the preceding segment for the current one. This
innovative approach not only mitigates issues inherent in the vanilla transformer model but also
effectively addresses the long-term dependency problem.
5. RoBERTa (Robustly Optimized BERT)
RoBERTa (Robustly Optimized BERT) is a variant of BERT
(Bidirectional Encoder Representations from
Transformers) developed by researchers at Facebook AI.
It is trained on a larger dataset and fine-tuned on a
variety of natural language processing (NLP) tasks,
making it a more powerful language representation
model than BERT. RoBERTa is a transformer-based
model, which means it uses self-attention mechanisms
to process input text.
Like BERT, RoBERTa is "bidirectional," meaning it
considers the context from both the left and the right
sides of a token, rather than just the left side as in
previous models.
Advantages:
Reduces training time and cost
Requires less data
Achieves high performance
Easy to apply to real-world tasks
Disadvantages:
May not be optimal for very different tasks
Large models can require high memory
Potential bias in source training data
Deep Q-Learning and Deep Q-Network
Working of Deep Q-Learning
Real-World Examples of Machine Learning (ML)
Here are examples of machine learning at work in our daily life that provide value in many
ways—some large and some small.
Markov Chain Monte Carlo (MCMC) methods
The Future of Machine Learning: Trends
1. Machine Learning Future – The Quantum Computing Effect
• Industry experts have high hopes for optimizing machine
learning speed through quantum computing. And rightfully so
—it makes simultaneous multi-stage operations possible,
which are then expected to reduce execution times in high-
dimensional vector processing significantly.
• Whether quantum computing will turn into the game-changer
everyone’s talking about, we are yet to find out! Currently,
there are no such models available on the market, but tech
giants are working hard to make that happen. With some much
uncertainty involved, the future of machine learning can be
difficult to predict.
2. Machine Learning Future – The Big Model Creation
The next few years are expected to mark the beginning of something big
—an all-purpose model that can perform various tasks at the same time.
You won’t have to worry about understanding the relevant applications
of a framework. Instead, you’ll train a model on a number of domains
according to your needs. How convenient would it be to have a system
that covers all bases—from diagnosing cancer to classifying dog images
by breed?
Of course, a well-designed quantum processor to enhance ML
capabilities will certainly give that development a boost. That’s why
great minds are now putting considerable effort into reinforcing the
scalability and structure of such a model. That’s one of the most exciting
future applications of machine learning!
3. Machine Learning Future – Distributed ML Portability
With the proliferation of databases and cloud storage, data teams want to
have more flexibility when it comes to using datasets in various systems.
We foresee a great advancement in the field of distributed machine
learning where scientists will no longer reinvent algorithms from scratch
for each platform. Rather, they will be able to immediately integrate their
work into the new systems, along with the user datasets. What does this
tell you about the future of machine learning?
In the coming years, we will likely experience some form of distributed
ML portability by running the tools natively on various platforms and
computer engines. In this way, we’ll eliminate the need for shifting to a
new toolkit. Experts in the field are already talking about adding
abstraction layers to make that technological leap.
4. Machine Learning Future – No-Code Environment
As open-source frameworks likeTensorFlow, scikitlearn, Caffe,
and Torch continue to evolve, machine learning technology is
likely to keep minimizing coding efforts for data teams.
In this way, non-programmers will have easy access to ML
with no postgraduate degree required; they can simply
download several packages and attend an online course on
how to work with these programs. Besides, automated ML
will improve the quality of results and analysis. So, in the near
future, machine learning will be classified as a major branch
of software engineering.
5. Machine Learning Future – The Power of Reinforcement Learning
Reinforcement learning (RL) is revolutionary—it enables companies to
make smart business decisions in a dynamic setting without being
specifically taught that.
With all that’s happening around us, unpredictability seems to have
become the new normal. Thus, we expect ground-breaking leaps in RL
to help us deal with unforeseen circumstances. And the future of
machine learning is linked with that of RL.
Everyone’s talking about the optimization of resources, but it is
reinforcement learning that can truly leverage data to maximize
rewards, where no other model can. RL is still in its early days, so we
will likely see several breakthroughs in the field within the next few
years in industries like economics, biology, and astronomy.