Natural Language Processing
Module 4
© 2024, ZAKA AI, Inc. All Rights Reserved.
OUTLINE
● What is Natural Language?
● NLP Applications
● Data preparation
● Feature Extraction
● Word Embeddings
● Deep Learning in NLP
● Recurrent Neural Networks
● The Future of NLP
© 2024, ZAKA AI, Inc. All Rights Reserved.
What is NLP?
© 2024, ZAKA AI, Inc. All Rights Reserved.
What is Natural Language?
Natural language refers to the way we, humans, communicate
with each other.
We are surrounded with text each day:
● Email
● Text messages
● Web pages
● Food menus
● Street signs
● etc..
© 2024, ZAKA AI, Inc. All Rights Reserved.
Voice is also Natural Language
© 2024, ZAKA AI, Inc. All Rights Reserved.
Natural Language Processing
● Field of study focused on making sense of language
● Uses computers to work with natural language data
© 2024, ZAKA AI, Inc. All Rights Reserved.
Why NLP?
● The presence of significant data (text) around!
● Automated way to process data & organize information
© 2024, ZAKA AI, Inc. All Rights Reserved.
Challenges of Natural Language
Human language is highly ambiguous. It’s changing and
evolving.
● Lexical Ambiguity – Words have multiple meanings
● Syntactic Ambiguity – Sentence is having multiple parse
trees.
● Semantic Ambiguity – Sentence having multiple meanings
● Anaphoric Ambiguity - Phrase or word which is previously
mentioned but has a different meaning.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Challenges - Lexical Ambiguity
Words have different meaning
© 2024, ZAKA AI, Inc. All Rights Reserved.
Challenges - Syntactic Ambiguity
Sentence is having multiple parse trees.
● The professor said on Monday he would give an exam.
● The government asks us to save soap and waste paper.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Challenges - Semantic Ambiguity
Sentences With Multiple Possible Meanings
© 2024, ZAKA AI, Inc. All Rights Reserved.
Semantic vs Syntactic Ambiguity
© 2024, ZAKA AI, Inc. All Rights Reserved.
Challenges - Anaphoric Ambiguity
Phrase or word which is previously mentioned but has a different
meaning.
© 2024, ZAKA AI, Inc. All Rights Reserved.
NLP
Applications
© 2024, ZAKA AI, Inc. All Rights Reserved.
Large Language Models
© 2024, ZAKA AI, Inc. All Rights Reserved.
Part-of-Speech (POS) tagging
She sells seashells on the seashore
© 2024, ZAKA AI, Inc. All Rights Reserved.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Named Entity Recognition (NER)
© 2024, ZAKA AI, Inc. All Rights Reserved.
Applications
Sentiment
Analysis
© 2024, ZAKA AI, Inc. All Rights Reserved.
Applications
Text
Translation
© 2024, ZAKA AI, Inc. All Rights Reserved.
Applications
Visual
Question
Answering
© 2024, ZAKA AI, Inc. All Rights Reserved.
Applications
Image
Captioning
© 2024, ZAKA AI, Inc. All Rights Reserved.
Applications
Automatic
Handwriting
Generation
© 2024, ZAKA AI, Inc. All Rights Reserved.
Data Preparation
© 2024, ZAKA AI, Inc. All Rights Reserved.
Text Data Preparation
● Text Data preprocessing is an essential step in building a
Machine Learning model and depending on how well the
data has been preprocessed; the results are seen.
● Text preprocessing includes:
○ Text Cleaning
○ Tokenization
○ Stemming & Lemmatization
Text Text Desired
Feature Modeling
Data Preprocessing Output
Extraction
© 2024, ZAKA AI, Inc. All Rights Reserved.
Clean the text
Text Cleaning is task specific
● Remove punctuation and special characters (@!.,><-+#$%^)
● Remove numbers and emojis
● Removing leading and trailing whitespaces
● Remove stopwords (of, at, by, for, with, etc..)
● Normalizing case (Apple vs apple)
● Remove HTML/XML tags
● Replacing accented characters (such as é)
● Correcting spelling errors
© 2024, ZAKA AI, Inc. All Rights Reserved.
Tokenization
● Turning a string or document into tokens (smaller chunks)
● Step in preparing text for NLP
● Different approaches:
○ Split by Whitespace
○ Split by word
○ Custom regex
© 2024, ZAKA AI, Inc. All Rights Reserved.
Stemming & Lemmatization
Stemming
● Stemming is the process of reducing Word Stemming
inflection in words to their root forms
Information Inform
such as mapping a group of words to the
same stem even if the stem itself is not Informative Inform
a valid word in the Language.
● In Python, you can choose between Computers Comput
PorterStemmer or LancasterStemmer
Feet Feet
for suffix stripping, each has its own
algorithm and sometimes display
different outputs.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Stemming & Lemmatization
Lemmatization
● Lemmatization, unlike Stemming,
reduces the inflected words properly Word Lemmatization
ensuring that the root word belongs to
Information Information
the language.
● In Lemmatization root word is called Informative Informative
Lemma: the dictionary form of a word.
Computers Computer
● In Python, you can use the
WordNetLemmatizer to turn a word Feet Foot
into its lemma form.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Natural Language Toolkit (NLTK)
● Python library written for working and modeling text.
● It provides good tools for loading and cleaning text
● Usually used to get data ready for Machine Learning and
Deep Learning algorithms.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Feature Extraction
© 2024, ZAKA AI, Inc. All Rights Reserved.
Feature Extraction
● Machine learning algorithms cannot work on the raw text
directly.
● We need some feature extraction techniques to convert text
into a matrix(or vector) of features.
● Feature Extraction approaches:
○ Bag-of-words
○ TF-IDF
○ Word Embeddings
Text Text Desired
Feature Modeling
Data Preprocessing Output
Extraction
© 2024, ZAKA AI, Inc. All Rights Reserved.
1. Bag of Words (BOW)
● It is one of the most fundamental methods to transform tokens
into a set of features.
● Generate a fixed-length vector of numbers
● Assign a unique number for each word
● We are only concerned with what words are present and not the
order in which they are present.
© 2024, ZAKA AI, Inc. All Rights Reserved.
2. Word Frequencies with TF-IDF
● Generate scores to highlight words that are more interesting: frequent in a
document but not across all documents.
● The TF–IDF value increases proportionally to the number of times a word
appears in the document and decreases with the number of documents in
the corpus that contain the word.
● It is composed of two terms:
○ Term Frequency: how often a given
word appears within a document.
○ Inverse Document Frequency: how
rare the word is across documents.
© 2024, ZAKA AI, Inc. All Rights Reserved.
3. The Word Embedding Model
● A learned representation for text where words that have the same
meaning have a similar representation.
● Individual words are represented as
real-valued vectors in a predefined
vector space.
● The vector values are learned in a
way that resembles a neural network,
and hence the technique is
considered one of the breakthroughs
of deep learning on NLP problems.
[Link]
ion-with-nlp-tf-idf-vs-word2vec-vs-bert-41ff868
d1794
© 2024, ZAKA AI, Inc. All Rights Reserved.
The Word Embedding Model
● A learned representation for text where
words that have the same meaning
have a similar representation.
● This is done mathematically by using
vector similarity
● Considered one of the breakthroughs of
deep learning on NLP problems.
Mathematically, the cosine
of the angle between such
vectors should be close to 1,
i.e. angle close to 0.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Word2Vec
● Statistical method for efficiently learning a standalone word
embedding from a text corpus.
● Developed by Tomas Mikolov, et al. at Google in 2013
● Has become the de facto standard for developing pre-trained
word embedding.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Embedding Layer
● Word embedding that is learned jointly with a neural
network.
● Slow approach but tailors the model to a specific training
dataset.
Integer
encoded data
Input Layer Embedding Layer
© 2024, ZAKA AI, Inc. All Rights Reserved.
Embedding Layer
● The Embedding layer is defined as the first hidden layer of a
network with 3 arguments:
○ input_dim: size of the vocabulary in the text data
○ output_dim: size of the vector space in which words will be embedded
○ input_length: length of input sequences
# define the model
model = Sequential()
[Link](Embedding(vocab_size, vector_size, input_length = max_length))
…
…
© 2024, ZAKA AI, Inc. All Rights Reserved.
How to use Embeddings?
● Learn an Embedding for your problem with a new layer
○ Learn it Standalone where a model is trained to learn the
embedding, which is saved and used as a part of another
model for your task later.
○ Learn Jointly with another task using Neural Network
● Reuse an existing Embedding
○ Static: Freeze the layer and use it without updating its
weights
○ Updated: Weights will be updated while training the
model
© 2024, ZAKA AI, Inc. All Rights Reserved.
Hands-on:
Sentiment Analysis
© 2024, Zaka AI, Inc. All Rights Reserved.
Deep Learning
in NLP
© 2024, ZAKA AI, Inc. All Rights Reserved.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Modeling Sequential Data
1. Multi-layer feed-forward Neural Network is only meant for
data points, which are independent of each other.
2. Convolutional Neural Network is used for images and 2
dimensional data
3. Recurrent Neural Network (RNN)
Text Text Feature Desired
Modeling
Data Preprocessing Extraction Output
© 2024, ZAKA AI, Inc. All Rights Reserved.
The Recurrent Neural Network (RNN)
© 2024, ZAKA AI, Inc. All Rights Reserved.
Unrolling a Recurrent Neural Network
The network takes both:
● The output of the network from the previous time step as input
● The internal state from the previous time step as a starting point
for the current time step.
[Link]
© 2024, ZAKA AI, Inc. All Rights Reserved.
Example
Vocab: “helo”
Train on “hello”
● “h” -> “e”
● “he” -> “l”
● “hel” -> “l”
● “hell” -> “o”
© 2024, ZAKA AI, Inc. All Rights Reserved.
Different types of architectures
© 2024, ZAKA AI, Inc. All Rights Reserved.
Recurrent Neural Networks in NLP
In the context of NLP, Recurrent Neural Networks have different
use-cases
● Predicting the next word in a sentence or
generating text based on the context
provided.
● Translating text from one language to
another
● Predicting sequences of phonemes or
words from spoken language to text.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Process Data as a Sequence
● Sequence prediction is different from other types of
supervised learning problems.
● The sequence imposes an order on the observations that
must be preserved when training models and making
predictions.
● Generally, prediction problems that involve sequence data are
referred to as sequence prediction problems.
Input Next item(s) in
Sequence(s) Model the sequence(s)
© 2024, ZAKA AI, Inc. All Rights Reserved.
Sequence Classification problems
● Sentiment Analysis. Given a sequence of text such as a
review or a tweet, predict whether the sentiment of the text is
positive or negative.
Sequence
Classification
Model
© 2024, ZAKA AI, Inc. All Rights Reserved.
Sequence Generation problems
● Text Generation. Given a corpus of text, generate new
sentences or paragraphs of text that read like they could have
been drawn from the corpus.
Sequence
Generation
Model
© 2024, ZAKA AI, Inc. All Rights Reserved.
Sequence-to-Sequence prediction
● Text Summarization. Given a document of text, predict a
shorter sequence of text that describes the salient parts of
the source document.
Sequence
To Sequence
Model
© 2024, ZAKA AI, Inc. All Rights Reserved.
Problems with
RNNs
© 2024, ZAKA AI, Inc. All Rights Reserved.
Problems of Long-Term Dependencies
● RNNs is fine when dealing with short term dependencies.
○ Example: The longest river on Earth is Nile
● If the sequence is long enough he’ll have a hard time carrying
information from earlier time steps to later ones.
○ Example: The man who ate my pizza has black hair
∂E/∂W = ∂E/∂y3 *∂y3/∂h3 *∂h3/∂y2 *∂y2/∂h1
All the gradients would rush to zero
● Backpropagation: exponentially fast due to the
multiplication:
Vanishing Gradient
© 2024, ZAKA AI, Inc. All Rights Reserved.
Exploding Gradient
● Exploding gradients are a problem when large error
gradients accumulate and result in very large updates to
neural network model weights during training.
● Gradients would rush to large values and eventually blow up
and crash the mode.
● In this case, RNNs assign stupidly high importance to the
weights without much reason.
© 2024, ZAKA AI, Inc. All Rights Reserved.
The Long Short-Term Memory Network (LSTM)
● RNNs can face issues:
○ vanishing gradients: weight changes that quickly became so small as to
have no effect
○ exploding gradients: weight changes that became so large as to result in
very large changes or even overflow
● LSTMs overcome this challenge by design
using the “forget gate”.
● Comprised of layers of neurons
● Have recurrent connections so that the
state from time steps is used as context for
formulating an output.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Data Preparation for LSTMs
● Numeric data: normalization and standardization
● Categorical data: One Hot Encoding
● Sequences with varied lengths:
○ Sequence padding: adding a value (usually 0) to the beginning
(pre-sequence) or the end of the sequence (post-sequence)
○ Sequence truncation: trim the length of sequences to a desired length.
Trimming can be applied to the beginning (pre-sequence) or the end of
the sequence (post-sequence)
© 2024, ZAKA AI, Inc. All Rights Reserved.
Hands-on:
Deep Sentiment
Analysis
© 2024, ZAKA AI, Inc. All Rights Reserved.
Hands-on:
Language Model
© 2024, ZAKA AI, Inc. All Rights Reserved.
The Transformer
Architecture
© 2024, ZAKA AI, Inc. All Rights Reserved.
What are Transformers?
● Transformers are a revolutionary Neural
Network architecture introduced in the paper
“Attention Is All You Need” in 2017
● Aimed to solve Seq2Seq tasks while handling
long corpuses of text.
● They depend on attention mechanisms,
without relying on sequence-aligned RNNS.
© 2024, ZAKA AI, Inc. All Rights Reserved.
Why Transformers?
● Explainable models to a certain degree, by
using the attention mechanism to generate
attention maps.
● Versatile models that can be adapted for
many use-cases
● Scaling capabilities and improved
performance with increasing size/number of
encoder and decoders.
● Parallelizable across different cores
© 2024, ZAKA AI, Inc. All Rights Reserved.
The Encoder-Decoder Architecture
© 2024, ZAKA AI, Inc. All Rights Reserved.
The Encoder Architecture
● Specialized in classification tasks,
such as sentiment analysis and
spam detection
● The input is received as a token
sequence and transformed into a
vector representation of fixed-size
● Tasked with the input sequence
● Self-attention is applied to the
input-tokens
© 2024, ZAKA AI, Inc. All Rights Reserved.
The Encoder Architecture
1. The encoder receives the entire sentence 5
2. A positional encoding is added to each word
3. The multi-head attention block does its magic
4. We get the original word embeddings and 4
mix them with new information that is
relatable
3
5. Normalize everything, feed forward the
information to add more understanding, and 2
normalize again.
The entire
1 sentence
© 2024, ZAKA AI, Inc. All Rights Reserved.
The Decoder Architecture
● Specialized in text generation
● Takes fixed-size vector
representations from the prompt and
generate sequences of words, one by
one
© 2024, ZAKA AI, Inc. All Rights Reserved.
The Decoder Architecture
● Specialized in text generation
● Takes fixed-size vector
representations from the prompt
and generate sequences of words,
one by one
● Triangle Masking for Attention
limits the model to only see the
previously generated tokens
© 2024, ZAKA AI, Inc. All Rights Reserved.
Why different Architectures?
© 2024, ZAKA AI, Inc. All Rights Reserved.
From
Transformers to
LLMs
© 2024, ZAKA AI, Inc. All Rights Reserved.
What are Large Language Models?
LLMs are advanced AI Models that understand and generate
human-like text. These models are:
● Based on Deep Learning Techniques (Transformers)
● Pre-trained on massive text datasets
● Flexible and can be fine-tuned
© 2024, ZAKA AI, Inc. All Rights Reserved.
LLM Applications
Large Language Models can be used in a variety of Language
related applications.
Machine Text Question
Translation Generation Answering
Summarization Content Sentiment
& Classification Creation Analysis
© 2024, ZAKA AI, Inc. All Rights Reserved.
State-of-the-art LLMs
© 2024, ZAKA AI, Inc. All Rights Reserved.
BERT
Bidirectional Encoder Representations from Transformers, also
known as BERT, is a language model developed by Google AI.
● Transformer model with Encoder only architecture
● Pre-trained on a massive dataset of text and code
● Tasks: Machine translation, text summarization, QA
© 2024, ZAKA AI, Inc. All Rights Reserved.
GPT and Variants
Generative Pre-trained Transformer, is a language
model developed by OpenAI.
● Pre-trained on massive amounts of text data
● Very effective at generating human-like text
● GPT-4 is constantly being updated and can
perform tasks such as image understanding,
graph drawing, and much more
● Taks: Text generation, Machine Translation, QA
© 2024, ZAKA AI, Inc. All Rights Reserved.
ChatGPT
An LLM chatbot by OpenAI, and the
most known variant of GPT-4.
● Trained to be informative and
comprehensive
● Follows instructions and completes
requests thoughtfully
© 2024, ZAKA AI, Inc. All Rights Reserved.
Llama
Llama stands for Large Language Model
Assistant, a chatbot developed by Meta AI.
Still under development.
● Llama 3 is open-source
● Currently being developed and applied in
different Meta products
● Optimized for assistant type use-cases
© 2024, ZAKA AI, Inc. All Rights Reserved.
Training Large
Language
Models
© 2024, ZAKA AI, Inc. All Rights Reserved.
LLM Training Approaches
Supervised Learning
● Machine Translation: Utilizing sentence pairs in different
languages, where the label corresponds to the translated
sentence in the target language.
● Classification Tasks: Including sentiment analysis, language
prediction, and fake news detection.
● Time Series: Replacing traditional models for forecasting.
© 2024, ZAKA AI, Inc. All Rights Reserved.
LLM Training Approaches
Unsupervised learning
● Predict the next part of a sequence without needing specific
labels.
● This training method uses Maximum Likelihood Estimation
(MLE).
● Not limited to text and can also handle code, time series, and
pixels.
© 2024, ZAKA AI, Inc. All Rights Reserved.
LLM Training Approaches
Transfer learning
● Fine-tuning
● Feature Extraction
● Adapting to specific domains
● Multimodal Tasks
© 2024, ZAKA AI, Inc. All Rights Reserved.
Step-by-step
Approach
© 2024, ZAKA AI, Inc. All Rights Reserved.
The Training Process
Model Hyper
Data Data
Architecture Parameter
Collection Preparation
Choice Tuning
Evaluation Testing Fine-tuning Pre-training
© 2024, ZAKA AI, Inc. All Rights Reserved.
Fine-tuning
● Fine-tuning involves adapting a pre-trained
model to perform a specific task
● It utilizes a smaller dataset specific to the
desired task.
● Can be done as:
○ Supervised fine tuning (SFT)
○ Reinforcement Learning with Human Feedback (RLHF)
© 2024, ZAKA AI, Inc. All Rights Reserved.
Fine-tuning
Fine-tuning in NLP is usually done for downstream NLP tasks,
which are tasks that can be done by using a pre-trained model.
Transformers
© 2024, ZAKA AI, Inc. All Rights Reserved.
Hands-on:
Arabic Hate Speech
Detection
© 2024, ZAKA AI, Inc. All Rights Reserved.
THANK
YOU
[Link]
© 2024, ZAKA AI, Inc. All Rights Reserved.