Human and Machine Language
CHAPTER 2
DEEP LEARNING FOR NLP
DEEP LEARNING NETWORKS LEARNS
REPRESENTATIONS AUTOMATICALLY
• Deep learning can be defined as the layering of simple algorithms called
artificial neurons into networks several layers deep.
• Representation learning is a subset of machine learning where the model
automatically discovers the best way to represent (or describe) the raw data
so it can perform a task — like classification or prediction — effectively.
• To make it more simpler: Representation learning means the model learns
features on its own, instead of relying on manually created features by
humans.
WHY ONLY “REPRESENTATION”?
• In machine learning, we rarely work with raw data directly we
work with representations.
Let’s See with an example
RELATION BETWEEN TRADITIONAL LEARNING
FAMILY AND REPRESENTATED LEARNING FAMILY
NATURAL LANGUAGE PROCESSING
• NLP involves taking the naturally spoken or naturally written
language of humans and processing it with machines to
automatically complete some task or to make a task easier for a
human to do.
• It is also the intersection of linguistics, AI, Computer Science.
A BRIEF HISTORY OF LEARNING FOR NLP
• In 2011, George Dahl and his team showed that deep neural networks could successfully
recognize words from speech - a big step for applying deep learning to language data.
• Then in 2012, the AlexNet model revolutionized computer vision by achieving amazing
accuracy on image datasets.
• Because of AlexNet’s success, most researchers focused on vision tasks for a while
(2013–2014).
• By 2015, these deep learning ideas started being applied to NLP tasks like translation.
The results were almost as good as traditional ML but with faster training and lower
computational costs - so efficient that Microsoft could even run translation on mobile
phones instead of cloud servers.
• Finally, by 2016–2017, deep learning began outperforming traditional machine learning
in NLP competitions — showing that deep learning was not just faster but also more
accurate for understanding and generating language.
2011: Deep learning enters NLP (speech recognition).
2012: Deep learning dominates image recognition.
2015: Deep learning starts catching up in NLP tasks.
2016–2017: Deep learning wins NLP competitions — becoming the new standard.
COMPUTATIONAL
REPRESENTATIONS OF
LANGUAGE
ONE HOT REPRESENTATIONS OF WORDS
• The traditional approach to encoding natural language
numerically for processing it with a machine is one-hot
encoding.
• As machine cannot understand the human language, it uses 0’s
and 1’s for computing the data.
• We will see the working of words encoding here:
The length of the sentence decides the length of the vector output
For Larger sentences larger vector outputs, for smaller sentences smaller
vector outputs.
WORD VECTORS
WORD VECTOR ARITHMETIC
UNDERSTANDING THE DIMENSIONAL SPACE
DIAGRAM TO SOLVE THE PROBLEM IS VERY
IMPORTANT.
Vector representations of words are the
information-dense alternative to one-hot
encodings of words. Whereas one-hot
representations capture information about
word location only, word vectors (also
known as word embeddings or vector-space
embeddings) capture information about
word meaning as well as location.
Vking = [-0.9, 1.9, 2.2]
Vman = [-1.1, 2.4, 3.0]
Vwoman = [-3.2, 2.5, 2.6]
we should find a location near the vector representing queen. To make this
arithmetic explicit by working through it dimension by dimension, we would
estimate the location of Vqueen.
LOCALIST VERSUS DISTRIBUTED REPRESENTATIONS
VECTOR BASED
ONE-HOT REPRESENTATIONS REPRESENTATIONS
• Treats every word as completely • Understands fine differences and
different it doesn’t understand that cat relationships it knows cat and dog are
and dog are related. more similar than cat and car.
• Needs humans to manually define • Learns relationships automatically from
large amounts of text data no manual
categories or relationships between labeling needed.
words. • Can easily fit new words by analyzing
• When a new word appears, the system how they appear in context with known
doesn’t know what to do it needs words.
retraining. • Based on how people actually use words
• Depends on human decisions how we in real sentences; learned from natural
define features or categories. data, not human bias.
• Doesn’t show any relationship cat and dog • Words with similar meanings have vectors
are as different as cat and banana. close together in space ,e.g., cat and dog
might be near each other, but car is farther
away.
ELEMENTS OF NATURAL HUMAN LANGUAGE
Phonology: Study of sounds in speech (like phonemes, which are the smallest sound units in
language).
Morphology: Study of word forms and structure (like roots, prefixes, suffixes).
Words: Actual vocabulary items.
Syntax: Grammar and structure how words combine into sentences.
Semantics: Meaning behind words and sentences.