mmmmmmm

for personal study of

* this notebook is an exam ,

I use in
you
naturally
: may skip some topics suggest
,

as a north star rather than a

right cokebook .

# S
NLD
automa
ting the
analysis,
what is NLP ? generation , and acquisition
of human
language .
2 Morphology explores the
word structures and

*
their

studies meaningful components .

Lexemes focuses on [

and clarifying word

normalizingwith meanings
especially words that have multiple
bank ,
meanings :

meaning
> Syntax focuses on sentence

Structures and rules

Semantics turns
language [
to actionable or understandable

representations like commands or logic

NLP models provide a

> framework for

understanding language
from basic text
processing EML

what makes NLP

Challenging comes down to
many factors .
languages involve
many layers (sound ,
grammar ,
syntax , etc .
) which interact with each
Human
in
many ways language is also
ambiguous words take multiple meanings
.

, can

and the richness of it allows an idea to be

expressed in multiple
ways.
BasicText
Processing Some basic text operations
normalization :

putting words , tokens in

describe Standard format (US vs . U S) which an

regex models
strings text
.

to
patterns
.

example of is Case Folding

.

lemmatization : reduce words to their

base form .

~> edit distance-

minimum edit distance is the min number of
operations to
go from text-1 to text 2 -
.
Levenstein

distance is the same but operations have fixed cost :

-
insertion 1
- deletion 1
- Substitution = 2

he distance is found using search (dynamic

programming (

Language
in the of error correction and
Models
topic spelling
errors there are two types :

correction for
Non-word spelling
tokenization is to break down text to words that are not correct at all (graffe >
-

giraffel
meaningful units .
We have
"type" and we Solution
the
: find words not in
dictionary suggest , words with

have least minimum distance prioritized by frequency

"tokens" that are an instance of a

type such as
ers . Cats
Real-word spelling
, for words that

are in
dictionary but used in
wrong context

the challenge is with numbers and

short/merged
Desert vs .

Jessert)
forms.
Noisy Channel
isn't , PhD , $63 51 .
another solution is

tokenization can be rule-based for faster inference

or data-driven for a flexible approach ·and tokens

data-driven
are subwords -
uses a
large corpus

to decide the tokens.

a common tokenizer :
BytePair originally a

compression algo , turns most frequent subwords

into a new token and comes
t er a deterministic

tokenizer .

(BPE used by OPT)

 given a misspelled words , find the correction The N-Gram is a simple model

In
il
w such that :
tha
M

W =
argmaxP(w(x) is able to capture
=
argmax P(x/w) P(w) .
Syntax

&
-

PXm) (like nouns Follow "the")

high
N-grams
memory
order
need much
Computation
&
-have zi
patterns

One metric that helps us evaluate a

model
language
perplexity
is

it evaluates how confident

the an LM is in a test Set
noisy channel is a probabilistic model that assumes

an observed word is the

nosy version of original data. lower perplexity means LM assigns higher
to correct words
It relies on
Baye's rule to find the most
likely original
word.
probability
The but it originates from
-log P
naming
Perplexity
can seem strange
communication systems :
"nosy" in our case means

rises which
misspelled
for us
=
2G ↓
or
wrong "Channel" is the way error total number ofwth
,

probability
.

of tokens
is how one misspelled a word The
.

goal is to model how channel token in test

Set

causes noice to mitigate it

.

~ N-Gram Probabilities -

Given the last sequences , estimate what word is most

Common to follow.

What kind of knowledge is captured by

model ? the whole formula is
this
language
Average Cross-entropy
unigram P(wi) 2

bigram P(wilWiz) => if model is

very
confident
·
= to if it is lame
the origin of the Formula goes back
Semantic
to information theory a
perplexity of
Vector
,

1 implies that the model

on
average

embedding
,

choose from 10 likely options

must
equally
at Step
each

words have of
multiplications of and many types association
log is used so that the
relatedness with each other.
doesn't cause numerical
many probabilities couch bark
underflow (the number would be super small)
I

& Sofa antonym

Synonym -
light
Problem : (Ms assign to unseen uypernymy
- >
-
has
wheel
vehicle car

Smoothi
ing mi &
the solution is
Sequences ,

ponymy part
of

(like add 1 to all frequencies and

read-lemma-reading
renormalized
word with these
how to represent a

is backoff relationships?
another solution to lower
word
order of is to
represent
N-gram sometimes lowering One
way
a
,

~ helps (better generalization ( as a

point in an N-dimentional
space so a wort=vector
-

lext Classification embedding

aka an as it

is embedded in this space .

and there is
>
- enables
computing similarity
there is Naive Bayes > Can be learned
defined
-
automatically
logistic Regression which uses

derived from
be
features /number of positive words , embeddings can

Co-occurance matrices : words that appear

# of Won't go into detail
"no's ,
etc. )
together often have similar meanings.
(really cool ! Search
it
up)

Now raw occurrence counts are not

informative on their own

Reweighting
give weight
)
more to

more
meaningful associations

reduce noise
I
one method of
Reweighting is tf-id ~ wordzvem
it takes into account both the
frequency how
trains a model to predict
uniqueness of
and the words .
two words close
likely are to appear
to each other The goal is to learn good
TF-IDF TF X IDF .

=
word model .
embettings with this

TF : how
frequent a wordt is document ↓ use two vectors per word :

1 when w is in context

TF(t ,
inAnd
d) 2 when w is an outside word

and the end . this helps

average/concat
in

log-scaling can be applied to reduce the

with optimization .

effect of high frequencies

classifier
In the end we throw away the
TF(tid)
log (Count (tid))
= 1 +
and use the word embettings .
IDF how word is

properties of
:
unique a

embettings
IDF(t) =

1090 toments
witht) -
similarity depends
on

Context size
during training
-words Can have first-order co-occurance
the problem with these
frequency-based or second-order (mutual neighbors
embettings are :

>
they capture relational
meanings
&
-

they are too

long -harderFora
Rome France Paris
- Italy
-
-

& they are

sparse not
generalize
well (curse of dim)

So it's best that

to use
embettings
are shorter and less dense

Neural

LanguageaG in
Simple form : use a feed forward network
and
that predicts next word probability
trains word embeddings .
Recurrent
RNNs can also be used for sentence

encoding or sentiment analysis by using the

hidden state of all last step .

Neural
or

Network
Stacked-RNN is a
type of architecture

but the cost of

training rises
rapidly
from wikipedia : RNN (CC)

Language modeling is a temporal problem.

models of fixed
The problem with
N-gram
Window sizes is that the context is limited .
Small

Bidirectional RNN
whole
and is to look at a
CW causes degrade in output predictions
sequence (not for and the outputs
big CW causes
huge computation . generation)
are
usually concatenated
RNNs can
theoretically preserve arbitery
CW The hidden state is used
long size.

to current taken.
from previous steps process

It * h provides
or Context
a sort of memory

fixed

y
ht -
1
t * the context has no size

&
-
& If
&
A model
Size of
does not

the Context
scale with the

& transformed hidden state of previous

* Same weights applied at each Step
Step is added to the transformed
input
-

like an
N-gram ,
RNN can be

he g (Uh + Way+ b)
=
used for

through
autoregressive generation
repeated sampling .

an activation function Start with <S) end with < /SY

LSTM Network
LSTM does not guarantee no

but makes it easier

vanishing gradient information

to
long-term
n.
preserve

Vanishing Gradient
gradient signals from far is lost
away
compared to close signals This . can be
RUNs and cause
a serious problem in

it to lose context from further tokens

content
new cell content : the new

of previous steps. ↓ be written to the cell

hidden state h and

LsTMs have a a

cell State C of the same size. C

last
content from
Cell State : keep some
information -
stores long-term cell state and write new content

ISTM can write , read , erase information

from each cell .

Deciding which is hidden State : read some content from the

of Gell state-
the same
controlled by three gates
Size as h and C .

forget gate Input Gate what Parts

: What is kept
:
Output Gate : what parts

of new cell content

are of cell are outputted to
or
from previous
forgotten written to cell the hidden
state
Cell State
 Gated
Recurrent
Units
(GRU)
a simpler alternative to LSTM

and
at each step has an input state a

hidden State (no Cell State)

update gate : which parts of hidden state

are updated vs. preserved

LSTM is powerful but GRU is Faster :

Reset gate : which

parts of previous hidden
new content
States are are used to compute

New Hidden State Content :

reset
gate
Selects parts of previous hidden state . Use this and
the input to compute new content

Hidden State : what is kept from previous

hidden and what is

state (update gate) updated
to new content .
Segaseq
also encoder-recoder is a of
CNNs for NLP
the idea is to use ID convolutions on

or
type of certain length- It also
network (in
subsequences
this case two RNs) that over time .

and output
includes max (or average) pooling
take a sequence as input
a
Sequence

S
this
max-pooling over time
goes through
the time dimension of each feature and selects the max
.

it deals of variable length

The reason is so that
-

Segzseq RNN with Attention is

and
very plausible
but it is not tends
Sequences ,

of network that in each

another type to
oversimplify the process .
On the other hand

and
step of the decoders ,
it's directly connected the CNN networks is parallelized
networks also
optimized for GPU
are
attention to certain These
encoder pay
.

to to

parts of encoder .

more difficult to interpret

This attention is so the encoder learns

attention to various parts

of
how to give

the input and forget less ,

it's much

more simple than the self-attention

mechanism .
 Transformers
the good thing about RNNs is that they have no
fixed-length
and
Context window (intheory) but suffer from vanishing gradient
The transformers support parallel
processing of
slow
training
.

inputSeq -

(from CS224N NLP

At their heart use self-attention which we use multi-head attention to get

,

of tokens in multiple ways :

Obtains a contextualized representation of context

token . There three roles

each are important :

query is the token we're

attending to

key other tokens we take into account

when
attending to
query (including the query

-value is what we compute attention

Scores for, can be anything.
the block has
trus other layers as well :

9 = we k - x
,
WK vi = x, W
V
- Feed forward layer for more process

the whole formula can be written as : - Residual Connections to flow the

it from
gradient and prevent vanishing

softmax(Rkg
.

· -
layer Norm to keep the weights in

limit and stabilize training

.
 If K= 1 then have greedy

Large
you
Search but Beam is a
general case

that lets look into potential paths

you

Language
down a search tree.

Sampling popular It is
.
is most

to tune between

Models
Probabilistic and allows

random or more factual.

more
generation

Conditional is when LLM

generation Top-K truncates the top
k most

response based given

a
outputs on
scored tokens , renormalize , and sample .

But be done
prompt .

many task can

and with word prediction

.
Top-p select vocals with at least
without a prompt
O probability
.

Decoding can be
greedy : select
Temperature reshapes the

: Deterministic
word of highest probability probability distribution by dividing logits
by T
Y = Softmax (u/T)
Beam Search is another method .

Vol
probabilities
-

E Softmax -

-- -
-

>

-
= !

- i

- -

skipped
In Beam Search we select the K highest Pretraining
tokens. If K= 2 we select types of Fine-tuning :

Probability Parameter
We and We and feet each to the model continual pretraining & Efficient FT (PEFT)
and sets of profs update some params
again get two new

③Task-specific FT by and keep the rest frozen

head to Supervised
adding
FT
Pe and P2 and we
multiply each a

the model (like for

which requires
VIP and a dataset of dedicated
~ with its p :
we get Classification)
WzPz prompt/responses
two rectors .
a type of language models is Encoder-decoder
bidirectional encoder like BERT decoders are trained to predict next words

encoders are trained to predict masked

that takes an entire
sequence +
words in a sequences .
benefit
These models
from bidirectional of
far Best for processing text.
generated output so .

translation ,
sentiment classification , etc

Does not use masked attention .

predict word
They aren't trained to next

but similar to
denoising tasks , predicts
the middle of .
tokens in
sequence

tokenizer fertility is a measure of

tokens needed
how
average
are

many
on

of
for a word . Impacts inference speed
models-

language models can also be used for

like
Classification Encoder-decoder
.
models

BERT can do this with a special [CLS]

token .
Also last layers can be changed

and fine-tuned .
problem with
large LLMs
Speech
Processing
these models often perform better ,

environmental footprint ,
high cast ,

&
Parameter Sharing
Some
methods -
Pruning
Distillation
multimodality is bridge
to
processing
a
> Knowledge
- speech
Reduce
Computation Mixed Precision
between NLP and
Signal Processing .

Sound is vibration of air molecules

Pitch is the
frequency and

loudness is the amplitude

Spectrum of Learning
Zero shot one-shot-few-shot-full
fine-tuning
-

example
basic
prompting single
Y
(Read about prompting , Chain-of-thought) higher
frequency
means
higher
Signal is the representation
RLHF
Pitch
of It's
sound waves .
analog by
with

model
nature but
sampling and quantization
1 pre-train can be converted into
digital form
-

reward model > map preferences

train
-

2 a In
into numbers
Analog Digital
3 fine-tune based on reward Conversion (ADC)
Signal is first
Sampled and
the quantized.

(Read about Fourier Transform and its variances)

Audio has
many features we can extract :

Pitch , patterns
rhythm , stress ,

of waveforms ,
zero-crossing
length and loudness,

etc ...
 of
typical applications Speech Foundational Models

modeling
Speech & sound transformer-based
general-purpose
speech
models for Speech- a
challenge with

Speech Recognition Sound >

-
Text is that it can be
very variable . SFTS

Text 2 Speech Sound

can
mitigate
this challenge be :

Text >
-

they learned robust representations

Speech Separation different sources
separating from huge data
Speech Enhancement increases
quality & heny leverage advanced
processing
techniques
The
typical ASR architecture is
These SFTs can be used as feature extractors

encoder-decoder with mounted them and

Sequence to sequence heat on
or a can be

fine-tuned for specific tasks ·

Transformers or RNNS

The Sound waves are represented as a

spectrum or
time-frequency that is feb int

which is then handled

Encoder to extract patterns

the Decoder
.
by

with TTS (tranfer text to audio) the pipeline

is takes
again encoder-decoder which

text , the encoder turns it into a latent representation

which is to decoder to produce a

given
a

mel-spectogram .