Text Pre-processing,

Tokenization &
Embeddings, RNN
AGENDA

1.1 Introduction to NLP Pipeline (30 mins)

1.2 Text Pre-processing (60 mins)

1.3 Tokenization Techniques (45 mins)

1.4 Word Embeddings (75 mins)

1.5 RNN (20 mins)

Confidential © Icanio Technology Labs Private Limited 2026

1.1 Introduction to NLP Pipeline

What is Natural Language Real-Life Example

Processing (NLP)?
When you type this in Google:
Natural Language Processing, or NLP, is a field
“Best phone under ₹15,000”
of Artificial Intelligence that helps computers
Google understands:
understand, read, listen to, and respond to
human language in a meaningful way. ● You are looking for products
● Your budget
In simple words:
● Your intention to compare options
NLP teaches machines how humans
talk, write, and express ideas — and
That understanding happens because of NLP.
how to reply like a human.

Confidential © Icanio Technology Labs Private Limited 2026

Why Do We Need NLP?

Computers naturally understand numbers and code, not human language. NLP acts as a bridge between
humans and machines.

With NLP, computers can:

● Read text messages

● Listen to speech
● Understand meaning
● Respond with useful answers

Confidential © Icanio Technology Labs Private Limited 2026

Daily Life Examples

Action You Do NLP Working Behind the Scenes

Ask Google Assistant a question Speech → Text → Understanding → Answer

Receive spam emails in spam folder Email → Analyze meaning → Classify

Translate a message in Google Translate Language detection → Translation → Output

Get movie recommendations Read reviews → Understand opinion → Suggest

Confidential © Icanio Technology Labs Private Limited 2026

Example: Chatbot Reply System

You type:

"I want to book a train ticket"

The system:

1. Reads your sentence

2. Breaks it into words
3. Understands your intention (booking)
4. Finds travel service
5. Responds with booking options

Confidential © Icanio Technology Labs Private Limited 2026

How NLP Understands Language

NLP doesn’t read full sentences like humans. It breaks text into small pieces
called tokens.

Example:
Tokens:
Sentence:
I | love | learning | AI
"I love learning AI"

Then it tries to:

● Find relationships
● Understand emotion
● Identify meaning

Confidential © Icanio Technology Labs Private Limited 2026

Common NLP Features (With Easy Examples)

1. Sentiment Analysis 2. Information Extraction

Finds the emotion in text. Finds important details from text.

Example: Example:

"This phone is amazing!" → Positive "My interview is on March 5th in

"This service is terrible" → Negative Chennai at Infosys"

Used in: Used in:

● Product reviews ● Date → March 5th

● Social media monitoring ● City → Chennai
● Feedback systems ● Company → Infosys

Confidential © Icanio Technology Labs Private Limited 2026

Common NLP Features (With Easy Examples)

3. Speech-to-Text & Text-to-Speech 4. Document Summarization

Converts: Turns long text into a short version.

● Voice → Text (Google Voice typing) Example:

● Text → Voice (Audiobooks, GPS navigation)
● A 10-page news article → 5-line summary

5. Machine Translation

Converts one language to another.

Example:

"Good Morning" → "Buenos Días"

Confidential © Icanio Technology Labs Private Limited 2026

Typical NLP Tasks (Real Use Cases)

Task Extraction Example

Searching “Best laptops for

Information Retrieval Finds what you search for.
students” on Google
Resume scanner extracting
Information Extraction Finds names, dates, places.
Name, Email, Skills

Language Generation Creates new text automatically. AI writing emails or stories

Positive vs Negative reviews

Text Classification Sorts text into categories.

Confidential © Icanio Technology Labs

Why Should You Learn NLP?

NLP powers many high-demand technologies today.

Real Systems Using NLP: Career Advantage:

● Spell check in Word If you know NLP, you can work in:
● Google Search
● Alexa, Siri, Google Assistant ● AI Engineering
● Gmail Spam Filter ● Chatbot Development
● Google Translate ● Data Science
● Search Systems
● Automation Tools

Confidential © Icanio Technology Labs Private Limited 2026

Popular NLP Tools & Libraries

Some commonly used tools:

● NLTK – Beginner-friendly Python NLP library
● Gensim – Topic modeling & text analysis
● Stanford NLP – Research-level NLP toolkit
● OpenNLP – Java-based NLP tools
● spaCy – Fast NLP for real-world apps

Confidential © Icanio Technology Labs Private Limited 2026

Why Is NLP Hard?
[Link] Matters
Human language is not simple or direct.
“He is cool”

[Link] Meaning, Different Could mean:

Sentences
● Friendly
“I’m tired” ● Calm
“I need rest” ● Stylish
“I’m exhausted”.

[Link] & Idioms [Link]-World Knowledge Needed

“Yeah, right!” “The glass fell and broke”

Means the opposite of what is NLP must know that glass is
said. fragile — not just a word.

Confidential © Icanio Technology Labs Private Limited 2026

Question

Which app do you use daily that works because of

NLP?

Confidential © Icanio Technology Labs Private Limited 2026

1.2 Text Pre-processing in NLP (45 Minutes)

Learning Goal
By the end of this session, learners will be able to:

● Understand why raw text cannot be used directly in AI models

● Clean and prepare text using real NLP techniques

● Write Python code to process messy real-world data

Confidential © Icanio Technology Labs Private Limited 2026

Why Need Text Pre-processing?

Example:

“I Looove this phone!!! 😍 Visit: [Link]”

Humans understand:

● Emotion → Happy
● Meaning → Likes the phone
● Extra junk → Website, emojis, symbols

But a computer sees:

I, Looove, this, phone, !!!, 😍, 🔥, Visit, [Link]

Confidential © Icanio Technology Labs Private Limited 2026

What is Text Pre-processing?

Text pre-processing means:

Cleaning and organizing raw text so machines can understand it better

Cooking Example

You don’t cook vegetables without: Same way, text must be:

● Washing ● Cleaned
● Cutting ● Simplified
● Removing dirt ● Structured

Confidential © Icanio Technology Labs Private Limited 2026

What is Text Pre-processing?

Text pre-processing means:

Cleaning and organizing raw text so machines can understand it better

Cooking Example

You don’t cook vegetables without: Same way, text must be:

● Washing ● Cleaned
● Cutting ● Simplified
● Removing dirt ● Structured

Confidential © Icanio Technology Labs Private Limited 2026

Step 1: Lowercasing & Punctuation Removal

Why This Matters What We Remove

Computers think: ● Symbols: ! @ # $ %

● Extra punctuation: . , ? !
“Apple” and “apple” are different words

Example:
Real-World Use:
Sentence:
Search engines do this so:
"This Phone Is AMAZING!"
“Buy Phone” = “buy phone” = “BUY PHONE”
After cleaning:

"this phone is amazing"

Confidential © Icanio Technology Labs Private Limited 2026

Step 2: Stop-word Removal

What Are Stop-Words? Example:

These are common words that don’t add much Original:

meaning:
"I am learning NLP in my college"
● is
● After removing stop-words:
am
● the “learning NLP college"
● in
● on Why This Helps Now the sentence keeps only important meaning
● and words
It makes:

● Models faster
● Meaning clearer
● Storage smaller

Confidential © Icanio Technology Labs Private Limited 2026

Step 3: Stemming vs Lemmatization

Both techniques reduce words to their base form, but they work differently.

Stemming (Rough Cutting) Lemmatization (Smart Dictionary Method)

It simply chops words. It finds the real root word using language rules.
Word Stemmed Word Stemmed

running run running run

studies studi better good

played play studies study

Real-World Example:
Google Translate and Grammarly use Lemmatization, not basic stemming.
Confidential © Icanio Technology Labs Private Limited 2026
Step 4: Handling Emojis, URLs & Special Characters

Real-world text is messy.

Example:

“I love this laptop 😍 Check here: [Link]

Problems:

● Emojis confuse models

● URLs don’t help sentiment
● Special symbols add noise

Clean Version:

"i love this laptop"

Confidential © Icanio Technology Labs Private Limited 2026

Hands-On Practice

Step 1: Install Libraries

pip install nltk spacy

python -m spacy download en_core_web_sm

Sample Raw Text

text = "I Looove this phone!!! 😍 Visit now: [Link]

Confidential © Icanio Technology Labs Private Limited 2026

Hands-On Practice

Step 2: Lowercase & Remove Symbols

import re

text = [Link]()

text = [Link](r'http\S+', '', text) # Remove URLs

text = [Link](r'[^a-z\s]', '', text) # Remove emojis & symbols

print(text)

Output:

i looove this phone visit now

Confidential © Icanio Technology Labs Private Limited 2026

Hands-On Practice

Step 3: Stop-word Removal (NLTK)

import nltk
Output:
from [Link] import stopwords
['looove', 'phone', 'visit']
from [Link] import word_tokenize

[Link]('punkt')

[Link]('stopwords')

words = word_tokenize(text)

stop_words = set([Link]('english'))

clean_words = [w for w in words if w not in stop_words]

print(clean_words)

Confidential © Icanio Technology Labs Private Limited 2026

Hands-On Practice

Step 4: Lemmatization (spaCy)

import spacy
Output:
nlp = [Link]("en_core_web_sm")
['love', 'phone', 'visit']
doc = nlp(" ".join(clean_words))

lemmas = [token.lemma_ for token in doc]

print(lemmas)

Confidential © Icanio Technology Labs Private Limited 2026

Real-World Systems That Use This

System Why It Uses Preprocessing

Spam Filters Removes useless words

Chatbots Understands intent

Search Engines Matches keywords

Review Analysis Finds emotions

Resume Screeners Extracts skills

Confidential © Icanio Technology Labs Private Limited 2026

Activity

“This movie was soooo bad 😡😡!!! Don’t watch it: [Link]”

Task

1. Lowercase it

2. Remove emojis and URL

3. Remove stop-words

4. Find base words

Confidential © Icanio Technology Labs Private Limited 2026

1.3 Tokenization Techniques (45 Minutes)

Learning Objective

By the end of this session, learners will:

● Understand what tokens are and why AI needs them

● Know the difference between word-level and subword tokenization

● Learn how modern AI models like GPT, BERT, and LLaMA read text

● Practice using real-world tokenizers from Hugging Face

Confidential © Icanio Technology Labs Private Limited 2026

Warm-Up Thought

Example
“If I give a book to a computer, how does it start reading it?”

Humans read sentences → words → meaning

Computers read characters → pieces → numbers

Tokenization is the step that cuts text into pieces the computer
can count and understand.

Confidential © Icanio Technology Labs Private Limited 2026

What is Tokenization?
Tokenization means:

Breaking text into small meaningful parts called tokens

Tokens can be:

● Full words
● Parts of words
● Even single characters

Real-World Example
These tokens later become numbers that AI models use
Sentence: to think.

"I love artificial intelligence"

Possible tokens:

I | love | artificial | intelligence

Confidential © Icanio Technology Labs Private Limited 2026
1. Word-Level Tokenization
What It Does

It splits text word by word, usually at spaces and punctuation.

Example: Advantages

Text: ● Easy to understand

"Chatbots are changing education"

● Good for basic NLP tasks
Tokens:
● Human-friendly
Chatbots | are | changing | education

Confidential © Icanio Technology Labs Private Limited 2026

1. Word-Level Tokenization
Problems Why We Need Smarter Tokenization
What about this word: Modern AI talks to:
"unbelievability" ● Millions of users
If the model has never seen it before, it gets confused. ● Multiple languages
● New words every day
Also: ● Emojis, hashtags, slang
● Large vocabulary
So instead of full words, models learn pieces of
● Doesn’t handle spelling mistakes well words.
“Helo” ≠ “Hello”

Confidential © Icanio Technology Labs Private Limited 2026

2. Subword Tokenization
This method breaks words into smaller meaningful parts.

Example:

Word:

"unhappiness"

Subwords:

un | happy | ness

Now the model understands:

● un → not
● happy → emotion
● ness → state

Even if it never saw the full word before!

Confidential © Icanio Technology Labs Private Limited 2026
Popular Subword Techniques

BPE (Byte Pair Encoding)

WordPiece SentencePiece
It learns common letter patterns.
Tries to create tokens that reduce Works at character level, not space
Example: confusion and mistakes. level.

"playing" → play + ing Example: This helps for:

It keeps merging letters that appear "smartphone" → smart + ● Languages without spaces
together often. phone (Chinese, Japanese)
● Mixed-language text
Used in: Used in:
Used in:
● GPT models ● BERT
● Many translation systems ● Google Search systems ● LLaMA
● Multilingual AI systems

Confidential © Icanio Technology Labs Private Limited 2026

Tokenization in Modern AI Models

GPT (ChatGPT) BERT LLaMA

Uses: Uses: Uses:

BPE-based WordPiece SentencePiece

tokenization
Text: It breaks text into flexible pieces, even
Text: across languages.
"unhappiness"
"ChatGPT is Important Fact
Becomes:
amazing" AI models do NOT see:
un | ##happy | ##ness
Might become: “Hello world”
The ## means “this part
Chat | GPT | is | amaz They see:
continues the previous word”.
| ing
[15496, 995]

Numbers represent tokens.

Confidential © Icanio Technology Labs Private Limited 2026
Hands-On: Hugging Face Tokenizers
Step 1: Install Library

pip install transformers sentencepiece

Step 2: Try Tokenizing Text

from transformers import AutoTokenizer

text = "Tokenization helps AI understand language"

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

tokens = [Link](text)
print(tokens) Output:

['token', '##ization', 'helps', 'ai', 'understand',

'language']

Confidential © Icanio Technology Labs Private Limited 2026

Real-World Applications

System Why Tokenization Matters

Chatbots Understand slang & typos

Translation Apps Break words across languages

Voice Assistants Process spoken text

Search Engines Match user queries

Social Media AI Read hashtags & emojis

Confidential © Icanio Technology Labs Private Limited 2026

Class Activity

“I’m learning #AI with ChatGPT!!!”

Task:

1. Word-level tokens

2. Subword tokens

3. Which one works better for social media text?

Confidential © Icanio Technology Labs Private Limited 2026

Word Embeddings

Learning Goals

By the end of this session, learners will be able to:

● Understand why AI cannot work with words directly

● Explain how words become numbers with meaning

● Describe how Word2Vec and GloVe learn relationships

● Compare old-style embeddings vs modern AI understanding

● Train a real embedding model in Python

Confidential © Icanio Technology Labs Private Limited 2026

Warm-Up: Human vs Machine Thinking

Question:

“Which words are more similar — king and queen OR king and banana?”

Humans instantly know:

● King ↔ Queen (related)

● King ↔ Banana (not related)

But computers see:

"king" = just letters

"banana" = just letters

So how does AI learn meaning?

That’s where Word Embeddings come in.

Confidential © Icanio Technology Labs Private Limited 2026

What Are Word Embeddings?

Word embeddings turn words into numbers, but not random numbers.

They turn words into:

Vectors — lists of numbers that store meaning, relationships, and similarity

Simple Idea:

Imagine a map of words.

● Similar words live close together

● Different words live far apart

Example:

● cat 🐱 is close to dog 🐶

● cat is far from airplane ✈️

Confidential © Icanio Technology Labs Private Limited 2026

Why Embeddings Matter

Without embeddings, AI only sees: With embeddings, AI sees:

Words = IDs like 1, 2, 3, 4 Meaning, emotion, topic, and usage

Real-World Example

Recommendation System Chatbots

Netflix doesn’t just recommend based on exact words. If you say:
It understands:
“I’m sad today”
● “romantic” is close to “love” “I feel unhappy”
● “thriller” is close to “suspense”
A chatbot understands both mean negative
This happens because of embedding similarity. emotion, not different topics.

Confidential © Icanio Technology Labs Private Limited 2026

How Embeddings Are Learned

AI learns word meaning by:

Watching which words appear near each other

Just like humans learn:

If “coffee” often appears near “cup”, “hot”, “morning”, “drink”

Then “coffee” must be a drink

Confidential © Icanio Technology Labs Private Limited 2026

Word2Vec – Learning from Context
Word2Vec is a method that learns word meaning by predicting neighbors

CBOW (Continuous Bag of Words) Skip-Gram

The model: The model:
Looks at surrounding words → Predicts the missing Takes one word → Predicts
word nearby words
Example: Example:
Sentence: Input:
"I drink ___ in the morning" coffee
Context: Predictions:
drink, hot, cup, morning
I, drink, in, the, morning
Prediction:
coffee
GloVe – Learning from Global Patterns

GloVe stands for:

Global Vectors Famous GloVe Example

Instead of looking at one sentence at a time, it:

AI learns:
● Scans the entire dataset
● Counts how often words appear together king - man + woman = queen
● Builds a big relationship table This works because embeddings capture
relationships, not just similarity
Analogy:

Word2Vec = Learning from conversations

GloVe = Learning from a dictionary of word relationships
Static vs Contextual Embeddings

Static Embeddings (Word2Vec, GloVe) Contextual Embeddings (BERT Preview)

One word = One meaning = One vector Meaning changes based on sentence.

Example: Example:

“bank” “I sat by the bank of the river”

Same vector for: “I deposited money in the bank”

● River bank BERT gives different vectors for “bank” in each

● Money bank sentence.
Hands-On: Train Your Own Word2Vec Model

Step 1: Install Libraries

pip install gensim nltk matplotlib scikit-learn

Step 2: Sample Data

sentences = [
"i love machine learning",
"i love artificial intelligence",
"machine learning is fun",
"artificial intelligence is
powerful"
]
Hands-On: Train Your Own Word2Vec Model

Step 3: Train Model

from [Link] import Word2Vec

data = [[Link]() for sentence in sentences]

model = Word2Vec(data, vector_size=50,

window=3, min_count=1, sg=1)

Step 4: Find Similar Words

print([Link].most_similar("machine"))
Real-World Systems Using Embeddings

System How It Uses Embeddings

Google Search Finds similar meaning

Amazon Product recommendations

ChatGPT Understands context

Grammarly Meaning-based corrections

Spotify Music similarity

Class Activity

Task:

If “doctor” is close to “nurse”, where should “hospital” appear on the word map?

Summary

Word embeddings:

● Turn language into math

● Store meaning inside numbers
● Power search, chatbots, translation, and AI assistants

Without embeddings:

AI would only read words — not understand them.

Introduction to Sequential Models

Learning Objective
By the end of this session, learners will:

● Understand why word order matters in language

● See the evolution from simple text methods to neural networks

● Recognize the limits of traditional machine learning in language understanding

Warm-Up: Simple Question

“What’s the difference between these two sentences?”

‘Dog bites man’
‘Man bites dog’

Same words — totally different meaning.

This shows:

Order creates meaning

Why Sequence Modeling Matters

Language is not just a list of words.

It’s a flow of ideas over time.

When humans speak or write:

● Each word depends on the previous one

● Meaning builds step by step

Sequential models try to teach AI:

How to remember past words while reading new ones

Real-World Examples

System Why Sequence Matters

Chatbots Understand full conversation

Translation Apps Keep sentence structure

Voice Assistants Follow spoken commands

Text Prediction Guess next word

Evolution of Language Models
Step 1: Bag of Words Step 2: Word Embeddings

Counts words, ignores order. Words become meaningful numbers.

Example: Now:

“I love AI” ● “King” is close to “Queen”

“AI love I”
● “Happy” is close to “Joy”
Both look the same to BoW.

Problem: But:
No understanding of Still doesn’t track sentence flow
meaning or grammar
Evolution of Language Models
Step 3: Neural Networks

Models that:

● Read words one by one

● Remember previous words
● Understand sequences

This is where:

RNNs, LSTMs, and

Transformers come in
Limitations of Traditional ML

Traditional models:

● Look at text as static data

● Don’t remember past words
● Can’t handle long sentences well

Example:

“The trophy doesn’t fit in the suitcase because it is too small.”

What is “it”?
Traditional ML can’t tell.
Sequential models can.
Recurrent Neural Networks (RNNs)

Learning Objective

By the end of this session, learners will:

● Understand what an RNN is and why it is used

● Explain time steps and hidden states
● Know how RNNs process sequences
● Identify limitations of basic RNNs
● Recognize real-world applications
Warm-Up: Memory & Meaning

Question
“If you forget the start of a sentence, can you understand the end?”

RNNs exist because:

Meaning depends on memory.

What Is an RNN?

An RNN is a neural network designed to:

Process sequences step by step and carry information forward

Unlike normal neural networks:

● They don’t treat inputs as independent

● Each step depends on the previous step

Real-World Example

Reading a Message

When you read:

“I ordered a pizza and it arrived cold”

You remember:

● What “it” refers to (pizza)

RNN tries to do this using its hidden state.

RNN Architecture

Time Steps

Each item in a sequence is one time step.

Example:

“AI is powerful”
Steps:

1. AI

2. is

3. powerful
RNN Architecture

Hidden State

The hidden state is:

The model’s memory

At every step:

● It reads the current input

● Combines it with previous memory

● Produces a new memory

Simple Flow
Limitations of Vanilla RNNs

Vanishing Gradient

In long sequences, the model:

Forgets early information

Example:

“The book on the table in the room near the window is

missing.”

By the time it reaches “missing,” it forgets “book.”

Real-World Use Cases

Application Why RNNs Are Used

Chatbots Understand message flow

Translation Keep word order

Speech Recognition Process audio over time

Text Prediction Guess next word

Time-Series Data Analyze trends