UNIT 1: Data Pre-processing and Data Representation

1. Introduction to Datasets and Data Formats

Dataset
A dataset is a collection of related data organized in a structured form for analysis.
Example:
Roll No Name Marks

101 Ram 85

102 Shyam 90

Key Terms:
 Record (Row) → One complete entry
 Attribute (Column) → Feature/variable
 Instance → Single data object

Types of Data
1. Structured Data
 Organized in tables
 Example: Excel, SQL tables
2. Unstructured Data
 No predefined format
 Example: Images, videos, text
3. Semi-Structured Data
 Partially organized
 Example: JSON, XML

Data Formats
1. CSV (Comma-Separated Values)
Definition
CSV (Comma-Separated Values) is a plain text file format used to store tabular
data.
Each row represents a record, and each value is separated by a comma.

Structure
 Rows → Records
  Columns → Attributes
 Separator → Comma (,)

Example
Name Age Marks
Ram 20 85
Shyam 21 90
Explanation:
 First row → Column names (Header)
 Remaining rows → Data values

Characteristics
 Stored as .csv file
 Can be opened in:
 Notepad
 Excel
 Python (Pandas)

Advantages
1. Lightweight
 Very small file size
 No extra formatting
2. Easy to Process
 Supported by almost all programming languages
 Easy to import/export
3. Human Readable
 Can be understood easily by users

Limitations
1. No Hierarchy
 Cannot store nested or complex data
 Only flat structure
2. No Data Type Support
 Everything is stored as text
 Cannot distinguish numbers, dates, etc.
3. Limited Features
 No formulas, charts, or formatting
Use Cases
 Data exchange between systems
 Machine learning datasets
 Simple data storage

2. JSON (JavaScript Object Notation)

Definition
JSON is a semi-structured data format that stores data as key-value pairs.

Structure
 Data is stored in:
 Objects { }
 Arrays [ ]
 Uses keys and values

Example
{
"Name": "Ram",
"Age": 20,
"Marks": 85
}

Nested Example
{
"Name": "Ram",
"Age": 20,
"Marks": {
"Math": 90,
"Science": 85
}
}

Here:
 "Marks" contains another object → nested structure
Characteristics
 Stored as .json
 Used in web applications and APIs
 Language-independent

Advantages
1. Flexible
 Supports complex and nested data
 Can represent real-world relationships
2. Widely Used in APIs
 Used for data exchange between:
 Server ↔ Client
 Web apps ↔ Databases
3. Lightweight & Readable
 Easier than XML
 Human-readable format

Limitations
1. Slightly Complex
 Harder than CSV for beginners
2. Larger Size than CSV
 Due to keys and structure
3. Parsing Required
 Needs a parser in programming languages

Use Cases
 Web APIs
 Configuration files
 Mobile and web applications

3. Excel (XLS / XLSX)

Definition
Excel is a spreadsheet-based file format used to store, analyze, and visualize data.

📌Structure
 Workbook → Entire file
 Worksheet → Individual sheet
  Cells → Intersection of rows and columns

📄 Example (Table Form)

Name Age Marks
Ram 20 85
Shyam 21 90

Characteristics
 File extensions:
 .xls (older)
 .xlsx (modern)
 Developed by Microsoft Excel

✔ Features
1. Multiple Sheets
 One file can contain many worksheets

2. Built-in Formulas
Examples:
 SUM()
 AVERAGE()
 IF()

3. Charts and Graphs

 Bar chart
 Pie chart
 Line graph

4. Data Formatting
 Colors, fonts, borders
 Conditional formatting

5. Data Analysis Tools

 Sorting
 Filtering
 Pivot Tables
Advantages
 Easy to use (GUI-based)
 Powerful for analysis
 Supports visualization

Limitations
1. Larger File Size
 Compared to CSV
2. Not Ideal for Big Data
 Slower with very large datasets
3. Requires Software
 Needs Excel or similar tools

Use Cases
 Business reports
 Data analysis
 Financial calculations
 Student records

Comparison Table
Feature CSV JSON Excel

Format Plain text Semi-structured Spreadsheet

Structure Flat Nested Tabular

Data Types Not supported Supported Supported

File Size Small Medium Large

Ease of Use Easy Moderate Very easy

Use Case Data exchange APIs/Web Analysis/Reports

Final Summary
 CSV → Simple, lightweight, best for basic data storage
 JSON → Flexible, supports nested data, used in APIs
 Excel → Powerful, user-friendly, used for analysis and reporting
Data Cleaning
Definition
Data Cleaning is the process of identifying and correcting errors, missing values,
duplicates, and inconsistencies in a dataset to improve its quality.

Importance
 Improves accuracy of results
 Ensures reliable analysis
 Essential for Machine Learning models
 Prevents wrong conclusions

Handling Missing Values

What are Missing Values?
Missing values occur when no data is stored for a variable in a dataset.

Example
Name Age Marks

Ram 20 85

Shyam — 90

Riya 19 —

Here, Age and Marks are missing.

Causes of Missing Data

 Data entry errors
 Equipment/sensor failure
 Incomplete forms
 Data corruption

Methods to Handle Missing Values

1. Deletion Method
(a) Row Deletion
Remove rows with missing values:
[Link]()
✔ Use when missing data is very small

(b) Column Deletion

Remove columns with too many missing values:
[Link](columns=['Age'])

2. Imputation Method (Filling Values)

(a) Mean Imputation
df['Age'].fillna(df['Age'].mean(), inplace=True)
✔ Best for numerical data

(b) Median Imputation

 Useful when data has outliers

(c) Mode Imputation

df['Gender'].fillna(df['Gender'].mode()[0], inplace=True)

3. Forward/Backward Fill
[Link](method='ffill') # forward fill
[Link](method='bfill') # backward fill

Handling Duplicates
What are Duplicate Values?
Duplicate data means same records appearing more than once.

Example
Name Age Marks

Ram 20 85

Ram 20 85

Problems Caused
 Biased analysis
 Incorrect statistics
  Increased data size

Detection and Removal

Check duplicates:
[Link]()

Remove duplicates:
df.drop_duplicates(inplace=True)

Types of Duplicates
 Full duplicates → Entire row same
 Partial duplicates → Some columns same

Handling Inconsistent Data

What is Inconsistent Data?
Data that is not uniform or standardized.

📌 Examples
Gender

Male

male

All represent same value but written differently.

Causes
 Human entry errors
 Different formats
 Multiple data sources

Methods to Handle Inconsistency

1. Standardization
Convert values into a standard format:
df['Gender'] = df['Gender'].[Link]()
2. Replace Values
df['Gender'].replace({'m': 'male', 'M': 'male'}, inplace=True)

3. Format Correction
Example: Date format
 Convert all dates to one format
df['Date'] = pd.to_datetime(df['Date'])

4. Removing Noise
 Remove unwanted characters
 Fix spelling errors

Summary Table
Problem Type Solution Methods

Missing Values Delete, Mean, Median, Mode

Duplicates Detect & Remove

Inconsistent Data Standardize, Replace, Format

Data Transformation and Normalization

Data Transformation
Definition
Data Transformation is the process of converting raw data into a clean,
structured, and suitable format for analysis, visualization, or machine learning.
It is a broad step in data preprocessing that includes multiple techniques.

Why Data Transformation is Important?

Real-world data is often:
 Unstructured
 Inconsistent
 In different formats
  Not suitable for algorithms
Transformation ensures:
 Data becomes usable
 Algorithms can understand and process it efficiently

Types of Data Transformation

1. Smoothing (Noise Removal)

Concept:
Removes random errors (noise) from data.
Example:
Marks: 45, 47, 100, 46 → 100 is noise
Techniques:
 Binning
 Moving average

2. Aggregation
Concept:
Combining multiple data points into a summary.
Example:
 Daily temperature → Monthly average
 Sales per day → Total monthly sales

3. Generalization
Concept:
Convert low-level data into higher-level concepts.
Example:
Age Category

22 Young

65 Senior

Used in:
 Data mining
 Decision making

4. Encoding Categorical Data

Machine learning models cannot understand text directly, so we convert it into
numbers.

(a) Label Encoding

Category Code

Male 1

Female 0

Simple
Can create false relationships

(b) One-Hot Encoding

Male Female

1 0

0 1

No ranking issue
Increases number of columns

5. Feature Construction
Concept:
Create new attributes from existing data.
Example:
 DOB → Age
 Price + Quantity → Total Cost
Helps improve model performance

6. Discretization
Concept:
Convert continuous data into categories
Example:
Marks Grade

85 A
 Marks Grade

65 B

Normalization

Definition
Normalization is the process of scaling numerical data into a specific range
without changing relationships between values.

Why Normalization is Required?

Problem:
Different features have different scales
Example:
 Age: 18–60
 Salary: 10,000–1,00,000
ML models give more importance to larger values (salary)

Solution:
Normalize data → bring all values to same scale

Types of Normalization

Min-Max Normalization is a technique used in data preprocessing (often in Machine

Learning and Data Science) to rescale values of a feature to a fixed range—usually 0 to
1.

Min-Max Scaling: scales data to a specific range (usually 0-1)

Formula: x' = (x - min) / (max - min)

Example:
Suppose you have exam scores: [40, 60, 80, 100]

1. Find min = 40, max = 100

2. Apply formula:
- x' = (40-40)/(100-40) = 0/60 = 0
- x' = (60-40)/(100-40) = 20/60 = 0.33
- x' = (80-40)/(100-40) = 40/60 = 0.67
 - x' = (100-40)/(100-40) = 60/60 = 1

Scaled scores: [0, 0.33, 0.67, 1]

Z-Score Normalization

Z-Score (Standardization): scales data to have mean=0, std=1

Formula: z = (x - μ) / σ

Example:

Suppose you have exam scores: [40, 60, 80, 100]

1. Calculate mean (μ) = (40+60+80+100)/4 = 70

2. Calculate std (σ) = sqrt(((40-70)^2 + (60-70)^2 + (80-70)^2 + (100-70)^2)/4) ≈ 22.36

3. Apply formula:

- z = (40-70)/22.36 ≈ -1.34

- z = (60-70)/22.36 ≈ -0.45

- z = (80-70)/22.36 ≈ 0.45

- z = (100-70)/22.36 ≈ 1.34

Z-scores: [-1.34, -0.45, 0.45, 1.34]

1. Find mean
2. Find standard deviation
3. Use formula: Z=(X−μ)/σ
4. Compute for each value

🔹 3. Transformation vs Normalization (Conceptual Clarity)

Aspect Transformation Normalization

Meaning Change format or structure Scale numerical values

Scope Broad (many techniques) Specific technique

Includes Encoding, aggregation, smoothing Min-Max, Z-score

 Aspect Transformation Normalization

Goal Make data usable Make data comparable

DATA REPRESENTATION USING TABLES

A table is a systematic arrangement of data in rows and columns, making it easy to

locate and compare values.

Components of a Table

Title – Describes what the table shows

Rows – Horizontal arrangement of data

Columns – Vertical arrangement of data

Headings – Labels for rows/columns

Body – Actual data values

Example Table
Scienc
Student Math Total
e

A 70 80 150

B 85 75 160

C 60 65 125

D 90 85 175

Advantages of Tables

 Present exact numerical values

 Easy to organize large data
 Helps in calculations and analysis
 Useful for reference
Limitations of Tables

 Difficult to identify patterns quickly

 Not visually attractive
 Time-consuming for large datasets

DATA REPRESENTATION USING CHARTS

Definition
Charts are graphical representations of data that help in quick understanding and
analysis.

Importance of Charts
 Simplifies complex data
 Shows trends and relationships
 Makes comparison easy
 Improves decision-making

TYPES OF CHARTS

1. Bar Chart
Description
Uses rectangular bars (vertical or horizontal) to represent data.

Example Use
Comparing marks of students.
Features
 Equal width bars
 Space between bars
 Height represents value
2. Line Chart
Description
Data points connected by straight lines.

Example Use
Temperature changes over time.
Features
 Shows trends
 Useful for continuous data

3. Pie Chart
Description
Circular chart divided into sectors.

Example Use
Budget distribution.
Features
 Total = 360°
 Each slice represents percentage

4. Histogram
Description
Graph showing frequency distribution using bars.
Example Use
Marks distribution in a class.
Features
 No gaps between bars
 Used for continuous data

5. Scatter Plot
Description
Uses points to show relationship between two variables.

Example Use
Height vs Weight.
Features
 Shows correlation
 Helps in prediction

TABLE VS CHART
Feature Table Chart
Nature Numerical Visual
Data accuracy Exact Approximate
Trend visibility Low High
Ease of
Moderate Easy
understanding
Feature Table Chart
Detailed
Best use Summary & patterns
data

WHEN TO USE
Use Tables When:
 Exact values are required
 Data is large and detailed
 Performing calculations
Use Charts When:
 Showing trends or patterns
 Comparing data visually
 Presenting to audience
Data preprocessing
Data preprocessing is a crucial step in data analysis and machine learning. Using
NumPy and Pandas (two popular Python libraries), you can efficiently clean,
transform, and prepare data for modeling.

It includes:

 Cleaning data
 Handling missing values
 Removing duplicates
 Scaling/normalizing data
 Transforming data

Libraries Used

 Pandas → Works with tables (DataFrames)

 NumPy → Performs numerical operations

import pandas as pd
import numpy as np

 pd→ shortcut for Pandas

 np→ shortcut for NumPy
 These libraries provide tools for data handling and calculations

What is Pandas?
Pandas is a powerful, fast, and open-source library built on NumPy. It is used for data
manipulation and real-world data analysis in Python.
Loading Data in Pandas DataFrame
Reading CSV file using pd.read_csv and loading data into a data frame. Import
pandas as using pd for the shorthand. You can download the data from here.
#Importing pandas library
importpandasaspd

#Loading data into a DataFrame

data_frame=pd.read_csv('Mall_Customers.csv')

Printing rows of the Data

By default, data_frame.head() displays the first five rows
and data_frame.tail() displays last five rows.
If we want to get first 'n' number of rows then we use, data_frame.head(n) similar
is the syntax to print the last n rows of the data frame.
#displaying first five rows
display(data_frame.head())

#displaying last five rows

display(data_frame.tail())

Load Dataset
df=pd.read_csv("[Link]")

 Reads a CSV file

 Stores it in a DataFrame (df) (like a table with rows & columns)

Handle Missing Values

[Link]([Link](), inplace=True)

 fillna() replaces missing values (NaN)

 [Link]() → calculates mean of each numeric column
 Missing values are replaced with column averages
 inplace=True → modifies original data directly

Example:
If age column = [20, 25, NaN] → mean = 22.5
Result → [20, 25, 22.5]

Encode Categorical Data

df['gender'] =df['gender'].map({'Male': 0, 'Female': 1})

 Converts text into numbers

 Machine learning models need numbers, not text

Mapping:

 Male → 0
 Female → 1
Example:
["Male", "Female", "Male"] → [0, 1, 0]

🔹 1. Import and Create Dataset

import pandas as pd

data = {
"Name": ["A", "B", "C", "D", "E"],
"Age": [20, 25, None, 30, 35],
"Salary": [20000, 25000, 30000, None, 50000]
}

df = [Link](data)
print(df)

👉 Creates a DataFrame with missing values

Handling missing values

import pandas as pd

# Step 1: Create DataFrame

data = {

"Age": [20, 25, None, 30, 35],

"Salary": [20000, 25000, 30000, None, 50000]

df = [Link](data)

# Step 2: Fill missing values with mean

df["Age"].fillna(df["Age"].mean(), inplace=True)

df["Salary"].fillna(df["Salary"].mean(), inplace=True)

print(df)

Remove missing values

import pandas as pd

data = {

"Name": ["A", "B", "C", "D", "E"],

"Age": [20, 25, None, 30, 35],

"Salary": [20000, 25000, 30000, None, 50000]

df = [Link](data)
df = [Link]()

print(df)

Removing Duplicates

import pandas as pd

data = {

"Name": ["A", "B", "C", "D", "E"],

"Age": [20, 25, 25, 30, 35],

"Salary": [20000, 25000, 30000, 2000, 50000]

df = [Link](data)

# Remove duplicate rows

df = df.drop_duplicates()

print(df)

Data Filtering

import pandas as pd

data = {

"Name":["A", "B", "C", "D", "E"],

"Age": [20, 25, 25, 30, 35],

"Salary": [20000, 25000, 30000, 2000, 50000]

df = [Link](data)

filtered = df[df["Salary"] > 30000]

print(filtered)

Data Normalization

import pandas as pd

# Step 1: Create dataset

data = {

"Name": ["A", "B", "C", "D", "E"],

"Age": [20, 25, 25, 30, 35],

"Salary": [20000, 25000, 30000, 2000, 50000]

}

df = [Link](data)

# Step 2: Apply normalization (0 to 1 scaling)

df["Salary_norm"] = (df["Salary"] - df["Salary"].min()) / (df["Salary"].max() - df["Salary"].min())

print(df)

import pandas as pd

# Step 1: Create dataset

data = {

"Name": ["A", "B", "C", "D", "E"],

"Age": [20, 25, 25, 30, 35],

"Salary": [20000, 25000, 30000, 2000, 50000]

df = [Link](data)

df["Salary_std"] = (df["Salary"] - df["Salary"].mean()) / df["Salary"].std()

print(df)

Common Functions:
import pandas as pd

df = pd.read_csv("[Link]")

[Link]() # first 5 rows

[Link]() # last 5 rows
[Link]() # structure
[Link]() # statistics
[Link]() # check missing
[Link]() # remove missing
[Link](0) # replace missing

What is NumPy?

NumPy (Numerical Python) is a powerful Python library used for:

 Fast numerical computations

 Working with arrays and matrices
 Performing data preprocessing operations efficiently

It is widely used in machine learning, data science, and data preprocessing.

. Create Dataset

import numpy as np

data = [Link]([10, 20, 30, 40, 50])

print(data)

👉 Creates a NumPy array for processing

OUTPUT:

[10 20 30 40 50]

🔹 . Handling Missing Values

import numpy as np

data = [Link]([10, 20, [Link], 40, 50])

mean = [Link](data)
data = [Link]([Link](data), mean, data)

print(data)

👉 Replaces missing values (NaN) with mean

[10. 20. 30. 40. 50.]

🔹 . Normalization (Min-Max Scaling)

import numpy as np

data = [Link]([10, 20, 30, 40, 50])

norm = (data - [Link](data)) / ([Link](data) - [Link](data))

print(norm)

👉 Scales values between 0 and 1

[0. 0.25 0.5 0.75 1. ]

🔹 . Standardization (Z-score)

import numpy as np

data = [Link]([10, 20, 30, 40, 50])

std = (data - [Link](data)) / [Link](data)

print(std)

👉 Converts data to mean = 0, std = 1

[-1.41421356 -0.70710678 0. 0.70710678 1.41421356]

🔹 Removing Outliers

import numpy as np

data = [Link]([10, 20, 30, 1000, 40, 50])

filtered = data[data < 100]

print(filtered)

👉 Removes extreme values

9. Data Filtering

import numpy as np

data = [Link]([10, 15, 20, 25, 30])

filtered = data[data > 20]

print(filtered)

👉 Selects values based on condition

[25 30]

🔹 . Sorting Data

import numpy as np

data = [Link]([50, 20, 40, 10, 30])

sorted_data = [Link](data)
print(sorted_data)

[10 20 30 40 50]

Data Visualization using Matplotlib

What is Matplotlib?
Matplotlib is a Python library used for creating graphs and plots to visualize data.

Purpose

 Convert data into visual form

 Easy understanding of patterns and trends
 Useful in data analysis and machine learning
 Types of Plots

1. Line Plot
A line plot connects data points using straight lines.

Use Case

 Showing trends over time

[Link]

x= [1, 2, 3]
y= [4, 5, 6]

[Link](x, y)
[Link]("Line Graph")
[Link]("X-axis")
[Link]("Y-axis")
[Link]()

2. Bar Chart

A bar chart represents data using rectangular bars.

Use Case

 Comparing values between categories

[Link]

categories= ['A', 'B']

values= [10, 20]

[Link](categories, values)
[Link]("Bar Chart")
[Link]()

[Link] Chart

A pie chart represents data in a circular form divided into slices.

Use Case

 Showing proportions

[Link]

data= [40, 60]

labels= ['A', 'B']

[Link](data, labels=labels)
[Link]("Pie Chart")
[Link]()
4. Histogram

A histogram shows frequency distribution of data.

Use Case

 Understanding data distribution

[Link]

data= [1, 2, 2, 3, 3, 3]

[Link](data)
[Link]("Histogram")
[Link]()

5. Scatter Plot

A scatter plot shows relationship between two variables using points.

Use Case

 Finding correlation

[Link]

x= [1, 2, 3, 4]
y= [10, 20, 25, 30]

[Link](x, y)
[Link]("Scatter Plot")
[Link]()

Introduction

Language is a method of communication with the help of which we can speak, read and write. For example, we think,
we make decisions, plans and more in natural language; precisely, in words. However, the big question that confronts
us in this AI era is that can we communicate in a similar manner with computers. In other words, can human beings
communicate with computers in their natural language? It is a challenge for us to develop NLP applications because
computers need structured data, but human speech is unstructured and often ambiguous in nature.

Natural Language Processing (NLP)

Natural Language Processing (NLP) is the sub-field of Computer Science especially Artificial Intelligence (AI) that is
concerned about enabling computers to understand and process human language. Technically, the main task of NLP
would be to program computers for analyzing and processing huge amount of natural language datalike text and
speech—and extract meaningful information from it.
In other words ,we can say that Natural Language Processing (NLP) helps computers understand, interpret and
produce human language. It studies language as data and develops a model that can analyse linguistic structure,
meaning and context in both written and spoken communication.

Examples

 Chatbots (like virtual assistantsSiri,Alexa)

 Language translation tools (google translator ,itranslator,Microsoft translator)

Key Components of NLP

1. Text Processing

Breaking down text into smaller parts:

 Tokenization (splitting sentences into words)

 Removing stop words (e.g., “is”, “the”)

 Stemming (running → run)& Lemmatization (better → good)

(reducing words to base form)

2. Syntax Analysis

Understanding grammatical structure:

 Parsing sentences

 Identifying parts of speech (noun, verb, etc.)

3. Semantic Analysis

Understanding meaning:

 Word sense disambiguation (e.g., “bank” as riverbank vs. financial institution)

 Named Entity Recognition (finding names, places, dates)

4. Pragmatics

Understanding context beyond literal meaning:

 Sarcasm detection

 Intent recognition

Applications

 Voice Assistants: Alexa, Siri and Google Assistant use NLP for voice recognition and interaction.

 Grammar and Text Analysis: Tools like Grammarly, Microsoft Word and Google Docs apply NLP for grammar
checking.

 Information Extraction: Search engines like Google and DuckDuckGo use NLP to extract relevant information.

 Chatbots: Website bots and customer support chatbots leverage NLP for automated conversations.
Real-World Applications

 Virtual assistants (Siri, Alexa)

 Search engines (Google)

 Email filtering

 Healthcare (clinical text analysis)

 Finance (market sentiment)

 Social media monitoring

Challenges in NLP

1. Ambiguity

 Same word, multiple meanings

2. Context Understanding

 Requires world knowledge

3. Sarcasm & Irony

 Hard for machines to detect

4. Multilingual Complexity

 Different grammar & scripts

5. Data Issues

 Noisy, unstructured, biased data

Advantages of NLP

✔ Automates language tasks

✔ Processes large-scale text data
✔ Improves human-computer interaction

Text Preprocessing Techniques in NLP

Text preprocessing is the first and most crucial step in any Natural Language Processing pipeline. It transforms raw,
messy text into a clean and structured format that machines can understand.

Why Text Preprocessing is Important

Raw text often contains:

 Noise (punctuation, HTML tags, emojis)

 Inconsistent formats (uppercase/lowercase)

 Irrelevant words (stopwords)

Preprocessing improves:

 Model accuracy
  Training speed

 Overall performance

Common Text Preprocessing Techniques

1. Lowercasing

Convert all text to lowercase to ensure uniformity.

Example:

 "NLP is Amazing" → "nlp is amazing"

✔ Helps avoid treating “NLP” and “nlp” as different words

2. Tokenization

Splitting text into smaller units (tokens).

Types:

 Sentence Tokenization

 Word Tokenization

Example:

 "I love NLP" → ["I", "love", "NLP"]

3. Stopword Removal

Removing common words that don’t add much meaning.

Examples of stopwords:

 is, the, and, in, of

Example:

 "I am learning NLP" → "learning NLP"

4. Stemming

Reducing words to their root form by chopping suffixes.

Example:

 playing → play

 studies → study

✔ Fast but sometimes inaccurate

5. Lemmatization

Reducing words to their base (dictionary) form using grammar rules.

Example:

 better → good

 running → run

✔ More accurate than stemming

6. Removing Punctuation

Eliminating punctuation marks.

Example:

 "Hello, world!" → "Hello world"

7. Removing Numbers

Removing digits if they are not useful.

Example:

 "Order 123 shipped" → "Order shipped"

8. Removing Special Characters

Cleaning symbols like:

 @, #, $, %, &, *

9. Removing HTML Tags

Useful when working with web data.

Example:

 "<p>Hello</p>" → "Hello"

10. Handling Emojis & Emoticons

 Remove or convert emojis into meaning

Example:

 😊 → "happy"

11. Normalization

Standardizing text:

 Expanding contractions:

o "don't" → "do not"

 Correcting spelling
12. Part-of-Speech (POS) Filtering

Keep only useful words like:

 Nouns

 Verbs

Removes less meaningful ones.

13. Named Entity Recognition (NER) (Optional Preprocessing)

Identifying entities like:

 Names

 Places

 Dates

Example:

 “Virat Kohli plays cricket in India”

Challenges in Preprocessing

 Losing important meaning (over-cleaning)

 Handling multilingual text

 Slang & informal language

 Context-dependent words

Tokenization in NLP

Tokenization is one of the most fundamental steps in Natural Language Processing. It involves breaking down text
into smaller units called tokens, which can be words, sentences, or subwords.

What is Tokenization?

Tokenization is the process of splitting a text into meaningful pieces so that machines can process it.

Example:

 Input:
"I love learning NLP!"

 Output:
["I", "love", "learning", "NLP", "!"]

Each piece is called a token.

Types of Tokenization

1. Sentence Tokenization

Splits text into sentences.

Example:

 Input:
"NLP is amazing. It is powerful."

 Output:
["NLP is amazing.", "It is powerful."]

Used in:

 Text summarization

 Document analysis

2. Word Tokenization

Splits sentences into words.

Example:

 "I love NLP" → ["I", "love", "NLP"]

Most commonly used method

3. Subword Tokenization

Breaks words into smaller meaningful units.

Example:

 "unhappiness" → ["un", "happi", "ness"]

Useful for:

 Rare words

 Handling unknown vocabulary

4. Character Tokenization

Splits text into individual characters.

Example:

 "NLP" → ["N", "L", "P"]

Useful for:

 Spelling correction

 Low-resource languages

5. N-gram Tokenization

Creates sequences of tokens.

Types:

 Unigram: one word → ["I", "love", "NLP"]

  Bigram: two words → ["I love", "love NLP"]

 Trigram: three words → ["I love NLP"]

Used in:

 Language modeling

 Predictive text

Challenges in Tokenization

1. Ambiguity

 "Let's eat, grandma" vs "Let's eat grandma"

2. Contractions

 "don't" → ["do", "n't"]

3. Languages Without Spaces

 Chinese, Japanese require special methods

4. Emojis & Special Characters

 Handling 😊, hashtags, mentions

5. Compound Words

 "state-of-the-art" → multiple tokens?

Importance of Tokenization

✔ Converts raw text into processable units

✔ Affects model performance directly
✔ Required for all NLP tasks

Stop Word Removal in NLP

Stop word removal is a key step in text preprocessing within Natural Language Processing. It involves removing
commonly used words that carry little or no meaningful information for analysis.

What are Stop Words?

Stop words are frequently occurring words in a language that:

 Do not contribute much to the meaning

 Are mainly used for grammar and sentence structure

Examples:

 English stop words:

 o is, am, are, the, a, an, in, on, at, of, for, to

Why Remove Stop Words?

Removing stop words helps:

✔ Improve Model Performance

 Reduces noise in the data

✔ Reduce Data Size

 Fewer words → faster processing

✔ Focus on Important Words

 Keeps meaningful terms

Example:

Original sentence:

"I am learning Natural Language Processing"

After stop word removal:

"learning Natural Language Processing"

Types of Stop Words

1. Standard Stop Words

Common function words:

 the, is, at, which

2. Domain-Specific Stop Words

Words that are common in a specific field but not useful.

Example:

 In news articles: “said”, “report”

3. Contextual Stop Words

Words that become irrelevant depending on context.

Step-by-Step Example

Sentence:

"This is a simple example to understand stop word removal"

Step 1: Tokenization
→ ["This", "is", "a", "simple", "example", "to", "understand", "stop", "word", "removal"]

Step 2: Remove Stop Words

→ ["simple", "example", "understand", "stop", "word", "removal"]

Advantages of Stop Word Removal

✔ Reduces dimensionality
✔ Speeds up training
✔ Improves efficiency

Disadvantages / Limitations

Loss of Meaning

Removing stop words can sometimes change meaning.

Example:

 "I do not like this"

 After removal → "like this" (wrong meaning)

Stemming and Lemmatization in NLP

Stemming and Lemmatization are core text normalization techniques used in Natural Language Processing to reduce
words to their base or root form. This helps machines treat similar words as the same, improving efficiency and
accuracy.

Why Do We Need Them?

In text data, the same word can appear in many forms:

 play, playing, played, plays

👉 Without normalization, a model treats them as different words

👉 With normalization, they map to a common base form

What is Stemming?

Stemming is a simple technique that removes suffixes (and sometimes prefixes) from words to get the root form.

🔧 How it Works:

 Uses rule-based cutting

 Does not consider grammar or meaning

Examples:

Word Stem

playing play
Word Stem

studies studi

happiness happi

✔ Characteristics:

 Fast

 Simple

 May produce incorrect or non-dictionary words

What is Lemmatization?

Lemmatization reduces words to their base or dictionary form (lemma) using vocabulary and grammar rules.

How it Works:

 Uses linguistic knowledge

 Considers:

o Part of speech (POS)

o Context

Examples:

Word Lemma

running run

better good

studies study

✔ Characteristics:

 More accurate

 Produces valid words

 Slower than stemming

Key Differences Between Stemming and Lemmatization

Feature Stemming Lemmatization

Approach Rule-based cutting Dictionary + grammar

Output May be invalid Always valid word

Speed Fast Slower

Feature Stemming Lemmatization

Accuracy Less accurate More accurate

Example studies → studi studies → study

Step-by-Step Example

Sentence:

"The children are playing and running faster"

After Stemming:

→ ["the", "children", "are", "play", "and", "run", "faster"] (approximate)

After Lemmatization:

→ ["the", "child", "be", "play", "and", "run", "fast"]

✔ Notice:

 “children” → “child”

 “are” → “be”

 “faster” → “fast”

When to Use What?

Use Stemming When:

 Speed is important

 Working with large datasets

 Slight inaccuracies are acceptable

Use Lemmatization When:

 Accuracy is important

 Meaning matters

Challenges

1. Over-Stemming

 Cuts too much

 Example: “university” → “univers”

2. Under-Stemming
  Doesn’t reduce enough

Word Frequency & Basic Text Analysis

Word frequency and basic text analysis are foundational techniques in Natural Language Processing used to
understand patterns, importance, and structure in text data.

Word Frequency

Word frequency refers to how often each word appears in a text or corpus.

Word frequency refers to the number of occurrences of each word in a text.

Example:
Text: “India is a great country. India has diversity.”

Word Frequency

India 2

is 1

a 1

great 1

country 1

has 1

diversity 1

2. Types of Word Frequency

1. Absolute Frequency

 Actual count of a word in text

Example: “India” appears 2 times → Absolute frequency = 2

2. Relative Frequency

 Frequency divided by total number of words

Formula:
Relative Frequency = (Word Count / Total Words)

3. Normalized Frequency

 Used to compare texts of different lengths

Example: per 1000 words or per 10,000 words

It helps identify:

 Important words
  Common themes

 Keywords in documents

Techniques for Word Frequency

1. Bag of Words (BoW)

 Represents text as word counts

 Ignores grammar and word order

Example:

 "I love NLP" → {I:1, love:1, NLP:1}

2. Term Frequency (TF)

Measures how frequently a word appears in a document.

Term Frequency (TF) refers to how often a particular word (term) appears in a document compared to
the total number of words in that document.

In simple words:
It tells you which words are important in a text based on how frequently they occur.

Formula
TF(t)=Number of times term t appears/Total number of terms in the document

Example
Text:
"Education is important. Education builds society."

Step 1: Count total words

Total words = 6

Step 2: Count term frequency

 Education = 2

Step 3: Apply formula

TF=2/6=0.33
So, TF of "Education" = 0.33

3. TF-IDF (Term Frequency–Inverse Document Frequency)

Balances importance of words across multiple documents.

Finds important words by reducing common words

 High score → important word

 Low score → common word

Words like “the”, “is” get low weight

✔️Widely used in:

 Search engines
 AI & NLP

Basic Text Analysis

Meaning
Basic text analysis is the process of examining and understanding text data to extract useful information.

Main Steps in Text Analysis

1. Text Collection

 Gather data (books, articles, speeches)

2. Cleaning the Text

 Convert to lowercase
 Remove punctuation (.,!?)
 Remove extra spaces

3. Tokenization

 Breaking text into words

Example:
“India is diverse” → [India, is, diverse]
4. Stop Word Removal

 Remove common words like is, the, and

5. Word Frequency Calculation

 Count each word (as discussed above)

6. Interpretation

 Find:
o Most frequent words
o Themes
o Patterns

Example (Full Process)

Text:
"India is a diverse country. India has rich culture."

Step 1: Clean Text

→ india is a diverse country india has rich culture

Step 2: Tokenize

→ [india, is, a, diverse, country, india, has, rich, culture]

Step 3: Remove Stop Words

→ [india, diverse, country, india, rich, culture]

Step 4: Frequency Table

Word Frequency

india 2

diverse 1

country 1

rich 1

culture 1
 Applications
 Research & surveys
 News analysis
 Social media trends
 Artificial Intelligence (NLP)
 Academic assignments

Key Terms
 Corpus → collection of texts
 Token → individual word
 Stop Words → common words removed
 Frequency Distribution → table of word counts

Basic Text Analysis Techniques

1. Frequency Distribution

 Shows how often words occur

 Can be visualized using charts

2. Word Cloud

 Visual representation of word frequency

 Bigger words = higher frequency

3. Keyword Extraction

 Identifies important words from text

4. N-gram Analysis

Analyzes sequences of words.

Example:

 Text: "I love NLP"

Type Output

Unigram I, love, NLP

Bigram I love, love NLP

Type Output

Trigram I love NLP

5. Sentiment Analysis (Basic Level)

 Determines emotion:

o Positive

o Negative

o Neutral

6. Text Length Analysis

 Number of:

o Words

o Sentences

o Characters

NLTK (Natural Language Toolkit)

What is NLTK?
NLTK (Natural Language Toolkit) is a popular Python library used for working with human language
data (text).

👉 It helps computers read, understand, and analyze text.

Developed By
 Developed by researchers at the University of Pennsylvania
 Widely used in education, research, and AI projects

Why NLTK is Used?

NLTK is used for Natural Language Processing (NLP) tasks such as:

 Tokenization (splitting text into words)

 Removing stop words
  Stemming and Lemmatization
 Word frequency analysis
 Text classification

Practical Implementation Using NLTK

1. Install NLTK

First install NLTK library.

pip install nltk

2. Import NLTK

import nltk

Download required datasets once:

import nltk
[Link]('punkt')
[Link]('stopwords')
[Link]('wordnet')
[Link]('averaged_perceptron_tagger')
[Link]('maxent_ne_chunker')
[Link]('words')

Tokenization

Tokenization means splitting text into words or sentences.

Word Tokenization

import nltk
from [Link] import word_tokenize

text = "I love Natural Language Processing."

words = word_tokenize(text)

print(words)

Output

['I', 'love', 'Natural', 'Language', 'Processing', '.']

1. Import NLTK

import nltk

Explanation

 Imports the NLTK library.

  NLTK stands for Natural Language Toolkit.

 It provides tools for NLP tasks.

2. Import word_tokenize

from [Link] import word_tokenize

Explanation

 Imports the word_tokenize() function.

 This function splits text into words and punctuation.

3. Store Text

text = "I love Natural Language Processing."

Explanation

 A sentence is stored in variable text.

4. Tokenize Text

words = word_tokenize(text)

Explanation

word_tokenize() breaks the sentence into tokens.

Input:

"I love Natural Language Processing."

Output:

['I', 'love', 'Natural', 'Language', 'Processing', '.']

Notice:

 Each word becomes a separate token

 Period . is also treated as a token

5. Print Result

print(words)

Explanation

Displays tokenized words.

Output

['I', 'love', 'Natural', 'Language', 'Processing', '.']

Stemming

Stemming reduces words to root form.

from [Link] import PorterStemmer

stemmer = PorterStemmer()

words = ["playing", "plays", "played"]

for word in words:

print([Link](word))

Output

play
play
play

1. Import PorterStemmer

from [Link] import PorterStemmer

Explanation

 Imports the PorterStemmer class from NLTK.

 Porter Stemmer is one of the most popular stemming algorithms.

2. Create Stemmer Object

stemmer = PorterStemmer()

Explanation

Creates a stemming object.

This object will perform stemming operations.

3. Create List of Words

words = ["playing", "plays", "played"]

Explanation

A list named words stores different forms of the word play.

4. Loop Through Words

for word in words:

Explanation
Loops through each word one by one.

Iteration process:

1. playing

2. plays

3. played

5. Apply Stemming

print([Link](word))

Explanation

stem() function reduces the word to its root form.

Lemmatization

Lemmatization converts words into meaningful root words.

from [Link] import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

words = ["running", "better", "cars"]

for word in words:

print([Link](word))

Output

running
better
car

1. Import WordNetLemmatizer

from [Link] import WordNetLemmatizer

Explanation

Imports the WordNetLemmatizer class from NLTK.

This class is used to perform lemmatization.

2. Create Lemmatizer Object

lemmatizer = WordNetLemmatizer()

Explanation

Creates a lemmatizer object.

This object will convert words into root forms.

3. Create Word List

words = ["running", "better", "cars"]

Explanation

Stores words in a list.

4. Loop Through Words

for word in words:

Explanation

Loops through each word one by one.

Iteration order:

1. running

2. better

3. cars

5. Apply Lemmatization

print([Link](word))

Explanation

lemmatize() converts the word into its root form.

Step-by-Step Working

First Iteration

[Link]("running")

Output

running

Part of Speech (POS) Tagging

Identifies grammar type of each word.

from [Link] import word_tokenize

from nltk import pos_tag

text = "Python is a powerful language"

words = word_tokenize(text)

tags = pos_tag(words)

print(tags)

Output

[('Python', 'NNP'),
('is', 'VBZ'),
('a', 'DT'),
('powerful', 'JJ'),
('language', 'NN')]

1. Import word_tokenize

from [Link] import word_tokenize

Explanation

Imports the word_tokenize() function.

Used to split sentence into words.

2. Import pos_tag

from nltk import pos_tag

Explanation

Imports pos_tag() function.

This function assigns grammar tags to words.

3. Store Sentence

text = "Python is a powerful language"

Explanation

Stores sentence inside variable text.

4. Tokenize Sentence

words = word_tokenize(text)

Explanation

Splits sentence into words.

Output

['Python', 'is', 'a', 'powerful', 'language']

5. Apply POS Tagging

tags = pos_tag(words)

Explanation

pos_tag() identifies grammar type of each word.

6. Print Result

print(tags)

Explanation

Displays words with grammar tags.

Output

[
('Python', 'NNP'),
('is', 'VBZ'),
('a', 'DT'),
('powerful', 'JJ'),
('language', 'NN')
]

Understanding Each Tag

Word Tag Meaning

Python NNP Proper Noun

is VBZ Verb

a DT Determiner

powerful JJ Adjective

language NN Noun

Named Entity Recognition (NER)

Finds names of people, places, organizations.

import nltk
from [Link] import word_tokenize
from nltk import pos_tag, ne_chunk

text = "Sachin Tendulkar lives in India"

words = word_tokenize(text)

tags = pos_tag(words)
entities = ne_chunk(tags)

print(entities)

Output

(S
(PERSON Sachin/NNP Tendulkar/NNP)
lives/VBZ
in/IN
(GPE India/NNP))

1. Import NLTK

import nltk

Explanation

Imports the NLTK library.

2. Import word_tokenize

from [Link] import word_tokenize

Explanation

Imports function used for word tokenization.

It splits sentence into words.

3. Import POS Tagging and NER

from nltk import pos_tag, ne_chunk

Explanation

 pos_tag() → identifies grammar type

 ne_chunk() → identifies named entities

4. Store Sentence

text = "Sachin Tendulkar lives in India"

Explanation

Stores sentence inside variable text.

5. Tokenization

words = word_tokenize(text)

Explanation

Splits sentence into tokens.

Output

['Sachin', 'Tendulkar', 'lives', 'in', 'India']

6. POS Tagging

tags = pos_tag(words)

Explanation

Assigns grammar tags to words.

Output

[
('Sachin', 'NNP'),
('Tendulkar', 'NNP'),
('lives', 'VBZ'),
('in', 'IN'),
('India', 'NNP')
]

Understanding POS Tags

Tag Meaning

NNP Proper Noun

VBZ Verb

IN Preposition

GPE Country/city/location

Text representation techniques are methods used in Natural Language Processing to convert text into numerical
form so machines can understand and process it

• Bag of Words (BoW)

It is one of the simplest and most widely used text representation techniques in Natural Language Processing. It
converts text into a numerical vector based on word occurrence, ignoring grammar and word order.

 Represents text as a collection of word counts.

 Ignores grammar and word order.

 Example:
“I love AI” → {I:1, love:1, AI:1}

BoW treats a document as a “bag” of words, meaning:

 The order of words doesn’t matter

  Only the frequency of words matters

Example:
Sentence: “I love AI and I love coding”
BoW representation counts each word:

Word Count

I 2

love 2

AI 1

and 1

coding 1

Steps to Create BoW

Step 1: Text Collection

Gather all documents (corpus).

Example:

 Doc1: “I love AI”

 Doc2: “AI is powerful”

Step 2: Text Preprocessing

 Lowercasing

 Removing punctuation

 Removing stopwords (optional)

 Tokenization

Step 3: Build Vocabulary

Create a list of unique words from all documents.

Vocabulary:
[I, love, AI, is, powerful]

Step 4: Vector Representation

Count occurrences of each word in every document.

Document I love AI is powerful

Doc1 1 1 1 0 0

Doc2 0 0 1 1 1
Each row is a feature vector.

Types of BoW Representations

• Binary BoW

 Only indicates presence (1) or absence (0)

Example:
“AI is AI” → [AI:1, is:1]

• Count-Based BoW

 Uses actual frequency (most common)

Instead of just marking whether a word exists (like binary BoW), count-based BoW records the exact frequency of
each word.

In simple terms:
“How many times does each word occur?”

Step-by-Step Example

Given Documents:

 Doc1: “I love AI and I love AI”

 Doc2: “AI is powerful”

Step 1: Build Vocabulary

List all unique words:

[I, love, AI, and, is, powerful]

Step 2: Count Word Frequencies

Document I love AI and is powerful

Doc1 22 2 1 0 0

Doc2 00 1 0 1 1

Each number = count of that word in the document

Normalized BoW

Normalized Bag of Words (BoW)

Normalized BoW is an improved version of the Bag of Words model in Natural Language Processing where word
counts are scaled (normalized) instead of using raw frequencies.
🔸 1. Why Normalization is Needed

In count-based BoW, longer documents naturally have higher word counts, which can bias the model.

Example:

 Doc1: 100 words

 Doc2: 10 words

Even if both talk about the same topic, Doc1 will have much larger counts.

Solution: Normalize counts so documents become comparable.

Instead of using raw counts:

Normalized value=Count of word/Total words in document

This converts counts into proportions (relative frequency).

3. Example

Documents:

 Doc1: “I love AI and I love AI”

 Doc2: “AI is powerful”

Step 1: Vocabulary

[I, love, AI, and, is, powerful]

Step 2: Count-Based BoW

Document I love AI and is powerful Total Words

Doc1 22 2 1 0 0 7

Doc2 00 1 0 1 1 3

Step 3: Normalize

Divide each value by total words:

Document I love AI and is powerful

Doc1 2/7 2/7 2/7 1/7 0 0

Doc2 0 0 1/3 0 1/3 1/3

4. Key Characteristics
✔ Values range between 0 and 1
✔ Represents relative importance of words
✔ Reduces bias due to document length

5. Advantages

✔ Fair comparison between documents of different lengths

✔ Prevents large documents from dominating
✔ Simple improvement over count-based BoW

6. Disadvantages

Still ignores:

 Word order

 Context/meaning

TF-IDF (Term Frequency – Inverse Document Frequency)

TF-IDF is a widely used text representation technique in Natural Language Processing that improves upon Bag of
Words by considering not just how often a word appears, but also how important it is across all documents.

A word is important if:

 It appears frequently in a document (high TF)

 But rarely in other documents (high IDF)

✔ Common words like “the”, “is” → low importance

✔ Unique words → high importance

2. Components of TF-IDF

Term Frequency (TF)

Measures how often a word appears in a document:

TF=Number of times word appears in document/Total words in document

Inverse Document Frequency (IDF)

Measures how unique or rare a word is across all documents:

IDF=log(Total number of documents/Number of documents containing the word)

If a word appears in many documents → IDF is low

If rare → IDF is high

TF-IDF Formula

TF-IDF=TF×IDF
 Step-by-Step Example

Documents:

 Doc1: “I love AI”

 Doc2: “AI is powerful”

Step 1: Vocabulary

[I, love, AI, is, powerful]

Step 2: Compute TF

Word TF (Doc1) TF (Doc2)

I 1/3 0

love 1/3 0

AI 1/3 1/3

is 0 1/3

powerful 0 1/3

Step 3: Compute IDF

Total documents = 2

Word Document Count IDF

I 1 log(2/1)

love 1 log(2/1)

AI 2 log(2/2) = 0

is 1 log(2/1)

powerful 1 log(2/1)

Step 4: TF-IDF Values

 Words like “AI” → IDF = 0 → TF-IDF = 0

 Rare words → higher TF-IDF

One-Hot Vector Encoding

One-Hot Encoding is a simple way to represent words as vectors in Natural Language Processing. Each word is
converted into a vector where only one position is “1” and all others are “0.”
1. Core Idea

Each word in the vocabulary gets a unique index

The vector length = size of vocabulary
Only one element is 1, rest are 0

2. Example

Vocabulary:

[I, love, AI, coding]

Assign index:

 I→0

 love → 1

 AI → 2

 coding → 3

One-Hot Representation:

Word Vector

I [1, 0, 0, 0]

love [0, 1, 0, 0]

AI [0, 0, 1, 0]

coding [0, 0, 0, 1]

3. Sentence Representation

Sentence: “I love AI”

→ Represent each word separately:

 I → [1,0,0,0]

 love → [0,1,0,0]

 AI → [0,0,1,0]

6. Advantages

✔ Simple and easy to understand

✔ No training required
✔ Ensures unique representation for each word
✔ Useful for:

 Basic ML models

 Small datasets
7. Disadvantages

No Semantic Meaning

 “king” ≠ “queen” (no relationship captured)

Curse of Dimensionality

 Large vocabulary → huge vectors

No Context Awareness

 Same word always has same vector

Memory Inefficient

 Too many zeros

Character Embeddings
Character embeddings are a text representation technique in Natural Language Processing where individual
characters (letters, digits, symbols) are converted into numerical vectors instead of whole words.

1. Core Idea

Instead of treating text as words:

Word-level:
“I love AI” → [“I”, “love”, “AI”]

Character-level:
“I love AI” → [‘I’, ‘ ’, ‘l’, ‘o’, ‘v’, ‘e’, ‘ ’, ‘A’, ‘I’]

Each character gets its own vector representation.

2. How It Works

Step 1: Build Character Vocabulary

Collect all unique characters.

Example:
[a, b, c, ..., z, A, B, ..., 0–9, punctuation, space]

Step 2: Assign Index

Each character gets an index.

Example:

 a→0

 b→1

 …

 z → 25
Step 3: Represent Characters

One-Hot Encoding

Each character → one-hot vector

Example:

 ‘a’ → [1,0,0,...]

 ‘b’ → [0,1,0,...]

3. Example

Word: “cat”

Character representation:

c → [0,0,1,...]
a → [1,0,0,...]
t → [0,0,0,...,1]

Introduction

Feature extraction from text data is the process of converting unstructured text into numerical features so
that machine learning models can understand and process it. Since algorithms cannot directly interpret text,
this step is essential in Natural Language Processing (NLP) tasks such as sentiment analysis, spam
detection, and text classification.

Why Feature Extraction is Needed

Text data:

 Contains words, not numbers

 Has variable length
 Is ambiguous and complex

Machine learning models require:

 Fixed-size numerical vectors

Therefore, feature extraction transforms text → numbers.

Text Preprocessing

Input Sentence:

“I am loving the Machine Learning course!”

Steps:

 Tokenization
→ [“I”, “am”, “loving”, “the”, “Machine”, “Learning”, “course”]
 Lowercasing
→ [“i”, “am”, “loving”, “the”, “machine”, “learning”, “course”]
 Stopword Removal
→ [“loving”, “machine”, “learning”, “course”]
 Stemming/Lemmatization
→ [“love”, “machine”, “learning”, “course”]

Feature Extraction Techniques

3.1 Bag of Words (BoW)

Concept:

Counts frequency of words in a document.

Example:

Documents:

 D1: “I love AI”

 D2: “I love ML”

Vocabulary:
[I, love, AI, ML]

Vectors:

 D1 → [1, 1, 1, 0]
 D2 → [1, 1, 0, 1]

Each position represents a word count.

Key Point:

 Simple but ignores word order and meaning

2 N-grams

Concept:

Captures sequences of words.

Example:

Sentence: “I love AI”

  Unigrams → I, love, AI
 Bigrams → “I love”, “love AI”

Helps capture context.

Use Case:

 “not good” vs “good” (important difference)

3 TF-IDF (Term Frequency–Inverse Document Frequency)

Concept:

Assigns importance to words.

Example:

Documents:

 D1: “AI is useful”

 D2: “AI is powerful”

Word “AI” appears in both → less important

Word “useful” appears once → more important

TF-IDF gives higher weight to unique words.

4. Word Embeddings (Semantic Features)

Word2Vec

Concept:

Represents words as dense vectors capturing meaning.

Example:

 “king” → [0.2, 0.8, …]

 “queen” → similar vector

Relationship:
king − man + woman ≈ queen

Insight:

Words with similar meanings are close in vector space.

GloVe

Concept:

Uses overall word co-occurrence in corpus.

Example:

 “ice” is closer to “cold” than “hot”

5. Contextual Feature Extraction

BERT

Concept:

Understands meaning based on context.

Example:

 “bank” in:
o “river bank” → land
o “bank account” → financial

Different vectors for same word!

Text classification in NLP is the automated process of assigning predefined categories or tags to
unstructured text, such as sentiment analysis, topic labeling, and spam detection. It transforms raw text into
structured information using machine learning, allowing for efficient content analysis and organization at
scale
For example, an email can be classified as spam or not spam, or a product review can be classified as
positive or negative.

Types of Text Classification

1. Binary Classification

In this type, the text is classified into two categories only.

Example:

 Spam vs Not Spam

 Positive vs Negative sentiment
This is the simplest form and widely used in applications like email filtering.

2. Multi-class Classification

Here, text is classified into more than two categories, but each text belongs to only one class.
Example:

 News articles → Sports, Politics, Technology

The model selects the most appropriate single category.

3. Multi-label Classification

In this case, a single text can belong to multiple categories simultaneously.

Example:

 “AI in healthcare” → Technology + Health

This is more complex because multiple outputs are predicted.

4. Hierarchical Classification

Categories are arranged in a tree-like structure.

Example:

 Sports → Cricket, Football

The classifier must consider relationships between categories.

Text Classification Process

1. Text Preprocessing

Raw text contains noise and irrelevant information, so it must be cleaned:

 Tokenization: Splitting text into words or tokens

 Lowercasing: Converting all text to lowercase
 Stopword Removal: Removing common words like “is”, “the”
 Stemming/Lemmatization: Reducing words to root form (e.g., “running” → “run”)

This step improves the quality of input data.

2. Feature Extraction
Computers cannot understand text directly, so it must be converted into numerical form.

🔹 Bag of Words (BoW)

This method represents text as a collection of word counts.

It ignores grammar and word order but captures frequency.

Example:
“I love AI” → counts of each word

Limitation: Does not understand meaning or context.

🔹 TF-IDF

This improves BoW by assigning importance to words.

 Words that appear frequently in a document get higher scores

 Words common across many documents get lower scores

Thus, it highlights meaningful words.

🔹 Word Embeddings

These are dense vector representations of words that capture semantic meaning.
Words with similar meanings have similar vectors.
Example: “king” and “queen” are closely related in vector space.

🔹 Contextual Embeddings

Modern techniques (like transformer models) consider the context of words.

For example, the word “bank” has different meanings in different sentences.
These models understand such differences, improving accuracy.

3. Model Building

Once features are extracted, machine learning models are trained.

🔹 Traditional Models

 Naive Bayes: Uses probability; simple and fast

 Logistic Regression: Predicts probability of classes
 SVM: Finds optimal boundary between classes

These work well for smaller datasets.

🔹 Deep Learning Models

 RNN/LSTM: Process sequences and remember context

 CNN: Capture important patterns in text

They perform better on large datasets.

🔹 Transformer Models

Modern models like BERT use attention mechanisms to understand full context of sentences.
They provide very high accuracy and are widely used today.

4. Training the Model

The dataset is divided into:

 Training data → to learn patterns

 Testing data → to evaluate performance

The model learns by minimizing error using techniques like gradient descent.

5. Evaluation Metrics

To measure performance:

 Accuracy: Overall correctness

 Precision: Correct positive predictions
 Recall: Ability to find all relevant cases
 F1-score: Balance between precision and recall

These metrics help determine how well the model performs.

Applications
Text classification is widely used in real life:

 Spam Detection: Identifying unwanted emails

 Sentiment Analysis: Understanding user opinions
 News Classification: Organizing articles
 Chatbots: Understanding user intent
 Healthcare: Classifying medical reports

Challenges
 Ambiguity: Same word has multiple meanings
  Sarcasm: Difficult for machines to detect
 Imbalanced Data: Some classes have more data than others
 Domain Dependency: Model trained in one domain may not work well in another

Stemmer
A stemmer in Natural Language Processing (NLP) is a tool used to reduce words to their root or base form (called a
“stem”).

A stemmer removes suffixes (and sometimes prefixes) from words so that different forms of a word are treated as
the same.

Examples:

 playing, played, plays → play

 connected, connection, connecting → connect

 studies → studi (not always a real word, just a stem)

Why it is used

Stemming helps in:

 Text normalization

 Improving search results (search engines match similar words)

 Reducing vocabulary size

 Information retrieval (like Google search)

from [Link] import PorterStemmer

ps = PorterStemmer()

words = ["playing", "played", "plays", "connection", "connected"]

for word in words:

print(word, "->", [Link](word))

Output:

playing -> play

played -> play
plays -> play
connection -> connect
connected -> connect

Morphological Analyzer

A morphological analyzer in Natural Language Processing (NLP) is a tool that analyzes the internal structure of
words and breaks them into meaningful components like root (stem), prefixes, suffixes, and grammatical features.
What it does

Instead of just cutting words like a stemmer, a morphological analyzer gives detailed linguistic information.

Example:

 unhappiness →
un + happy + ness
(prefix + root + suffix)

 running →
root: run
tense: present participle

🔹 Key Functions

A morphological analyzer performs:

 Segmentation → splitting words into morphemes

 Root identification → finding base word

 Feature extraction → tense, number, gender, etc.

 Normalization → mapping words to base form

🔹 Types of Morphology

1. Inflectional Morphology

o Changes form but not meaning

o play → playing, played

2. Derivational Morphology

o Changes meaning or word class

o happy → happiness

Applications

 Machine Translation

 Spell Checking

 Information Retrieval

 Speech Recognition

 Chatbots

Sentiment Analysis
Sentiment Analysis in Natural Language Processing (NLP) is the process of determining the emotional tone or
opinion expressed in text.
🔹 What it does

It classifies text into categories like:

 😊 Positive → “This product is amazing!”

 😐 Neutral → “The product is okay.”

 😞 Negative → “This is a bad product.”

🔹 Types of Sentiment Analysis

1. Polarity-based

o Positive / Negative / Neutral

2. Emotion-based

o Happy, Sad, Angry, etc.

3. Aspect-based

o Analyzes specific parts

o Example: “Camera is good but battery is poor”

→ Camera: Positive, Battery: Negative

Applications

 Product reviews analysis

 Social media monitoring

 Customer feedback

 Chatbots

 Market research

What is Named Entity Recognition?

Named Entity Recognition in Natural Language Processing (NLP) is the task of identifying and classifying important
entities in text into predefined categories.

Example

Sentence:
“Virat Kohli plays for Royal Challengers Bangalore in Bangalore.”

NER Output:

 Virat Kohli → Person

 Royal Challengers Bangalore → Organization/Team

  Bangalore → Location

Common Entity Types

 Person

 Location

 Organization

 Date & Time

 Money

 Percentage

Applications

 Information extraction

 Chatbots

 Search engines

 Resume parsing

 News analysis

. Text Similarity

✅ What is Text Similarity?

Text similarity in Natural Language Processing (NLP) measures how similar two pieces of text are.

🔹 Example

 Sentence 1: “I love machine learning”

 Sentence 2: “I like AI and ML”

👉 Even though words differ, meaning is similar → high similarity

🔹 Types of Text Similarity

1. Lexical Similarity (word-based)

 Compares words directly

 Example methods:

o Cosine Similarity

o Jaccard Similarity

2. Semantic Similarity (meaning-based)

  Understands meaning

 Uses embeddings (Word2Vec, BERT)

🔹 Applications

 Plagiarism detection

 Search engines

 Chatbots

 Document clustering

What are Keywords?

Keywords are significant terms that summarize the content.

👉 Example:
Text: “Machine learning is widely used in data science and artificial intelligence.”

Keywords:

 machine learning

 data science

 artificial intelligence

🔹 2. Why Keyword Extraction is Important

 Helps in summarizing documents

 Improves search engines (SEO)

 Used in chatbots and AI systems

 Supports topic detection

🔹 3. Types of Keywords

1. Single-word keywords → “AI”, “data”

2. Multi-word phrases → “machine learning”, “data science”

🔹 4. Methods of Keyword Extraction

🔸 (A) Frequency-Based Method

 Count how often words appear

 Remove stopwords (like is, the, and)

👉 Simple but not very accurate

🔸 (B) TF-IDF (Most Important Method)

TF-IDF=TF×log⁡(NDF)TF\text{-}IDF = TF \times \log\left(\frac{N}{DF}\right)TF-IDF=TF×log(DFN)

✔️Explanation:

 TF (Term Frequency) → how often a word appears in a document

 IDF (Inverse Document Frequency) → how rare the word is across documents

 Words with high TF and low DF are important

🔸 (C) RAKE (Rapid Automatic Keyword Extraction)

 Splits text into phrases

 Scores based on word co-occurrence

 Works well for short text

🔸 (D) TextRank

 Graph-based algorithm

 Words are nodes, relationships are edges

 Ranks keywords like Google PageRank

🔸 (E) Machine Learning / Deep Learning

 Uses models like:

o BERT

o Word embeddings

 Captures context and meaning

🔹 5. Steps in Keyword Extraction

1. Text Preprocessing

o Lowercasing

o Removing punctuation

o Removing stopwords

2. Tokenization

o Split text into words

3. Scoring

o Use TF-IDF / RAKE / TextRank

4. Ranking

o Select top keywords

Applications
  🔍 Search engines

 📄 Document summarization

 📊 Topic modeling

 🛒 Product tagging

 📱 Social media analysis

🔹 8. Advantages

 Saves time in reading large text

 Identifies important information quickly

 Useful in automation

🔹 9. Limitations

 May ignore context

 Can extract irrelevant words

 Depends on preprocessing quality

What is Model Deployment?

Model deployment in Natural Language Processing (NLP) means making a trained NLP model available for real-world
use.
After training and testing the model, deployment allows users or applications to interact with it and get predictions
automatically.

For example:

 A chatbot replying to customers

 Language translation apps

 Spam email detection

 Sentiment analysis on reviews

 Voice assistants like Siri or Alexa

The trained model is deployed on a server, cloud platform, website, or mobile app so it can process new text data in
real time.

NLP Model Lifecycle

The basic lifecycle of an NLP model is:

1. Data Collection

2. Data Preprocessing

3. Feature Extraction

4. Model Training
 5. Model Evaluation

6. Model Deployment

7. Monitoring and Updating

Deployment is the final stage where the model becomes usable by end users.

Why Model Deployment is Important

Without deployment, a model stays only inside a notebook or local system.

Deployment helps to:

 Use the model in real applications

 Automate predictions

 Serve multiple users

 Integrate with websites and apps

 Process live data

Steps in NLP Model Deployment

Step 1: Train the Model

Step 2: Save the Model

Python libraries:

Step 3: Create API

Step 4: Deploy on Server or Cloud

The API is uploaded to cloud hosting platforms.

Users can now send requests from:

 Websites

 Mobile apps

 Chatbots

Common Tools for Deployment

Tool Purpose

spaCy NLP processing

Flask Create API

FastAPI High-speed APIs

Docker Containerization

 Amazon Web Services Cloud deployment and hosting

Tool Purpose

 Google Cloud

 Microsoft Azure

 Heroku

Train Model
↓
Save Model
↓
Create API
↓
Deploy on Server/Cloud
↓
Users Send Requests
↓
Model Returns Predictions

1. Saving and Loading Trained Models

We use joblib

other: pickle, torch etc

What is Joblib?

joblib is a Python library used to:

 Save trained machine learning models

 Load saved models later

 Store large NumPy arrays efficiently

It is commonly used with:

 Scikit-learn

 Machine learning projects

 NLP models

Why Do We Use Joblib?

Training a model again and again takes time.

Instead of retraining:

1. Train once

2. Save the model

 3. Load it whenever needed

This is called model persistence.

Installing Joblib

pip install joblib

Importing Joblib

from joblib import dump, load

Function Purpose

dump() Save object

load() Load saved object

mport libraries and models  sklearn is the library called Scikit-learn used for
Machine Learning. feature_extraction.text
m sklearn.feature_extraction.text import CountVectorizer
contains tools for processing text.
m sklearn.naive_bayes import MultinomialNB
CountVectorizer converts text into numbers so the
m joblib import dump
machine learning model can understand it.

aining data  Imports the MultinomialNB algorithm.

ts = [
NB means Naive Bayes.
ove NLP",
ate bugs", It is commonly used for:
thon is amazing",
is is bad"  Sentiment analysis ,Spam detection ,Text
classification

 imports dump function from Joblib.

els = [
sitive", dump() is used to save trained models into files.
gative", # Training data
sitive",
gative" Creates a list called texts.

Each sentence is training data.

onvert text into numerical vectors Creates labels for each sentence.
torizer = CountVectorizer()
Labels are the correct answers for training.
vectorizer.fit_transform(texts) These sentences will teach the model positive and
negative sentiment.
ain model
Data inside list
del = MultinomialNB()
Sentence Meaning
[Link](X, labels)
I love NLP Positive
ave model
I hate bugs Negative
mp(model, "sentiment_model.joblib")
ave vectorizer Sentence Meaning
mp(vectorizer, "[Link]")
Python is amazing Positive
nt("Model Saved Successfully") This is bad Negative

# Convert text into numerical vectors

tput Creates an object of CountVectorizer.

del Saved Successfully This object will:

 Split sentences into words

 Build vocabulary

 Count word frequency

It performs 2 tasks:

1. fit()

Learns all unique words from training data.

Vocabulary becomes:

Word Index

love 0

nlp 1

hate 2

bugs 3

python 4

amazing 5

bad 6

2. transform()

Converts sentences into numerical vectors.

Sentence Vector

I love NLP [1,1,0,0,0,0,0]

I hate bugs [0,0,1,1,0,0,0]

Python is amazing [0,0,0,0,1,1,0]

This is bad [0,0,0,0,0,0,1]

# Train model

[Link] the Naive Bayes model.

 The model is empty right now.

It has not learned anything yet.

2. Input:

 X → numerical vectors

 labels → correct outputs

The model learns:

 Which words indicate positive sentiment

 Which words indicate negative sentiment

Example Learning

Word Learned Sentiment

love Positive

amazing Positive

hate Negative

bad Negative

Save Model

 Saves trained model into a file.

 Saves the vocabulary and text-processing rules.

Text Data
↓
CountVectorizer
↓
Numerical Vectors
↓
Naive Bayes Model
↓
Training
↓
Save Model using Joblib
important Terms

Term Meaning

Serialization Converting object into file

Deserialization Loading object back

Model Persistence Saving trained model

.joblib Saved model file extension

2. Loading Saved Models

Program

from joblib import load #Used to load saved files.)

# Load model
model = load("sentiment_model.joblib") #Loads trained model from disk.

# Load vectorizer
vectorizer = load("[Link]") # Loads saved vocabulary.

# New text
text = ["I love Python"]

# Convert text
X = [Link](text) # Converts new text into numerical form.

# Prediction
prediction = [Link](X) # Model predicts sentiment.

print(prediction)

Output

['Positive']

3. Creating Simple APIs for Models

API ( Application Programming Interface) allows communication between:

 Frontend

 NLP model

We use:

 Flask

Other: FastAPI, TorchServe, TensorFlow Serving, Ray Serve etc

Flask is a lightweight Python web framework used to:

 Build APIs
  Create web applications

Required Libraries

Install required packages:

pip install flask scikit-learn joblib

from flask import Flask, request, jsonify Imports important things from Flask.
from joblib import load
Item Purpose
# Create Flask app Flask Creates web application
app = Flask(__name__)
request Gets data sent by user
# Load model and vectorizer
jsonify Converts Python data into JSON response
model = load("sentiment_model.joblib")
vectorizer = load("[Link]")

# API route # Create Flask app

@[Link]("/predict", methods=["POST"]) Creates Flask app object.
def predict():
Flask(__name__)
# Get text from user
Part Meaning
text = [Link]["text"]
Flask Flask class
# Convert into vector
X = [Link]([text]) __name__ Current Python file name

Flask uses this to:

# Predict
prediction = [Link](X)  locate files

 configure application
# Return result
return jsonify({ API Route
"prediction": prediction[0]
@[Link]("/predict", methods=["POST"])
})
Explanation
# Run application
if __name__ == "__main__": Creates API endpoint.
[Link](debug=True) Route

/predict

This URL will receive prediction requests.

methods=["POST"]

Means:
  only POST requests are allowed

POST is used for:

 sending data securely to server

def predict():

Defines function called when user accesses:

# Get text from user

Reads JSON data sent by user.

# Return result

Sends prediction back to user in JSON format

API response:

{
"prediction": "Positive"
}

API response:

{
"prediction": "Positive"
}

if __name__ == "__main__":

Explanation

Checks:

 Is this file being run directly?

If yes:

 start Flask server

[Link](debug=True)

Starts local server.

Output

Running on [Link]

save this code as:

[Link]

4. Deploying NLP Models

Deployment means hosting the model so users can access it online.

Common Deployment Platforms

Platform Purpose

Render Free deployment

Railway Cloud deployment

Heroku Web hosting

PythonAnywhere Python hosting

[Link] Simple Web Interface for Predictions

A web interface allows users to:

 Enter text

 Click button

 View prediction

HTML Interface Example

Save File as: [Link]

<!DOCTYPE html>
<html>
<head>
<title>Sentiment Analysis</title>
</head>
<body>

<h1>Sentiment Analysis</h1>

<form action="/predict" method="post"> #Sends data to Flask route.

<input type="text" name="text">

<button type="submit">Predict</button>

</form>

</body>
</html>

output
Flask Backend with HTML

Program

from flask import Flask, render_template, request  Imports important things from Flask.
from joblib import load
Item Purpose
app = Flask(__name__) Flask Creates web application

model = load("sentiment_model.joblib") render_template Displays HTML pages

vectorizer = load("[Link]")
request Gets data from user
@[Link]("/")
def home():
return render_template("[Link]")  Imports load() function from Joblib.

Purpose:
@[Link]("/predict", methods=["POST"])
def predict():  loads saved model files

text = [Link]["text"]
 Imports load() function from Joblib.
X = [Link]([text])
Purpose:

prediction = [Link](X)  loads saved model files

 Creates Flask application object.

return f"Prediction: {prediction[0]}"
 Loads trained sentiment analysis
if __name__ == "__main__": model
[Link](debug=True)
 Loads saved CountVectorizer.

 Creates route for homepage.

means main page of website.

Defines homepage function.

Displays HTML webpage.

 Creates prediction API route.

methods=["POST"]

Means:

 this route accepts POST requests

only

 Defines prediction function.

 Runs when user submits form.

 Gets text entered by user from

HTML form.

 Converts text into numerical vector

 Uses trained model to predict

sentiment.

 Displays prediction result on

browser.

Folder Structure

project/
│
├── [Link]
├── sentiment_model.joblib
├── [Link]
│
└── templates/
└── [Link]

User Interaction

User opens:

[Link]

Meaning

Part Meaning

[Link] Local computer

5000 Flask port number

/ Home page

Enters:
I love python

Model predicts:

Positive

User Sends Text

↓
Flask API Receives Request
↓
Load Vectorizer
↓
Convert Text into Vector
↓
Load Trained Model
↓
Predict Sentiment
↓
Return JSON Response

What is Scikit-learn?

Scikit-learn is an open-source Python library used for:

 Machine Learning

 Data Analysis

 Predictive Modeling

It provides simple and efficient tools for:

 Classification

 Regression

 Clustering

 Data preprocessing

 Model evaluation

Why Scikit-learn is Used

Scikit-learn helps computers:

 Learn from data

 Make predictions

 Find patterns automatically

Example:

 Predict house prices

  Detect spam emails

 Recognize faces

 Analyze customer behavior

Program 1

from sklearn.feature_extraction.text import CountVectorizer

text = [
"I love NLP",
"I love Python"
]

vectorizer = CountVectorizer()

result = vectorizer.fit_transform(text)

print(vectorizer.get_feature_names_out())
print([Link]())

OUTPUT

['love' 'nlp' 'python']

[[1 1 0]
[1 0 1]]

This program uses Scikit-learn CountVectorizer to convert text into numerical form using the Bag of Words (BoW)
technique.

Code Explanation

from sklearn.feature_extraction.text import CountVectorizer

Imports CountVectorizer from Scikit-learn.

It is used to:

 Convert text into vectors

 Count word frequency

text = [
"I love NLP",
"I love Python"
]

Creates two text documents.

Document 1:

I love NLP
Document 2:

I love Python

vectorizer = CountVectorizer()

Creates an object of CountVectorizer.

 Converts text to lowercase

 Removes special characters

 Tokenizes words

result = vectorizer.fit_transform(text)

Two operations happen here:

1. fit()

Learns all unique words (vocabulary).

Vocabulary:

love
nlp
python

(I is ignored because single-letter words are removed by default.)

2. transform()

Converts sentences into numerical vectors.

print(vectorizer.get_feature_names_out())

Prints vocabulary words.

Output

['love' 'nlp' 'python']

print([Link]())

Converts sparse matrix into normal array.

Output

[[1 1 0]
[1 0 1]]

How Output is Generated

Vocabulary order:

Index Word

0 love

1 nlp

2 python

Sentence 1: "I love NLP"

love nlp python

1 1 0

Vector:

[1 1 0]

Sentence 2: "I love Python"

love nlp python

1 0 1

Vector:

[1 0 1]

Final Output

['love' 'nlp' 'python']

[[1 1 0]
[1 0 1]]

Program 2

from sklearn.feature_extraction.text import TfidfVectorizer

text = [

"I love NLP",

"I love Python"

vectorizer = TfidfVectorizer()

result = vectorizer.fit_transform(text)
print(vectorizer.get_feature_names_out())

print([Link]())

Output

['love' 'nlp' 'python']

[[0.57973867 0.81480247 0. ] [0.57973867 0. 0.81480247]]

Explatation

This program uses Scikit-learn TfidfVectorizer to convert text into numerical values using the TF-IDF technique.

TF-IDF means:

TF-IDF=TF×IDF

It measures:

 How important a word is in a document

 Compared to all other documents

1. Import Library

from sklearn.feature_extraction.text import TfidfVectorizer

Imports TfidfVectorizer.

Used for:

 Text representation

 Feature extraction

 NLP preprocessing

2. Create Text Data

text = [
"I love NLP",
"I love Python"
]

Creates two documents.

Document 1

I love NLP

Document 2

I love Python

3. Create TF-IDF Object

vectorizer = TfidfVectorizer()

Creates TF-IDF vectorizer object.

Functions:

 Converts text to lowercase

 Removes punctuation

 Calculates TF-IDF scores

4. Learn Vocabulary + Transform Text

result = vectorizer.fit_transform(text)

This performs two tasks:

fit()

Learns unique words (vocabulary).

Vocabulary becomes:

love
nlp
python

(I is ignored because single-letter words are removed by default.)

transform()

Converts text into TF-IDF numerical vectors.

5. Print Feature Names

print(vectorizer.get_feature_names_out())

Displays vocabulary words.

Output

['love' 'nlp' 'python']

6. Print TF-IDF Matrix

print([Link]())

Converts sparse matrix into normal array.

Output

Approximate output:
[[0.57973867 0.81480247 0. ]
[0.57973867 0. 0.81480247]]

How Output is Calculated

Vocabulary:

Index Word

0 love

1 nlp

2 python

TF-IDF Meaning

Word: "love"

Appears in both documents.

So its importance is lower.

TF-IDF value becomes smaller.

Word: "nlp"

Appears only in Document 1.

Higher importance.

TF-IDF value becomes larger.

Word: "python"

Appears only in Document 2.

Higher importance.

TF-IDF value becomes larger.

Document Representation

Document 1: "I love NLP"

love nlp python

0.57 0.81 0

Vector:

[0.57, 0.81, 0]

Document 2: "I love Python"

love nlp python

0.57 0 0.81

Vector:

[0.57, 0, 0.81]

Program 3

from [Link] import OneHotEncoder

import numpy as np

data = [Link]([["cat"], ["dog"], ["fish"]])

encoder = OneHotEncoder()

result = encoder.fit_transform(data)

print([Link]())

Output

[[1. 0. 0.]

[0. 1. 0.]

[0. 0. 1.]]

This program uses Scikit-learn OneHotEncoder to convert categorical text data into numerical format.

Machine learning models cannot understand text directly, so categories are converted into binary vectors.

1. Import Libraries

from [Link] import OneHotEncoder

Imports OneHotEncoder.

Used for:

 Converting categorical data into numbers

 Data preprocessing

import numpy as np

Imports NumPy.

Used for:
  Arrays

 Numerical operations

2. Create Data

data = [Link]([["cat"], ["dog"], ["fish"]])

Creates categorical dataset.

Data contains 3 categories:

Animal

cat

dog

fish

Why Double Brackets?

[["cat"], ["dog"], ["fish"]]

Scikit-learn expects data in 2D format:

 Rows = samples

 Columns = features

Shape of data:

(3,1)

Meaning:

 3 rows

 1 column

3. Create Encoder Object

encoder = OneHotEncoder()

Creates One Hot Encoder object.

Purpose:

 Detect unique categories

 Convert them into binary vectors

4. Fit and Transform Data

result = encoder.fit_transform(data)

This performs two operations:

fit()

Learns all unique categories.

Categories found:

cat
dog
fish

transform()

Converts categories into binary vectors.

One Hot Encoding Logic

Each category gets its own column.

Category cat dog fish

cat 1 0 0

dog 0 1 0

fish 0 0 1

Only one value is 1.

Others are 0.

That is why it is called:

One Hot Encoding

5. Print Result

print([Link]())

fit_transform() returns a sparse matrix.

toarray() converts it into normal array format.

Final Output

[[1. 0. 0.]
[0. 1. 0.]
[0. 0. 1.]]

Output Explanation

cat

[1 0 0]

Meaning:
  cat = Yes

 dog = No

 fish = No

dog

[0 1 0]

Meaning:

 cat = No

 dog = Yes

 fish = No

fish

[0 0 1]

Meaning:

 cat = No

 dog = No

 fish = Yes

spaCy

spaCy is an open-source Python library used for Natural Language Processing (NLP).
It is designed for fast, efficient, and real-world text processing.

It helps computers understand human language.

Features of spaCy

 Tokenization

 Part-of-Speech Tagging

 Named Entity Recognition (NER)

 Lemmatization

 Stop Removal Words

 Text Similarity

Practical Implementation Using spaCy

1. Install spaCy

pip install spacy

Download English language model:

python -m spacy download en_core_web_sm

Basic NLP Processing

Tokenization

Breaking sentence into words.

Program

import spacy

nlp = [Link]("en_core_web_sm")

text = "I love learning NLP using spaCy."

doc = nlp(text)

for token in doc:

print([Link])

Output

I
love
learning
NLP
using
spaCy
.

Explanation:

1. Import spaCy Library

import spacy

Explanation

 Imports the spaCy library into Python.

 Now we can use NLP functions provided by spaCy.

2. Load English Language Model

nlp = [Link]("en_core_web_sm")

Explanation

 Loads the English small model named "en_core_web_sm".

This model contains:

 Vocabulary
  Grammar rules

 Tokenizer

 POS tagger

 Named Entity Recognizer

Meaning of Name

Part Meaning

en English language

core Core model

web Trained on web text

sm Small model

The loaded model is stored in variable nlp.

3. Store Input Text

text = "I love learning NLP using spaCy."

Explanation

 Stores the sentence in variable text.

4. Process the Text

doc = nlp(text)

Explanation

 Sends the text to spaCy pipeline.

 spaCy analyzes the sentence.

It performs:

 Tokenization

 POS tagging

 Parsing

 NER

 Lemmatization

The processed result is stored in doc.

5. Loop Through Tokens

for token in doc:

Explanation

 doc contains tokens (words and punctuation).

  Loop accesses one token at a time.

Tokens in sentence:

Token

love

learning

NLP

using

spaCy

6. Print Each Token

print([Link])

Explanation

 [Link] gives the actual word.

 Prints one token per line.

Output

I
love
learning
NLP
using
spaCy
.

Part of Speech (POS) Tagging

Identifies noun, verb, adjective, etc.

Program

import spacy

nlp = [Link]("en_core_web_sm")

text = "Python is very powerful."

doc = nlp(text)
for token in doc:
print([Link], " --> ", token.pos_)

Output

Python --> PROPN

is --> AUX
very --> ADV
powerful --> ADJ
. --> PUNCT

Meaning of POS Tags

Word POS Tag Meaning

Python PROPN Proper Noun

is AUX Auxiliary Verb

a DET Determiner

Powerful ADJ Adjective

spaCy vs NLTK
Feature spaCy NLTK

Speed Fast Slower

Use Industry Education

Ease Moderate Easy

Data Handling Large Small

5. Named Entity Recognition (NER)

Finds names of people, countries, organizations, etc.

Program

import spacy

nlp = [Link]("en_core_web_sm")

text = "Sachin Tendulkar lives in India and works with Google."

doc = nlp(text)
for ent in [Link]:
print([Link], " --> ", ent.label_)

Output

Sachin Tendulkar --> PERSON

India --> GPE
Google --> ORG

5. Access Named Entities

for ent in [Link]:

Explanation

 [Link] contains all detected entities.

 ent means one entity at a time.

Entities detected:

Text Entity Type

Sachin Tendulkar PERSON

India GPE

Google ORG

6. Print Entity and Label

print([Link], " --> ", ent.label_)

Explanation

Part Meaning

[Link] Actual entity name

ent.label_ Category of entity

Output

Sachin Tendulkar --> PERSON

India --> GPE
Google --> ORG

Meaning of Labels

Label Meaning

PERSON Name of a person

GPE Geopolitical entity (country/city/state)

Label Meaning

ORG Organization/company

6. Lemmatization

Converts words to root/base form.

Program

import spacy

nlp = [Link]("en_core_web_sm")

text = "running played studies"

doc = nlp(text)

for token in doc:

print([Link], " --> ", token.lemma_)

Output

running --> run

played --> play
studies --> study

Stop Word Removal

Removing common words like is, the, am.

Program

import spacy

nlp = [Link]("en_core_web_sm")

text = "This is a simple example of stop word removal."

doc = nlp(text)

for token in doc:

if not token.is_stop:
print([Link])

Output

simple
example
stop
word
removal

Which Words Were Removed?

Removed Stop Words

This

is

of

These are common words with less importance.

10Text Similarity

Compare similarity between two sentences.

Program

import spacy

nlp = [Link]("en_core_web_sm")

doc1 = nlp("I like Python")

doc2 = nlp("I love Python")

print([Link](doc2))

Example Output

0.89