TEXT DATA

ANALYTICS
UNIT - I
Introduction to Natural Language Processing Basics, Language Syntax and
Structure (Words, Phrases, Clauses, & Grammar), Language Semantics
Processing, (Lexical Semantic Relations, Homonyms, Homographs, and
Homophones, Capitonyms, Hyponyms and Hypernyms), Text Corpora (Corpora
Annotation and Utilities), Accessing Text Corpora (Brown Corpus, WordNet
Corpus) and NLP Applications (Machine Translation, Text Summarization and
Text categorization)
Introduction to Natural Language Processing Basics
What Is Natural Language?
Natural language is any language that has developed naturally over time through
human communication, such as English, Japanese, or Sanskrit. Unlike
programming languages, which are artificially constructed, natural languages
evolve through social interaction and can be expressed through speech, writing,
or signs.

Philosophy of Language
The study of language philosophy deals with four key areas: the nature of
meaning, the use of language, language cognition, and the relationship between
language and reality. These areas help us understand how language conveys
meaning, how it is used in communication, how the human brain processes it, and
how language relates to truth and reality.

Nature of Meaning
Semantics, or the meaning of words and sentences, is a major focus of linguistics.
It explores how meaning is formed, how words relate to each other, and how
syntax contributes to the structure of meaningful expressions. Beyond words,
factors like body language and past experiences influence how meaning is
interpreted.

Use of Language
Language is not only about words but also about context, intent, and
communication. Speech acts, tone, and body language all play a role in how
language is used. Additionally, language acquisition and cognitive activities
shape how humans learn and apply language in daily life.

Language Cognition
Cognition studies how the human brain processes language, from structuring
words into sentences to understanding messages. It also examines how language
and thought interact, shaping both communication and perception.
Language and Reality
Language helps humans describe reality, but philosophers debate the extent to
which language accurately represents truth. Some expressions directly reflect
reality, while others may be more abstract or subjective. Understanding this
relationship is essential for areas like text analytics and natural language
processing (NLP).

Language Syntax and Structure (Words, Phrases, Clauses, & Grammar)

Language Syntax and Structure
Language syntax and structure define the rules that govern how words form
meaningful sentences. Syntax dictates the correct arrangement of words, while
structure organizes language into hierarchical units like phrases and clauses. A
well-structured sentence follows grammatical conventions, ensuring clarity and
coherence in communication. Understanding syntax is crucial in natural language
processing (NLP) for tasks like text parsing, classification, and summarization.

Words and Parts of Speech

Words are the fundamental building blocks of language, each with its own lexical
meaning. While words are the smallest independent units, they consist of smaller
morphemes that contribute to their meaning. Words are classified into parts of
speech (POS) such as nouns (N), verbs (V), adjectives (ADJ), and adverbs
(ADV). Some words belong to open classes, meaning they allow new additions
(e.g., nouns and verbs), while others belong to closed classes, with a fixed set of
words (e.g., pronouns and prepositions). POS tagging helps in analyzing language
structure and meaning in NLP applications.
Phrases and Their Types
Phrases are groups of words that function as a single unit within a sentence,
forming an intermediate step between words and clauses. There are five major
types of phrases:
• Noun Phrase (NP): A phrase centered around a noun, often acting as a
subject or object (e.g., The quick brown fox).
• Verb Phrase (VP): A phrase containing a main verb and possibly auxiliary
verbs (e.g., is jumping over the dog).
• Adjective Phrase (ADJP): A phrase with an adjective as its head, modifying
a noun (e.g., very beautiful).
• Adverb Phrase (ADVP): A phrase with an adverb as its head, modifying
verbs, adjectives, or other adverbs (e.g., quite slowly).
• Prepositional Phrase (PP): A phrase beginning with a preposition, followed
by a noun phrase (e.g., on the table).
Phrases contribute to sentence structure by grouping related words, ensuring
proper meaning and readability.

Clauses: The Building Blocks of Sentences

A clause is a larger syntactic unit than a phrase and contains a subject and
predicate. Clauses can be classified as main (independent) clauses, which can
stand alone as complete sentences, and subordinate (dependent) clauses, which
rely on a main clause for meaning. Clauses also vary by function:
• Declarative Clauses: Make statements (The sky is blue).
• Imperative Clauses: Give commands or requests (Please close the door).
• Interrogative Clauses: Ask questions (Did you finish your work?).
• Exclamative Clauses: Express strong emotions (What a beautiful sunset!).
Clauses can be combined to form complex and compound sentences using
conjunctions and dependent words, enhancing sentence variety and meaning.

Grammar and Sentence Structure

Grammar provides the foundational rules for organizing words, phrases, and
clauses into well-formed sentences. It helps in maintaining syntactic accuracy and
ensuring effective communication. The evolution of grammar has led to various
frameworks, but the two most widely used are dependency grammar and
constituency grammar.

Dependency Grammar
Dependency grammar focuses on the relationships between words rather than
grouping them into hierarchical structures. Each word in a sentence, except for
the root (usually a verb), depends on another word, forming directed acyclic
graphs (DAGs). Dependencies define how words relate, such as subjects linking
to verbs and objects linking to actions. In dependency grammar, the meaning is
derived from the connections between words rather than their order in a sentence.

Constituency Grammar (Phrase Structure Grammar)

Constituency grammar represents sentence structure hierarchically by breaking it
into constituents—words, phrases, and clauses. It follows phrase structure rules,
where a sentence (S) is split into a noun phrase (NP) and a verb phrase (VP),
which can be further divided into smaller components. This recursive structure
allows for detailed analysis of sentence composition. Unlike dependency
grammar, constituency grammar focuses on the hierarchical order of words in a
sentence, making it useful for syntactic tree parsing.

Conclusion
Language syntax and structure play a fundamental role in organizing words into
meaningful expressions. Words form phrases, which combine into clauses, and
clause’s structure sentences following grammatical rules. Dependency and
constituency grammars offer different approaches to understanding language,
both of which are crucial in natural language processing (NLP) for tasks like
machine translation, text summarization, and speech recognition. Understanding
these linguistic principles helps in building better computational models for
analyzing and generating human language.
Language Semantics Processing, (Lexical Semantic Relations, Homonyms,
Homographs, and Homophones, Capitonyms, Hyponyms and Hypernyms)
Language Semantics Processing
Language semantics is the study of meaning in words, phrases, and sentences. It
plays a crucial role in natural language understanding (NLU) by analyzing how
words relate to each other and how they convey meaning based on their structure
and context. Lexical semantics, a subfield of linguistics, focuses on these
relationships by examining how words are connected and how their meanings
change depending on usage.

Lexical Semantic Relations

Words derive meaning not only from their definitions but also from their
relationship with other words. These relationships are fundamental to lexical
semantics and are essential for text analysis, machine translation, sentiment
analysis, and NLP applications. Words are structured within a lexicon, a
vocabulary collection where each word is a lexical unit that carries meaning
independently or in combination with others. Understanding these relationships
helps machines disambiguate word meanings, classify text, and enhance search
engine accuracy.

Homonyms, Homographs, and Homophones

Homonyms are words that share the same spelling or pronunciation but have
different meanings. For example, bat (the animal) and bat (used in baseball) are
homonyms. This ambiguity makes homonyms particularly challenging for natural
language processing (NLP) because meaning depends on context.

Homographs have the same spelling but different meanings and sometimes
different pronunciations. For example, lead (the metal) and lead (to guide) or bass
(a type of fish) and bass (a low musical sound). NLP models must analyze
sentence structure and context to determine the correct meaning.

Homophones sound the same but have different meanings and spellings, such as
pair (a set of two) and pear (a fruit). Homophones can cause speech-to-text errors
in voice recognition systems, making disambiguation crucial for AI assistants and
transcription software.

Capitonyms
Capitonyms are words that change meaning when capitalized. Examples include
March (the month) vs. march (to walk in rhythm) and Turkey (the country) vs.
turkey (the bird). These differences are essential in text processing, as incorrect
capitalization can alter meaning, especially in named entity recognition (NER)
tasks in NLP.

Hyponyms and Hypernyms

Hyponyms are specific words that belong to a broader category represented by a
hypernym. For example, dog, cat, and rabbit are hyponyms of the hypernym
animal. Similarly, rose and tulip are hyponyms of flower. These hierarchical
relationships help build taxonomies and ontologies, enabling AI to classify words
accurately, improve recommendation systems, and enhance search engine results.

Hypernyms, on the other hand, are more general terms encompassing multiple
hyponyms. Understanding hyponymy and hypernymy allows NLP models to
perform abstraction, synonym expansion, and context-based retrieval in search
engines.

Lexical semantic relations help machines interpret language, resolve ambiguities,

and enhance text analysis. Understanding homonyms, homographs, homophones,
capitonyms, hyponyms, and hypernyms is crucial for machine translation, speech
recognition, and sentiment analysis. By leveraging these relationships, NLP
systems improve their ability to process human language with greater accuracy
and contextual understanding.

Text Corpora (Corpora Annotation and Utilities)

Text Corpora
A text corpus is a large and structured collection of written or spoken text, often
stored in electronic form for linguistic and statistical analysis. Text corpora (plural
of corpus) are essential in natural language processing (NLP) as they provide the
data necessary for training language models, analyzing syntax, and understanding
word usage patterns. Monolingual corpora contain text in a single language, while
multilingual corpora include multiple languages, enabling cross-linguistic
analysis and translation tasks.

Corpora Annotation
Text corpora are annotated with metadata to enrich the text with useful linguistic
and structural information. Annotation involves adding various linguistic markers
to facilitate syntactic, semantic, and grammatical analysis. Common types of
annotation include:
• POS tagging: Assigns parts of speech (e.g., noun, verb, adjective) to each
word.
• Word stemming: Identifies the base form of words by removing affixes.
• Word lemmatization: Maps words to their canonical (dictionary) form.
• Dependency grammar annotation: Captures syntactic relationships
between words.
• Constituency grammar annotation: Breaks sentences into hierarchical
structures like phrases and clauses.
• Semantic role labeling: Tags words with their roles, such as agent, theme,
location, or recipient.

Treebanks and Advanced Annotations

Advanced annotated corpora, known as treebanks, store structured syntactic
information using dependency and constituency parse trees. These are crucial for
training POS taggers, syntax analyzers, and semantic parsers in NLP. Treebanks
help develop linguistic models that understand sentence structure, word
dependencies, and grammatical correctness.

Corpora Utilities in NLP

Text corpora serve as foundational datasets for numerous NLP applications. They
enable tasks such as:
• Machine translation (e.g., Google Translate)
• Speech recognition (e.g., voice assistants like Siri or Alexa)
 • Grammar and spell checking (e.g., Grammarly)
• Text-to-speech and speech-to-text synthesis
• Entity recognition and knowledge extraction
Additionally, corpora provide lexical insights through concordance, collocations,
and frequency counts, helping linguists analyze word usage patterns, create
dictionaries, and improve language learning models. The rich annotation and
structured data in corpora make them indispensable in both linguistic research
and NLP advancements.

Accessing Text Corpora (Brown Corpus, WordNet Corpus)

Accessing Text Corpora
Text corpora are large collections of structured text used for linguistic research
and natural language processing (NLP). The Natural Language Toolkit (NLTK)
in Python provides built-in support for accessing and analyzing various corpora,
including the Brown Corpus and the WordNet Corpus. These corpora enable tasks
like text classification, part-of-speech (POS) tagging, syntactic parsing, and
semantic analysis.

Installing NLTK and Downloading Corpora

To access text corpora, you need to install the NLTK library using the command:
pip install nltk
After installation, corpora and related resources can be downloaded with:
import nltk
[Link]('all')
This ensures that all necessary text resources are available for NLP tasks. Users
can also selectively download specific corpora using the `[Link]()`
GUI interface.

Accessing the Brown Corpus

The Brown Corpus, developed at Brown University in 1961, is one of the earliest
structured text corpora. It consists of texts from 500 sources, categorized into 15
genres, such as news, fiction, mystery, and science fiction.
To load the Brown Corpus in Python:

from [Link] import brown

print("Total Categories:", len([Link]()))
print([Link]())

This returns a list of all available genres. To access sentences from a specific
category (e.g., mystery):

[Link](categories='mystery')

This outputs tokenized sentences from the mystery genre. Additionally, POS-
tagged sentences can be retrieved using:

brown.tagged_sents(categories='mystery')

Analyzing the most common nouns in the mystery genre can be done by
extracting words tagged as nouns (NN, NP) and computing their frequency
distribution.

Accessing the WordNet Corpus

WordNet is a large lexical database of English that groups words into synsets
(synonym sets) and provides semantic relationships between words, such as
hyponymy, hypernymy, antonymy, and meronymy. It is widely used in semantic
analysis, word sense disambiguation, and information retrieval.

To access WordNet in Python:

from [Link] import wordnet as wn

word_synsets = [Link]('hike')
print(word_synsets)

This returns a list of synsets (word senses) for "hike", including both noun and
verb meanings. To explore detailed meanings, definitions, and example
sentences:

for synset in word_synsets:

print("Synset Name:", [Link]())
print("POS Tag:", [Link]())
print("Definition:", [Link]())
print("Examples:", [Link]())

For instance, "hike" can refer to a long walk (noun) or an increase in price (verb).

Importance of Brown and WordNet Corpora

The Brown Corpus helps in text categorization, POS tagging, and linguistic
research, while WordNet is essential for semantic analysis, lexical relationships,
and machine learning applications. Together, these corpora provide valuable
insights for NLP tasks such as word sense disambiguation, search engines, and
automated text processing.

NLP Applications (Machine Translation, Text Summarization and Text

categorization)
NLP Applications
Natural Language Processing (NLP) enables machines to understand, process,
and generate human language. It plays a crucial role in various applications,
including machine translation, text summarization, and text categorization, which
help automate language-related tasks and improve human-computer interaction.
These applications rely on a combination of linguistic, statistical, and deep
learning techniques to enhance accuracy and efficiency.
Machine Translation
Machine translation (MT) refers to the automated conversion of text from one
language to another while maintaining grammatical structure, meaning, and
context. Early machine translation relied on rule-based approaches and simple
word-for-word substitution, which often resulted in inaccurate translations. Over
time, more sophisticated techniques, such as statistical machine translation
(SMT) and neural machine translation (NMT), have improved the quality of
translations by using large-scale linguistic datasets and deep learning models.

Modern MT systems, like Google Translate and DeepL, utilize transformer-based

architectures (e.g., GPT, BERT, and T5) to generate context-aware translations.
These models analyze entire sentences rather than individual words, improving
translation accuracy. Additionally, real-time translation systems now enable
speech-to-text and text-to-speech translation, enhancing global communication.

Text Summarization
Text summarization is the process of reducing lengthy documents or articles into
concise summaries while preserving key information. This is particularly useful
in news aggregation, legal documents, research papers, and AI-driven content
generation. There are two primary types of summarizations:
• Extraction-based summarization identifies and extracts the most relevant
sentences from the source text. It is widely used in news summarization
and search engine snippets.
• Abstraction-based summarization generates entirely new sentences based
on the meaning of the original text. It uses deep learning and NLP models
to create human-like summaries, similar to how a person would rewrite a
passage.
With the explosion of digital content, AI-powered text summarization tools help
process large volumes of information efficiently, providing quick insights from
books, articles, and research papers.
Text Categorization
Text categorization (text classification) involves assigning a document to one or
more predefined categories based on its content. It is widely used in spam
filtering, sentiment analysis, email classification, and document organization.

This task can be performed using various machine learning techniques:

• Supervised learning trains a model on labeled datasets, where each
document is tagged with a category (e.g., spam vs. non-spam). Algorithms
like Naïve Bayes, Support Vector Machines (SVM), and deep learning
(CNNs, RNNs) are commonly used.
• Unsupervised learning applies clustering techniques, such as k-means or
topic modeling, to automatically group similar documents without
predefined labels.
Advanced deep learning-based text classification models, like BERT, RoBERTa,
and XLNet, can analyze complex sentence structures and improve accuracy in
large-scale classification tasks.

Conclusion
NLP applications like machine translation, text summarization, and text
categorization are revolutionizing communication, content management, and
information retrieval. These technologies are now integrated into search engines,
virtual assistants, business automation, and AI-driven analytics, making language
processing more efficient and accessible across industries.
UNIT – II
Concept of the Tokenization, Sentence Tokenization, Word Tokenization,
Concept of the Text Normalization, (Cleaning Text, Removing Special characters,
Removing stop words,..etc) correcting words using stemming and Lemmatization
and Understanding text syntax and structure (POS tagging and Parsing).
Concept of the Tokenization, Sentence Tokenization, Word Tokenization
Concept of Tokenization
Tokenization is a crucial step in Natural Language Processing (NLP) that involves
splitting text into smaller components called tokens. These tokens can be words,
sentences, or subwords, depending on the level of granularity required for a
particular NLP task. Tokenization helps convert raw text into a structured form
that can be analyzed, processed, and understood by machine learning models. It
is widely used in tasks like text classification, named entity recognition (NER),
sentiment analysis, and machine translation.

Tokenization methods vary depending on the language and writing system. While
English uses spaces as natural delimiters, languages like Chinese, Japanese, and
Thai require specialized tokenization techniques since they do not use spaces
between words. Advanced NLP models, such as Byte Pair Encoding (BPE) and
WordPiece tokenization, further refine the process for neural language models
like BERT and GPT.

Sentence Tokenization
Sentence tokenization, also known as sentence segmentation, involves breaking
a paragraph or document into individual sentences. This step is important for
tasks like machine translation, summarization, and sentiment analysis, where the
sentence structure affects the meaning.

The basic approach to sentence tokenization involves using punctuation marks

such as periods (.), question marks (?), and exclamation points (!) to determine
sentence boundaries. However, rule-based methods may fail when dealing with
abbreviations (e.g., "Dr.", "U.S.A.") or informal writing.

To improve accuracy, NLTK’s `sent_tokenize` function is commonly used.

It relies on a pre-trained `PunktSentenceTokenizer`, which applies
machine learning-based heuristics to detect sentence boundaries more effectively.
Pre-trained tokenizers for different languages, such as German, French, and
Spanish, can also be loaded in NLTK to handle multilingual text segmentation.
Additionally, regular expressions (`RegexpTokenizer`) allow custom rules
for segmenting sentences.

Word Tokenization
Word tokenization involves splitting a sentence into individual words or
subwords, enabling deeper linguistic analysis and text processing. This is
essential for text normalization, stemming, lemmatization, and POS tagging,
which are necessary for building NLP applications like chatbots, search engines,
and AI assistants.

Some of the main approaches to word tokenization include:

• `word_tokenize` (NLTK’s default tokenizer): Uses
TreebankWordTokenizer, which follows Penn Treebank rules for
handling contractions (e.g., wasn't → was + n't).
• `TreebankWordTokenizer`: Separates punctuation from words and
ensures correct segmentation of contractions.
• `RegexpTokenizer`: Uses regular expressions to define tokenization
patterns, allowing flexibility in extracting words from text.
• `WordPunctTokenizer`: Splits text into words while treating
punctuation as separate tokens (e.g., isn't → is + ' + t).
• `WhitespaceTokenizer`: Simply splits text based on spaces, which
is useful for structured data but fails for complex linguistic structures.

Importance of Tokenization
Tokenization is a fundamental step in text preprocessing and directly impacts the
performance of NLP models. Poor tokenization can lead to incorrect syntactic
and semantic analysis, affecting tasks like named entity recognition (NER),
machine translation, and text generation. Advanced tokenization techniques, such
as subword tokenization (BPE, WordPiece), are now widely used in deep
learning models to handle out-of-vocabulary (OOV) words effectively.

Tokenization plays a vital role in structuring text for NLP applications. Sentence
tokenization enables meaningful segmentation of text, while word tokenization
helps analyze language at a granular level. With advancements in rule-based,
statistical, and deep learning-based tokenization, modern NLP systems can
process and interpret text more efficiently, enabling better search engines,
chatbots, and AI-driven content analysis tools.

Concept of the Text Normalization, (Cleaning Text, Removing Special

characters, Removing stop words)
Concept of Text Normalization
Text normalization is a vital preprocessing step in Natural Language Processing
(NLP) that involves cleaning, structuring, and standardizing text to make it
suitable for analysis. Raw text often contains inconsistencies, redundant
characters, and unnecessary elements, which can hinder machine learning models
from effectively understanding language. The key steps in text normalization
include tokenization, text cleaning, removing special characters, and eliminating
stopwords. These steps improve the accuracy and efficiency of text classification,
search engines, sentiment analysis, and chatbot responses.

Cleaning Text
Raw text often contains unwanted characters, extra spaces, formatting symbols,
and HTML/XML tags that do not contribute to meaningful analysis. Cleaning text
ensures that only relevant and meaningful words remain, making it easier for NLP
models to process language.

There are multiple ways to clean text:

Removing HTML tags: Using tools like NLTK’s `clean_html()` function or
BeautifulSoup to extract text from web pages.
Stripping XML and JSON data: Using XPath, lxml, or regex to extract text fields.
Removing extra whitespace and special formatting: Converting irregular text into
a uniform format.

Cleaning text is a crucial step before tokenization, stemming, and lemmatization,

ensuring that text is free from unnecessary noise before further processing.
Removing Special Characters
Special characters, such as punctuation marks, symbols, and emojis, are often
removed to reduce noise and improve text analysis. Special character removal
helps in tasks like text classification, keyword extraction, and machine learning
model training.

There are two main approaches:

1. Removing special characters after tokenization: This method ensures
that individual words are clean and contain only meaningful characters.
2. Removing special characters before tokenization: Using regular
expressions (regex) to clean text before breaking it into tokens ensures that
unnecessary symbols are eliminated early.
However, in certain applications like sentiment analysis and named entity
recognition (NER), punctuation and symbols (e.g., `@username` in social media
posts) may be contextually important and should be preserved when necessary.

Removing Stopwords
Stopwords are frequently occurring words, such as "the," "is," "and," and "in,"
that add little value to NLP tasks but increase computational complexity.
Removing stopwords improves text analysis, reduces data size, and enhances
model efficiency.

Stopwords can be filtered using NLTK’s built-in stopword list:

from [Link] import stopwords

stop_words = set([Link]('english'))
filtered_tokens = [word for word in tokens if word not
in stop_words]

However, stopwords should be handled carefully in context-sensitive

applications:
In sentiment analysis, removing words like "not" or "never" can change the
meaning of a sentence ("not good" vs. "good").
In domain-specific applications, such as medical or legal NLP, common words
may hold significant meaning and should not be removed.

Conclusion
Text normalization is an essential step in NLP, ensuring that raw text is clean,
structured, and optimized for analysis. Cleaning text, removing special
characters, and eliminating stopwords help improve data quality, model
performance, and computational efficiency. By carefully selecting which
elements to retain or remove, NLP systems can achieve better accuracy in
applications like search engines, speech recognition, and AI-driven text
processing.

Correcting words using stemming and Lemmatization

Correcting words is an essential step in text normalization, helping to standardize
variations of words to their base form. This process is important for improving
text analysis, search engine results, and NLP applications like chatbots and
sentiment analysis. Two key techniques used for this are stemming and
lemmatization. While both methods reduce words to their root form, they differ
in accuracy and linguistic correctness.

Stemming
Stemming is the process of reducing a word to its root stem by removing prefixes
and suffixes. The resulting stem may not always be a valid dictionary word, as it
follows rule-based truncation rather than linguistic correctness. Stemming is
useful for information retrieval, text classification, and clustering, where slight
variations of a word can be treated as the same term.

Popular stemming algorithms include:

• Porter Stemmer – A widely used algorithm that applies five phases of suffix
stripping to remove common endings (e.g., jumping → jump).
• Lancaster Stemmer – A more aggressive stemmer that removes larger
portions of words but may over-stem (e.g., lying → ly).
 • Snowball Stemmer – An improved version of the Porter Stemmer,
supporting multiple languages.
• Regexp Stemmer – Uses regular expressions to remove specific word
endings based on predefined rules.
Example of Stemming:

from [Link] import PorterStemmer

ps = PorterStemmer()
print([Link]('running'))# Output: run
print([Link]('jumped')) # Output: jump
print([Link]('strange'))# Output: strang

While stemming is computationally efficient, it can sometimes produce incorrect

stems (strange → strang), which can lead to loss of meaning in NLP applications.

Lemmatization
Lemmatization is a more advanced approach that reduces words to their base
dictionary form (lemma) by considering word meaning, context, and part of
speech (POS). Unlike stemming, lemmatization always returns valid words,
making it more suitable for tasks like machine translation and sentiment analysis.

The WordNet Lemmatizer in NLTK uses semantic analysis and POS tagging to
find the correct root form of a word. If the POS is incorrect, lemmatization may
not be effective.
Example of Lemmatization:

from [Link] import WordNetLemmatizer

wnl = WordNetLemmatizer()
print([Link]('running', 'v'))# Output: run
print([Link]('ate', 'v'))# Output: eat
print([Link]('cars', 'n')) # Output: car
print([Link]('fancier', 'a'))# Output: fancy

If no matching lemma is found in the WordNet corpus, the input word remains
unchanged.
Stemming vs. Lemmatization
Feature Stemming Lemmatization
Definition Removes prefixes and Converts words to their base
suffixes to get the root form dictionary form (lemma)
Method Uses rule-based truncation Uses linguistic rules and dictionary
lookup
Output May not be a real word Always returns a valid word (e.g.,
(e.g., strange → strang) running → run)
Context Does not consider word Considers word meaning and part of
meaning or POS speech
Speed Faster (simple rules) Slower (requires dictionary lookup)
Use Cases Search engines, keyword Machine translation, sentiment
extraction, text clustering analysis, named entity recognition

Conclusion
Both stemming and lemmatization play crucial roles in text preprocessing, but
lemmatization is more accurate due to its reliance on word semantics and
dictionary validation. However, stemming is faster and more suitable for large-
scale NLP tasks like search indexing and text classification. Choosing between
them depends on the specific NLP application and accuracy requirements.

Understanding text syntax and structure (POS tagging and Parsing)

Introduction to POS Tagging
Parts of Speech (POS) tagging is a fundamental NLP technique that assigns
grammatical labels to words based on their role in a sentence. It helps in
understanding language structure and is widely used in applications such as text
analysis, sentiment detection, and machine translation. POS tagging categorizes
words into predefined classes such as nouns, verbs, adjectives, and adverbs. The
Penn Treebank Tagset is one of the most commonly used tagging schemes,
providing detailed linguistic annotations for each word.

Recommended POS Taggers

Several pre-built POS taggers are available to simplify text processing. The
NLTK pos_tag() function is a widely used method that applies POS tags using
pre-trained models. Additionally, the pattern module offers another way to tag
words efficiently. These tools are useful for quick and reliable tagging without
requiring manual training.

Building Custom POS Taggers

For greater flexibility and improved accuracy, custom POS taggers can be
developed using NLTK’s built-in taggers. A DefaultTagger assigns a single
POS tag (such as ‘NN’ for nouns) to all words, serving as a baseline model. More
advanced approaches include:
• RegexpTagger – Uses regular expressions to identify and tag words based
on predefined patterns.
• N-gram Taggers – Unigram, Bigram, and Trigram taggers analyze word
sequences to predict POS based on context. A Unigram tagger tags words
based on their most frequent occurrence, while Bigram and Trigram
taggers consider adjacent words for better accuracy.
Combining Multiple Taggers for Better Accuracy
To improve tagging precision, multiple taggers can be combined using a backoff
approach. A combined tagger starts with a trigram tagger, which falls back to a
bigram tagger, then to a unigram tagger, and finally to a regex-based tagger if
previous models fail. This hierarchical approach increases accuracy and ensures
that unknown words are still assigned reasonable tags.

Classifier-Based POS Tagging (Machine Learning Approach)

A more sophisticated approach involves training a machine learning classifier for
POS tagging. The ClassifierBasedPOSTagger in NLTK allows training
with algorithms like Naïve Bayes or Maximum Entropy. By analyzing linguistic
features such as word context, sentence structure, and grammatical rules,
classifier-based taggers achieve high accuracy, often exceeding 93% on large
datasets.

Conclusion
POS tagging plays a critical role in NLP by providing structural insights into text
data. While pre-trained taggers offer quick solutions, custom models and machine
learning-based approaches deliver higher accuracy and adaptability. The best
approach depends on the specific NLP task, the required accuracy, and the
available training data.

Understanding Text Syntax and Structure

Introduction to Syntax and Structure
Text syntax and structure define the rules and organization of words, phrases, and
sentences in a language. Syntax refers to grammatical correctness and word
arrangement, while structure ensures clarity and coherence in communication.
Understanding syntax and structure is crucial in natural language processing
(NLP) for analyzing and processing textual data effectively.

Key Techniques for Analyzing Syntax and Structure

To analyze text syntax and structure, several NLP techniques are used:
• Parts of Speech (POS) Tagging: Identifies the grammatical categories
(nouns, verbs, adjectives, etc.) of words in a sentence.
• Shallow Parsing (Chunking): Groups words into phrases (noun phrases,
verb phrases) without full sentence parsing.
• Dependency Parsing: Examines the grammatical relationships between
words, showing which words depend on others.
• Constituency Parsing: Breaks a sentence into hierarchical structures (like
subject, predicate) to analyze the grammatical structure.
Implementation and Practical Use
These techniques are implemented using libraries like NLTK and spaCy, which
provide efficient tools for analyzing text. These methods are widely applied in
chatbots, text analytics, grammar checkers, and search engines.
Understanding text syntax and structure is essential for improving machine
comprehension and language understanding. By leveraging POS tagging, parsing
techniques, and structured text processing, NLP models can extract meaningful
insights, improving the accuracy and effectiveness of language-based
applications.

Parts of Speech (POS) Tagging

Definition and Importance
Parts of Speech (POS) tagging is a fundamental Natural Language Processing
(NLP) technique that assigns grammatical categories such as nouns, verbs,
adjectives, and adverbs to words in a sentence based on their syntactic role. POS
tagging helps in text analysis, machine translation, sentiment analysis, and speech
recognition by providing structured information about words.

POS Tagging Using Penn Treebank

The Penn Treebank tagset is a widely used standard for POS tagging. It consists
of specific tags representing various word categories. For example, "NN" denotes
a singular noun (e.g., "dog"), "VBZ" represents a verb in third-person singular
form (e.g., "runs"), and "JJ" is used for adjectives (e.g., "quick"). These tags
provide clarity in sentence structure analysis.

POS Tagging Methods

1. Rule-Based Tagging: Uses predefined grammatical rules to assign POS
tags.
2. Statistical Tagging: Employs probabilistic models like Hidden Markov
Models (HMM) to determine POS tags based on word sequences.
3. Machine Learning-Based Tagging: Utilizes algorithms such as Naïve
Bayes, Decision Trees, and Deep Learning to predict word categories.
POS Tagging in Python (NLTK Library)
The Natural Language Toolkit (NLTK) provides an easy-to-use function
`pos_tag()` for POS tagging.
import nltk
[Link]('averaged_perceptron_tagger')
sentence = "The quick brown fox jumps over the lazy
dog."
tokens = nltk.word_tokenize(sentence)
tagged_sentence = nltk.pos_tag(tokens)
print(tagged_sentence)

Output:
[('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'),
('fox', 'NN'),
('jumps', 'VBZ'), ('over', 'IN'), ('the', 'DT'),
('lazy', 'JJ'), ('dog', 'NN')]
Each word in the sentence is assigned a POS tag, helping in better text
comprehension and processing.
Challenges in POS Tagging
• Ambiguity: Words like "run" can function as both a noun and a verb,
making accurate tagging challenging.
• Context Sensitivity: The meaning of a word changes based on sentence
structure.
• Proper Noun Recognition: Identifying names and places accurately is often
difficult.
POS tagging plays a crucial role in NLP applications by enhancing text
classification, machine translation, and AI-driven language models. By
leveraging rule-based, statistical, and machine learning approaches, modern POS
tagging techniques improve accuracy and efficiency in language processing tasks.

Parsing
Introduction to Parsing
Parsing is the process of analyzing the structure of a sentence to determine its
grammatical components and relationships. It is a fundamental step in Natural
Language Processing (NLP), helping in syntax analysis, machine translation, and
text understanding. Parsing techniques fall into two major categories: shallow
parsing and deep parsing (which includes dependency-based and constituency-
based parsing).

Shallow Parsing
Shallow parsing, also called chunking, involves breaking a sentence into
meaningful phrases without analyzing their internal structure. It focuses on
grouping words into noun phrases (NP), verb phrases (VP), and prepositional
phrases (PP) rather than performing a full syntactic analysis. Shallow parsing is
efficient and is commonly used for named entity recognition (NER) and
information extraction.

Dependency-Based Parsing
Dependency parsing examines relationships between words in a sentence based
on grammatical dependencies. Each word is assigned a head (root word) and
dependent words connected through directed edges, forming a dependency tree.
This technique is widely used in semantic analysis, machine translation, and
chatbot development. The labels on dependency edges define grammatical roles
such as subject, object, and modifier.

Constituency-Based Parsing
Constituency parsing represents sentence structure as a tree based on phrase-
based grammar rules. The sentence is divided into constituents (phrases) which
are recursively broken down into smaller components. This technique is based on
Context-Free Grammar (CFG) and is useful in machine translation and speech
recognition. Popular parsing algorithms include Recursive Descent, Shift
Reduce, and Probabilistic Context-Free Grammar (PCFG).

Parsing is crucial in linguistic analysis, speech processing, and AI-driven text

understanding. While shallow parsing provides quick phrase segmentation, deep
parsing (dependency and constituency-based) helps in understanding complex
syntactic relationships. Modern NLP systems often combine both methods to
achieve better accuracy in text processing applications.
UNIT – III
Concepts of feature extraction, Methods of Feature extraction (Bag of words
Model, TF-IDF Models, Advanced word Factorization Models likes Word2vec),
Strengths and weakness of models and Word cloud. etc, Concepts of Document
term matrix, Term Document Matrix
Concepts of Feature Extraction
Feature extraction is a crucial step in natural language processing (NLP) that
converts unstructured text data into numerical representations that machine
learning (ML) models can process. Since ML algorithms work with mathematical
operations, raw text must be transformed into structured features while preserving
its meaning and context.

The goal of feature extraction is to capture meaningful patterns, relationships, and

distinctions within the data to improve the performance of ML models. Features
can be simple, like word counts, or more complex, like dense word embeddings
that capture semantic relationships between words. There are multiple methods
for text feature extraction, ranging from basic frequency-based approaches to
advanced neural network-based vectorization techniques.

Methods of Feature Extraction

1. Bag of Words (BoW) Model
The Bag of Words (BoW) model is one of the simplest text feature
extraction techniques. It represents a document as a vector of word
occurrences, disregarding grammar and word order. The BoW model
works by:
1. Creating a vocabulary from all unique words in the dataset.
2. Representing each document as a numerical vector where each word's
value corresponds to its frequency in that document.

For example, given a corpus:

['the sky is blue',
'sky is blue and beautiful',
'the beautiful sky is so blue']
The vocabulary could be:
`['the', 'sky', 'is', 'blue', 'and', 'beautiful', 'so']`

Each document is then represented as a vector of word counts.

 Key Characteristics of BoW:
• Simple and easy to implement.
• Ignores word order, semantics, and context.
• Can be extended to n-grams to capture word sequences.
Advantages of BoW:
• Works well for text classification and document clustering.
• Straightforward and computationally efficient.
Limitations of BoW:
• Produces high-dimensional sparse vectors.
• Assigns equal importance to all words, even common ones like "the"
and "is."
• Does not capture relationships between words.

2. TF-IDF (Term Frequency-Inverse Document Frequency) Model

The TF-IDF model improves upon BoW by assigning weights to words
based on their importance within a document relative to the entire corpus.
It reduces the influence of commonly occurring words and highlights more
distinctive terms.

Components of TF-IDF:
1. Term Frequency (TF): Measures how often a word appears in a
document.
Frequency of word w in document D
𝑇𝐹(𝑤, 𝐷) =
Total words in document D

2. Inverse Document Frequency (IDF): Reduces the weight of words that

appear frequently across many documents.
Total number of documents + 1
𝐼𝐷𝐹(𝑤) = 1 + log ( )
Number of documents containing word w + 1
3. TF-IDF Score: The final score is computed by multiplying TF and IDF.
𝑇𝐹 − 𝐼𝐷𝐹(𝑤, 𝐷) = 𝑇𝐹(𝑤, 𝐷) × 𝐼𝐷𝐹(𝑤)
 Key Characteristics of TF-IDF:
• Words that are frequent in a document but rare in the corpus get
higher weights.
• Common words (e.g., "the", "is") have lower weights, reducing their
influence.
• Helps in keyword extraction, search ranking, and text classification.

Advantages of TF-IDF:
• Highlights important words while filtering out less meaningful ones.
• More effective than BoW in capturing document-specific
information.

Limitations of TF-IDF:
• Still ignores word order and relationships.
• Less effective when words have multiple meanings (polysemy).
• Does not capture deep semantic meaning.

3. Advanced Word Vectorization Models

Advanced feature extraction techniques go beyond frequency-based
approaches to capture the semantic meaning of words using word
embeddings. One of the most popular models for this is word2vec,
developed by Google in 2013.
Word2Vec Model:
• Uses neural networks to learn dense vector representations of
words.
• Captures word relationships, allowing words with similar
meanings to have similar vector representations.
Works based on two architectures:
• Continuous Bag of Words (CBOW): Predicts a word based on its
surrounding words.
• Skip-Gram: Predicts surrounding words given a single word.
For example, word2vec can capture relationships like:
king man + woman = queen
This means that the model understands the analogy between gendered
terms based on vector relationships.

Key Parameters in Word2Vec:

• Size: Defines the dimensionality of word vectors (e.g., 100, 300).
• Window: Determines the number of surrounding words to consider
for context.
• Min_count: Filters out rare words to reduce noise.
• Sample: Downsamples high-frequency words to prevent bias.

Feature Extraction Techniques Using Word Vectors:

Once a word2vec model is trained, document-level feature vectors can be
obtained using:
1. Averaged Word Vectors: Computes the mean of all word vectors
in a document to create a single vector representation.
2. TF-IDF Weighted Word Vectors: Assigns weights to word vectors
based on their TF-IDF scores, giving more importance to rare but
meaningful words.
Advantages of Word2Vec:
• Captures word meaning, relationships, and context better than BoW
and TF-IDF.
• Enables semantic reasoning in NLP tasks.
• Reduces dimensionality, making computations more efficient.
Limitations of Word2Vec:
• Requires large amounts of training data to produce high-quality
embeddings.
• Can be computationally expensive to train.
• Struggles with out-of-vocabulary (OOV) words that were not seen
during training.
Comparison of Feature Extraction Methods
Feature Bag of Words TF-IDF Word2Vec
(BoW)
Word Uses raw word Uses weighted Uses dense vector
Representation frequency word frequency embeddings
Captures Word No No Partially, through
Order? context
Handles Common No, treats all Yes, reduces Yes, understands
Words? words equally weight of semantic
common words relationships
Captures Word No, based only on Yes, based on Yes, captures
Importance? frequency rarity and word meaning and
importance analogies
Sparse Yes, large feature Yes, but more No, uses dense,
Representation? vectors refined lower-
dimensional
vectors
Computational Low Medium High
Complexity
Use Cases Text Search ranking, Sentiment
classification, keyword analysis, machine
clustering extraction translation

Conclusion
Feature extraction is a fundamental step in NLP, converting raw text into
numerical representations for ML models. The Bag of Words (BoW) model is
simple and effective for basic tasks but ignores meaning. TF-IDF improves on
BoW by weighting words based on importance but still lacks semantic
understanding. Advanced word vectorization models like word2vec capture
deeper meaning and relationships, making them suitable for tasks requiring high-
level language understanding. Depending on the application, choosing the right
feature extraction method is crucial for achieving optimal results in text-based
ML tasks.
Word Cloud
A word cloud is a text visualization technique where word size represents
frequency in a dataset. It helps identify key themes in documents, reviews, and
social media.

How Word Clouds Work

1. Text Preprocessing – Remove punctuation, special symbols, and stopwords.
2. Word Frequency Calculation – Count occurrences of each word.
3. Visualization – Display words with larger text for higher frequency.

Creating Word Clouds in R

Using the `wordcloud` package:
library(wordcloud)
tidy_books %>%
anti_join(stop_words) %>%
count(word) %>%
with(wordcloud(word, n, [Link] = 100))
For sentiment-based word clouds:
library(reshape2)
tidy_books %>%
inner_join(get_sentiments("bing")) %>%
count(word, sentiment) %>%
acast(word ~ sentiment, [Link] = "n", fill = 0) %>%
[Link](colors = c("gray20", "gray80"), [Link] = 100)

Advantages
• Quick insights into frequent words.
• Easy interpretation with customizable visuals.
• Sentiment analysis by categorizing positive and negative words.
Limitations
• Lacks context – word relationships are ignored.
• Size misinterpretation – not all large words are important.
• Sentiment clouds – word sizes are not comparable across categories.

Applications
• Text mining (books, articles)
• Social media monitoring (tweets, reviews)
• Market research (customer feedback)
• News & political analysis

Conclusion
Word clouds are a simple, effective way to visualize word importance but should
be used with other text analysis techniques for deeper insights.

Document-Term Matrix (DTM)

A Document-Term Matrix (DTM) is a structured representation of text data where
rows correspond to documents, columns correspond to terms (words), and each
cell contains the frequency of a word in a given document. It is widely used in
natural language processing (NLP), text mining, and topic modeling.

Since most words do not appear in every document, a DTM is typically sparse,
meaning a large portion of the matrix contains zeros.

Steps to Create a DTM in R

1. Preprocessing the Text
• Remove unnecessary words, punctuation, and stopwords.
• Convert text to lowercase.
• Tokenize words (split text into individual words).
2. Counting Word Frequencies
• Calculate the number of times each word appears in each document.
3. Casting to a Document-Term Matrix
• Convert the structured word count data into a sparse matrix format using
`cast_dtm()`.

Example in R
library(tidytext)
library(dplyr)

Remove stopwords and count word occurrences

word_counts <- nasa_desc %>%
anti_join(my_stop_words) %>%
count(id, word, sort = TRUE) %>%
ungroup()

Convert to Document-Term Matrix

desc_dtm <- word_counts %>%
cast_dtm(id, word, n)

This creates a matrix where:

• Rows represent NASA descriptions (documents).
• Columns represent unique words (terms).
• Values indicate the frequency of a word in a given document.

Key Characteristics of DTM

• Sparse Representation – Since most words do not appear in every
document, the matrix contains many zeros.
• Used for Topic Modeling – DTMs are essential in Latent Dirichlet
Allocation (LDA) for discovering hidden topics.
 • Term Frequency-Based – The values in the matrix typically represent raw
word counts but can be adjusted using TF-IDF for improved weighting.
• Efficient for NLP Tasks – Used in text classification, clustering, and
sentiment analysis.
Conclusion
A Document-Term Matrix is a fundamental tool in text analysis, converting raw
text into a numerical format for machine learning models. It enables tasks like
topic modeling, document clustering, and sentiment analysis, making it essential
in NLP and text mining applications.
UNIT – IV
Concepts of Topic Modelling, Algorithms of Topic Modelling (Latent Semantic
Indexing (LSI), Latent Dirichlet Allocation (LDA), Non-Negative Matrix
Factorization (NMF) and Similarity based text clustering models), Text
Classification using supervised methods (Like Multinomial Naïve Bayes,
Support vector machines, Random Forest …), concept of Sentiment Analysis and
its applications. Sentence Subjectivity and Sentiment Classification; Sentiment
Lexicon generation and Summarization
Concepts of Topic Modeling
Topic modeling is an unsupervised machine learning technique used to uncover
hidden themes or topics in a large collection of documents. Unlike keyword
extraction, which simply identifies frequently occurring words, topic modeling
groups words into coherent topics based on their statistical co-occurrence. These
topics help in understanding the overall structure of a corpus, categorizing
documents, summarizing content, and improving search and recommendation
systems.

Topic models assume that each document is a mixture of topics and that each
topic consists of a set of words with different probabilities. This technique is
widely used in applications such as:
• Document clustering: Grouping similar documents together.
• Recommendation systems: Suggesting content based on shared topics.
• Sentiment analysis: Identifying themes in customer feedback.
• Search engines: Enhancing search relevance by understanding document
themes.

There are several algorithms for topic modeling, each with different approaches
and strengths. The three most widely used techniques are Latent Semantic
Indexing (LSI), Latent Dirichlet Allocation (LDA), and Non-Negative Matrix
Factorization (NMF).

Algorithms of Topic Modeling

1. Latent Semantic Indexing (LSI)
Latent Semantic Indexing (LSI), also known as Latent Semantic Analysis
(LSA), is one of the earliest topic modeling techniques. It is based on
Singular Value Decomposition (SVD), a matrix factorization method that
reduces the dimensionality of a document-term matrix to identify latent
patterns in word usage.

How LSI Works:

1. Convert text into a document-term matrix (DTM):
 • Rows represent documents, columns represent words, and values
indicate term frequencies.
2. Apply Singular Value Decomposition (SVD):
• SVD decomposes the matrix into three smaller matrices:
𝐴 = 𝑈 ⋅ 𝑆 ⋅ 𝑉𝑇
where:
• U represents documents in a reduced topic space.
• S contains singular values that capture the importance of topics.
• V represents terms in the topic space.
3. Reduce dimensions:
• By keeping only the top k singular values, LSI captures the most
important patterns in the data.
4. Extract topics:
• Topics are identified based on word associations in the reduced
space.

Advantages of LSI:
• Captures synonymy (similar words in the same context), improving
search accuracy.
• Useful for information retrieval, search engines, and text
summarization.

Disadvantages of LSI:
• Computationally expensive, especially for large datasets.
• Does not handle polysemy (words with multiple meanings)
effectively.
• Lacks a probabilistic framework, making interpretation harder.

2. Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation (LDA) is a generative probabilistic model that
assumes each document is a mixture of topics and that each topic is a
distribution of words. Unlike LSI, LDA explicitly models the probability
of topics within each document, making it more interpretable.

How LDA Works:

1. Initialize parameters:
• Define the number of topics K and assume a Dirichlet prior
distribution over them.
2. Assign words randomly to topics.
3. Iterate using Gibbs sampling:
• For each word w in each document d, update topic assignment T
based on:
• P(T | D): Proportion of words in document D assigned to topic T.
• P(W | T): Proportion of times word W is assigned to topic T across
all documents.
• This step refines topic assignments iteratively.
4. Generate final topics:
• Topics are represented by the most frequently occurring words.

Advantages of LDA:
• Handles polysemy well (assigns words different meanings based on
context).
• Provides a probabilistic interpretation of topics.
• Works well with large-scale text corpora.

Disadvantages of LDA:
• Requires careful hyperparameter tuning (e.g., number of topics).
• Computationally intensive for very large datasets.
• The topics may not always be coherent, requiring manual
interpretation.
3. Non-Negative Matrix Factorization (NMF)
Non-Negative Matrix Factorization (NMF) is a matrix factorization
technique that decomposes a document-term matrix into two lower-rank
matrices. Unlike LSI, which allows negative values, NMF constrains all
values to be non-negative, ensuring that topics are additive and easier to
interpret.

How NMF Works:

1. Convert text into a document-term matrix (DTM).
2. Factorize the matrix into two smaller matrices:
𝑉 ≈𝑊 ⋅𝐻
where:
• V is the original document-term matrix.
• W is the document-topic matrix (indicates the importance of topics
per document).
• H is the topic-term matrix (indicates the importance of words per
topic).
3. Optimize using a cost function (e.g., Frobenius norm):
• Minimize the difference between V and WH using iterative
optimization methods.
4. Extract topics:
• Topics are identified from the H matrix, which contains word
distributions.

Advantages of NMF:
• Produces more coherent and interpretable topics compared to LSI
and LDA.
• Works well even on small datasets.
• No negative values, making results easy to understand.
 Disadvantages of NMF:
• Requires choosing the right number of topics.
• Sensitive to initialization (different runs may give different results).
• Works best when datasets are preprocessed well (removing stop
words, stemming, etc.).

Comparison of LSI, LDA, and NMF

Algorithm Technique Strengths Weaknesses Best Used For
LSI (Latent Matrix Captures Computational Information
Semantic factorization hidden ly expensive, retrieval, search
Indexing) (SVD) semantic does not engines, text
structures, handle summarization
improves polysemy well
search
engines
LDA (Latent Probabilistic Handles Requires Document
Dirichlet model polysemy, hyperparamete classification,
Allocation) assigns r tuning, recommendatio
words computationall n systems, topic
probabilistic y intensive discovery
ally to topics
NMF (Non- Matrix Produces Sensitive to Text mining,
Negative factorization interpretable initialization, bioinformatics,
Matrix with non- topics, requires document
Factorization negative works well preprocessed clustering
) constraints for small data
datasets

Choosing the Right Algorithm:

• Use LSI if you need fast topic extraction and work with search engines.
• Use LDA if you need clear probabilistic topic assignments and work with
large corpora.
• Use NMF if you need interpretable, distinct topics, especially for small
datasets.
Each method has its own strengths and is suitable for different tasks. The choice
depends on dataset size, computational resources, and the need for
interpretability.

Text Classification Using Supervised Methods

Text classification is a supervised learning task where a model learns from labeled
training data to assign new text data to predefined categories. This process
involves:
1. Training – The model learns from labeled text examples by identifying
patterns in word distributions.
2. Evaluation – The model's performance is tested on unseen data using
metrics like accuracy, precision, and recall.
3. Tuning – Hyperparameters are optimized to improve model accuracy and
reduce errors.
Two widely used supervised classification techniques in Natural Language
Processing (NLP) are Multinomial Naïve Bayes (MNB) and Support Vector
Machines (SVM). These algorithms are effective for text classification tasks such
as spam filtering, sentiment analysis, document categorization, and news
classification.

1. Multinomial Naïve Bayes (MNB)

Multinomial Naïve Bayes (MNB) is a probabilistic classifier based on
Bayes' theorem, which calculates the probability of a text document
belonging to a particular class given its word distribution. It assumes that
words in a document are conditionally independent of each other (the
“naïve” assumption).

Mathematical Formulation
Using Bayes' Theorem, the probability of a document belonging to a class
Ck is given by:
𝑛

𝑃(𝐶𝑘 |𝑋) ∝ 𝑃(𝐶𝑘 ) ∏ 𝑃(𝑥𝑖 |𝐶𝑘 )

𝑖=1
 where:
• (𝑃(𝐶𝑘 )) is the prior probability of class (𝐶𝑘 ).
• (𝑃(𝑥𝑖 |𝐶𝑘 )) is the likelihood of word (𝑥𝑖 ) occurring in class (𝐶𝑘 ).
• ( 𝑋 ) represents a document with multiple words.

The Multinomial Naïve Bayes classifier extends this formula by

considering word frequencies (or TF-IDF scores), making it ideal for text
classification.

Advantages of MNB:
• Fast and efficient, even with large datasets.
• Works well with word count-based features (Bag of Words, TF-
IDF).
• Performs well with limited training data.

Limitations of MNB:
• Assumes word independence, which is not always true.
• Struggles with complex language structures and long phrases.
• Cannot handle negative feature values (e.g., word embeddings).

Use Cases of MNB:

• Spam classification (e.g., detecting spam emails).
• Sentiment analysis (e.g., positive vs. negative movie reviews).
• News categorization (e.g., sports vs. politics articles).

2. Support Vector Machines (SVM)

Support Vector Machines (SVM) is a powerful classification algorithm that
finds an optimal hyperplane to separate different classes in high-
dimensional space. Unlike probabilistic classifiers like Naïve Bayes, SVM
is a discriminative model that directly finds decision boundaries between
classes.
Mathematical Formulation
For a binary classification problem, the decision boundary is defined by a
hyperplane:
𝑤 ⋅𝑥 + 𝑏 = 0
where:
• ( 𝑤 ) is the weight vector.
• ( 𝑥 ) is the feature vector.
• ( 𝑏 ) is the bias term.

The algorithm aims to maximize the margin between the closest points
(support vectors) of each class. The larger the margin, the better the
generalization.
For non-linearly separable data, SVM uses a kernel trick (e.g., polynomial
or RBF kernels) to transform data into higher dimensions, making it
separable.

Advantages of SVM:
• Works well with high-dimensional text data.
• Effective for both linear and non-linear classification.
• Less prone to overfitting, unlike decision trees.

Limitations of SVM:
• Computationally expensive for large datasets.
• Requires careful tuning of kernel parameters.
• Difficult to interpret compared to probabilistic models like Naïve
Bayes.

Use Cases of SVM:

• Document classification (e.g., categorizing research papers).
• Fake news detection (e.g., identifying misleading articles).
• Opinion mining (e.g., classifying user reviews as positive/negative).
Comparison of MNB and SVM
Feature Multinomial Naïve Support Vector Machines
Bayes (MNB) (SVM)
Type Probabilistic model Discriminative model
Works best with Word frequency-based High-dimensional word
features (TF-IDF, embeddings
BoW)
Performance on small Good (less prone to Needs sufficient training
data overfitting) data
Speed Fast and efficient Computationally
expensive
Handles non-linearity? No Yes (with kernel trick)
Best suited for Spam filtering, news Complex NLP tasks, fake
categorization news detection

Choosing the Right Model

• Use Multinomial Naïve Bayes if you have text-based categorical features
(e.g., word counts, TF-IDF) and need fast, scalable classification.
• Use Support Vector Machines if you have high-dimensional data (e.g.,
embeddings, word vectors) and require better accuracy in complex NLP
tasks.
Both models are widely used in text classification, sentiment analysis, and NLP
applications, with their effectiveness depending on dataset size, complexity, and
feature representation.

Concept of Sentiment Analysis

Sentiment analysis, also known as opinion mining, is the process of analyzing
people’s opinions, emotions, attitudes, and sentiments toward various entities,
such as products, services, events, or political figures. It is an NLP (Natural
Language Processing) task that determines whether a given text expresses a
positive, negative, or neutral sentiment.
The field of sentiment analysis has gained prominence due to the rise of social
media, where vast amounts of user-generated opinions are shared daily.
Researchers and businesses use sentiment analysis to extract meaningful insights
from reviews, social media posts, blogs, and news articles. It plays a crucial role
in understanding customer opinions, monitoring brand reputation, and assessing
public sentiment in politics, finance, and social events.

Applications of Sentiment Analysis

1. Business and Market Analysis
• Companies analyze customer reviews and feedback to improve products
and services.
• Sentiment analysis helps in brand reputation management by monitoring
social media mentions.
• Businesses predict sales trends based on consumer sentiment toward
products.

2. Social Media Monitoring

• Governments and organizations track public sentiment on social issues and
policies.
• Sentiment analysis helps detect emerging trends and measure public
reactions to events.
• Social media platforms use it to filter inappropriate content and detect hate
speech.

3. Political Analysis and Election Predictions

• Sentiment analysis is used to analyze public opinions on political
candidates before elections.
• Researchers link social media sentiments (e.g., Twitter, Facebook) with
election outcomes.
• Politicians and policymakers assess public response to policies and
speeches.
4. Stock Market and Financial Analysis
• Sentiment analysis of news articles, tweets, and blogs helps predict stock
market trends.
• Investors analyze market sentiment before making investment decisions.

5. Healthcare and Customer Support

• Sentiment analysis is applied to patient feedback and health forums to
improve healthcare services.
• Chatbots and virtual assistants use sentiment analysis to understand
customer emotions and improve interactions.

6. Entertainment Industry and Movie Reviews

• Film studios and producers analyze audience reactions to predict box office
performance.
• Music and TV streaming platforms recommend content based on user
sentiment toward genres and artists.

Sentiment analysis has expanded across industries, helping businesses,

governments, and researchers make data-driven decisions by extracting insights
from textual data.

Sentence Subjectivity and Sentiment Classification

Traditional document-level sentiment classification is often too broad, making
sentence-level classification more effective for certain applications. The goal is
to determine whether each sentence expresses a positive, negative, or neutral
sentiment. This process can be approached through three-class classification
(positive, negative, neutral) or a two-step classification, where a sentence is first
classified as opinionated (subjective) or not, and then assigned a sentiment label.

Sentence Subjectivity Classification

Subjectivity classification distinguishes between subjective sentences
(expressing opinions, emotions, or beliefs) and objective sentences (stating
factual information). Early approaches used supervised learning methods like
Naïve Bayes and bootstrapping to classify sentences. Some objective sentences,
however, can imply sentiment, such as *"Then, it stopped working yesterday."*
Advanced classification models incorporate lexical features, linguistic patterns,
and deep learning to improve accuracy.

Sentence Sentiment Classification

Once a sentence is classified as subjective, the next step is to determine whether
it expresses positive or negative sentiment. Early methods relied on lexicon-based
approaches, assigning sentiment scores to words, while more advanced methods
used machine learning techniques like Naïve Bayes, Conditional Random Fields
(CRF), and deep learning. The challenge arises in compound and complex
sentences, where different parts of a sentence may express conflicting sentiments.

Dealing with Conditional Sentences

Conditional sentences express hypothetical scenarios, making sentiment analysis
difficult. For example, *"If someone makes a reliable car, I will buy it."* contains
the positive word *"reliable"*, but does not express an actual opinion. Specialized
supervised learning methods analyze sentence structure, conditional words (if,
unless), and linguistic patterns to improve sentiment classification in such cases.

Dealing with Sarcastic Sentences

Sarcasm detection is a major challenge in sentiment analysis since sarcastic
statements often mean the opposite of their literal words. Studies have used
pattern-based learning, lexical markers (exclamations, punctuation), and machine
learning techniques like SVM and logistic regression to detect sarcasm, though
accuracy remains low. Future research aims to leverage contextual deep learning
models to improve sarcasm detection.

Cross-Language Subjectivity and Sentiment Classification

Since sentiment analysis resources are mostly in English, cross-language
sentiment classification is important for non-English languages. Approaches
include translating text into English for classification, training classifiers in the
target language, or using multilingual sentiment lexicons. Translation quality
significantly impacts accuracy, and domain adaptation techniques are being
explored to improve cross-language sentiment classification.

Using Discourse Information for Sentiment Classification

Most sentiment classifiers analyze individual sentences without considering
discourse structure. However, sentiment is often influenced by rhetorical
relations, contrast words, and contextual dependencies. Advanced methods use
graph-based models, discourse parsing, and Markov Logic Networks (MLN) to
integrate sentence relationships for more accurate sentiment classification.

Conclusion
Sentence-level sentiment classification is crucial for fine-grained sentiment
analysis, especially in complex texts. Challenges such as sarcasm, conditional
statements, and discourse structure require advanced machine learning
techniques and contextual analysis. Future research will focus on deep learning
models and linguistic context integration to enhance sentiment classification
accuracy.

Sentiment Lexicon Generation

A sentiment lexicon is a collection of words and phrases that express positive or
negative sentiment. These words, also known as opinion words, help in sentiment
analysis by categorizing text into emotional tones. Sentiment lexicons include not
only individual words (e.g., beautiful, terrible) but also phrases and idioms (e.g.,
cost an arm and a leg). Words can be classified as base-type (expressing direct
sentiment) or comparative-type (indicating relative comparison, e.g., better,
worse).

There are three primary methods to compile a sentiment lexicon:

1. Manual Approach – Accurate but time-consuming.
2. Dictionary-Based Approach – Expands words using synonyms and
antonyms from dictionaries.
3. Corpus-Based Approach – Extracts sentiment words based on patterns and
statistical relationships in large text datasets.
Since automated methods may introduce errors, manual verification is often
necessary to improve lexicon accuracy.

Dictionary-Based Approach
The dictionary-based approach builds a sentiment lexicon by expanding an initial
set of words using WordNet or similar resources. It starts with a seed list of
positive and negative words and iteratively finds related words until no new
words are discovered.

To improve accuracy, researchers introduced graph-based methods and statistical

techniques:

• Kamps et al. (2004) measured word sentiment based on WordNet distance

from reference words like *good* and *bad*.
• Blair-Goldensohn et al. (2008) refined the approach using a neutral seed
set to prevent errors.
• Rao & Ravichandran (2009) applied semi-supervised learning to enhance
sentiment word classification.
• Turney & Littman (2003) used PMI (Pointwise Mutual Information) to
determine a word’s sentiment by measuring its association with positive or
negative reference words.

Although dictionary-based methods are effective, they struggle with domain-

specific sentiment variations. Words like *quiet* can be positive for a car but
negative for a speakerphone. To address this, corpus-based methods are often
used alongside dictionary-based approaches.

Corpus-Based Approach
The corpus-based approach adapts sentiment lexicons to specific domains by
analyzing word usage in text corpora. This is crucial because sentiment words
can change meaning depending on context.
Early research used linguistic rules to extract sentiment words:
• Hatzivassiloglou & McKeown (1997) found that adjectives linked by
*AND* often share the same sentiment, while *BUT* signals sentiment
shifts.
• Kanayama & Nasukawa (2006) analyzed sentiment propagation across
sentences to identify domain-specific sentiment words.
• Ding, Liu & Yu (2008) introduced aspect-based sentiment analysis,
recognizing that words like *long* can be positive (battery life is long) or
negative (takes a long time to focus).

Other methods used machine learning to refine sentiment classification:

• Wilson, Wiebe & Hoffmann (2005) developed context-aware sentiment
classifiers.
• Du et al. (2010) proposed domain adaptation using mutual reinforcement
learning, aligning sentiment words between different domains.
Although corpus-based approaches improve domain specificity, they require
large labeled datasets and complex linguistic rules, making them computationally
intensive.

Desirable and Undesirable Facts in Sentiment Analysis

Not all sentiment words are explicitly positive or negative; some objective words
imply sentiment depending on context. For example, in mattress reviews, valley
is negative (e.g., A valley formed in the mattress), even though it is neutral in
general use.
To address implied sentiment detection, Zhang & Liu (2011b) developed a two-
step identification process:
1. Candidate Identification – Analyzing word sentiment context to infer
sentiment orientation.
2. Pruning – Removing words with inconsistent sentiment associations.

Dependency relations were used to analyze sentence structures, identifying

implicit sentiment-bearing words. While this method showed promise, accuracy
remains a challenge, requiring further research to improve context-dependent
sentiment detection.