ADMIRALTY UNIVERSITY OF NIGERIA

IBUSA, DELTA STATE - NIGERIA

FACULTY OF SCIENCE
Department of COMPUTING SCIENCE
(SEN 207- CYBER & SOFT ENG)
[Link](FULLTIME)
(200L – I SEM)

2025-26
NATURAL LANGUAGE PROCESSING WITH PYTHON
(NLP WITH PYTHON)

LECTURE NOTES
Vision

To be a premier center for academic excellence and research through innovative interdisciplinary
collaborations and making significant contributions to the community, organizations, and society as a
whole.
(NLP) Unit-I
Natural Language Processing – Introduction
Objectives
The main objectives of this course are to:
1. Explain the underlying structure of natural language, linguistics, language syntax and structure;
2. know python syntax, structure, data structures and types;
3. Describe the various requirements for setting up a python application;
4. Provide basic knowledge of text tokenization, text normalization text syntax and structure. text
classification, text classification blueprint and text normalization;
5. Equip students with basic knowledge of feature extraction, classification algorithms and
evaluation classification models
6. Equip students with the basic knowledge of text summarization, information extraction and
feature matrix. single value decomposition;
7. Demonstrate the use of key phrase extraction and topic modelling;
8. Build a simple project for natural language processing in python;

Learning Outcomes
At the end of this course, the students should be able to:
1. Describe the main components of language structure;
2. Perform pre-processing (cleaning) operations on text;
3. Apply methods from Corpus Linguistics to garner greater insights on a corpus;
4. Produce word-clouds, bar charts and other basic visualizations on variables of interest;
5. Produce clusters using the k-means algorithm to uncover patterns in a corpus;
6. Transform text to vectors using approaches delineated;
7. Produce word embedding on a corpus;
8. Calculate the probability of a sentence using a language modelling approach.

Course Content
underlying structure of natural language, linguistics, language syntax and structure, Language Syntax
and Structure. Text Corpora and Natural Language Processing. Python Syntax, Structure, Data Structures
and Types. Control flow, Functional Programming and Classes. Regex in Python, Text Tokenization, Text
Normalization Text Syntax and Structure. Text Classification, Text Classification Blueprint and Text
Normalization. Feature Extraction, Classification Algorithms and Evaluation Classification Models. Text
Summarization, Information Extraction and Feature Matrix. Single Value Decomposition, Keyphrase
Extraction and Topic Modelling. Automated Document Summarization, Information Retrieval and
Feature Engineering. Text Similarity, Analyzing Term Similarity and Analyzing Document Similarity.
Document Clustering, Clustering Greatest Movies of All Time and Semantic Analysis. Exploring
WordNet, Word Sense Disambiguation, and Named Entity Recognition. Analyzing Semantic
Representation and Sentiment Analysis.
Introduction
✓ Humans communicate through some form of language either by text or speech.
✓ To make interactions between computers and humans, computers need to understand natural
languages used by humans.
✓ Natural language processing is all about making computers learn, understand, analyze, manipulate
and interpret natural(human) languages.
✓ NLP stands for Natural Language Processing, which is a part of Computer Science, Human
languages or Linguistics, and Artificial Intelligence.
✓ Processing of Natural Language is required when you want an intelligent system like robot to
perform as per your instructions, when you want to hear decision from a dialogue based clinical
expert system, etc.
✓ The ability of machines to interpret human language is now at the core of many applications that we
use every day - chatbots, Email classification and spam filters, search engines, grammar checkers,
voice assistants, and social language translators.
✓ The input and output of an NLP system can be Speech or Written Text.
Natural Language Processing
Natural Language: By natural language we mean a language that is used for everyday communication by
humans; languages like English, Hindi or Portuguese. In contrast to artificial languages such as
programming languages and logical formalisms, natural languages have evolved as they pass from
generation to generation, and are hard to pin down with explicit rules. We will take Natural Language
Processing (or NLP for short) in a wide sense to cover any kind of computer
manipulation of natural language. At one extreme, it could be as simple as counting the number of times
the letter t occurs in a paragraph of text. At the other extreme, NLP might involve “understanding”
complete human utterances, at least to the extent of being able to give useful responses to them. Most
human knowledge and most human communication are represented and expressed using language.
Technologies based on NLP are becoming increasingly widespread. For example, handheld computers
(PDAs) support predictive text and handwriting recognition; web search engines give access to
information locked up in unstructured text; machine translation allows us to retrieve texts written in
Chinese and read them in Spanish. By providing more natural human-machine interfaces, and more
sophisticated access to stored information, language processing has come to play a central role in the
multilingual information society.

Natural Language Processing

Natural Language Processing (NLP) is a branch of Artificial Intelligence (AI) that helps machines to
understand and process human languages either in text or audio form. It is used across a variety of
applications from speech recognition to language translation and text summarization.
Natural Language Processing can be categorized into two components:
1. Natural Language Understanding: It involves interpreting the meaning of the text.
2. Natural Language Generation: It involves generating human-like text based on processed data.
 NLP Tasks NLP Technique

NLP is used by many applications that use language, such as text translation, voice recognition, text
summarization and chatbots. You may have used some of these applications yourself, such as voice-
operated GPS systems, digital assistants, speech-to-text software and customer service bots. NLP also
helps businesses improve their efficiency, productivity and performance by simplifying complex tasks
that involve language.

Linguistics
In Natural Language Processing (NLP), linguistics is the foundational study of language structure that
provides the rules and theories computers use to understand, interpret, and generate human language. It
involves analyzing language at different levels, such as morphology (word structure), syntax (sentence
structure), semantics (meaning), and pragmatics (language in use), to build computational models and
develop technologies like translation apps, voice assistants, and sentiment analysis tools.
Key linguistic concepts in NLP
• Morphology: The study of word structure and formation. NLP uses this to understand how words
are built and to group words with similar roots. Morphology concerns the structure and meaning
of words. Some words, such as send, appear to be ‘atomic’ or monomorphemic others, such as
sends, sending, resend appear to be constructed from several atoms or morphemes. We know these
‘bits of words’ are morphemes because they crop up a lot in other words too – thinks, thinking,
reprogram, rethink.
• Syntax: The rules that govern how words are arranged to form sentences. NLP relies on syntax
to perform tasks like part-of-speech tagging and parsing to understand a sentence's grammatical
structure. Syntax concerns the way in which words can be combined together to form
(grammatical) sentences. (Linguists use asterisks to indicate ‘ungrammaticality’, or illegality
given the rules of a language.) Words combine syntactically in certain orders in a way which
mirrors the meaning conveyed; eg. John loves Mary means something different from Mary loves
John.

• Semantics: The study of meaning in language. NLP uses semantic analysis to understand the
meaning of words, phrases, and sentences, which is crucial for tasks like question answering and
machine translation. Semantics is about the manner in which lexical meaning is combined
morpho- logically and syntactically to form the meaning of a sentence. Mostly, this is regular,
 productive and rule governed; eg. the meaning of John gave Mary a dog can be represented as
(some (x) (dog x) & (past-time (give (john, mary, x)))), but sometimes it is idiomatic as in the
meaning of John kicked the bucket, which can be (past-time (die (john))). (To make this notation
useful we also need to know the meaning of these capitalised words and brackets too.)
• Phonetics and Phonology: Phonetics deals with the physical production and perception of
speech sounds, while phonology focuses on how sounds function within a language. This is
critical for speech recognition and text-to-speech systems. Phonology concerns the use of sounds
in a particular language. English makes use of about 45 phonemes – contrastive sounds, eg. /p/
and /b/ are contrastive because pat and bat mean different things. (Note the use of [x] for a phone
and /x/ for the related phoneme).
• Lexicon: The lexicon contains information about particular idiosyncratic properties of words; eg.
what sound or orthography goes with what meaning – pat or/pat/ means pat, irregular
morphological forms – sent (not sended), what part-of-speech a word is, eg. storm can be noun
or verb, semi-productive meaning extensions and relations, eg. many animal denoting nouns can
be used to refer to the edible flesh of the animal (chicken, haddock etc) but some can’t (easily)
cow, deer, pig etc., and so forth.
• Pragmatics: The study of how context influences the meaning of language. This is important for
advanced NLP tasks that require understanding the nuances of a conversation, such as intent
recognition. Pragmatics is about the use of language in context, where context includes both the
linguistic and situational context of an utterance; eg. if I say Draw the curtains in a situation where
the curtains are open this is likely to be a command to someone present to shut the curtains (and
vice versa if they are closed).

NLP Techniques
NLP encompasses a wide array of techniques that aimed at enabling computers to process and understand
human language. These tasks can be categorized into several broad areas, each addressing different
aspects of language processing. Here are some of the key NLP techniques:
1. Text Processing and Preprocessing
• Tokenization: Dividing text into smaller units, such as words or sentences.
• Stemming and Lemmatization: Reducing words to their base or root forms.
• Stopword Removal: Removing common words (like "and", "the", "is") that may not carry
significant meaning.
• Text Normalization: Standardizing text, including case normalization, removing punctuation and
correcting spelling errors.
2. Syntax and Parsing
• Part of Speech (POS) Tagging: Assigning parts of speech to each word in a sentence (e.g., noun,
verb, adjective).
• Dependency Parsing: Analyzing the grammatical structure of a sentence to identify relationships
between words.
• Constituency parsing: Breaking down a sentence into its constituent parts or phrases (e.g., noun
phrases, verb phrases).
3. Semantic Analysis
• Named Entity Recognition: Identifying and classifying entities in text, such as names of people
organizations, locations, dates, etc.
 • Word Sense Disambiguation: Determining which meaning of a word is used in a given context.
• Coreference Resolution: Identifying when different words refer to the same entity in a text (e.g.,
"he" refers to "John").
4. Information Extraction
• Entity Extraction: Identifying specific entities and their relationships within the text.
• Relation Extraction: Identifying and categorizing the relationships between entities in a text.
5. Text Classification in NLP
• Sentiment Analysis: Determining the sentiment or emotional tone expressed in a text (e.g.,
positive, negative, neutral).
• Topic Modeling: Identifying topics or themes within a large collection of documents.
• Spam Detection: Classifying text as spam or not spam.
6. Language Generation
• Machine Translation: Translating text from one language to another.
• Text Summarization: Producing a concise summary of a larger text.
• Text Generation: Automatically generating coherent and contextually relevant text.
7. Speech Processing
• Speech Recognition: Converting spoken language into text.
• Text to Speech (TTS) Synthesis: Converting written text into spoken language.
8. Question Answering
• Retrieval Based QA: Finding and returning the most relevant text passage in response to a query.
• Generative QA: Generating an answer based on the information available in a text corpus.
9. Dialogue Systems
• Chartbots and Virtual Assistants: Enabling systems to engage in conversations with users,
providing responses and performing tasks based on user input.
10. Sentiment and Emotion Analysis in NLP
• Emotion Detection: Identifying and categorizing emotions expressed in text.
• Opinion Mining: Analyzing opinions or reviews to understand public sentiment toward products,
services or topics.
How Natural Language Processing (NLP) Works

NLP Working
Working in NLP typically involves using computational techniques to analyze and understand human
language. This can include tasks such as language understanding, language generation and language
interaction.
1. Text Input and Data Collection
• Data Collection: Gathering text data from various sources such as websites, books, social media
or proprietary databases.
• Data Storage: Storing the collected text data in a structured format, such as a database or a
collection of documents.
2. Text Preprocessing
Preprocessing is crucial to clean and prepare the raw text data for analysis. Common preprocessing steps
include:
• Tokenization: Splitting text into smaller units like words or sentences.
• Lowercasing: Converting all text to lowercase to ensure uniformity.
• Stopword Removal: Removing common words that do not contribute significant meaning, such
as "and," "the," "is."
• Punctuation Removal: Removing punctuation marks.
• Stemming and Lemmatization: Reducing words to their base or root forms. Stemming cuts off
suffixes, while lemmatization considers the context and converts words to their meaningful base
form.
• Text Normalization: Standardizing text format, including correcting spelling errors, expanding
contractions and handling special characters.
3. Text Representation
• Bag of words (BoW): Representing text as a collection of words, ignoring grammar and word
order but keeping track of word frequency.
• Term Frequency-Inverse Document Frequency: A statistic that reflects the importance of a word
in a document relative to a collection of documents.
• Word Embeddings: Using dense vector representations of words where semantically similar
words are closer together in the vector space (e.g., Word2Vec, GloVe).
4. Feature Extraction
Extracting meaningful features from the text data that can be used for various NLP tasks.
• N-grams: Capturing sequences of N words to preserve some context and word order.
• Syntactic Features: Using parts of speech tags, syntactic dependencies and parse trees.
• Semantic Features: Leveraging word embeddings and other representations to capture word
meaning and context.
5. Model Selection and Training
Selecting and training a machine learning or deep learning model to perform specific NLP tasks.
• Supervised Learning: Using labeled data to train models like Support Vector Machines (SVM),
Random Forests or deep learning models like Convolutional Neural Networks (CNNs) and
Recurrent Neural Networks (RNNs).
• Unsupervised Learning: Applying techniques like clustering or topic modeling (e.g., Latent
Dirichlet Allocation) on unlabeled data.
 • Pre-trained Models: Utilizing pre-trained language models such as BERT GPT or transformer-
based models that have been trained on large corpora.
6. Model Deployment and Inference
Deploying the trained model and using it to make predictions or extract insights from new text data.
• Text Classification: Categorizing text into predefined classes (e.g., spam detection, sentiment
analysis).
• Named Entity Recognition: Identifying and classifying entities in the text.
• Machine Translation: Translating text from one language to another.
• Question Answering: Providing answers to questions based on the context provided by text data.
7. Evaluation and Optimization
Evaluating the performance of the NLP algorithm using metrics such as accuracy, precision, recall, F1-
score and others.
• Hyperparameter Tuning: Adjusting model parameters to improve performance.
• Error Analysis: Analyzing errors to understand model weaknesses and improve robustness.
Technologies related to Natural Language Processing
There are a variety of technologies related to natural language processing (NLP) that are used to analyze
and understand human language. Some of the most common include:
1. Machine learning: NLP relies heavily on machine learning techniques such as supervised and
unsupervised learning, deep learning and reinforcement learning to train models to understand
and generate human language.
2. Natural Language Toolkits (NLTK) and other libraries: NLTK is a popular open-source library
in Python that provides tools for NLP tasks such as tokenization, stemming and part-of-speech
tagging. Other popular libraries include spaCy, OpenNLP and CoreNLP.
3. Parsers: Parsers are used to analyze the syntactic structure of sentences, such as dependency
parsing and constituency parsing.
4. Text-to-Speech (TTS) and Speech-to-Text (STT) systems: TTS systems convert written text
into spoken words, while STT systems convert spoken words into written text.
5. Named Entity Recognition (NER) systems: NER systems identify and extract named entities
such as people, places and organizations from the text.
6. Sentiment Analysis: A technique to understand the emotions or opinions expressed in a piece of
text, by using various techniques like Lexicon-Based, Machine Learning-Based and Deep
Learning-based methods
7. Machine Translation: NLP is used for language translation from one language to another
through a computer.
 8. Chatbots: NLP is used for chatbots that communicate with other chatbots or humans through
auditory or textual methods.
9. AI Software: NLP is used in question-answering software for knowledge representation,
analytical reasoning as well as information retrieval.
Applications of Natural Language Processing (NLP)
• Spam Filters: One of the most irritating things about email is spam. Gmail uses natural language
processing (NLP) to discern which emails are legitimate and which are spam. These spam filters
look at the text in all the emails you receive and try to figure out what it means to see if it's spam
or not.
• Algorithmic Trading: Algorithmic trading is used for predicting stock market conditions. Using
NLP, this technology examines news headlines about companies and stocks and attempts to
comprehend their meaning in order to determine if you should buy, sell or hold certain stocks.
• Questions Answering: NLP can be seen in action by using Google Search or Siri Services. A
major use of NLP is to make search engines understand the meaning of what we are asking and
generate natural language in return to give us the answers.
• Summarizing Information: On the internet, there is a lot of information and a lot of it comes in
the form of long documents or articles. NLP is used to decipher the meaning of the data and then
provides shorter summaries of the data so that humans can comprehend it more quickly.
Future Scope
NLP is shaping the future of technology in several ways:
• Chatbots and Virtual Assistants: NLP enables chatbots to quickly understand and respond to
user queries, providing 24/7 assistance across text or voice interactions.
• Invisible User Interfaces (UI): With NLP, devices like Amazon Echo allow for seamless
communication through voice or text, making technology more accessible without traditional
interfaces.
• Smarter Search: NLP is improving search by allowing users to ask questions in natural language,
as seen with Google Drive's recent update, making it easier to find documents.
• Multilingual NLP: Expanding NLP to support more languages, including regional and minority
languages, broadens accessibility.
Future Enhancements: NLP is evolving with the use of Deep Neural Networks (DNNs) to make human-
machine interactions more natural. Future advancements include improved semantics for word
understanding and broader language support, enabling accurate translations and better NLP models for
languages not yet supported.
Applications of NLP or Use cases of NLP

1. Sentiment analysis
➢ Sentiment analysis, also referred to as opinion mining, is an approach to natural language
processing (NLP) that identifies the emotional tone behind a body of text.
 ➢ This is a popular way for organizations to determine and categorize opinions about a product,
service or idea.
➢ Sentiment analysis systems help organizations gather insights into real-time customer sentiment,
customer experience and brand reputation.
➢ Generally, these tools use text analytics to analyze online sources such as emails, blog posts,
online reviews, news articles, survey responses, case studies, web chats, tweets, forums and
comments.
➢ Sentiment analysis uses machine learning models to perform text analysis of human language.
The metrics used are designed to detect whether the overall sentiment of a piece of text is positive,
negative or neutral.
2. Machine Translation
➢ Machine translation, sometimes referred to by the abbreviation MT, is a sub-field of
computational linguistics that investigates the use of software to translate text or speech from one
language to another.
➢ On a basic level, MT performs mechanical substitution of words in one language for words in
another, but that alone rarely produces a good translation because recognition of whole phrases
and their closest counterparts in the target language is needed.
➢ Not all words in one language have equivalent words in another language, and many words have
more than one meaning.

➢ Solving this problem with corpus statistical and neural techniques is a rapidly growing field
that is leading to better translations, handling differences in linguistic typology, translation
of idioms, and the isolation of anomalies.
➢ Corpus: A collection of written texts, especially the entire works of a particular

author.

3. Text Extraction

➢ There are a number of natural language processing techniques that can be used to extract
information from text or unstructured data.
➢ These techniques can be used to extract information such as entity names, locations,
quantities, and more.
➢ With the help of natural language processing, computers can make sense of the vast amount
of unstructured text data that is generated every day, and humans can reap the benefits of
having this information readily available.

➢ Industries such as healthcare, finance, and e-commerce are already using natural language
processing techniques to extract information and improve business processes.
 ➢ As the machine learning technology continues to develop, we will only see more and more
information extraction use cases covered.

4. Text Classification

➢ Unstructured text is everywhere, such as emails, chat conversations, websites, and social
media. Nevertheless, it’s hard to extract value from this data unless it’s organized in a certain
way.
➢ Text classification also known as text tagging or text categorization is the process of
categorizing text into organized groups. By using Natural Language Processing
(NLP), text classifiers can automatically analyze text and then assign a set of pre-defined
tags or categories based on its content.
➢ Text classification is becoming an increasingly important part of businesses as it allows to
easily get insights from data and automate business processes.

5. Speech Recognition
Speech recognition is an interdisciplinary subfield of computer science and computational
linguistics that develops methodologies and technologies that enable the recognition and
translation of spoken language into text by computers.
It is also known as automatic speech recognition (ASR), computer speech recognition or
speech to text (STT).
It incorporates knowledge and research in the computer science, linguistics and computer
engineering fields. The reverse process is speech synthesis.

Speech recognition use cases

A wide number of industries are utilizing different applications of speech technology today, helping
businesses and consumers save time and even lives. Some examples include:
Automotive: Speech recognizers improves driver safety by enabling voice-activated navigation
systems and search capabilities in car radios.
Technology: Virtual agents are increasingly becoming integrated within our daily lives,
particularly on our mobile devices. We use voice commands to access them through our
smartphones, such as through Google Assistant or Apple’s Siri, for tasks, such as voice search, or
through our speakers, via Amazon’s Alexa or Microsoft’s Cortana, to play music. They’ll only
continue to integrate into the everyday products that we use, fueling the “Internet of Things”
movement.
Healthcare: Doctors and nurses leverage dictation applications to capture and log patient
diagnoses and treatment notes.
Sales: Speech recognition technology has a couple of applications in sales. It can help a call center
transcribe thousands of phone calls between customers and agents to identify common call
patterns and issues. AI chatbots can also talk to people via a webpage, answering common queries
and solving basic requests without needing to wait for a contact center agent to be available. In
both instances speech recognition systems help reduce time to resolution for consumer issues.
6. Chatbot
Chatbots are computer programs that conduct automatic conversations with people. They are
mainly used in customer service for information acquisition. As the name implies, these are bots
designed with the purpose of chatting and are also simply referred to as “bots.”
You’ll come across chatbots on business websites or messengers that give pre-scripted replies to
your questions. As the entire process is automated, bots can provide quick assistance 24/7 without
human intervention.

7. Email Filter
One of the most fundamental and essential applications of NLP online is email filtering. It began
with spam filters, which identified specific words or phrases that indicate a spam message. But,
like early NLP adaptations, filtering has been improved.
Gmail's email categorization is one of the more common, newer implementations of NLP. Based
on the contents of emails, the algorithm determines whether they belong in one of three categories
(main, social, or promotional).
This maintains your inbox manageable for all Gmail users, with critical, relevant emails you want
to see and reply to fast.
8. Search Autocorrect and Autocomplete
When you type 2-3 letters into Google to search for anything, it displays a list of probable search
keywords. Alternatively, if you search for anything with mistakes, it corrects them for you while
still returning relevant results. Isn't it incredible?
Everyone uses Google search autocorrect autocomplete on a regular basis but seldom gives it any
thought. It's a fantastic illustration of how natural language processing is touching millions of
people across the world, including you and me.
Both, search autocomplete and autocorrect make it much easier to locate accurate results.
3. Components of NLP
There are two components of NLP, Natural Language Understanding (NLU)and Natural Language
Generation (NLG).

Natural Language Understanding (NLU) which involves transforming human language into a
machine-readable format. It helps the machine to understand and analyze human language by
extracting the text from large data such as keywords, emotions, relations, and semantics.
Natural Language Generation (NLG) acts as a translator that converts the computerized data into
natural language representation. It mainly involves Text planning, Sentence planning, and Text
realization.
The NLU is harder than NLG.

4. Steps in NLP
There are general five steps :
1. Lexical Analysis
2. Syntactic Analysis (Parsing)
3. Semantic Analysis
4. Discourse Integration  5. Pragmatic Analysis
Lexical Analysis:
The first phase of NLP is the Lexical Analysis.
This phase scans the source code as a stream of characters and converts it into meaningful lexemes.

It divides the whole text into paragraphs, sentences, and words.

Lexeme: A lexeme is a basic unit of meaning. In linguistics, the abstract unit of morphological analysis
that corresponds to a set of forms taken by a single word is called lexeme.
The way in which a lexeme is used in a sentence is determined by its grammatical category.
Lexeme can be individual word or multiword.
For example, the word talk is an example of an individual word lexeme, which may have many
grammatical variants like talks, talked and talking.

Multiword lexeme can be made up of more than one orthographic word. For example, speak up, pull
through, etc. are the examples of multiword lexemes.

Syntax Analysis (Parsing)

 Syntactic Analysis is used to check grammar, word arrangements, and shows the relationship among
the words.
 The sentence such as “The school goes to boy” is rejected by English syntactic analyzer.

Semantic Analysis
Semantic analysis is concerned with the meaning representation.
It mainly focuses on the literal meaning of words, phrases, and sentences.
The semantic analyzer disregards sentence such as “hot ice-cream”.
Another Example is “Asaba calls out to Dave” passes a syntactic analysis because it’s a grammatically
correct sentence. However, it fails a semantic analysis. Because Asaba is a place (and can’t literally call
out to people), the sentence’s meaning doesn’t make sense.

Discourse Integration

 Discourse Integration depends upon the sentences that precedes it and also invokes the meaning of the
sentences that follow it.

 For instance, if one sentence reads, “Manhattan speaks to all its people,” and the following sentence
reads, “It calls out to Dave,” discourse integration checks the first sentence for context to understand that
“It” in the latter sentence refers to Manhattan.

Pragmatic Analysis
During this, what was said is re-interpreted on what it actually meant.

It involves deriving those aspects of language which require real world knowledge.

For instance, a pragmatic analysis can uncover the intended meaning of “Manhattan speaks to all its
people.” Methods like neural networks assess the context to understand that the sentence isn’t literal, and
most people won’t interpret it as such. A pragmatic analysis deduces that this sentence is a metaphor for
how people emotionally connect with place.

5. Finding the structure of Words

Words and Their Components

 Words are defined in most languages as the smallest linguistic units that can form a complete utterance
by themselves.
 The minimal parts of words that deliver aspects of meaning to them are called morphemes.

Tokens:
Suppose, for a moment, that words in English are delimited only by whitespace and punctuation (the
marks, such as full stop, comma, and brackets)
 Example: Will you read the newspaper? Will you read it? I won’t
read it. If we confront our assumption with insights from syntax, we notice two words here: words
newspaper and won’t.
Being a compound word, newspaper has an interesting derivational structure.

In writing, newspaper and the associated concept is distinguished from the isolated news and paper.
For reasons of generality, linguists prefer to analyze won’t as two syntactic words, or tokens, each of
which has its independent role and can be reverted to its normalized form.
The structure of won’t could be parsed as will followed by not.
In English, this kind of tokenization and normalization may apply to just a limited set of cases, but in
other languages, these phenomena have to be treated different way.

Lexemes
By the term word, we often denote not just the one linguistic form in the given context but also the
concept behind the form and the set of alternative forms that can express it.
Such sets are called lexemes or lexical items, and they constitute the lexicon of a language.
Lexemes can be divided by their behaviour into the lexical categories of verbs, nouns, adjectives,
conjunctions or other parts of speech.
The citation form of a lexeme, by which it is commonly identified, is also called its lemma.
When we convert a word into its other forms, such as turning the singular mouse into the plural mice
or mouses, we say we inflect the lexeme.
When we transform a lexeme into another one that is morphologically related, regardless of its lexical
category, we say we derive the lexeme: for instance, the nouns receiver and reception are derived from
the verb receive.
Example: Did you see him?
didn’tsee him. Ididn’tsee anyone Example presents the problem of tokenization of didn’t and the
investigation of the internal structure of anyone.
The difficulty with the definition of what counts as a word need not pose a problem for the syntactic
description if we understand no one as two closely connected tokens treated as one fixed element.
Morphemes: These components are usually called segments or morphs.
Morphology
Morphology is the domain of linguistics that analyses the internal structure of words.
Morphological analysis – exploring the structure of words
Words are built up of minimal meaningful elements called morphemes:
played = play-ed cats = cat-s
unfriendly = un-friend-ly
Two types of morphemes: i Stems: play, cat, friend ii Affixes: -ed, -s, un-, -ly Two main types of affixes:
i Prefixes precede the stem: un ii Suffixes follow the stem: -ed, -s, un-, -ly
Stemming = find the stem by stripping off affixes play = play replayed = re-play-ed computerized =
comput-er-ize-d

Problems in morphological processing

Inflectional morphology: inflected forms are constructed from base forms and inflectional Affixes.
Inflection relates different forms of the same word
Lemma Singular Plural
Cat cat Cats
Mouse mouse mice
Derivational morphology: words are constructed from roots (or stems) and derivational affixes:
inter+national = international international+ize = internationalize internationalize+ation =
internationalization

The simplest morphological process concatenates morphs one by one, as in disagree- ment-s, where agree
is a free lexical morpheme and the other elements are bound grammatical morphemes contributing some
partial meaning to the whole word.
In a more complex scheme, morphs can interact with each other, and their forms may become subject to
additional phonological and orthographic changes denoted as morphophonemic.
The alternative forms of a morpheme are termed allomorphs.
The ending -s, indicating plural in “cats,” “dogs,” the -es in “dishes,” and the -en of “oxen” are all
allomorphs of the plural morpheme.
Typology
✓ Morphological typology divides languages into groups by characterizing the prevalent
morphological phenomena in those languages.
✓ It can consider various criteria, and during the history of linguistics, different classifications have
been proposed.
✓ Let us outline the typology that is based on quantitative relations between words, their
morphemes, and their features:
✓ Isolating, or analytic, languages include no or relatively few words that would comprise more
than one morpheme (typical members are Chinese, Vietnamese, and Thai; analytic tendencies are
also found in English).
✓ Synthetic languages can combine more morphemes in one word and are further divided into
agglutinative and fusional languages.
✓ Agglutinative languages have morphemes associated with only a single function at a time (as in
Korean, Japanese, Finnish, and Tamil, etc.)
 ✓ Fusional languages are defined by their feature-per-morpheme ratio higher than one (as in Arabic,
Czech, Latin, Sanskrit, German, etc.).
✓ In accordance with the notions about word formation processes mentioned earlier, we can also
find out using concatenative and nonlinear:
✓ Concatenative languages linking morphs and morphemes one after another.
✓ Nonlinear languages allowing structural components to merge nonsequentially to apply tonal
morphemes or change the consonantal or vocalic templates of words.

6. Natural Language Processing With Python's NLTK Package

NLTK, or Natural Language Toolkit, is a Python package that you can use for NLP.
A lot of the data that you could be analyzing is unstructured data and contains human- readable text.
Before you can analyze that data programmatically, you first need to preprocess it.
Now we are going to see kinds of text preprocessing tasks you can do with NLTK so that you’ll be ready
to apply them in future projects.
1. Tokenizing
✓ By tokenizing, you can conveniently split up text by word or by sentence.
✓ This will allow you to work with smaller pieces of text that are still relatively coherent and
meaningful even outside of the context of the rest of the text.
✓ It’s your first step in turning unstructured data into structured data, which is easier to analyze.
✓ When you’re analyzing text, you’ll be tokenizing by word and tokenizing by sentence.

Tokenizing by word
Words are like the atoms of natural language. They’re the smallest unit of meaning that still makes sense
on its own. Tokenizing your text by word allows you to identify words that come up particularly often.
For example, if you were analyzing a group of job ads, then you might find that the word “Python” comes
up often. That could suggest high demand for Python knowledge, but you’d need to look deeper to know
more.
Tokenizing by sentence
When you tokenize by sentence, you can analyze how those words relate to one another and see more
context. Are there a lot of negative words around the word “Python” because the hiring manager doesn’t
like Python? Are there more terms from the domain of herpetology than the domain of software
development, suggesting that you may be dealing with an entirely different kind of python than you were
expecting?
Python Program for Tokenizing by Sentence
from [Link] import sent_tokenize, word_tokenize example_string = """
Muad'Dib learned rapidly because his first training was in how to learn. And the first lesson of all was
the basic trust that he could [Link]'s shocking to find how many people do not believe
theycan learn, and how many more believe learning to be difficult."""
sent_tokenize(example_string)

Output
["\n Muad'Dib learned rapidly because his first training was in how to learn.",
'And the first lesson of all was the basic trust that he could learn.’,
"It's shocking to find how many people do not believe they can learn,\n and how many more believe
learning to be difficult."] Note:
import nltk [Link]('punkt')
Python Program for Tokenizing by Word
from [Link] import sent_tokenize, word_tokenize example_string = """
Muad'Dib learned rapidly because his first training was in how to learn. And the first lesson of all was
the basic trust that he could [Link]'s shocking to find how many people do not believe they can learn,and
how many more believe learning to be difficult."""
word_tokenize(example_string)
Output:
["Muad'Dib", 'learned', 'rapidly', 'because', 'his', 'first', 'training', 'was', 'in', 'how', 'to',
'learn', '.', 'And', 'the', 'first', 'lesson', 'of', 'all', 'was', 'the', 'basic', 'trust', 'that', 'he',
'could', 'learn', '.', 'It', "'s", 'shocking', 'to', 'find', 'how', 'many', 'people', 'do', 'not',
'believe', 'they', 'can', 'learn', ',', 'and', 'how', 'many', 'more', 'believe', 'learning', 'to', 'be',
'difficult', '.']

2. Filtering Stop Words

 Stop words are words that you want to ignore, so you filter them out of your text when you’re
processing it. Very common words like 'in', 'is', and 'an' are often used as stop words since they don’t add
a lot of meaning to a text in and of themselves.
 Note: [Link]("stopwords")
Python program to eliminate stopwords
from [Link] import stopwords from [Link] import word_tokenize worf_quote = "Sir, I protest.
I am not a merry man!"
words_in_quote = word_tokenize(worf_quote)
print(words_in_quote)
stop_words = set([Link]("english")) filtered_list = []
for word in words_in_quote:
if [Link]() not in stop_words:
filtered_list.append(word)
print(filtered_list) Output:
['Sir', ',', 'I', 'protest', '.', 'I', 'am', 'not', 'a', 'merry', 'man', '!’]

['Sir', ',', 'protest', '.', 'merry', 'man', '!’]

‘I’ is pronoun and it is context word

Content words give you information about the topics covered in the text or the sentiment that the author
has about those topics.
Context words give you information about writing style. You can observe patterns in how authors use
context words in order to quantify their writing style.
Once you’ve quantified their writing style, you can analyze a text written by an
unknown author to see how closely it follows a particular writing style so you can try to identify who the
author is.
3. Stemming

✓ Stemming is a text processing task in which you reduce words to their root, which is the core part
of a word.
✓ For example, the words “helping” and “helper” share the root “help.”
✓ Stemming allows you to zero in on the basic meaning of a word rather than all the details of how
it’s being used.
✓ NLTK has more than one stemmer, but we’ll be using the Porter stemmer.

Python program for Stemming

from [Link] import PorterStemmer

from [Link] import word_tokenize stemmer = PorterStemmer()

string_for_stemming = "The crew of the USS Discovery discovered many discoveries. Discovering is
what explorers do."
words = word_tokenize(string_for_stemming)
print(words)
stemmed_words = [[Link](word) for word in words]
print(stemmed_words)
Output
['The', 'crew', 'of', 'the', 'USS', 'Discovery', 'discovered', 'many', 'discoveries', '.', 'Discovering', 'is', 'what',
'explorers', 'do', '.’]
['the', 'crew', 'of', 'the', 'uss', 'discoveri', 'discov', 'mani', 'discoveri', '.', 'discov', 'is', 'what', 'explor', 'do', '.’]

Original Stemmed
word version
'Discovery' 'discoveri'
'discovered' 'discov'
'discoveries' 'discoveri'
'Discovering' 'discov'
4. Tagging Parts of Speech
Part of speech is a grammatical term that deals with the role’s words play when you use them together
in sentences. Tagging parts of speech, or POS tagging, is the task of labeling the words in your text
according to their part of speech.

Part of speech Role Examples

Noun Is a person, place, or thing mountain, bagel, Poland
Pronoun Replaces a noun you, she, we
Adjective Gives information about what a noun is like efficient, windy, colorful
Verb Is an action or a state of being learn, is, go
 Adverb Gives information about a verb, an efficiently, always, very
adjective, or another adverb
Preposition Gives information about how a noun or from, about, at
pronoun is connected to another word
Conjunction Connects two other words or phrases so, because, and

Interjection Is an exclamation yay, ow, wow

• Some sources also include the category articles (like “a” or “the”) in the list of parts of speech, but
other sources consider them to be adjectives. NLTK uses the word determiner to refer to articles.

Python program for Tagging Parts of Speech

import nltk [Link]('punkt')
[Link]('averaged_perceptron_tagger')
from [Link] import word_tokenize sagan_quote = """ If you wish to make an apple pie from
scratch, you must first invent the universe."""
words_in_sagan_quote = word_tokenize(sagan_quote) nltk.pos_tag(words_in_sagan_quote)
Output:
• [('If', 'IN'), ('you', 'PRP'), ('wish', 'VBP'), ('to', 'TO'), ('make', 'VB'), ('an', 'DT'), ('apple',
'NN'), ('pie', 'NN'), ('from', 'IN'), ('scratch', 'NN'), (',', ','), ('you', 'PRP'), ('must', 'MD'), ('first', 'VB'),
('invent', 'VB'), ('the', 'DT'), ('universe', 'NN'), ('.', '.')]

5. Lemmatizing
Like stemming, lemmatizing reduces words to their core meaning, but it will give you a complete English
word that makes sense on its own instead of just a fragment of a word like 'discoveri'.

A lemma is a word that represents a whole group of words, and that group of words is called a lexeme.

For example, if you were to look up the word “blending” in a dictionary, then you’d need to look at the
entry for “blend,” but you would find “blending” listed in that entry.
In this example, “blend” is the lemma, and “blending” is part of the lexeme. So when you lemmatize a
word, you are reducing it to its lemma.

5. Python Program for Lemmatization

import nltk
[Link]('punkt') [Link]('wordnet')
from [Link] import WordNetLemmatizer
from [Link] import word_tokenize
lemmatizer = WordNetLemmatizer()
string_for_lemmatizing = "The friends of DeSoto love scarves." words =
word_tokenize(string_for_lemmatizing)
lemmatized_words = [[Link](word) for word in words] print(lemmatized_words)
Output:
[Link]("worst") o/p: 'worst’
[Link]("worst", pos="a") o/p: 'bad'
6. Chunking
✓ chunking allows you to identify phrases.
✓ A phrase is a word or group of words that works as a single unit to perform a grammatical
function. Noun phrases are built around a noun.
✓ Here are some examples: “A planet”, “A tilting planet”, “A swiftly tilting planet”
✓ Chunking makes use of POS tags to group words and apply chunk tags to those groups. Chunks
don’t overlap, so one instance of a word can be in only one chunk at a time.
✓ After getting a list of tuples of all the words in the quote, along with their POS tag. In order to
chunk, you first need to define a chunk grammar.
Note: A chunk grammar is a combination of rules on how sentences should be chunked. It often uses
regular expressions, or regexes.
 Create a chunk grammar with one regular expression rule:
 grammar = "NP: {<DT>?<JJ>*<NN>}“
 Create a chunk parser with this grammar:
Python program for chuncking
import nltk [Link]('puckt')
from [Link] import word_tokenize

quote = "It's a dangerous business, Frodo, going out your door."

words_quote = word_tokenize(quote)
print(words_quote)
[Link]("averaged_perceptron_tagger") tags = nltk.pos_tag(words_quote)
print(tags)
#Regular expression for Noun Phrase
grammar = "NP: {<DT>?<JJ>*<NN>}"
#Create a chunk parser with this grammar:
chunk_parser = [Link](grammar)
tree = chunk_parser.parse(tags)
print(tree)
Output:
['It', "'s", 'a', 'dangerous', 'business', ',', 'Frodo', ',', 'going', 'out', 'your', 'door', '.']

[('It', 'PRP'), ("'s", 'VBZ'), ('a', 'DT'), ('dangerous', 'JJ'), ('business', 'NN'), (',', ','),
('Frodo', 'NNP'), (',', ','), ('going', 'VBG'), ('out', 'RP'), ('your', 'PRP$'), ('door', 'NN'),
('.', '.')]

(S

It/PRP

's/VBZ

(NP a/DT dangerous/JJ business/NN)

,/, Frodo/NNP
,/, going/VBG

out/RP

your/PRP$

(NP door/NN)

./.)
Tree Representation

TEXT CORPORA
Text corpora are large, structured collections of real-world text used as foundational datasets in Natural
Language Processing (NLP) to train models for tasks like translation, sentiment analysis, and speech
recognition, providing the authentic language examples that NLP algorithms learn from to understand,
interpret, and generate human language. These machine-readable collections can be diverse, ranging
from books and news to web pages and chat logs, often annotated for specific linguistic features.
What is a Text Corpus?
• A corpus (plural: corpora) is a vast collection of authentic texts, like a database of language
examples.
• It serves as the raw material for NLP, similar to a dataset in general machine learning.
• It can include digital texts, transcribed speech, and even social media data, often organized by
genre or style.
How Corpora are Used in NLP
• Training Models: Corpora train machine learning models to recognize patterns, grammar, and
meaning in human language.
• Statistical Analysis: Linguists and data scientists use corpora to test hypotheses, count word
frequencies, and tag parts of speech.
• Developing Applications:
o Machine Translation: Parallel corpora (aligned texts in multiple languages) are crucial.
o Speech Recognition: Corpora help systems understand spoken words.
o Sentiment Analysis: Analyzing web text corpora reveals online language styles.
o Autocomplete/Predictive Text: Models learn common word sequences (n-grams) from
corpora.
Types of Corpora
• Brown Corpus: An early, genre-categorized English corpus (news, fiction).
• Web Corpora: Large collections from the internet (e.g., wikipedia, forums).
• Parallel Corpora: Texts aligned in different languages for translation.
• Chat Corpora: Dialogue records from instant messaging.
The Process
 1. Data Collection: Gathering texts from diverse sources.
2. Preprocessing: Cleaning, tokenizing (splitting into words/sentences), and normalizing text
(e.g., lowercasing).
3. Analysis/Modeling: Applying NLP techniques to the prepared corpus.
REGEX IN PYTHON
Python RegEx
A Regular Expression or RegEx is a special sequence of characters that uses a search pattern to find a
string or set of strings.
It can detect the presence or absence of a text by matching it with a particular pattern and also can split
a pattern into one or more sub-patterns.
Regex Module in Python
Python has a built-in module named "re" that is used for regular expressions in Python. We can import
this module by using import statement.
Importing re module in Python using following command: import re
RegEx Functions
The re module in Python provides various functions that help search, match, and manipulate strings
using regular expressions.
Below are main functions available in the re module:
FUNCTION DESCRIPTION
[Link]() finds and returns all matching occurrences in a list
[Link]() Regular expressions are compiled into pattern objects
[Link]() Split string by the occurrences of a character or a pattern.
[Link]() Replaces all occurrences of a character or patter with a replacement
string.
RESUBN It's similar to [Link]() method but it returns a tuple: (new_string,
number_of_substitutions)
[Link]() Escapes special character
[Link]() Searches for first occurrence of character or pattern

Let's see the working of these RegEx functions with definition and examples:
1. [Link]()
Returns all non-overlapping matches of a pattern in the string as a list. It scans the string from left to
right.
Example: This code uses regular expression \d+ to find all sequences of one or more digits in the given
string.
import re
string = """Hello my Number is 123456789 and
my friend's number is 987654321"""

regex = '\d+'
match = [Link](regex, string)
print(match)

Output
['123456789', '987654321']
2. [Link]()
Compiles a regex into a pattern object, which can be reused for matching or substitutions.
Example 1: This pattern [a-e] matches all lowercase letters between 'a' and 'e', in the input string "Aye,
said Mr. Gibenson Stark". The output should be ['e', 'a', 'd', 'b', 'e'], which are matching characters.
import re
p = [Link]('[a-e]')
print([Link]("Aye, said Mr. Gibenson Stark"))

Output
['e', 'a', 'd', 'b', 'e', 'a']
Explanation:
• First occurrence is 'e' in "Aye" and not 'A', as it is Case Sensitive.
• Next Occurrence is 'a' in "said", then 'd' in "said", followed by 'b' and 'e' in "Gibenson", the Last
'a' matches with "Stark".
• Metacharacter backslash '\' has a very important role as it signals various sequences. If the
backslash is to be used without its special meaning as metacharacter, use'\\'
Example 2: The code uses regular expressions to find and list all single digits and sequences of digits
in the given input strings. It finds single digits with \d and sequences of digits with \d+.
import re
p = [Link]('\d')
print([Link]("I went to him at 11 A.M. on 4th July 1886"))

p = [Link]('\d+')
print([Link]("I went to him at 11 A.M. on 4th July 1886"))

Output
['1', '1', '4', '1', '8', '8', '6']
['11', '4', '1886']
Example 3: Word and non-word characters
• \w matches a single word character.
• \w+ matches a group of word characters.
• \W matches non-word characters.
import re

p = [Link]('\w')
print([Link]("He said * in some_lang."))

p = [Link]('\w+')
print([Link]("I went to him at 11 A.M., he \
said *** in some_language."))

p = [Link]('\W')
print([Link]("he said *** in some_language."))

Output
['H', 'e', 's', 'a', 'i', 'd', 'i', 'n', 's', 'o', 'm', 'e', '_', 'l', 'a', 'n', 'g']
['I', 'went', 'to', 'him', 'at', '11', 'A', 'M', 'he', 'said', 'in', 'some_language']
[' ', ' ', '*', '*', '*', ' ', ' ', '.']
Example 4: The regular expression pattern 'ab*' to find and list all occurrences of 'ab' followed by zero
or more 'b' characters. In the input string "ababbaabbb". It returns the following list of matches: ['ab',
'abb', 'abbb'].
import re
p = [Link]('ab*')
print([Link]("ababbaabbb"))

Output
['ab', 'abb', 'a', 'abbb']
Explanation:
• Output 'ab', is valid because of single 'a' accompanied by single 'b'.
• Output 'abb', is valid because of single 'a' accompanied by 2 'b'.
• Output 'a', is valid because of single 'a' accompanied by 0 'b'.
• Output 'abbb', is valid because of single 'a' accompanied by 3 'b'.
3. [Link]()
Splits a string wherever the pattern matches. The remaining characters are returned as list elements.
Syntax:
[Link](pattern, string, maxsplit=0, flags=0)
• pattern: Regular expression to match split points.
• string: The input string to split.
• maxsplit (optional): Limits the number of splits. Default is 0 (no limit).
• flags (optional): Apply regex flags like [Link].
Meta-characters
Metacharacters are special characters in regular expressions used to define search patterns. The re
module in Python supports several metacharacters that help you perform powerful pattern matching.
Below is a quick reference table:
METACHARACTERS DESCRIPTION
\ Used to drop the special meaning of character following it
[] Represent a character class
^ Matches the beginning
$ Matches the end
. Matches any character except newline
| Means OR (Matches with any of the characters separated by it.
? Matches zero or one occurrence
* Any number of occurrences (including 0 occurrences)
+ One or more occurrences
{} Indicate the number of occurrences of a preceding regex to match.
() Enclose a group of Regex
Let's discuss each of these metacharacters in detail:
1. \ - Backslash
The backslash (\) makes sure that the character is not treated in a special way. This can be considered a
way of escaping metacharacters.
For example, if you want to search for the dot(.) in the string then you will find that dot(.) will be
treated as a special character as is one of the metacharacters (as shown in the above table). So for this
case, we will use the backslash(\) just before the dot(.) so that it will lose its specialty. See the below
example for a better understanding.
Example: The first search ([Link](r'.', s)) matches any character, not just the period, while the
second search ([Link](r'\.', s)) specifically looks for and matches the period character.
import re
s = '[Link]'
# without using \
match = [Link](r'.', s)
print(match)
# using \
match = [Link](r'\.', s)
print(match)

Output
<[Link] object; span=(0, 1), match='g'>
<[Link] object; span=(5, 6), match='.'>
2. [] - Square Brackets
Square Brackets ([]) represent a character class consisting of a set of characters that we wish to match.
For example, the character class [abc] will match any single a, b, or c.
We can also specify a range of characters using - inside the square brackets. For example,
• [0, 3] is sample as [0123]
• [a-c] is same as [abc]
We can also invert the character class using the caret(^) symbol. For example,
• [^0-3] means any character except 0, 1, 2, or 3
• [^a-c] means any character except a, b, or c
Example: In this code, you're using regular expressions to find all the characters in the string that fall
within the range of 'a' to 'm'. The [Link]() function returns a list of all such characters. In the given
string, the characters that match this pattern are: 'c', 'k', 'b', 'f', 'j', 'e', 'h', 'l', 'd', 'g'.
import re

string = "The quick brown fox jumps over the lazy dog"
pattern = "[a-m]"
result = [Link](pattern, string)

print(result)

Output
['h', 'e', 'i', 'c', 'k', 'b', 'f', 'j', 'm', 'e', 'h', 'e', 'l', 'a', 'd', 'g']
3. ^ - Caret
Caret (^) symbol matches the beginning of the string i.e. checks whether the string starts with the given
character(s) or not. For example -
• ^g will check if the string starts with g such as geeks, globe, girl, g, etc.
• ^ge will check if the string starts with ge such as geeks, geeksforgeeks, etc.
Example: This code uses regular expressions to check if a list of strings starts with "The". If a string
begins with "The," it's marked as "Matched" otherwise, it's labeled as "Not matched".
import re
regex = r'^The'
strings = ['The quick brown fox', 'The lazy dog', 'A quick brown fox']
for string in strings:
if [Link](regex, string):
 print(f'Matched: {string}')
else:
print(f'Not matched: {string}')

Output
Matched: The quick brown fox
Matched: The lazy dog
Not matched: A quick brown fox
4. $ - Dollar
Dollar($) symbol matches the end of the string i.e checks whether the string ends with the given
character(s) or not. For example-
• s$ will check for the string that ends with a such as geeks, ends, s, etc.
• ks$ will check for the string that ends with ks such as geeks, geeksforgeeks, ks, etc.
Example: This code uses a regular expression to check if the string ends with "World!". If a match is
found, it prints "Match found!" otherwise, it prints "Match not found".
import re

string = "Hello World!"

pattern = r"World!$"

match = [Link](pattern, string)

if match:
print("Match found!")
else:
print("Match not found.")

Output
Match found!
5. . - Dot
Dot(.) symbol matches only a single character except for the newline character (\n). For example -
• a.b will check for the string that contains any character at the place of the dot such as acb, acbd,
abbb, etc
• .. will check if the string contains at least 2 characters
Example: This code uses a regular expression to search for the pattern "[Link]" within the string.
The dot (.) in the pattern represents any character. If a match is found, it prints "Match found!"
otherwise, it prints "Match not found".
import re

string = "The quick brown fox jumps over the lazy dog."
pattern = r"[Link]"

match = [Link](pattern, string)

if match:
print("Match found!")
else:
print("Match not found.")
Output
Match found!
6. | - Or
The | operator means either pattern on its left or right can match. a|b will match any string that contains
a or b such as acd, bcd, abcd, etc.
7. ? - Question Mark
The question mark (?) indicates that the preceding element should be matched zero or one time. It
allows you to specify that the element is optional, meaning it may occur once or not at all.
For example, ab?c will be matched for the string ac, acb, dabc but will not be matched for abbc
because there are two b. Similarly, it will not be matched for abdc because b is not followed by c.
8.* - Star
Star (*) symbol matches zero or more occurrences of the regex preceding the * symbol.
For example, ab*c will be matched for the string ac, abc, abbbc, dabc, etc. but will not be matched for
abdc because b is not followed by c.
9. + - Plus
Plus (+) symbol matches one or more occurrences of the regex preceding the + symbol.
For example, ab+c will be matched for the string abc, abbc, dabc, but will not be matched for ac, abdc,
because there is no b in ac and b, is not followed by c in abdc.
10. {m, n} - Braces
Braces match any repetitions preceding regex from m to n both inclusive.
For example, a{2, 4} will be matched for the string aaab, baaaac, gaad, but will not be matched for
strings like abc, bc because there is only one a or no a in both the cases.
11. (<regex>) - Group
Group symbol is used to group sub-patterns.
For example, (a|b)cd will match for strings like acd, abcd, gacd, etc.

Tokenization in NLP
Tokenization is a fundamental step in Natural Language Processing (NLP). It involves dividing a Textual
input into smaller units known as tokens. These tokens can be in the form of words, characters, sub-
words, or sentences. It helps in improving interpretability of text by different models. Let's understand
How Tokenization Works.

Representation of Tokenization
What is Tokenization in NLP?
Natural Language Processing (NLP) is a subfield of, information engineering, and human-computer
interaction. It focuses on how to process and analyze large amounts of natural language data efficiently.
It is difficult to perform as the process of reading and understanding languages is far more complex than
it seems at first glance.
• Tokenization is a foundation step in NLP pipeline that shapes the entire workflow.
• Involves dividing a string or text into a list of smaller units known as tokens.
 • Uses a tokenizer to segment unstructured data and natural language text into distinct chunks of
information, treating them as different elements.
• Tokens: Words or Sub-words in the context of natural language processing. Example: A word is
a token in a sentence, A character is a token in a word, etc.
• Application: Multiple NLP tasks, text processing, language modelling, and machine translation.

Types of Tokenization
Tokenization can be classified into several types based on how the text is segmented. Here are some types
of tokenization:
1. Word Tokenization: Word tokenization is the most commonly used method where text is divided into
individual words. It works well for languages with clear word boundaries, like English. For example,
"Machine learning is fascinating" becomes:
Input before tokenization: ["Machine Learning is fascinating"]
Output when tokenized by words: ["Machine", "learning", "is", "fascinating"]
2. Character Tokenization: In Character Tokenization, the textual data is split and converted to a
sequence of individual characters. This is beneficial for tasks that require a detailed analysis, such as
spelling correction or for tasks with unclear boundaries. It can also be useful for modelling character-
level language.
Example
Input before tokenization: ["You are helpful"]
Output when tokenized by characters: ["Y", "o", "u", " ", "a", "r", "e", " ", "h", "e", "l", "p", "f", "u",
"l"]
3. Sub-word Tokenization: This strikes a balance between word and character tokenization by breaking
down text into units that are larger than a single character but smaller than a full word. This is useful
when dealing with morphologically rich languages or rare words.
Example
["Time", "table"]
["Rain", "coat"]
["Grace", "fully"]
["Run", "way"]
Sub-word tokenization helps to handle out-of-vocabulary words in NLP tasks and for languages that form
words by combining smaller units.
4. Sentence Tokenization: Sentence tokenization is also a common technique used to make a division
of paragraphs or large set of sentences into separated sentences as tokens. This is useful for tasks requiring
individual sentence analysis or processing.
Input before tokenization: ["Artificial Intelligence is an emerging technology. Machine learning is
fascinating. Computer Vision handles images. "]
Output when tokenized by sentences ["Artificial Intelligence is an emerging technology.", "Machine
learning is fascinating.", "Computer Vision handles images."]
5. N-gram Tokenization: N-gram tokenization splits words into fixed-sized chunks (size = n) of data.
Input before tokenization: ["Machine learning is powerful"]
Output when tokenized by bigrams: [('Machine', 'learning'), ('learning', 'is'), ('is', 'powerful')]
Need of Tokenization
Tokenization is an essential step in text processing and natural language processing (NLP) for
several reasons. Some of these are listed below:
 • Effective Text Processing: Reduces the size of raw text, resulting in easy and efficient statistical
and computational analysis.
• Feature extraction: Text data can be represented numerically for algorithmic comprehension by
using tokens as features in ML models.
• Information Retrieval: Tokenization is essential for indexing and searching in systems that store
and retrieve information efficiently based on words or phrases.
• Text Analysis: Used in sentiment analysis and named entity recognition, to determine the
function and context of individual words in a sentence.
• Vocabulary Management: Generates a list of distinct tokens, Helps manage a corpus's
vocabulary.
• Task-Specific Adaptation: Adapts to need of particular NLP task, Good for summarization and
machine translation.
Implementation for Tokenization
Sentence Tokenization using sent_tokenize
The code snippet uses sent_tokenize function from NLTK library. The sent_tokenize function is used to
segment a given text into a list of sentences.
from [Link] import sent_tokenize
text = "Hello everyone. Welcome to GeeksforGeeks. You are studying NLP article."
sent_tokenize(text)
Output:
['Hello everyone.', 'Welcome to GeeksforGeeks.' ,'You are studying NLP article']
How sent_tokenize works: The sent_tokenize function uses an instance of PunktSentenceTokenizer from
the [Link] module, which is already been trained and thus very well knows to mark the end
and beginning of sentence at what characters and punctuation.
Sentence Tokenization using PunktSentenceTokenizer
It is efficient to use 'PunktSentenceTokenizer' to from the NLTK library. The Punkt tokenizer is a data-
driven sentence tokenizer that comes with NLTK. It is trained on large corpus of text to identify sentence
boundaries.
import [Link]

# Loading PunktSentenceTokenizer using English pickle file

tokenizer = [Link]('tokenizers/punkt/PY3/[Link]')
[Link](text)
Output:
['Hello everyone.', 'Welcome to GeeksforGeeks.', 'You are studying NLP article']
Tokenize sentence of different language
Sentences from different languages can also be tokenized using different pickle file other than English.
• In the following code snippet, we have used NLTK library to tokenize a Spanish text into
sentences using pre-trained Punkt tokenizer for Spanish.
• The Punkt tokenizer: Data-driven ML-based tokenizer to identify sentence boundaries.
import [Link]
spanish_tokenizer = [Link]('tokenizers/punkt/PY3/[Link]')

text = 'Hola amigo. Estoy bien.'

spanish_tokenizer.tokenize(text)
Output:
['Hola amigo.',
 'Estoy bien.']
Word Tokenization using work_tokenize
The code snipped uses the word_tokenize function from NLTK library to tokenize a given text into
individual words.
• The word_tokenize function is helpful for breaking down a sentence or text into its constituent
words.
• Eases analysis or processing at the word level in natural language processing tasks.
from [Link] import word_tokenize
text = "Hello everyone. Welcome to GeeksforGeeks."
word_tokenize(text)
Output:
['Hello', 'everyone', '.', 'Welcome', 'to', 'GeeksforGeeks', '.']
How word_tokenize works: word_tokenize() function is a wrapper function that calls tokenize() on an
instance of the TreebankWordTokenizer class.
Word Tokenization Using TreebankWordTokenizer
The code snippet uses the TreebankWordTokenizer from the Natural Language Toolkit (NLTK) to
tokenize a given text into individual words.
from [Link] import TreebankWordTokenizer
tokenizer = TreebankWordTokenizer()
[Link](text)
Output:
['Hello', 'everyone.', 'Welcome', 'to', 'GeeksforGeeks', '.']
These tokenizers work by separating the words using punctuation and spaces. And as mentioned in the
code outputs above, it doesn't discard the punctuation, allowing a user to decide what to do with the
punctuations at the time of pre-processing.
Word Tokenization using WordPunctTokenizer
The WordPunctTokenizer is one of the NLTK tokenizers that splits words based on punctuation
boundaries. Each punctuation mark is treated as a separate token.
from [Link] import WordPunctTokenizer
tokenizer = WordPunctTokenizer()
[Link]("Let's see how it's working.")
Output:
['Let', "'", 's', 'see', 'how', 'it', "'", 's', 'working', '.']
Word Tokenization using Regular Expression
The code snippet uses the RegexpTokenizer from the NLTK to tokenize a given text based on a regular
expression pattern.
from [Link] import RegexpTokenizer
tokenizer = RegexpTokenizer(r'\w+')
text = "Let's see how it's working."
[Link](text)
Output:
['Let', 's', 'see', 'how', 'it', 's', 'working']
Using regular expressions allows for more fine-grained control over tokenization, and you can customize
the pattern based on your specific requirements.
More Techniques for Tokenization
We have discussed the ways to implement how can we perform tokenization using NLTK library. We can
also implement tokenization using following methods and libraries:
• Spacy: Spacy is NLP library that provide robust tokenization capabilities.
 • BERT tokenizer: uses Word Piece tokenizer, which is a type of sub-word tokenizer for tokenizing
input text. Using regular expressions allows for more fine-grained control over tokenization, and
you can customize the pattern based on your specific requirements.
• Byte-Pair Encoding: is a data compression algorithm that has also found applications in the field
of natural language processing, specifically for tokenization. It is a Sub-word Tokenization
technique that works by iteratively merging the most frequent pairs of consecutive bytes (or
characters) in a given corpus.
• Sentence Piece: is another sub-word tokenization algorithm commonly used for natural language
processing tasks. It is designed to be language-agnostic and works by iteratively merging frequent
sequences of characters or sub words in a given corpus.
Limitations of Tokenization
• Unable to capture the meaning of the sentence hence, results in ambiguity.
• Chinese, Japanese, Arabic, lack distinct spaces between words. Hence, absence of clear
boundaries that complicates the process of tokenization.
• Tough to decide how to tokenize text that may include more than one word, for example email
address, URLs and special symbols

NORMALIZING TEXTUAL DATA WITH PYTHON

We will learn How to Normalizing Textual Data with Python. Let's discuss some concepts :
• Textual data ask systematically collected material consisting of written, printed, or electronically
published words, typically either purposefully written or transcribed from speech.
• Text normalization is that the method of transforming text into one canonical form that it'd not
have had before. Normalizing text before storing or processing it allows for separation of concerns
since the input is sure to be consistent before operations are performed thereon. Text
normalization requires being conscious of what sort of text is to be normalized and the way it's to
be processed afterwards; there's no all-purpose normalization procedure.
Steps Required
Here, we will discuss some basic steps need for Text normalization.
• Input text String,
• Convert all letters of the string to one case (either lower or upper case),
• If numbers are essential to convert to words else remove all numbers,
• Remove punctuations, other formalities of grammar,
• Remove white spaces,
• Remove stop words,
• And any other computations.
We are doing Text normalization with above-mentioned steps, every step can be done in some ways. So
we will discuss each and everything in this whole process.
Text String
# input string
string = " Python 3.0, released in 2008, was a major revision of the language that is not completely
backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's
end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."
print(string)
Output:
" Python 3.0, released in 2008, was a major revision of the language that is not completely backward
compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's end-of-life,
only Python 3.6.x[30] and later are supported, with older versions still supporting e.g. Windows 7 (and
old installers not restricted to 64-bit Windows)."
Case Conversion
In Python, lower() is a built-in method used for string handling. The lower() methods returns the
lowercased string from the given string. It converts all uppercase characters to lowercase. If no uppercase
characters exist, it returns the original string.
# input string
string = " Python 3.0, released in 2008, was a major revision of the language that is not completely
backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's
end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."

# convert to lower case

lower_string = [Link]()
print(lower_string)
Output:
" python 3.0, released in 2008, was a major revision of the language that is not completely backward
compatible and much python 2 code does not run unmodified on python 3. with python 2's end-of-life,
only python 3.6.x[30] and later are supported, with older versions still supporting e.g. windows 7 (and
old installers not restricted to 64-bit windows)."
Remove numbers if they're not relevant to your analyses. Usually, regular expressions are used to remove
numbers.
# import regex
import re

# input string
string = " Python 3.0, released in 2008, was a major revision of the language that is not completely
backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's
end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."

# convert to lower case

lower_string = [Link]()

# remove numbers
no_number_string = [Link](r'\d+','',lower_string)
print(no_number_string)
Output:
"python., released in, was a major revision of the language that is not completely backward compatible
and much python code does not run unmodified on python. with python 's end-of-life, only python ..x[]
and later are supported, with older versions still supporting e.g. windows (and old installers not restricted
to -bit windows).

The part of replacing with punctuation can also be performed using regex. In this, we replace all
punctuation by empty string using certain regex.
# import regex
import re

# input string
string = " Python 3.0, released in 2008, was a major revision of the language that is not completely
backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's
end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."

# convert to lower case

lower_string = [Link]()

# remove numbers
no_number_string = [Link](r'\d+','',lower_string)

# remove all punctuation except words and space

no_punc_string = [Link](r'[^\w\s]','', no_number_string)
print(no_punc_string)
Output:
'python released in was a major revision of the language that is not completely backward compatible and
much python code does not run unmodified on python with python s end of life only python x and later
are supported with older versions still supporting eg windows and old installers not restricted to bit
windows'
Python
The strip() function is an inbuilt function in Python programming language that returns a copy of the
string with both leading and trailing characters removed (based on the string argument passed).
# import regex
import re

# input string
string = " Python 3.0, released in 2008, was a major revision of the language that is not completely
backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's
end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."

# convert to lower case

lower_string = [Link]()

# remove numbers
no_number_string = [Link](r'\d+','',lower_string)

# remove all punctuation except words and space

no_punc_string = [Link](r'[^\w\s]','', no_number_string)

# remove white spaces

no_wspace_string = no_punc_string.strip()
print(no_wspace_string)
Output:
'python released in was a major revision of the language that is not completely backward compatible and
much python code does not run unmodified on python with python s endoflife only python x and later
are supported with older versions still supporting eg windows and old installers not restricted to bit
windows'
Removing Stop Words
Stop words” are the foremost common words during a language like “the”, “a”, “on”, “is”, “all”. These
words don't carry important meaning and are usually faraway from texts. It is possible to get rid of stop
words using tongue Toolkit (NLTK), a set of libraries and programs for symbolic and statistical tongue
processing.
# download stopwords
import nltk
[Link]('stopwords')

# import nltk for stopwords

from [Link] import stopwords
stop_words = set([Link]('english'))
print(stop_words)

# assign string
no_wspace_string='python released in was a major revision of the language that is not completely
backward compatible and much python code does not run unmodified on python with python s endoflife
only python x and later are supported with older versions still supporting eg windows and old installers
not restricted to bit windows'

# convert string to list of words

lst_string = [no_wspace_string][0].split()
print(lst_string)

# remove stopwords
no_stpwords_string=""
for i in lst_string:
if not i in stop_words:
no_stpwords_string += i+' '

# removing last space

no_stpwords_string = no_stpwords_string[:-1]
print(no_stpwords_string)
Output:

In this, we can normalize the textual data using Python. Below is the complete python program:
# import regex
import re

# download stopwords
import nltk
[Link]('stopwords')

# import nltk for stopwords

from [Link] import stopwords
stop_words = set([Link]('english'))

# input string
string = " Python 3.0, released in 2008, was a major revision of the language that is not completely
backward compatible and much Python 2 code does not run unmodified on Python 3. With Python 2's
end-of-life, only Python 3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."

# convert to lower case

lower_string = [Link]()

# remove numbers
no_number_string = [Link](r'\d+','',lower_string)

# remove all punctuation except words and space

no_punc_string = [Link](r'[^\w\s]','', no_number_string)

# remove white spaces

no_wspace_string = no_punc_string.strip()
no_wspace_string

# convert string to list of words

lst_string = [no_wspace_string][0].split()
print(lst_string)

# remove stopwords
no_stpwords_string=""
for i in lst_string:
if not i in stop_words:
no_stpwords_string += i+' '

# removing last space

no_stpwords_string = no_stpwords_string[:-1]

# output
print(no_stpwords_string)
Output:
 UNDERSTANDING TEXT CLASSIFICATION IN PYTHON
Text Classification: Text classification is a common NLP task used to solve business problems in various
fields. The goal of text classification is to categorize or predict a class of unseen text documents, often
with the help of supervised machine learning. Similar to a classification algorithm that has been trained
on a tabular dataset to predict a class, text classification also uses supervised machine learning. The fact
that text is involved in text classification is the main distinction between the two.
You can also perform text classification without using supervised machine learning. Instead of
algorithms, a manual rule-based system can be designed to perform the task of text classification. We’ll
compare and review the pros and cons of rule-based and machine learning-based text classification
systems in the next section.

Text Classification Use-Cases and Applications

Spam classification: There are many practical use cases for text classification across many
industries. For example, a spam filter is a common application that uses text classification to sort
emails into spam and non-spam categories.

Classifying news articles and blogs: Another use case is to automatically assign text documents into
predetermined categories. A supervised machine learning model is trained on labeled data, which
includes both the raw text and the target. Once a model is trained, it is then used in production to
obtain a category (label) on the new and unseen data (articles/blogs written in the future).
Categorize customer support requests: A company might use text classification to automatically
categorize customer support requests by topic or to prioritize and route requests to the appropriate
department.

Hate speech detection: With over 1.7 billion daily active users, Facebook inevitably has content
created on the site that is against the rules. Hate speech is included in this undesirable content.
Facebook tackles this issue by requesting a manual review of postings that an AI text classifier has
identified as hate speech. Postings that were flagged by AI are examined in the same manner as posts that
users have reported. In fact, in just the first three months of 2020, the platform removed 9.6 million items
of content that had been classified as hate speech.

Types of Text Classification Systems

There are mainly two types of text classification systems; rule-based and machine learning-based text
classification.

1. Rule-based text classification: Rule-based techniques use a set of manually constructed

language rules to categorize text into categories or groups. These rules tell the system to classify
text into a particular category based on the content of a text by using semantically relevant textual
elements. An antecedent or pattern and a projected category make up each rule.
For example, imagine you have tons of new articles, and your goal is to assign them to relevant categories
such as Sports, Politics, Economy, etc.
With a rule-based classification system, you will do a human review of a couple of documents to come
up with linguistic rules like this one:
• If the document contains words such as money, dollar, GDP, or inflation, it belongs to the Politics
group (class).
Rule-based systems can be refined over time and are understandable to humans. However, there are
certain drawbacks to this strategy.
These systems, to begin with, demand in-depth expertise in the field. They take a lot of time since creating
rules for a complicated system can be difficult and frequently necessitates extensive study and testing.
Given that adding new rules can alter the outcomes of the pre-existing rules, rule-based systems are also
challenging to maintain and do not scale effectively.

2. Machine learning-based text classification: Machine learning-based text classification is a

supervised machine learning problem. It learns the mapping of input data (raw text) with the labels (also
known as target variables). This is similar to non-text classification problems where we train a supervised
classification algorithm on a tabular dataset to predict a class, with the exception that in text classification,
the input data is raw text instead of numeric features.
Like any other supervised machine learning, text classification machine learning has two phases; training
and prediction.

Training phase: A supervised machine learning algorithm is trained on the input-labeled dataset during
the training phase. At the end of this process, we get a trained model that we can use to obtain predictions
(labels) on new and unseen data.
Prediction phase: Once a machine learning model is trained, it can be used to predict labels on new and
unseen data. This is usually done by deploying the best model from an earlier phase as an API on the
server.
Text Preprocessing Pipeline
Preprocessing text data is an important step in any natural language processing task. It helps in cleaning
and preparing the text data for further processing or analysis.
A text preprocessing pipeline is a series of processing steps that are applied to raw text data in order to
prepare it for use in natural language processing tasks.
The steps in a text preprocessing pipeline can vary, but they typically include tasks such as tokenization,
stop word removal, stemming, and lemmatization. These steps help reduce the size of the text data and
also improve the accuracy of NLP tasks such as text classification and information extraction.
Text data is difficult to process because it is unstructured and often contains a lot of noise. This noise can
be in the form of misspellings, grammatical errors, and non-standard formatting. A text preprocessing
pipeline aims to clean up this noise so that the text data can be more easily analyzed.
Feature Extraction
The two most common methods for extracting feature from text or in other words converting text data
(strings) into numeric features so machine learning model can be trained are: Bag of Words (a.k.a
CountVectorizer) and Tf-IDF.

Bag of Words: A bag of words (BoW) model is a simple way of representing text data as numeric
features. It involves creating a vocabulary of known words in the corpus and then creating a vector for
each document that contains counts of how often each word appears.

TF-IDF: TF-IDF stands for term frequency-inverse document frequency, and it is another way of
representing text as numeric features. There are some shortcomings of the Bag of Words (BoW) model
that Tf-IDF overcomes. The TF-IDF model is different from the bag of words model in that it takes into
account the frequency of the words in the document, as well as the inverse document frequency. This
means that the TF-IDF model is more likely to identify the important words in a document than the bag
of words model.
Text Preprocessing
This is where all text cleaning takes place. It’s a loop that iterates through all 5,572 documents and does
the following:
• Remove all special characters
• Lowercase all the words
• Tokenize
• Remove stopwords
• Lemmatize
text preprocessing
# download nltk
import nltk
[Link](all')
# create a list text text =
list(data['text'])
 # preprocessing loop
import re from [Link]
import stopwords from [Link] import
WordNetLemmatizer lemmatizer =
WordNetLemmatizer()
corpus = []
for i in range(len(text)):
r = [Link]('[^a-zA-Z]', ' ', text[i])

r = [Link]()
r = [Link]()
r = [word for word in r if word not in [Link]('english')]
r = [[Link](word) for word in r]

r = ' '.join(r)

[Link](r)
#assign corpus to data['text']
data['text'] = corpus [Link]()

TEXT SUMMARIZATION
Automatic Text Summarization is a key technique in Natural Language Processing (NLP) that uses
algorithms to reduce large texts while preserving essential information. Although it doesn’t receive as
much attention as other machine learning breakthroughs, text summarization technology has seen
continuous improvements. By extracting key concepts and maintaining the original meaning, these
systems can revolutionize industries such as banking, law, and healthcare, enabling faster decision-
making and information retrieval.
There are two primary types of text summarization techniques:
1. Extractive Summarization
2. Abstractive Summarization
Extractive summarization algorithms automatically generate summaries by selecting and combining
key passages from the original text. Unlike human summarizers, these models focus on extracting the
most important sentences without creating new content. The goal is to preserve the meaning of the
original text while condensing it.
The TextRank algorithm is widely used for extractive summarization tasks. By ranking sentences based
on their relevance and importance, it can generate a concise summary. Let's explore how this algorithm
works with a sample text.
Utilizing TextRank Algorithm for Extractive Text Summarization
TextRank is implemented in the spaCy library. With the help of PyTextRank, a spaCy extension, we can
efficiently apply the TextRank algorithm to summarize text. While extractive summarization provides a
modified version of the original text by retaining key phrases, it does not generate entirely new content.
Prerequisites:
1. spaCy: A Python library for NLP tasks.
2. PyTextRank: A spaCy extension that implements the TextRank algorithm.
To install spaCy and the required language model, run the following commands:
!pip install spacy
!python3 -m spacy download en_core_web_lg
To install PyTextRank, run:
!pip install pytextrank
Here’s a simple implementation of spaCy and PyTextRank for automatic text summarization. The code
installs the required packages, downloads the spaCy language model, and processes a lengthy text to
extract key phrases and sentences. The summary is limited to two key phrases and two sentences.
import spacy
import pytextrank

nlp = [Link]("en_core_web_lg")
nlp.add_pipe("textrank")

example_text = """
Deep learning (also known as deep structured learning) is part of a broader family of machine learning
methods based on artificial neural networks with representation learning. Learning can be supervised,
semi-supervised or unsupervised. Deep-learning architectures such as deep neural networks, deep belief
networks, deep reinforcement learning, recurrent neural networks and convolutional neural networks
have been applied to fields including computer vision, speech recognition, natural language processing,
machine translation, bioinformatics, drug design, medical image analysis, material inspection and board
game programs, where they have produced results comparable to and in some cases surpassing human
expert performance. Artificial neural networks (ANNs) were inspired by information processing and
distributed communication nodes in biological systems. ANNs have various differences from biological
brains. Specifically, neural networks tend to be static and symbolic, while the biological brain of most
living organisms is dynamic (plastic) and analogue. The adjective "deep" in deep learning refers to the
use of multiple layers in the network. Early work showed that a linear perceptron cannot be a universal
classifier, but that a network with a nonpolynomial activation function with one hidden layer of
unbounded width can.
Deep learning is a modern variation which is concerned with an unbounded number of layers of bounded
size, which permits practical application and optimized implementation, while retaining theoretical
universality under mild conditions. In deep learning the layers are also permitted to be heterogeneous and
to deviate widely from biologically informed connectionist models, for the sake of efficiency, trainability
and understandability, whence the structured part."""
print('Original Document Size:',len(example_text))
doc = nlp(example_text)

for sent in doc._.[Link](limit_phrases=2, limit_sentences=2):

print(sent)
print('Summary Length:',len(sent))

Output:
Original Document Size: 1808
Deep-learning architectures such as deep neural networks, deep belief networks, deep reinforcement
learning, recurrent neural networks and convolutional neural networks have been applied to fields
including computer vision, speech recognition, natural language processing, machine translation,
bioinformatics, drug design, medical image analysis, material inspection and board game programs,
where they have produced results comparable to and in some cases surpassing human expert
performance.
Summary Length: 76
Specifically, neural networks tend to be static and symbolic, while the biological brain of most living
organisms is dynamic (plastic) and analogue.
Summary Length: 27

Abstractive summarization generates entirely new sentences to convey key ideas from the original text.
Unlike extractive summarization, which selects and rearranges sentences from the original content,
abstractive methods rephrase information in a more concise and coherent manner, often using new
vocabulary that wasn't present in the original.
Abstractive summarization has gained prominence with the advent of transformer, which have
revolutionized NLP tasks. Initially, models based on recurrent neural network were used for text
summarization, but Transformers introduced a unique architecture that significantly improved
performance.
Note: Not all Transformer models are designed for text summarization. One of the most notable models
in this domain is PEGASUS, which has shown superior performance in generating high-quality
summaries.
PEGASUS: A Transformer Model for Text Summarization
PEGASUS is a Transformer-based model designed specifically for text summarization. Unlike other
models, PEGASUS uses a unique pre-training strategy where critical sentences are masked during
training. The model is then tasked with generating these hidden sentences, which enables it to create more
accurate and coherent summaries.
To use the PEGASUS model for text summarization, you need to install the following libraries and
frameworks:
!pip install git+[Link]
!pip install git+[Link]
!pip install sentencepiece
!pip install git+[Link]
!pip install pegasus
Once the dependencies are installed, you can begin summarizing text with the PEGASUS model. Below
is an example code snippet that uses the hugging face library to load the model, tokenize the input text,
generate a summary, and display it.
from transformers import pipeline
from transformers import PegasusForConditionalGeneration, PegasusTokenizer

# Pick model
model_name = "google/pegasus-xsum"
# Load pretrained tokenizer
pegasus_tokenizer = PegasusTokenizer.from_pretrained(model_name)

example_text = """Deep learning (also known as deep structured learning) is part of a broader family of
machine learning methods based on artificial neural networks with representation learning. Learning can
be supervised, semi-supervised or unsupervised. Deep-learning architectures such as deep neural
networks, deep belief networks, deep reinforcement learning, recurrent neural networks and
convolutional neural networks have been applied to fields including computer vision, speech recognition,
natural language processing, machine translation, bioinformatics, drug design, medical image analysis,
material inspection and board game programs, where they have produced results comparable to and in
some cases surpassing human expert performance. Artificial neural networks (ANNs) were inspired by
information processing and distributed communication nodes in biological systems. ANNs have various
differences from biological brains. Specifically, neural networks tend to be static and symbolic, while the
biological brain of most living organisms is dynamic (plastic) and analogue. The adjective "deep" in deep
learning refers to the use of multiple layers in the network.
Early work showed that a linear perceptron cannot be a universal classifier, but that a network with a
nonpolynomial activation function with one hidden layer of unbounded width can. Deep learning is a
modern variation which is concerned with an unbounded number of layers of bounded size, which permits
practical application and optimized implementation, while retaining theoretical universality under mild
conditions. In deep learning the layers are also permitted to be heterogeneous and to deviate widely from
biologically informed connectionist models, for the sake of efficiency, trainability and understandability,
whence the structured part."""

print('Original Document Size:',len(example_text))

# Define PEGASUS model
pegasus_model = PegasusForConditionalGeneration.from_pretrained(model_name)
# Create tokens
tokens = pegasus_tokenizer(example_text, truncation=True, padding="longest", return_tensors="pt")

# Generate the summary

encoded_summary = pegasus_model.generate(**tokens)

# Decode the summarized text

decoded_summary = pegasus_tokenizer.decode(encoded_summary[0], skip_special_tokens=True)

# Print the summary

print('Decoded Summary :',decoded_summary)

summarizer = pipeline(
"summarization",
model=model_name,
tokenizer=pegasus_tokenizer,
framework="pt"
)

summary = summarizer(example_text, min_length=30, max_length=150)

summary[0]["summary_text"]
Output:
Original Document Size: 1825 Decoded Summary : Deep learning is a branch of computer science that
deals with the study and training of machine learning. 'Deep learning is a branch of computer science
which deals with the study and training of complex systems such as speech recognition, natural language
processing, machine translation and medical image analysis. Deep-learning architectures such as deep
neural networks, deep belief networks, deep reinforcement learning, recurrent neural networks and
neuralal networks have been applied to fields including computer vision, speech recognition, natural
language processing, machine translation, bioinformatics, drug design, medical image analysis, material
inspection and board game programs, where they have produced results comparable to and in some cases
surpassing human expert performance.'
Conclusion
The future of text summarization looks promising, with advancements in both extractive and abstractive
methods, powered by models like PEGASUS. As these techniques evolve, they will enable more accurate
and intuitive summarization, transforming how we process vast amounts of information. This progress
highlights the growing potential of AI in enhancing human comprehension and knowledge management.

INFORMATION EXTRACTION AND FEATURE METRIX

Information Extraction (IE) finds useful, structured data (like names, dates, relations) from unstructured
text, while
Feature extraction converts this (and other raw data like images) into numerical formats (features) for
machines, often using techniques like matric factorization to create a Feature Matrix, which is a table of
data points (rows) and their characteristics (columns), essential for tasks like pattern recognition and
classification. Essentially, IE pulls out what to look at, and Feature Extraction transforms it into a usable
matrix for algorithms.
Information Extraction (IE)
• Goal: Identify and extract specific, structured information from unstructured data (text, audio).
• Examples:
o Named Entity Recognition (NER): Finding names, locations, organizations.
o Relation Extraction: Discovering relationships between entities (e.g., "X works at Y").
o Opinion Mining: Determining sentiment (positive/negative).
• Applications: Search engines, databases, business intelligence, healthcare.
•
Feature Extraction & The Feature Matrix
• Feature Extraction: Transforming raw data into a concise set of numerical features, reducing
dimensionality while preserving key information.
• Feature Matrix: A table (often numerical) where rows are data instances (e.g., documents,
images) and columns are extracted features (e.g., word counts, pixel values, derived metrics).
• How it works with IE: After IE identifies entities, Feature Extraction converts these textual
mentions (like "Barack Obama") into numerical vectors or attributes that a machine learning
model can process.
• Key Techniques:
o Vectorization (e.g., DictVectorizer): Converts dictionaries of features into matrices.
o Matrix Factorization (e.g., PCA, SVD, NMF): Decomposes large matrices to find
underlying patterns and create fewer, more meaningful features.
• Purpose: Makes data manageable for algorithms, improves accuracy, and speeds up processing

SINGULAR VALUE DECOMPOSITION

Singular Value Decomposition (SVD) Singular Value Decomposition (SVD) is a factorization method
in linear algebra that decomposes a matrix into three other matrices, providing a way to represent data in
terms of its singular values.
SVD helps you split that table into three parts:
• U: This part tells you about the people (like their general preferences).
• Σ: This part shows how important each factor is (how much each rating matters).
• Vᵀ: This part tells you about the products (how similar they are to each other)
Lets understand this with help of an example: Suppose you have a small table of people’s ratings for two
movies,
Name Movie 1 Rating Movie 2 Rating
Amit 5 3
Sanket 4 2
 Harsh 2 5
• SVD breaks this table into three smaller parts: one that shows people’s preferences, one that shows
the importance of each movie, and one that shows how similar the movies are to each other
• Mathematically, the SVD of a matrix 𝐴(of size 𝑚 × 𝑛) is represented as: 𝐴 = 𝑈Σ𝑉 𝑇
Here:
• 𝑈: An 𝑚 × 𝑚orthogonal matrix whose columns are the left singular vectors of 𝐴.
• Σ: A diagonal 𝑚 × 𝑛matrix containing the singular values of 𝐴in descending order.
• 𝑉 𝑇 : The transpose of an 𝑛 × 𝑛orthogonal matrix, where the columns are the right singular vectors
of 𝐴.
KEYWORD EXTRACTION AND TOPIC MODELLING
Keyword Extraction Methods in NLP
Keyword extraction is a vital task in Natural Language Processing (NLP) for identifying the most relevant
words or phrases from text, and enhancing insights into its content.
Significance of Keyword Extraction in NLP
Keyword extraction is a technique used to identify and extract the most relevant words or phrases from
a piece of text. The significance of keyword extraction in NLP discussed below:
• Information Retrieval: Keywords function as queries to retrieve pertinent items from extensive
text collections or databases.
• Document summarization involves utilizing extracted keywords to create succinct summaries
of documents, effectively encapsulating their fundamental substance.
• Text categorization and classification involve the use of keywords to determine the primary
subjects or categories of documents, which aids in the process of classifying them.
• Search Engine Optimization (SEO): Keywords are essential for enhancing the visibility and
ranking of site content in search engine results pages.
• Keyword extraction is an essential stage in topic modeling techniques as it aids in identifying
the fundamental themes or topics present in a collection of documents.
Libraries Required for Keyword Extraction
• NLTK: provides a range of modules for text processing. These modules include routines for TF-
IDF and TextRank-based keyword extraction.
• YAKE library: offers a Python version of the YAKE algorithm, which is used for unsupervised
keyword extraction.
• RAKE is not a library, but it may be constructed easily using Python's string manipulation tools
and basic text processing techniques.
• KeyBert: It works on a transformer model (BERT) which uses bert embeddings to identify the
most similar keywords and phrases within a large document.

KeyWord Extraction using TestRank

TestRank is an algorithm used for keyword extraction in the context of natural language processing (NLP)
and information retrieval. It was developed to improve upon traditional methods of keyword extraction
by considering the structure and content of a document in a more relevant way.
• Text Preprocessing: The algorithm starts by preprocessing the text, which involves tasks like
tokenization (breaking the text into words or phrases), removing stop words (common words like
"the," "and," "is" that don't carry significant meaning), and possibly stemming or lemmatization
(reducing words to their root form).
• Building a Graph: TestRank constructs a graph representation of the text, where nodes
represent words or phrases, and edges represent relationships between them. These
 relationships could be based on co-occurrence within sentences or paragraphs, semantic
similarity, or other linguistic features.
• Scoring Nodes: Each node (word or phrase) in the graph is assigned a score based on various
factors. TestRank typically considers factors such as the node's degree (how many other nodes it
is connected to), its centrality within the graph, its position within the document, and possibly
its semantic relevance to the overall content.
• Ranking Keywords: After scoring all nodes in the graph, TestRank ranks them based on their
scores. The highest-scoring nodes are considered the most important keywords or phrases in
the document.
• Keyword Extraction: Finally, TestRank selects the top-ranked nodes as the extracted
keywords for the document. These keywords are representative of the main themes or topics
discussed in the text.
Implementation of Textrank Using Python
1. Install PyTextRank: !pip3 install pytextrank installs the PyTextRank package.
2. Setup spaCy and PyTextRank: Load spaCy's English model and add PyTextRank to the pipeline
with nlp = [Link]("en_core_web_sm") and nlp.add_pipe("textrank").
3. Process Text: Process the text using doc = nlp("TextRank is a keyword extraction algorithm...")
to apply TextRank.
4. Extract Keywords: Iterate over doc._.phrases[:10] to print the top-ranked phrases, which are the
extracted keywords.
# Installation
!pip3 install pytextrank

import spacy
import pytextrank

# example text
text = "TextRank is a keyword extraction algorithm based on PageRank and has been widely used in
natural language processing tasks."

# load a spaCy model, depending on language, scale, etc.

nlp = [Link]("en_core_web_sm")

# add PyTextRank to the spaCy pipeline

nlp.add_pipe("textrank")
doc = nlp(text)

# examine the top-ranked phrases in the document

for phrase in doc._.phrases[:10]:
print([Link])
Output:
PageRank
a keyword extraction algorithm
TextRank

TOPIC MODELLING
TOPIC MODELING - TYPES, WORKING, APPLICATIONS
As the extent and complexity of records continue to grow exponentially, traditional evaluation strategies
are falling quickly when it comes to making experience of unstructured information, along with text, snap
shots, and audio. This is wherein the importance of advanced analytics techniques, like topic modelling,
comes into play.

Understanding Topic Modelling

Topic modeling is a technique in NLP and machine learning that aims to uncover latent thematic
structures within a collection of texts. Topic modelling is a system learning technique that robotically
discovers the principle themes or "topics" that represents a huge collection of documents. The intention
of topic modelling is to discover the hidden semantic systems within textual content facts, permitting
customers to arrange, apprehend, and summarize the data in a manner that is each green and insightful.
At the coronary heart of topic modelling, the concepts of "topics" and "topic models" comes into mind.
A 'topic' is defined as a recurring pattern of words that best represents a theme within the
documents. Topic models are algorithms that scan the document collection to discover these topics.
They provide a way to quantify the structure of topics within the text and how these topics are
related to each other.
Imagine you have a big pile of books, however you don't know what they may be about. Topic modeling
allows you go through them. It seems for words that regularly dangle out together, like "pizza" and
"cheese" or "dog" and "bark." By recognizing these phrase together, subject matter modeling figures out
which book is especially speaking about.
Importance of Topic Modelling
Topic modelling is a powerful text mining approach that allows researchers, businesses, and selection-
makers to discover the hidden thematic structures within big collections of unstructured textual content
facts. Its importance may be summarized as follows:
• Extracting Insights from Unstructured Data : Topic modelling enables the evaluation of
unstructured records, inclusive of files, articles, and social media posts, which make up 80-90%
of all new company facts. It lets in companies to derive precious insights from this enormous
trove of unstructured statistics that would in any other case be tough to procedure manually.
• Improving Content Organization and Retrieval: By robotically figuring out the primary
subjects within a corpus of text, subject matter modelling may be used to cluster and prepare big
report collections, making it simpler to look, navigate, and retrieve applicable statistics.
• Enhancing Customer Experience and Personalization: Topic modelling can be carried out to
patron feedback, evaluations, and social media information to uncover the important thing topics
and sentiments which might be essential to clients. This data can then be used to improve
merchandise, offerings, and personalised suggestions.
• Accelerating Research and Discovery: In educational and scientific domains, subject matter
modelling has been used to research massive bodies of literature, discover rising research trends,
and discover connections between disparate fields, accelerating the pace of studies and
innovation.
• Automating Repetitive Tasks
• : By mechanically categorizing and organizing text information based on subjects, topic
modelling can help automate many time-eating and repetitive duties, inclusive of customer
service ticket tagging, file class, and content material summarization.
 • Enabling Trend Analysis and Monitoring: Topic modelling may be used to music modifications
in subject matter distributions over the years, allowing groups to locate rising developments, shifts
in public opinion, and other patterns that can be applicable for strategic selection-making.
In summary, the importance of subject matter modelling lies in its capability to extract significant insights
from unstructured records, enhance information enterprise and retrieval, enhance client stories, accelerate
studies and discovery, automate repetitive tasks, and allow trend evaluation – all of that may have a large
effect on commercial enterprise operations, choice-making, and innovation.

How do Topic Modeling Works?

Topic modeling work by means of studying the co-occurrence styles of phrases inside a corpus of
documents. By identifying the phrases that frequently appear together, the algorithm can infer the latent
topics that are gift inside the information. This method is normally performed in an unmanaged way,
which means that the model discovers the topics without any prior understanding or labeling of the files.

Imagine a detective tasked with unraveling a mystery with none prior clues or suspects. Topic modeling
operates in a comparable fashion, piecing collectively the narrative hidden in the textual content, guided
completely by the subtle cues embedded within the co-incidence patterns of words. Through this
unsupervised exploration, the set of rules unveils the underlying shape of the corpus, illuminating the
hidden topics and subjects that outline its essence.
Types of Topic Modeling Techniques
While there are numerous topic modelling techniques to be had, of the most broadly used and properly-
mounted techniques are Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA).
Latent Semantic Analysis (LSA): is a topic modelling method that makes use of a mathematical method
known as singular value decomposition to identify the underlying semantic standards inside a corpus of
text. LSA assumes that there's an inherent shape in word utilization that may be captured via the
relationships between words and documents.
The LSA algorithm works via building a term-file matrix, which represents the frequency of every word
in each record. It then applies SVD to this matrix, decomposing it into 3 matrices that seize the
relationships among phrases, documents, and the latent topics then ensuing topic representations may be
used to apprehend the thematic structure of the textual content corpus and to perform duties which include
record clustering, records retrieval, and text summarization.
Latent Dirichlet Allocation (LDA): is some other extensively used subject matter modelling technique
that takes a probabilistic method to discovering the hidden thematic shape of a textual content corpus.
Unlike LSA, which makes use of a linear algebraic method, LDA is a generative probabilistic version
that assumes each report is a combination of a small number of subjects, and that every word's creation
is as a result of one of the record's subjects.
The LDA algorithm works by means of assuming that each file in the corpus is composed of a
combination of subjects, and that each topic is characterised by means of a distribution over the
vocabulary. The version then iteratively updates the topic-phrase and report-subject matter distributions
to maximise the probability of the found facts. The resulting topic representations can be used to
understand the thematic shape of the textual content corpus and to carry out tasks which include file type,
advice, and exploratory analysis.
How Topic Modeling is Implemented?
Implementing topic modelling in practice involves several key steps, such as statistics evaluation,
preprocessing, and model fitting. For this tutorial we'll proceed with random generated dataset, and see
how can we implement topic modeling. The steps are followed below:
Step 1. Data Preparation: The first step in implementing topic modelling is to put together the text
documents. This usually entails amassing and organizing the applicable documents, making sure that the
records is in a appropriate layout for analysis.
Step 2. Preprocessing Steps: Before proceeding to model fitting, it's far vital to process and textual
context to enhance the exceptional of the consequences. Common preprocessing steps include:
• Stopword Removal: Removing not unusual words that do not carry any meaning, which includes
"the," "a," and "is."
• Punctuation Removal: Removing punctuation marks and special characters from the text.
• Lemmatization: Reducing phrases to their base or dictionary form, to improve the consistency of
the vocabulary.
Step 3. Creating Document-Term Matrix: After preprocessing the textual content, the following step
is to create a document-time-matrix which represents the frequency of every phrase in every report. This
matrix serves because the input to the topic modelling algorithms.
Step 4: Model Fitting: Once the data is prepared, the next step is to match the topic modelling algorithm
to the facts. This includes specifying the number of subjects to be observed and going for walks the
algorithm to reap the topic representations.
• For LSA, this entails applying Singular Value Decomposition (SVD) to the document-term matrix
to extract the latent subjects.
• For LDA, this involves iteratively updating the subject-phrase and record-subject matter
distributions to maximise the probability of the discovered facts.
Applications of Topic Modeling
Topic modeling has numerous applications across various fields:
• Content Recommendation: By understanding the topics within documents, content
recommendation system can suggest articles, books, or media that match a user's interests.
• Document Classification: It helps in automatically classifying documents into predefined
categories based on their content.
• Summarization: Topic modeling can assist in summarizing large collections of documents by
highlighting the main themes.
• Trend Analysis: In business and social media, topic modeling can identify trends and shifts in
public opinion by analyzing textual data over time.
• Customer Feedback Analysis: Companies use topic modeling to analyze customer reviews and
feedback to identify common issues and areas for improvement.
Advantages of Topic Modeling
• Unsupervised Learning: Topic modeling does not require labeled data, making it suitable for
exploring unknown corpora.
 • Scalability: It can handle large volumes of text data efficiently.
• Insight Generation: Provides meaningful insights by uncovering hidden structures in the data.
Challenges in Topic Modeling
• Interpretability: The extracted topics might not always be easily interpretable, requiring human
intervention to label and understand.
• Parameter Sensitivity: Algorithms like LDA require setting several hyperparameters (e.g.,
number of topics), which can significantly impact results.
• Quality of Text: The effectiveness of topic modeling depends on the quality and cleanliness of
the input text.
Conclusion
Topic modelling has emerged as a powerful device for extracting meaningful insights from large and
unstructured datasets, records of text information. By uncovering the hidden thematic structures within
documents, topic modelling allows researchers, entrepreneurs, and decision-makers to benefit a deeper
information of the underlying patterns and trends, ultimately using extra knowledgeable and strategic
decision-making. As the volume and complexity of records keep growing, the importance of advanced
analytics strategies like subject matter modelling will most effective hold to increase, making it an
essential skill for everyone interested by leveraging the electricity of data to pressure innovation and
development.

AUTOMATED DOCUMENT SUMMARIZATION

Automated document summarization is
an application of Natural Language Processing (NLP) and Large Language Models (LLMs) that uses
computer programs to condense large volumes of text into concise, coherent summaries while retaining
essential information. This process is crucial for managing information overload and speeding up
decision-making.
Key Approaches
The primary methods for automated document summarization are extractive and abstractive.
• Extractive Summarization: This method identifies and extracts the most important sentences or
phrases directly from the original document and concatenates them to form a summary. The
extracted content is not modified, which helps preserve factual accuracy and grammatical
correctness.
• Abstractive Summarization: This technique involves generating new sentences and phrases that
capture the core meaning of the original text, much like a human would. It builds an internal
semantic representation of the content and then uses natural language generation to create a novel,
coherent summary, which may contain words not present in the original document.
• Hybrid/Multi-level Summarization: This approach combines both methods, often by first
extracting key information and then using abstractive techniques to refine the result into a more
concise and readable form. This is particularly useful for very long documents that exceed the
input limits of some models.

Technologies and Techniques

Modern automated summarization heavily relies on artificial intelligence algorithms, particularly deep
learning models like transformers (e.g., BERT, BART, T5). These models use various techniques:
• Feature-based methods: Early methods used linguistic or statistical features like word
frequency, sentence position, and title similarity to score sentence importance.
 • Graph-based methods: These model text as a graph where sentences are nodes and edges
represent semantic or lexical similarity (e.g., TextRank, LexRank). A ranking algorithm then
selects the most important sentences.
• Machine Learning/Deep Learning: Supervised and unsupervised learning, along with neural
network architectures (like RNNs, LSTMs, and Transformers), are used to learn from large
datasets and generate summaries.
Applications
Automated document summarization is used across various domains to improve efficiency:
• News Aggregation: Generating headlines or short summaries of news articles.
• Academic Research: Annotating, indexing, and condensing research papers.
• Business and Finance: Creating executive summaries of business or financial reports for quicker
decision-making.
• Legal Documents: Extracting key terms and information from lengthy contracts or court
decisions.
• Healthcare: Summarizing patient records or medical literature for doctors and researchers.
Available Tools
Various AI-powered tools and software libraries allow users to implement or utilize automated
summarization. Examples include:
• Integrated tools: Features like Microsoft Copilot in Word or Google's NotebookLM can
automatically summarize documents or research materials.
• Online platforms: Tools like QuillBot’s Summarizer offer easy, customizable summarization for
general use.
• Libraries for developers: Libraries such as HuggingFace's Transformers provide pre-trained
models for building custom summarization solutions in Python
WHAT IS INFORMATION RETRIEVAL?
Information Retrieval: helps to find relevant information from large collections of documents. It can
be defined as a software program that deals with the organization, storage, retrieval and evaluation
of information from documents. It is like a smart librarian who doesn’t give you direct answers but tells
you where to find the right book like this IR system scans them and pulls out the ones that match your
query.
When you search for something Information Retrieval (IR) model helps find the most relevant
documents and ranks them based on your query. It works by comparing your query with documents in
the system using a matching function. This function gives each document a retrieval status value
(RSV) which helps rank the most relevant results first. To do this IR systems represent documents using
descriptors i.e most important keywords from vocabulary (V).
Estimation of the probability of user’s relevance rel for each document d and query q with respect to a
set R q of training documents: Prob(rel ∣ 𝑑, 𝑞, 𝑅𝑞 )
Components of Information Retrieval/ IR Model
The Information Retrieval (IR) model can be broken down into key components that involve both the
system and the user. Here’s how it works in a simple flow:

1. User Side (Search Process)

• Problem Identification: A student wants to learn about machine learning and types a query into
a search engine.
• Representation: The user converts their need into a search query using keywords or phrases like
instead of asking "How do machines learn?" the student types "machine learning basics" into
Google and the problem is converted into a query (keywords or phrases).
• Query: The user submits the search query into IR system.
• Feedback: User can refine or modify the search based on the retrieved results.
2. System Side (Retrieval Process)
• Acquisition: The system collects and stores a large number of documents or data sources. It can
includes web pages, books, research papers or any text-based information.
• Representation: Each document in the system is analyzed and represented in a structured way
using keywords (terms). Example: If the document talks about "machine learning" it is tagged
with relevant terms like "AI, deep learning, algorithms, models" to help retrieval.
• File Organization: The documents are indexed and stored efficiently so the system can quickly
find relevant ones. Like organizing a library so books can be found easily based on topics.
• Matching: The system compares the user's search query with stored documents to find the
best matches. It uses matching functions that rank documents based on relevance.
 • Retrieved Object: The system returns the most relevant documents to the user. These
documents are ranked so the most useful ones appear at the top.
3. Interaction Between User & System
• The user reviews the retrieved results and may provide feedback to refine the search. The system
then processes the updated query and retrieves better results.

• Acquisition: In this step the selection of documents and other objects from various web resources
that consist of text-based documents takes place. The required data is collected by web crawlers
and stored in the database.
• Representation: It consists of indexing that contains free-text terms, controlled vocabulary,
manual and automatic techniques as well. Example: Abstracting contains summarizing and
Bibliographic description that contains author, title, sources, data and metadata.
• File Organization: There are two types of file organization methods. i.e. Sequential that contains
documents by document data and Inverted: that contains list of records under each term.
• Query: An IR process starts when a user enters a query into the system. Queries are formal
statements of information needs. For example, search strings in web search engines. In IR a query
does not uniquely identify a single object in the collection. Instead several objects may match the
query, perhaps with different degrees of relevancy.
Difference Between Information Retrieval and Data Retrieval
Information Retrieval Data Retrieval
The software program that deals with the Data retrieval deals with obtaining data from a database
organization, storage, retrieval and management system such as ODBMS. It is A process of
evaluation of information from document identifying and retrieving the data from the database
repositories particularly textual information. based on the query provided by user or application.
Retrieves information about a subject. Determines the keywords in the user query and
retrieves the data.
Small errors are likely to go unnoticed. A single error object means total failure.
Not always well structured and is Has a well-defined structure and semantics.
semantically ambiguous.
Does not provide a solution to the user of the Provides solutions to the user of the database system.
database system.
The results obtained are approximate The results obtained are exact matches.
matches.
Results are ordered by relevance. Results are unordered by relevance.
It is a probabilistic model. It is a deterministic model.
Advantages of Information Retrieval
• Efficient Access: Information retrieval techniques make it possible for users to easily locate and
retrieve vast amounts of data or information.
• Personalization of Results: User profiling and personalization techniques are used to tailor
search results to individual preferences and behaviors.
• Scalability: They are capable of handling increasing data volumes.
• Precision: These systems can provide highly accurate and relevant search results and reducing
the likelihood of irrelevant information appearing in search results.
Disadvantages of Information Retrieval
• Information Overload: When a lot of information is available users often face information
overload making it difficult to find most useful and relevant material.
• Lack of Context: They may fail to understand the context of a user's query leading to inaccurate
results.
• Privacy and Security Concerns: They often access sensitive user data that can raise privacy and
security concerns.
• Maintenance Challenges: Keeping these systems up-to-date and effective requires a lot of efforts
including regular updates, data cleaning and algorithm adjustments.
• Bias and fairness: Ensure that systems do not exhibit biases and provide fair and unbiased results.
 FEATURE ENGINEERING
Feature Engineering is the process of selecting, creating or modifying features like input variables or
data to help machine learning models learn patterns more effectively. It involves transforming raw data
into meaningful inputs that improve model accuracy and performance.
Feature Engineering Architecture
This step may include handling missing values, encoding categories, scaling numbers, creating new
features or combining existing ones. It helps turn messy real-world data into a form that models can
understand and use for better predictions.
Importance of Feature Engineering
Feature engineering can significantly influence model performance. By refining features, we can:
• Improve accuracy: Choosing the right features helps the model learn better, leading to more
accurate predictions.
• Reduce overfitting: Using fewer, more important features helps the model avoid memorizing the
data and perform better on new data.
• Boost interpretability: Well-chosen features make it easier to understand how the model makes
its predictions.
• Enhance efficiency: Focusing on key features speeds up the model’s training and prediction
process, saving time and resources.
Processes Involved in Feature Engineering
Processes involved in Feature Engineering
1. Feature Creation: Feature creation involves generating new features from domain knowledge or by
observing patterns in the data. It can be:
1. Domain-specific: Created based on industry knowledge like business rules.
2. Data-driven: Derived by recognizing patterns in data.
3. Synthetic: Formed by combining existing features.
2. Feature Transformation: Transformation adjusts features to improve model learning:
1. Normalization & Scaling: Adjust the range of features for consistency.
2. Encoding: Converts categorical data to numerical form i.e one-hot encoding.
3. Mathematical transformations: Like logarithmic transformations for skewed data.
3. Feature Extraction: Extracting meaningful features can reduce dimensionality and improve model
accuracy:
• Dimensionality reduction: Techniques like PCA reduce features while preserving important
information.
• Aggregation & Combination: Summing or averaging features to simplify the model.
4. Feature Selection: Feature selection involves choosing a subset of relevant features to use:
• Filter methods: Based on statistical measures like correlation.
• Wrapper methods: Select based on model performance.
• Embedded methods: Feature selection integrated within model training.
5. Feature Scaling: Scaling ensures that all features contribute equally to the model:
• Min-Max scaling: Rescales values to a fixed range like 0 to 1.
• Standard scaling: Normalizes to have a mean of 0 and variance of 1.
Steps in Feature Engineering
Feature engineering can vary depending on the specific problem but the general steps are:
1. Data Cleaning: Identify and correct errors or inconsistencies in the dataset to ensure data quality
and reliability.
 2. Data Transformation: Transform raw data into a format suitable for modeling including scaling,
normalization and encoding.
3. Feature Extraction: Create new features by combining or deriving information from existing
ones to provide more meaningful input to the model.
4. Feature Selection: Choose the most relevant features for the model using techniques like
correlation analysis, mutual information and stepwise regression.
5. Feature Iteration: Continuously refine features based on model performance by adding,
removing or modifying features for improvement.
Common Techniques in Feature Engineering
1. One-Hot Encoding: converts categorical variables into binary indicators, allowing them to be used by
machine learning models.
import pandas as pd

data = {'Color': ['Red', 'Blue', 'Green', 'Blue']}

df = [Link](data)

df_encoded = pd.get_dummies(df, columns=['Color'], prefix='Color')

print(df_encoded)

Output
Color_Blue Color_Green Color_Red
0 False False True
1 True False False
2 False True False
3 True False False
2. Binning: transforms continuous variables into discrete bins, making them categorical for easier
analysis.
import pandas as pd

data = {'Age': [23, 45, 18, 34, 67, 50, 21]}

df = [Link](data)

bins = [0, 20, 40, 60, 100]

labels = ['0-20', '21-40', '41-60', '61+']

df['Age_Group'] = [Link](df['Age'], bins=bins, labels=labels, right=False)

print(df)

Output
Age Age_Group
0 23 21-40
1 45 41-60
2 18 0-20
3 34 21-40
4 67 61+
5 50 41-60
6 21 21-40
3. Text Data Preprocessing: Involves removing stop word, stemming and vectorization text data to
prepare it for machine learning models.
import nltk
from [Link] import stopwords
from [Link] import PorterStemmer
from sklearn.feature_extraction.text import CountVectorizer

texts = ["This is a sample sentence.", "Text data preprocessing is important."]

stop_words = set([Link]('english'))
stemmer = PorterStemmer()
vectorizer = CountVectorizer()

def preprocess_text(text):
words = [Link]()
words = [[Link](word)
for word in words if [Link]() not in stop_words]
return " ".join(words)

cleaned_texts = [preprocess_text(text) for text in texts]

X = vectorizer.fit_transform(cleaned_texts)

print("Cleaned Texts:", cleaned_texts)

print("Vectorized Text:", [Link]())

4. Feature Splitting: Divides a single feature into multiple sub-features, uncovering valuable insights
and improving model performance.
import pandas as pd

data = {'Full_Address': [
'123 Elm St, Springfield, 12345', '456 Oak Rd, Shelbyville, 67890']}
df = [Link](data)

df[['Street', 'City', 'Zipcode']] = df['Full_Address'].[Link](

r'([0-9]+\s[\w\s]+),\s([\w\s]+),\s(\d+)')

print(df)

Output
Full_Address Street City Zipcode
0 123 Elm St, Springfield, 12345 123 Elm St Springfield 12345
1 456 Oak Rd, Shelbyville, 67890 456 Oak Rd Shelbyville 67890...
Tools for Feature Engineering
There are several tools available for feature engineering. Here are some popular ones:
• Featuretools: Automates feature engineering by extracting and transforming features from
structured data. It integrates well with libraries like pandas and scikit-learn making it easy to
create complex features without extensive coding.
• TPOT: Uses genetic algorithms to optimize machine learning pipelines, automating feature
selection and model optimization. It visualizes the entire process, helping you identify the best
combination of features and algorithms.
• DataRobot: Automates machine learning workflows including feature engineering, model
selection and optimization. It supports time-dependent and text data and offers collaborative tools
for teams to efficiently work on projects.
• Alteryx: Offers a visual interface for building data workflows, simplifying feature extraction,
transformation and cleaning. It integrates with popular data sources and its drag-and-drop
interface makes it accessible for non-programmers.
• [Link]: Provides both automated and manual feature engineering tools for a variety of data types.
It includes features for scaling, imputation and encoding and offers interactive visualizations to
better understand model results.
TEXT SIMILARITY, ANALYZING TERM SIMILARITY AND ANALYZING DOCUMENT
SIMILARITY.
Text similarity analysis, a core NLP task, measures how alike texts are, distinguishing between Lexical
similarity (word/character match) and Semantic Similarity (meaning), using techniques like TF-IDF &
Cosine Similarity for documents and semantic embeddings (Word2Vec) for deeper meaning, crucial for
tasks like plagiarism detection, search, and summarization.
Analyzing Term Similarity
• Definition: How similar individual words or short phrases are in meaning or usage, even if they
aren't identical.
• Methods:
o Word embedding (Word2Vec, GloVe): Represent words as dense vectors, where similar
words have close vectors in a multi-dimensional space.
o Synonyms Dictionaries/Knowledge Graphs: Use external resources (like WordNet) to
find semantic relationships.
o Contextual Analysis: Analyze surrounding words to understand meaning (e.g., using
Transformers).
Analyzing Document Similarity
• Definition: Measuring the overall closeness in content or meaning between two or more
documents.
• Steps & Techniques:
1. Preprocessing: Clean text (remove stop words, punctuation).
2. Representation: Convert text into numerical vectors.
▪ Bag-of-Words/TF-IDF: Counts word occurrences, weighting important terms
(Term Frequency-Inverse Document Frequency).
▪ Embeddings: Use document-level embeddings (from Word2Vec or advanced
models).
3. Similarity Metric: Calculate a score (usually 0 to 1).
▪ Vosine Similarity: Measures the angle between document vectors (common with
TF-IDF/embeddings).
▪ Jaccard Similarity: Based on shared unique words.
▪ Euclidean Distance: Measures straight-line distance between vectors.
• Applications: Plagiarism detection, recommendation systems, document clustering, information
retrieval, and summarization.
•
Key Approaches
• Lexical: Focuses on exact words (e.g., Jaccard, N-grams).
• Statistical/Vector-Based: Uses term frequencies (TF-IDF, Cosine Similarity).
• Semantic/Deep Learning: Uses embeddings to capture meaning (Word2Vec, BERT).

DOCUMENT CLUSTERING
Document clustering (or text clustering) is an unsupervised learning technique in data mining and NLP
that automatically groups similar documents into clusters (categories) based on shared content, without
prior knowledge of the categories, helping organize large text collections, summarize topics, and improve
information retrieval. It works by converting documents into numerical vectors (like TF-IDF), then using
algorithms (like K-Means) to find groupings where internal documents are highly similar, and external
documents are dissimilar.
Key Concepts
• Unsupervised Learning: Unlike classification (supervised), clustering discovers hidden
structures without labeled examples, finding its own groups.
• Similarity: Documents are grouped by shared properties, often based on word usage (e.g., "bag-
of-words").
• Clusters: Subsets of documents where members are alike, but different from members of other
clusters.
• Vectorization: Converting text into numerical vectors (e.g., using TF-IDF) for mathematical
processing.
•
Process
1. Text Preprocessing: Clean text by removing stop words ("the," "a"), stemming words (e.g.,
"running" to "run"), and handling punctuation.
2. Vectorization: Convert processed text into numerical vectors (e.g., using TF-IDF) where each
dimension represents a word's importance.
3. Clustering Algorithm: Apply algorithms like K-Means, Hierarchical, or DBSCAN to group
similar vectors.
4. Evaluation: Assess cluster quality (e.g., using silhouette scores).
4.
Applications
• Automatic Organization: Structuring large document repositories.
• Popic Extraction: Identifying prevalent themes in text.
• Information Retrieval: Improving search results.
• Recommendation Systems: Suggesting related articles or content.
•
Types of Clustering
• Disjoint: Each document belongs to exactly one cluster.
• Overlaping: A document can belong to multiple clusters (e.g., fuzzy clustering).
Clustering Greatest Movies of All Time and Semantic Analysis.
Clustering the "greatest movies of all time" using semantic analysis involves applying machine learning
techniques to group films based on the deeper meaning and context of their descriptions, rather than just
keywords or basic genres
.
Methodology
The process typically involves the following steps:
1. Data Collection: Gathering data on highly-rated movies from sources like IMDb or Wikipedia
top lists, including plot summaries, genre tags, and critical reviews.
2. Text Pre-processing: Cleaning the textual data (removing stop words, punctuation, etc.) to
prepare it for analysis.
3. Semantic Analysis (Vectorization): Converting the text into numerical vectors that capture the
meaning of the words. This goes beyond traditional methods like TF-IDF (Term Frequency-
Inverse Document Frequency), which only considers word frequency. Modern approaches use
techniques like:
o Word Embeddings: Using pre-trained models (like Word2Vec or GloVe) that understand
word relationships (e.g., "king" is close to "queen" in the vector space).
 o Sentence/Document Embeddings: Using models (like BERT or Universal Sentence
Encoder) to create vectors for entire plot summaries, capturing the overall theme and
context.
4. Clustering Algorithm: Applying a clustering algorithm, commonly a semantic-based K-means
clustering algorithm, which is a modified version of the standard K-means, to group similar movie
vectors into "k" number of clusters.
5. Evaluation: Assessing the quality of the clusters using metrics like precision, recall, and F-
measure to ensure the groupings are meaningful.
Expected Outcomes
By using a semantic-based approach, the clustering can reveal nuanced groupings that might be missed
by simple keyword analysis. For example, movies might cluster around themes such as:
• "Justice and Morality": Grouping films like 12 Angry Men and Schindler's List based on their
focus on ethical dilemmas and the legal system.
• "Epic Sagas and Journeys": Grouping films like The Godfather and The Return of the King
which focus on multi-generational narratives or expansive quests.
• "Wartime Experiences": Clustering films like Apocalypse Now that explore the psychological
impact and chaos of conflict.
This method improves the quality of movie recommendations and categorization by considering context
and underlying themes rather than just surface-level similarities.

EXPLORING WORDNET
Exploring WordNet means diving into a massive lexical database that groups English words into sets of
cognitive synonyms (synsets) and links them with conceptual-semantic relationships, acting like a digital
dictionary and thesaurus but focused on word meanings and concepts, powering applications in AI, NLP,
and text analysis by providing rich semantic context. You can explore its hierarchical structures (like
'vehicle' is a 'conveyance') and relations (hypernym, hyponym, meronym) through online interfaces or
programming libraries like NLTK.
What WordNet Is
• A Lexical Database: Organizes nouns, verbs, adjectives, and adverbs into synonym sets
(synsets).
• Concept-Based: Focuses on distinct concepts rather than just words, linking them through
relationships.
• Network Structure: Interlinks synsets with semantic relations (e.g., "is-a", "part-of").
• More Than a Dictionary/Thesaurus: Provides deeper semantic connections and hierarchies,
enabling tasks like word sense disambiguation.
•
Key Concepts to Explore
• Synsets: Groups of synonyms representing a single concept (e.g., {car, automobile, motorcar}).
• Semantic relation:
o Hypernym/Hyponym: "is-a" (e.g., car -> vehicle; vehicle -> conveyance).
o Meronym/Holonym: "part-of" (e.g., wheel -> car).
• Word sense disambiguation(WSD): Using WordNet to determine the correct meaning of a word
in context.
•
How to Explore It
• Online Interface: Visit the Princeton WordNet website to browse words and their connections.
 • Programming Libraries: Use tools like NLTK (Natural Language Toolkit) in Python to access
WordNet programmatically.
• Applications: See how it's used in text summarization, sentiment analysis, machine translation,
and building knowledge graphs.

Example of Exploration
Searching for "car" might show its synset {car, auto, automobile} and link it up the hierarchy to "motor
vehicle," "wheeled vehicle," and "vehicle," and down to specific types like "roadster," demonstrating its
rich conceptual map

WORD SENSE DISAMBIGUATION IN NATURAL LANGUAGE PROCESSING

Word Sense Disambiguation (WSD) is the process of determining which meaning of a word is intended
in a particular context. It addresses the challenge of polysemy, where many words have multiple related
meanings. For example, when we encounter the sentence "I went to the bank," we automatically
understand whether "bank" refers to a financial institution or the edge of a river based on surrounding
context clues.
WSD is important in many natural language processing (NLP) applications because if word meanings
are misidentified, the results can be problematic:
• Machine translation systems may produce incorrect translations due to misinterpretation of word
meanings.
• Information retrieval systems might return irrelevant results because of semantic uncertainty or
confusion.
• Question-answering systems require precise word meanings to provide accurate and relevant
answers.
Primary Approaches to Word Sense Disambiguation
WSD techniques can be categorized into three main approaches, each with distinct methodologies and
use cases.
1. Knowledge-Based Methods
Knowledge-based approaches utilize lexical resources such as dictionaries and semantic networks to
determine word meanings. The algorithm works over this approach.
• Compare context words with dictionary definitions of candidate senses
• Calculate overlap between contextual words and definitional content
• Select the sense with maximum overlap score
Advantages:
• Does not require annotated training data
• Leverages existing linguistic knowledge bases
• Provides interpretable disambiguation decisions
The Lesk algorithm assumes that words used together in coherent text will have semantic relationships
reflected in their dictionary definitions.
2. Supervised Learning Methods
Supervised approaches treat WSD as a classification problem, training machine learning models on
datasets where word instances have been manually annotated with correct senses.
Key characteristics:
• Requires substantial amounts of sense-annotated training data
• Employs standard machine learning algorithms such as support vector machines, decision trees
or neural networks
• Uses contextual features including surrounding words and syntactic relationships
Training process:
• Extract features from annotated examples
• Train classifier to map feature vectors to sense labels
• Apply trained model to disambiguate new instances
While supervised methods achieve high accuracy, they face the challenge of obtaining sufficient
annotated data for all word-sense combinations.
3. Unsupervised Learning Methods
Unsupervised approaches operate without sense-labeled training data, instead relying on distributional
patterns in large text corpora.
Fundamental principle:
• Words appearing in similar contexts tend to have similar meanings
• Cluster word occurrences based on contextual similarity
• Assign sense labels to resulting clusters
Modern techniques:
• Utilize word embeddings and contextualized representations
• Employ clustering algorithms to group similar contexts
• Leverage large-scale language models for contextual understanding
These methods are particularly valuable when annotated data is scarce or unavailable for specific domains
or languages.
Implementation: Basic WSD System
1. Creating the Class and Sense Inventory
We create a BasicWSD class which stores a sense inventory for target words. Each word has multiple
meanings and each sense is associated with keywords that help identify it.
• self.sense_inventory: Stores each ambiguous word along with its senses and their associated
keywords.
• self.stop_words: Stores common words (e.g., the and, of) to be ignored during processing.
class BasicWSD:
def __init__(self):
self.sense_inventory = {
'bank': {
'financial': ['money', 'deposit', 'account', 'loan', 'cash', 'credit', 'savings'],
'geographical': ['river', 'water', 'shore', 'stream', 'fishing', 'boat']
},
'mouse': {
'computer': ['click', 'computer', 'cursor', 'button', 'screen', 'software'],
'animal': ['cheese', 'cat', 'rodent', 'small', 'trap', 'pet']
},
'star': {
'celebrity': ['famous', 'actor', 'movie', 'film', 'hollywood', 'performance'],
'celestial': ['sky', 'night', 'bright', 'constellation', 'galaxy', 'space']
}
}
self.stop_words = {'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at', 'to', 'for', 'of', 'with', 'by'}
2. Preprocessing the Input Sentence
We define a method to clean up the input sentence. It removes unnecessary words and punctuation so
that only meaningful context remains.
• [Link](): Converts all characters to lowercase for consistency.
 • [Link](ch, ""): Removes punctuation symbols.
• [Link](): Splits the sentence into words and filters out stop words and single-character
tokens.
def preprocess(self, sentence):
sentence = [Link]()
for ch in ".,!?;:":
sentence = [Link](ch, "")
return [word for word in [Link]() if word not in self.stop_words and len(word) > 1]
3. Disambiguating the Target Word
We now add the method that predicts the correct sense of the target word. It compares context words with
keywords for each sense.
• context = . : Extracts all context words except the target word.
• scores[sense] = len(set(context) & set(keywords)): Counts how many context words match each
sense's keywords.
• max(scores, key=[Link]): Selects the sense with the highest overlap score.
def disambiguate(self, target, sentence):
if target not in self.sense_inventory:
return "Target word not found in sense inventory."

context = [w for w in [Link](sentence) if w != target]

scores = {}

for sense, keywords in self.sense_inventory[target].items():

scores[sense] = len(set(context) & set(keywords))

best_sense = max(scores, key=[Link])

return best_sense, scores
4. Testing the Implementation
We create an object of the class and test it with sample sentences.
• [Link](word, sentence): Returns the predicted sense and the overlap scores for each
possible sense.
• The output displays the original sentence, the target word, predicted sense and a breakdown of
scores.
wsd = BasicWSD()

examples = [
("bank", "I need to deposit money into my savings account at the bank"),
("bank", "The fisherman stood on the river bank casting his line"),
("mouse", "The computer mouse stopped responding to clicks")
]

for word, sentence in examples:

sense, score = [Link](word, sentence)
print(f"\n• Sentence: {sentence}")
print(f"• Target Word: '{word}'")
print(f"• Predicted Sense: {sense}")
print("• Overlap Scores:")
 for s, val in [Link]():
print(f" - {s}: {val}")
Output:

WSD using Knowledge Transfer

We can see from the output that:
1. Financial context example:
• Sentence: "I need to deposit money into my savings account at the bank"
• Predicted sense: "financial" (overlaps: money, deposit, account, savings)
• Confidence score: 4 matching words
2. Geographical context example:
• Sentence: "The fisherman stood on the river bank casting his line"
• Predicted sense: "geographical" (overlaps: river, fishing)
• Confidence score: 2 matching words
Challenges and Limitations
The basic approach faces several constraints:
• Limited coverage: Only handles predefined words with manually curated sense inventories
• Shallow semantic understanding: Simple word overlap cannot capture deeper semantic
relationships
• Context dependency: Requires sufficient contextual clues for accurate disambiguation
Broader WSD Challenges
• Data sparsity: As many word-sense combinations appear infrequently in training corpora,
making supervised learning difficult for rare senses.
• Sense granularity: Different lexical resources may define sense boundaries differently. Fine-
grained sense distinctions are typically more difficult to disambiguate than coarse-grained
categories.
• Domain adaptation: Models trained on general text often perform poorly when applied to
specialized domains such as medical, legal or technical texts.
Applications and Future Directions
WSD technology finds practical application across numerous domains:
• Machine Translation: Accurate sense identification improves translation quality by selecting
appropriate target language equivalents for ambiguous source words.
• Information Retrieval: Search engines employ WSD to better understand user query intent and
retrieve more relevant documents.
 • Content Analysis: Text processing systems benefit from precise word meanings for tasks such
as sentiment analysis, topic modeling and document classification.

NAMED ENTITY RECOGNITION

Named Entity Recognition (NER) in NLP focuses on identifying and categorizing important information
known as entities in text. These entities can be names of people, places, organizations, dates, etc. It helps
in transforming unstructured text into structured information which helps in tasks like text
summarization, knowledge graph creation and question answering.
Understanding Named Entity Recognition
NER helps in detecting specific information and sort it into predefined categories. It plays an important
role in enhancing other NLP tasks like part of speech and perser-set. Examples of Common Entity Types:
• Person Names: Albert Einstein
• Organizations: GeeksforGeeks
• Locations: Paris
• Dates and Times: 5th May 2025
• Quantities and Percentages: 50%, $100
It helps in handling ambiguity by analyzing surrounding words, structure of sentence and the overall
context to make the correct classification. It means context can change based on entity’s meaning.
Example 1:
• Amazon is expanding rapidly (Organization)
• The Amazon is the largest rainforest (Location)
Example 2:
• Jordan won the MVP award (Person)
• Jordan is a country in the Middle East (Location)
Working of Named Entity Recognition (NER)
Various steps involves in NER and are as follows:
1. Analyzing the Text: It processes entire text to locate words or phrases that could represent
entities.
2. Finding Sentence Boundaries: It identifies starting and ending of sentences using punctuation
and capitalization which helps in maintaining meaning and context of entities.
3. Tokenizing and Part-of-Speech Tagging: Text is broken into tokens (words) and each token is
tagged with its grammatical role which provides important clues for identifying entities.
4. Entity Detection and Classification: Tokens or groups of tokens that match patterns of known
entities are recognized and classified into predefined categories like Person, Organization,
Location etc.
5. Model Training and Refinement: Machine learning models are trained using labeled datasets
and they improve over time by learning patterns and relationships between words.
6. Adapting to New Contexts: A well-trained model can generalize to different languages, styles
and unseen types of entities by learning from context.
Methods of Named Entity Recognition
There are different methods present in NER which are:
1. Lexicon Based Method
This method uses a dictionary of known entity names. This process involves checking if any of these
words are present in a given text. However, this approach isn't commonly used because it requires
constant updating and careful maintenance of the dictionary to stay accurate and effective.
2. Rule Based Method
It uses a set of predefined rules which helps in extraction of information. These rules are based on patterns
and context. Pattern-based rules focus on the structure and form of words helps in looking at their
morphological patterns. On the other hand context-based rules focus on the surrounding words or the
context in which a word appears within the text document. This combination of pattern-based and
context-based rules increases the accuracy of information extraction in NER.
3. Machine Learning-Based Method
There are two main types of category in this:
• Multi-Class Classification: Trains model on labeled examples where each entity is categorized.
In addition to labelling model also requires a deep understanding of context which makes it a
challenging task for a simple machine learning algorithm.
• Conditional Random Field (CRF): It is implemented by both NLP Speech Tagger and NLTK.
It is a probabilistic model that understands the sequence and context of words which helps in
making entity prediction more accurate.
4. Deep Learning Based Method
• Word Embeddings: Captures the meaning of words in context.
• Automatic Learning: Deep models learn complex patterns without manual feature engineering.
• Higher Accuracy: Performs well on large varied datasets.
Implementation of NER in Python
Step 1: Installing Libraries
Firts we need to install necessary libraries. You can run the following commands in command prompt to
install them.
!pip install spacy
!pip install nltk
!python -m spacy download en_core_web_sm
Step 2: Importing and Loading data
We will be using pandas and spacy libraries to implement this.
• nlp = [Link]("en_core_web_sm"): Loads the pre-trained "en_core_web_sm" SpaCy model
and stores it in the variable nlp for text processing tasks.
import pandas as pd
import spacy
import requests
from bs4 import BeautifulSoup
nlp = [Link]("en_core_web_sm")
pd.set_option("display.max_rows", 200)
Step 3: Applying NER to a Sample Text
We have created some random content to implement this you can use any text based on your choice.
• doc = nlp(content): Processes text stored in content using the nlp model and stores resulting
document object in the variable doc for further analysis.
• for ent in [Link]: Iterates through the named entities ([Link]) identified in the processed
document and performs actions for each entity.
content = "Trinamool Congress leader Mahua Moitra has moved the Supreme Court against her expulsion
from the Lok Sabha over the cash-for-query allegations against her. Moitra was ousted from the
Parliament last week after the Ethics Committee of the Lok Sabha found her guilty of jeopardising
national security by sharing her parliamentary portal's login credentials with businessman Darshan
Hiranandani."
doc = nlp(content)
for ent in [Link]:
 print([Link], ent.start_char, ent.end_char, ent.label_)
Output:

Resulting document
It displays the names of the entities, their start and end positions in the text and their predicted labels.
Step 4: Visualizing Entities
We will highlight the text with their categories using visualizing technique for better understanding.
• [Link](doc, style="ent"): Visualizing named entities in the processed doc object by
highlighting them in the text with their respective categories such as person, organization, location
etc.
from spacy import displacy
[Link](doc, style="ent")
Output:

Highlighted text with their categories

Step 5: Creating a DataFrame for Entities
• entities = [([Link], ent.label_, ent.lemma_) for ent in [Link]]: Creating a list of tuples where
each tuple contains the text, label (type) and lemma (base form) of each named entity found in
the processed doc object.
entities = [([Link], ent.label_, ent.lemma_) for ent in [Link]]
df = [Link](entities, columns=['text', 'type', 'lemma'])
print(df)
Output:

Text after categorization

Here dataframe provides a structured representation of the named entities, their types and lemmatized
forms. NER helps organize unstructured text into structured information making it a useful for a wide
range of NLP applications.

UNDERSTANDING SEMANTIC ANALYSIS - NLP

Introduction to Semantic Analysis
Semantic Analysis is a subfield of Natural Language Processing (NLP) that attempts to understand the
meaning of Natural Language. Understanding Natural Language might seem a straightforward process
to us as humans. However, due to the vast complexity and subjectivity involved in human language,
interpreting it is quite a complicated task for machines. Semantic Analysis of Natural Language captures
the meaning of the given text while taking into account context, logical structuring of sentences and
grammar roles.
Parts of Semantic Analysis
Semantic Analysis of Natural Language can be classified into two broad parts:
1. Lexical Semantic Analysis: Lexical Semantic Analysis involves understanding the meaning of each
word of the text individually. It basically refers to fetching the dictionary meaning that a word in the text
is deputed to carry.
2. Compositional Semantics Analysis: Although knowing the meaning of each word of the text is
essential, it is not sufficient to completely understand the meaning of the text.
For example, consider the following two sentences:
• Sentence 1: Students love GeeksforGeeks.
• Sentence 2: GeeksforGeeks loves Students.
Although both these sentences 1 and 2 use the same set of root words {student, love, geeksforgeeks},
they convey entirely different meanings.
Hence, under Compositional Semantics Analysis, we try to understand how combinations of individual
words form the meaning of the text.
Tasks involved in Semantic Analysis
In order to understand the meaning of a sentence, the following are the major processes involved in
Semantic Analysis:
1. Word Sense Disambiguation
2. Relationship Extraction
Word Sense Disambiguation:
In Natural Language, the meaning of a word may vary as per its usage in sentences and the context of the
text. Word Sense Disambiguation involves interpreting the meaning of a word based upon the context of
its occurrence in a text.
For example, the word 'Bark' may mean 'the sound made by a dog' or 'the outermost layer of a tree.'
Likewise, the word 'rock' may mean 'a stone' or 'a genre of music' - hence, the accurate meaning of the
word is highly dependent upon its context and usage in the text.
Thus, the ability of a machine to overcome the ambiguity involved in identifying the meaning of a word
based on its usage and context is called Word Sense Disambiguation.
Relationship Extraction:
Another important task involved in Semantic Analysis is Relationship Extracting. It involves firstly
identifying various entities present in the sentence and then extracting the relationships between those
entities.
For example, consider the following sentence:
Semantic Analysis is a topic of NLP which is explained on the GeeksforGeeks blog. The entities involved
in this text, along with their relationships, are shown below.

Entities
Relationships

Elements of Semantic Analysis

Some of the critical elements of Semantic Analysis that must be scrutinized and taken into account while
processing Natural Language are:
• Hyponymy: Hyponymys refers to a term that is an instance of a generic term. They can be
understood by taking class-object as an analogy. For example: 'Color' is a hypernymy while 'grey',
'blue', 'red', etc, are its hyponyms.
• Homonymy: Homonymy refers to two or more lexical terms with the same spellings but
completely distinct in meaning. For example: 'Rose' might mean 'the past form of rise' or 'a
flower', - same spelling but different meanings; hence, 'rose' is a homonymy.
• Synonymy: When two or more lexical terms that might be spelt distinctly have the same or
similar meaning, they are called Synonymy. For example: (Job, Occupation), (Large, Big), (Stop,
Halt).
• Antonymy: Antonymy refers to a pair of lexical terms that have contrasting meanings - they are
symmetric to a semantic axis. For example: (Day, Night), (Hot, Cold), (Large, Small).
• Polysemy: Polysemy refers to lexical terms that have the same spelling but multiple closely
related meanings. It differs from homonymy because the meanings of the terms need not be
closely related in the case of homonymy. For example: 'man' may mean 'the human species' or 'a
male human' or 'an adult male human' - since all these different meanings bear a close association,
the lexical term 'man' is a polysemy.
• Meronomy: Meronomy refers to a relationship wherein one lexical term is a constituent of some
larger entity. For example: 'Wheel' is a meronym of 'Automobile'
Meaning Representation
While, as humans, it is pretty simple for us to understand the meaning of textual information, it is not so
in the case of machines. Thus, machines tend to represent the text in specific formats in order to interpret
its meaning. This formal structure that is used to understand the meaning of a text is called meaning
representation.
Basic Units of Semantic System:
In order to accomplish Meaning Representation in Semantic Analysis, it is vital to understand the building
units of such representations. The basic units of semantic systems are explained below:
1. Entity: An entity refers to a particular unit or individual in specific such as a person or a
location. For example GeeksforGeeks, Delhi, etc.
2. Concept: A Concept may be understood as a generalization of entities. It refers to a broad class
of individual units. For example Learning Portals, City, Students.
3. Relations: Relations help establish relationships between various entities and concepts. For
example: 'GeeksforGeeks is a Learning Portal', 'Delhi is a City.', etc.
4. Predicate: Predicates represent the verb structures of the sentences.
In Meaning Representation, we employ these basic units to represent textual information.
Approaches to Meaning Representations:
Now that we are familiar with the basic understanding of Meaning Representations, here are some of the
most popular approaches to meaning representation:
 1. First-order predicate logic (FOPL)
2. Semantic Nets
3. Frames
4. Conceptual dependency (CD)
5. Rule-based architecture
6. Case Grammar
7. Conceptual Graphs
Semantic Analysis Techniques
Based upon the end goal one is trying to accomplish, Semantic Analysis can be used in various ways.
Two of the most common Semantic Analysis techniques are:
Text Classification
In-Text Classification, our aim is to label the text according to the insights we intend to gain from the
textual data.
For example:
• In Sentiment Analysis, we try to label the text with the prominent emotion they convey. It is
highly beneficial when analyzing customer reviews for improvement.
• In Topic Classification, we try to categories our text into some predefined categories. For
example: Identifying whether a research paper is of Physics, Chemistry or Maths
• In Intent Classification, we try to determine the intent behind a text message. For example:
Identifying whether an e-mail received at customer care service is a query, complaint or request.
Text Extraction
In-Text Extraction, we aim at obtaining specific information from our text.
For Example,
• In Keyword Extraction, we try to obtain the essential words that define the entire document.
• In Entity Extraction, we try to obtain all the entities involved in a document.
Significance of Semantics Analysis
Semantics Analysis is a crucial part of Natural Language Processing (NLP). In the ever-expanding era of
textual information, it is important for organizations to draw insights from such data to fuel businesses.
Semantic Analysis helps machines interpret the meaning of texts and extract useful information, thus
providing invaluable data while reducing manual efforts.
Besides, Semantics Analysis is also widely employed to facilitate the processes of automated answering
systems such as chatbots - that answer user queries without any human interventions.

SENTIMENT ANALYSIS
Sentiment Analysis is the process of analyzing textual data to determine the emotional tone expressed in
it. It classifies text as positive, negative or neutral and can also detect more nuanced emotions like happy,
sad, angry or frustrated. It is widely used to analyze large volumes of text such as:
• Customer reviews
• Social media posts
• Emails and chat transcripts
• Survey responses
The goal of sentiment analysis is to understand customer opinions, monitor brand reputation and improve
business decisions based on real insights.
Why is Sentiment Analysis Important?
Sentiment analysis provides actionable insights to businesses and is essential for understanding customer
opinions. Key Benefits of Sentiment Analysis are:
 • Customer Feedback Analysis: Analyse reviews, comments and survey responses to identify
issues and improve satisfaction.
• Brand Reputation Management: Monitor mentions on social media, forums and review
platforms in real-time.
• Product Development and Innovation: Understand which features are well-received and which
need improvement.
• Competitor Analysis: Compare sentiment around your products with competitors products.
• Marketing Campaign Evaluation: Measure the effectiveness of campaigns based on positive or
negative reactions.
Types of Sentiment Analysis
1. Fine-Grained Sentiment Analysis
• Fine-grained sentiment analysis rates sentiment on multiple levels rather than just positive,
negative or neutral.
• It can categorise text as very positive, positive, neutral, negative or very negative, often using a
numerical scale like 1–5 stars.
• For example, a Product review ratings on e-commerce platforms.
2. Emotion Detection
• Emotion detection goes beyond polarity and identifies specific emotions in text such as joy,
sadness, anger, fear or excitement.
• It is also called the lexicon-based method of sentiment analysis and helps understand deeper
emotional context.
3. Aspect-Based Sentiment Analysis
• Aspect-based sentiment analysis focuses on specific features or attributes of a product or service.
• For a smartphone review, it separately analyzes battery, screen, camera and performance to
understand customer sentiment for each aspect.
4. Multilingual Sentiment Analysis
• Multilingual sentiment analysis works on text written in multiple languages.
• It is highly challenging due to variations in grammar, syntax and cultural expressions across
languages but it is essential for global applications.
5. Intent-Based Sentiment Analysis
• Intent-based sentiment analysis identifies the underlying intention behind the text in addition to
its sentiment.
• For example, Detecting purchase intent from reviews mentioning discounts, deals or offers in e-
commerce.
Approaches to Sentiment Analysis
1. Rule-Based Approach
The rule-based approach relies on predefined lexicons and rules to classify text as positive, negative or
neutral.
• Counts positive and negative words using a sentiment dictionary.
• Handles simple phrases well, including some negations like “not bad”.
• Easy to implement, interpretable, no training required.
• Hard to scale, limited accuracy for complex sentences, requires continuous lexicon updates.
2. Machine Learning Approach
The machine learning (ML) approach trains models to automatically learn sentiment patterns from
labeled data.
• Algorithms include Naive Bayes, Support Vector Machines (SVM), Random Forest and others.
 • Text is converted into numeric features using TF-IDF or Bag-of-Words.
• Can handle large datasets, captures complex patterns and relationship.
• Requires large labeled datasets, domain-specific models and retraining needed for new domains.
3. Neural Network / Deep Learning Approach
This approach uses neural networks to capture contextual and sequential information in text.
• Common architectures include RNN ,LSTM, GRU and Transformers.
• Excels at handling long sentences and context-aware sentiment.
• High accuracy, captures context and nuances, state-of-the-art performance.
• Computationally expensive and hence requires significant training data.
4. Hybrid Approach
The hybrid approach combines rule-based and ML/deep learning methods to improve both speed and
accuracy.
• Uses the lexicon-based rules for quick initial classification.
• Uses ML or deep learning to refine predictions and handle complex sentences.
• Better accuracy than individual approaches, adaptable.
• Complex to implement, requires integration of multiple systems.
How Sentiment Analysis Works
Step 1: Preprocessing
Preprocessing ensures text is clean and standardized for analysis:
• Text Cleaning: Remove HTML tags, special characters, numbers and emojis.
• Tokenization: Split sentences into words or tokens.
• Stopword: Filter out common words like "and", "the", "is."
• stemming/lemmatization: Reduce words to root forms.
• Handling Emojis and Slang: Convert emojis or slang to standard words for analysis.
Step 2: Feature Extraction
Convert text to numeric representation using:
• BagofWord: Converts text into word-count vectors based on vocabulary.
• Tf-idf: Gives higher weight to important words and lower weight to common ones.
• Word embedding: Represent words as dense vectors that capture meaning and relationships.
Similar words get similar vectors, helping models understand context better.
Step 3: Sentiment Classification
• Model Training: Use labeled datasets to train models.
• Prediction: Assign sentiment labels to new text based on learned patterns.
• Post-processing: Aggregate sentiment scores across multiple sentences or aspects.
Difference Between Sentiment Analysis vs Semantic Analysis
Here we compare sentiment analysis with semantic analysis:
FEATURE SENTIMENT ANALYSIS SEMANTIC ANALYSIS
PURPOSE Determines emotional tone in text Understands the actual meaning and
relationships in text.
FOCUS Focuses on opinions, emotions and Focuses on context, concepts and word
feelings. sense.
TECHNIQUES Lexicon-based methods, ML NER, semantic role labeling,
USED models, deep learning dependency parsing, ontologies
CONTEXT Limited ability to interpret context Strong emphasis on context and word
HANDLING and sarcasm. relationships.
 USE CASES Social media monitoring, reviews, Chatbots, search engines, QA systems,
customer feedback. knowledge graphs.
Tools and Libraries for Sentiment Analysis
• Python Libraries:NLTK, textblob, vader, spacy
• Deep Learning Frameworks:Tensorflow, pytorch
• Pretrained Models:BERT, Roberta, distilbert, GPT (Generative pre-trained transformer)
• Cloud Services: AWS Comprehend, Google Cloud Natural Language, IBM Watson NLP
Advantages
• Real-Time Insights: Helps businesses instantly understand customer opinions from reviews,
tweets and feedback.
• Scalable Analysis: Can process thousands of text entries much faster than manual evaluation.
• Improves Decision-Making: Supports product improvement, marketing strategy and customer
satisfaction analysis.
• Cost-Effective: Reduces the need for large teams to manually analyze feedback.
• Enhances Customer Experience: Enables personalized response and support based on
sentiment.
Limitations
• Domain Dependency: Models trained for one domain may fail in another.
• Handling Slang and Emojis: Informal language and emoticons are often difficult to interpret
accurately.
• Neutral or Mixed Sentiments: Hard to classify statements expressing both positive and negative
opinions.
• Data Quality Issues: Noisy or unstructured text can reduce accuracy significantly.