1

Natural Language Processing (NLP) Unit-I

1. Natural Language Processing – Introduction

 Humans communicate through some form of language either by text or speech.

 To make interactions between computers and humans, computers need to understand
natural languages used by humans.
 Natural language processing is all about making computers learn, understand,
analyze, manipulate and interpret natural(human) languages.
 NLP stands for Natural Language Processing, which is a part of Computer Science,
Human languages or Linguistics, and Artificial Intelligence.
 Processing of Natural Language is required when you want an intelligent system like
robot to perform as per your instructions, when you want to hear decision from a
dialogue based clinical expert system, etc.
 The ability of machines to interpret human language is now at the core of many
applications that we use every day - chatbots, Email classification and spam filters,
search engines, grammar checkers, voice assistants, and social language translators.
 The input and output of an NLP system can be Speech or Written Text.

2. Applications of NLP or Use cases of NLP

1. Sentiment analysis
 Sentiment analysis, also referred to as opinion mining, is an approach to natural
language processing (NLP) that identifies the emotional tone behind a body of text.
 This is a popular way for organizations to determine and categorize opinions about a
product, service or idea.
 Sentiment analysis systems help organizations gather insights into real-time customer
sentiment, customer experience and brand reputation.
 Generally, these tools use text analytics to analyze online sources such as emails, blog
posts, online reviews, news articles, survey responses, case studies, web chats, tweets,
forums and comments.
 Sentiment analysis uses machine learning models to perform text analysis of human
language. The metrics used are designed to detect whether the overall sentiment of a
piece of text is positive, negative or neutral.
2. Machine Translation
 Machine translation, sometimes referred to by the abbreviation MT, is a sub-field
of computational linguistics that investigates the use of software to translate text or
speech from one language to another.
 On a basic level, MT performs mechanical substitution of words in one language for
words in another, but that alone rarely produces a good translation because
recognition of whole phrases and their closest counterparts in the target language is
needed.
 Not all words in one language have equivalent words in another language, and many
words have more than one meaning.
 2

 Solving this problem with corpus statistical and neural techniques is a rapidly growing
field that is leading to better translations, handling differences in linguistic typology,
translation of idioms, and the isolation of anomalies.
 Corpus: A collection of written texts, especially the entire works of a particular
author.

3. Text Extraction
 There are a number of natural language processing techniques that can be
used to extract information from text or unstructured data.
 These techniques can be used to extract information such as entity names,
locations, quantities, and more.
 With the help of natural language processing, computers can make sense
of the vast amount of unstructured text data that is generated every day,
and humans can reap the benefits of having this information readily
available.
 Industries such as healthcare, finance, and e-commerce are already using
natural language processing techniques to extract information and
improve business processes.
 As the machine learning technology continues to develop, we will only
see more and more information extraction use cases covered.

4. Text Classification

 Unstructured text is everywhere, such as emails, chat conversations, websites, and

social media. Nevertheless, it’s hard to extract value from this data unless it’s
organized in a certain way.
 Text classification also known as text tagging or text categorization is the process of
categorizing text into organized groups. By using Natural Language
Processing (NLP), text classifiers can automatically analyze text and then assign a set
of pre-defined tags or categories based on its content.
 Text classification is becoming an increasingly important part of businesses as it
allows to easily get insights from data and automate business processes.

5. Speech Recognition
 Speech recognition is an interdisciplinary subfield of computer
science and computational linguistics that develops methodologies and technologies
that enable the recognition and translation of spoken language into text by computers.
 It is also known as automatic speech recognition (ASR), computer speech
recognition or speech to text (STT).
 It incorporates knowledge and research in the computer
science, linguistics and computer engineering fields. The reverse process is speech
synthesis.
 3

Speech recognition use cases

 A wide number of industries are utilizing different applications of speech technology
today, helping businesses and consumers save time and even lives. Some examples
include:
 Automotive: Speech recognizers improves driver safety by enabling voice-activated
navigation systems and search capabilities in car radios.
 Technology: Virtual agents are increasingly becoming integrated within our daily
lives, particularly on our mobile devices. We use voice commands to access them
through our smartphones, such as through Google Assistant or Apple’s Siri, for tasks,
such as voice search, or through our speakers, via Amazon’s Alexa or Microsoft’s
Cortana, to play music. They’ll only continue to integrate into the everyday products
that we use, fueling the “Internet of Things” movement.
 Healthcare: Doctors and nurses leverage dictation applications to capture and log
patient diagnoses and treatment notes.
 Sales: Speech recognition technology has a couple of applications in sales. It can help
a call center transcribe thousands of phone calls between customers and agents to
identify common call patterns and issues. AI chatbots can also talk to people via a
webpage, answering common queries and solving basic requests without needing to
wait for a contact center agent to be available. In both instances speech recognition
systems help reduce time to resolution for consumer issues.
6. Chatbot
 Chatbots are computer programs that conduct automatic conversations with people.
They are mainly used in customer service for information acquisition. As the name
implies, these are bots designed with the purpose of chatting and are also simply
referred to as “bots.”

 You’ll come across chatbots on business websites or messengers that give pre-scripted
replies to your questions. As the entire process is automated, bots can provide quick
assistance 24/7 without human intervention.

7. Email Filter
 One of the most fundamental and essential applications of NLP online is email
filtering. It began with spam filters, which identified specific words or phrases that
indicate a spam message. But, like early NLP adaptations, filtering has been
improved.
 Gmail's email categorization is one of the more common, newer implementations of
NLP. Based on the contents of emails, the algorithm determines whether they belong
in one of three categories (main, social, or promotional).
 This maintains your inbox manageable for all Gmail users, with critical, relevant
emails you want to see and reply to fast.
8. Search Autocorrect and Autocomplete
 When you type 2-3 letters into Google to search for anything, it displays a list of
probable search keywords. Alternatively, if you search for anything with mistakes, it
corrects them for you while still returning relevant results. Isn't it incredible?
 4

 Everyone uses Google search autocorrect autocomplete on a regular basis but seldom
gives it any thought. It's a fantastic illustration of how natural language processing is
touching millions of people across the world, including you and me.
 Both, search autocomplete and autocorrect make it much easier to locate accurate
results.
3. Components of NLP
 There are two components of NLP, Natural Language Understanding (NLU)and
Natural Language Generation (NLG).
 Natural Language Understanding (NLU) which involves transforming
humanlanguage into a machine-readable [Link] helps the machine to understand
and analyze human language by extracting the text from large data such as keywords,
emotions, relations, and semantics.
 Natural Language Generation (NLG) acts as a translator that converts
thecomputerized data into natural language representation.
 It mainly involves Text planning, Sentence planning, and Text realization.
 The NLU is harder than NLG.

4. Steps in NLP
There are general five steps :
 1. Lexical Analysis
 2. Syntactic Analysis (Parsing)
 3. Semantic Analysis
 4. Discourse Integration
 5. Pragmatic Analysis

Lexical Analysis:
 The first phase of NLP is the Lexical Analysis.
 This phase scans the source code as a stream of characters and converts it into
meaningful lexemes.
 It divides the whole text into paragraphs, sentences, and words.
 Lexeme: A lexeme is a basic unit of meaning. In linguistics, the abstract unit of
morphological analysis that corresponds to a set of forms taken by a single word is
called lexeme.
 The way in which a lexeme is used in a sentence is determined by its grammatical
category.
 5

 Lexeme can be individual word or multiword.

 For example, the word talk is an example of an individual word lexeme,
which mayhave many grammatical variants like talks, talked and talking.
 Multiword lexeme can be made up of more than one orthographic word. For
example, speak up, pull through, etc. are the examples of multiword lexemes.

Syntax Analysis (Parsing)

 Syntactic Analysis is used to check grammar, word arrangements, and
shows therelationship among the words.
 The sentence such as “The school goes to boy” is rejected by English
syntactic analyzer.

Semantic Analysis
 Semantic analysis is concerned with the meaning representation.
 It mainly focuses on the literal meaning of words, phrases, and sentences.
 The semantic analyzer disregards sentence such as “hot ice-cream”.
 Another Example is “Manhattan calls out to Dave” passes a syntactic analysis because it’s
a grammatically correct sentence. However, it fails a semantic analysis. Because
Manhattan is a place (and can’t literally call out to people), the sentence’s meaning doesn’t
make sense.

Discourse Integration
 Discourse Integration depends upon the sentences that precedes it and also
invokesthe meaning of the sentences that follow it.

 For instance, if one sentence reads, “Manhattan speaks to all its people,” and the
following sentence reads, “It calls out to Dave,” discourse integration checks the first
sentence for context to understand that “It” in the latter sentence refers to Manhattan.

Pragmatic Analysis
 During this, what was said is re-interpreted on what it actually meant.
 It involves deriving those aspects of language which require real world knowledge.
 For instance, a pragmatic analysis can uncover the intended meaning of “Manhattan
speaks to all its people.” Methods like neural networks assess the context to
understand that the sentence isn’t literal, and most people won’t interpret it as such. A
pragmatic analysis deduces that this sentence is a metaphor for how people
emotionally connect with place.

5. Finding the structure of Words

Words and Their Components
 Words are defined in most languages as the smallest linguistic units that
can form acomplete utterance by themselves.
 The minimal parts of words that deliver aspects of meaning to them are called
morphemes.
 6

Tokens:
Suppose, for a moment, that words in English are delimited only by
whitespace and punctuation (the marks, such as full stop, comma, and
brackets)
 Example: Will you read the newspaper? Will you read it? I won’t
read it. If we confront our assumption with insights from syntax,
we notice twowords here: words newspaper and won’t.
 7

Being a compound word, newspaper has an interesting derivational

structure.
In writing, newspaper and the associated concept is distinguished from
the isolated news and paper.
For reasons of generality, linguists prefer to analyze won’t as two
syntactic words, or tokens, each of which has its independent role and can
be reverted to its normalized form.
 The structure of won’t could be parsed as will followed by not.
 In English, this kind of tokenization and normalization may apply to just
a limited set of cases, but in other languages, these phenomena have to be
treated different way.

Lexemes
 By the term word, we often denote not just the one linguistic form in the given
context but also the concept behind the form and the set of alternative forms that can
express it.
 Such sets are called lexemes or lexical items, and they constitute the lexicon of a
language.
 Lexemes can be divided by their behaviour into the lexical categories of verbs, nouns,
adjectives, conjunctions or other parts of speech.
 The citation form of a lexeme, by which it is commonly identified, is also called its
lemma.
 When we convert a word into its other forms, such as turning the singular mouse into
the plural mice or mouses, we say we inflect the lexeme.
 When we transform a lexeme into another one that is morphologically related,
regardless of its lexical category, we say we derive the lexeme: for instance, the nouns
receiver and reception are derived from the verb receive.
 Example: Did you see him? I didn’t see him. I didn’t see anyone
Example presents the problem of tokenization of didn’t and the investigation of the
internal structure of anyone.
 The difficulty with the definition of what counts as a word need not pose a problem
for the syntactic description if we understand no one as two closely connected tokens
treated as one fixed element.

Morphemes
These components are usually called segments or morphs.
Morphology
Morphology is the domain of linguistics that analyses the internal structure of words.
 Morphological analysis – exploring the structure of words
 Words are built up of minimal meaningful elements called morphemes:
played = play-ed
cats = cat-s
unfriendly = un-friend-ly
 8

Two types of morphemes:

i Stems: play, cat, friend
ii Affixes: -ed, -s, un-, -ly
Two main types of affixes:
i Prefixes precede the stem: un
ii Suffixes follow the stem: -ed, -s, un-, -ly
Stemming = find the stem by stripping off affixes
play = play
replayed = re-play-ed
computerized = comput-er-ize-d

Problems in morphological processing

Inflectional morphology: inflected forms are constructed from base forms
and inflectional
Affixes.
Inflection relates different forms of the same word
Lemma Singular Plural
Cat cat Cats
Mouse mouse mice
Derivational morphology: words are constructed from roots (or stems)
and derivational
affixes:
inter+national = international
international+ize = internationalize
internationalize+ation = internationalization

 The simplest morphological process concatenates morphs one by one, as in disagree-

ment-s, where agree is a free lexical morpheme and the other elements are bound
grammatical morphemes contributing some partial meaning to the whole word.
 In a more complex scheme, morphs can interact with each other, and their forms may
become subject to additional phonological and orthographic changes denoted as
morphophonemic.
 The alternative forms of a morpheme are termed allomorphs.
 The ending -s, indicating plural in “cats,” “dogs,” the -es in “dishes,” and the -en of
“oxen” are all allomorphs of the plural morpheme.
Typology
 Morphological typology divides languages into groups by characterizing the prevalent
morphological phenomena in those languages.
 It can consider various criteria, and during the history of linguistics, different
classifications have been proposed.
 Let us outline the typology that is based on quantitative relations between words, their
morphemes, and their features:
 9

 Isolating, or analytic, languages include no or relatively few words that would

comprise more than one morpheme (typical members are Chinese, Vietnamese, and
Thai; analytic tendencies are also found in English).
 Synthetic languages can combine more morphemes in one word and are further
divided into agglutinative and fusional languages.
 Agglutinative languages have morphemes associated with only a single function at a
time (as in Korean, Japanese, Finnish, and Tamil, etc.)
 Fusional languages are defined by their feature-per-morpheme ratio higher than one
(as in Arabic, Czech, Latin, Sanskrit, German, etc.).
 In accordance with the notions about word formation processes mentioned earlier, we
can also find out using concatenative and nonlinear:
 Concatenative languages linking morphs and morphemes one after another.
 Nonlinear languages allowing structural components to merge nonsequentially to
apply tonal morphemes or change the consonantal or vocalic templates of words.

Morphological Typology
 Morphological typology is a way of classifying the languages of the world that groups
languages according to their common morphological structures.
 The field organizes languages on the basis of how those languages form words by
combining morphemes.
 The morphological typology classifies languages into two broad classes like synthetic
languages and analytical languages.
 The synthetic class is then further sub classified as either agglutinative languages or
fusional languages.
 Analytic languages contain very little inflection, instead relying on features like word
order and auxiliary words to convey meaning.
 Synthetic languages, ones that are not analytic, are divided into two categories:
agglutinative and fusional languages.
 Agglutinative languages rely primarily on discrete particles(prefixes, suffixes, and
infixes) for inflection, ex: inter+national = international, international+ize =
internationalize.
 While fusional languages "fuse" inflectional categories together, often allowing one
word ending to contain several categories, such that the original root can be difficult
to extract (anybody, newspaper).

6. Natural Language Processing With Python's NLTK Package

• NLTK, or Natural Language Toolkit, is a Python package that you can use for NLP.
• A lot of the data that you could be analyzing is unstructured data and contains human-
readable text.
• Before you can analyze that data programmatically, you first need to preprocess it.
• Now we are going to see kinds of text preprocessing tasks you can do with NLTK so
that you’ll be ready to apply them in future projects.
 10

1. Tokenizing
 By tokenizing, you can conveniently split up text by word or by sentence.
 This will allow you to work with smaller pieces of text that are still relatively
coherent and meaningful even outside of the context of the rest of the text.
 It’s your first step in turning unstructured data into structured data, which is easier
to analyze.
 When you’re analyzing text, you’ll be tokenizing by word and tokenizing by
sentence.

Tokenizing by word
• Words are like the atoms of natural language. They’re the smallest unit of meaning
that still makes sense on its own.
• Tokenizing your text by word allows you to identify words that come up particularly
often.
• For example, if you were analyzing a group of job ads, then you might find that the
word “Python” comes up often.
• That could suggest high demand for Python knowledge, but you’d need to look deeper
to know more.
Tokenizing by sentence
• When you tokenize by sentence, you can analyze how those words relate to one
another and see more context.
• Are there a lot of negative words around the word “Python” because the hiring
manager doesn’t like Python?
• Are there more terms from the domain of herpetology than the domain of software
development, suggesting that you may be dealing with an entirely different kind
of python than you were expecting?

Python Program for Tokenizing by Sentence

from [Link] import sent_tokenize, word_tokenize
example_string = """
Muad'Dib learned rapidly because his first training was in how to
learn. And the first lesson of all was the basic trust that he could
[Link]'s shocking to find how many people do not believe
they can learn,and how many more believe learning to be
difficult."""
sent_tokenize(example_string)
Output
["\n Muad'Dib learned rapidly because his first training was in how to learn.",
 11

'And the first lesson of all was the basic trust that he could learn.’,
"It's shocking to find how many people do not believe they can learn,\n and how
many more believe learning to be difficult."]
Note:
import nltk
[Link]('punkt')

Python Program for Tokenizing by Word

from [Link] import sent_tokenize, word_tokenize
example_string = """
Muad'Dib learned rapidly because his first training was in how to learn. And
the first lesson of all was the basic trust that he could [Link]'s
shocking to find how many people do not believe they can learn,and
how many more believe learning to be difficult."""
word_tokenize(example_string)
Output:
["Muad'Dib", 'learned', 'rapidly', 'because', 'his', 'first', 'training', 'was', 'in', 'how', 'to',
'learn', '.', 'And', 'the', 'first', 'lesson', 'of', 'all', 'was', 'the', 'basic', 'trust', 'that', 'he',
'could', 'learn', '.', 'It', "'s", 'shocking', 'to', 'find', 'how', 'many', 'people', 'do', 'not',
'believe', 'they', 'can', 'learn', ',', 'and', 'how', 'many', 'more', 'believe', 'learning', 'to', 'be',
'difficult', '.']

2. Filtering Stop Words

 Stop words are words that you want to ignore, so you filter them out of your text
when you’re processing it. Very common words like 'in', 'is', and 'an' are often
used as stop words since they don’t add a lot of meaning to a text in and of
themselves.
 Note: [Link]("stopwords")

Python program to eliminate stopwords

from [Link] import stopwords
from [Link] import word_tokenize
worf_quote = "Sir, I protest. I am not a merry man!"
words_in_quote = word_tokenize(worf_quote)
print(words_in_quote)
stop_words = set([Link]("english"))
filtered_list = []
 12

for word in words_in_quote:

if [Link]() not in stop_words:
filtered_list.append(word)
print(filtered_list)
Output:
• ['Sir', ',', 'I', 'protest', '.', 'I', 'am', 'not', 'a', 'merry', 'man', '!’]
• ['Sir', ',', 'protest', '.', 'merry', 'man', '!’]
• ‘I’ is pronoun and it is context word
• Content words give you information about the topics covered in the text or the
sentiment that the author has about those topics.
• Context words give you information about writing style. You can observe patterns in
how authors use context words in order to quantify their writing style.
• Once you’ve quantified their writing style, you can analyze a text written by an
unknown author to see how closely it follows a particular writing style so you can try
to identify who the author is.

3. Stemming
 Stemming is a text processing task in which you reduce words to their root, which
is the core part of a word.
 For example, the words “helping” and “helper” share the root “help.”
 Stemming allows you to zero in on the basic meaning of a word rather than all the
details of how it’s being used.
 NLTK has more than one stemmer, but we’ll be using the Porter stemmer.
Python program for Stemming
from [Link] import PorterStemmer
from [Link] import word_tokenize
stemmer = PorterStemmer()
string_for_stemming = "The crew of the USS Discovery discovered many
discoveries. Discovering is what explorers do."
words = word_tokenize(string_for_stemming)
print(words)
stemmed_words = [[Link](word) for word in words]
print(stemmed_words)
Output
 13

• ['The', 'crew', 'of', 'the', 'USS', 'Discovery', 'discovered', 'many', 'discoveries', '.',
'Discovering', 'is', 'what', 'explorers', 'do', '.’]
• ['the', 'crew', 'of', 'the', 'uss', 'discoveri', 'discov', 'mani', 'discoveri', '.', 'discov', 'is',
'what', 'explor', 'do', '.’]

Original word Stemmed version

'Discovery' 'discoveri'

'discovered' 'discov'

'discoveries' 'discoveri'

'Discovering' 'discov'

4. Tagging Parts of Speech

Part of speech is a grammatical term that deals with the roles words play when
you use them together in sentences. Tagging parts of speech, or POS tagging, is
the task of labeling the words in your text according to their part of speech.

Part of speech Role Examples

Noun Is a person, place, or thing mountain, bagel,

Poland

Pronoun Replaces a noun you, she, we

Adjective Gives information about what a noun is efficient, windy,

like colorful

Verb Is an action or a state of being learn, is, go

Adverb Gives information about a verb, an efficiently, always,

adjective, or another adverb very

Preposition Gives information about how a noun or from, about, at

pronoun is connected to another word

Conjunction Connects two other words or phrases so, because, and

 14

Interjection Is an exclamation yay, ow, wow

• Some sources also include the category articles (like “a” or “the”) in the list of parts
of speech, but other sources consider them to be adjectives. NLTK uses the
word determiner to refer to articles.

Python program for Tagging Parts of Speech

import nltk
[Link]('punkt')
[Link]('averaged_perceptron_tagger')
from [Link] import word_tokenize
sagan_quote = """
If you wish to make an apple pie from scratch,
you must first invent the universe."""
words_in_sagan_quote = word_tokenize(sagan_quote)
nltk.pos_tag(words_in_sagan_quote)
Output:
• [('If', 'IN'), ('you', 'PRP'), ('wish', 'VBP'), ('to', 'TO'), ('make', 'VB'), ('an', 'DT'), ('apple',
'NN'), ('pie', 'NN'), ('from', 'IN'), ('scratch', 'NN'), (',', ','), ('you', 'PRP'), ('must', 'MD'),
('first', 'VB'), ('invent', 'VB'), ('the', 'DT'), ('universe', 'NN'), ('.', '.')]
POS Tag information
• nltk uses The Penn Treebank's POS tags
[Link]('tagsets')
[Link].upenn_tagset()

5. Lemmatizing
• Like stemming, lemmatizing reduces words to their core meaning, but it will give
you a complete English word that makes sense on its own instead of just a fragment of
a word like 'discoveri'.
• A lemma is a word that represents a whole group of words, and that group of words is
called a lexeme.
• For example, if you were to look up the word “blending” in a dictionary, then you’d
need to look at the entry for “blend,” but you would find “blending” listed in that
entry.
• In this example, “blend” is the lemma, and “blending” is part of the lexeme. So when
you lemmatize a word, you are reducing it to its lemma.
 15

5. Python Program for Lemmatization

import nltk
[Link]('punkt')
[Link]('wordnet')
from [Link] import WordNetLemmatizer
from [Link] import word_tokenize
lemmatizer = WordNetLemmatizer()
string_for_lemmatizing = "The friends of DeSoto love scarves."
words = word_tokenize(string_for_lemmatizing)
lemmatized_words = [[Link](word) for word in words]
print(lemmatized_words)
Output:
 [Link]("worst")
o/p: 'worst’
 [Link]("worst", pos="a")
o/p: 'bad'

6. Chunking
 chunking allows you to identify phrases.
 A phrase is a word or group of words that works as a single unit to perform a
grammatical function. Noun phrases are built around a noun.
 Here are some examples:
 “A planet”
 “A tilting planet”
 “A swiftly tilting planet”
 Chunking makes use of POS tags to group words and apply chunk tags to those
groups. Chunks don’t overlap, so one instance of a word can be in only one chunk
at a time.

 After getting a list of tuples of all the words in the quote, along with their POS
tag. In order to chunk, you first need to define a chunk grammar.

 Note: A chunk grammar is a combination of rules on how sentences should be

chunked. It often uses regular expressions, or regexes.
 Create a chunk grammar with one regular expression rule:
 grammar = "NP: {<DT>?<JJ>*<NN>}“
 Create a chunk parser with this grammar:
 16

Python program for chuncking

import nltk
[Link]('puckt')
from [Link] import word_tokenize
quote = "It's a dangerous business, Frodo, going out your door."
words_quote = word_tokenize(quote)
print(words_quote)
[Link]("averaged_perceptron_tagger")
tags = nltk.pos_tag(words_quote)
print(tags)
#Regular expression for Noun Phrase
grammar = "NP: {<DT>?<JJ>*<NN>}"
#Create a chunk parser with this grammar:
chunk_parser = [Link](grammar)
tree = chunk_parser.parse(tags)
print(tree)

Output:
• ['It', "'s", 'a', 'dangerous', 'business', ',', 'Frodo', ',', 'going', 'out', 'your', 'door', '.']
• [('It', 'PRP'), ("'s", 'VBZ'), ('a', 'DT'), ('dangerous', 'JJ'), ('business', 'NN'), (',', ','),
('Frodo', 'NNP'), (',', ','), ('going', 'VBG'), ('out', 'RP'), ('your', 'PRP$'), ('door', 'NN'),
('.', '.')]
• (S
• It/PRP
• 's/VBZ
• (NP a/DT dangerous/JJ business/NN)
• ,/, Frodo/NNP
• ,/, going/VBG
• out/RP
• your/PRP$
• (NP door/NN)
• ./.)
 17

Tree Representation

7. Chinking
• Chinking is used together with chunking, but while chunking is used to include a
pattern, chinking is used to exclude a pattern.
Python program to perform chinking
import nltk
[Link]('puckt')
from [Link] import word_tokenize
quote = "It's a dangerous business, Frodo, going out your door."
words_quote = word_tokenize(quote)
print(words_quote)
[Link]("averaged_perceptron_tagger")
tags = nltk.pos_tag(words_quote)
print(tags)
#Regular expression
grammar = """
Chunk: {<.*>+}
}<JJ>{""“
chunk_parser = [Link](grammar)
tree = chunk_parser.parse(tags)
print(tree)
Output:
• ['It', "'s", 'a', 'dangerous', 'business', ',', 'Frodo', ',', 'going', 'out', 'your', 'door', '.']
• [('It', 'PRP'), ("'s", 'VBZ'), ('a', 'DT'), ('dangerous', 'JJ'), ('business', 'NN'), (',', ','),
('Frodo', 'NNP'), (',', ','), ('going', 'VBG'), ('out', 'RP'), ('your', 'PRP$'), ('door', 'NN'),
('.', '.')]
 18

• (S
• (Chunk It/PRP 's/VBZ a/DT)
• dangerous/JJ
• (Chunk business/NN ,/, Frodo/NNP ,/, going/VBG out/RP your/PRP$ door/NN ./.))
Tree Representation

8. Using Named Entity Recognition (NER)

Some Examples of Named Entity Recognition (NER)

Python Program to Name Entity Recognition

import nltk
[Link]('punkt')
from [Link] import word_tokenize
quote = "It's a dangerous business, Frodo, going out your door."
words_quote = word_tokenize(quote)
print(words_quote)
[Link]("averaged_perceptron_tagger")
tags = nltk.pos_tag(words_quote)
[Link]("maxent_ne_chunker")
[Link]("words")
tree = nltk.ne_chunk(tags)
print(tree)

Output
['It', "'s", 'a', 'dangerous', 'business', ',', 'Frodo', ',', 'going', 'out', 'your', 'door', '.']
 19

(S
It/PRP
's/VBZ
a/DT
dangerous/JJ
business/NN
,/,
(PERSON Frodo/NNP)
,/,
going/VBG
out/RP
your/PRP$
door/NN
./.)

Note: If we use this code it simply specifies that it is a Named Entity with out
giving the specification.

• tree = nltk.ne_chunk(tags, binary=True)

• print(tree)
Output

9. Term Frequency - Inverse Document Frequency (TF-IDF)

• Term Frequency - Inverse Document Frequency (TF-IDF) is a widely used
statistical method in natural language processing and information retrieval.
• It measures how important a term is within a document relative to a collection of
documents (i.e., relative to a corpus).
• Words within a text document are transformed into important numbers by a text
vectorization process.
• There are many different text vectorization scoring schemes, with TF-IDF being
one of the most common.
20
21
22
23