MODULE 1 VI SEM NLP [BAI601]

MODULE 1
SYLLABUS:
Introduction: What is Natural Language Processing? Origins of NLP,
Language and Knowledge, The Challenges of NLP, Language and Grammar,
Processing Indian Languages, NLP Applications.
Language Modeling: Statistical Language Model - N-gram model (unigram,
bigram), Paninion Framework, Karaka theory.
_________________________________________________________________
Contents
1.1 What is Natural Language Processing? .......................................................1
1.2 Origin of NLP ..............................................................................................2
1.3 Language and Knowledge............................................................................3
1.4 The Challenges of NLP ................................................................................5
1.5 Language and Grammar ...............................................................................7
1.5.1 Surface Structure and Deep Structure of Sentences ...................... 7
1.5.2 Phrase Structure Grammar and Parsing with Examples .................... 8

____________________LECTURE 01______________________

1.1 What is Natural Language Processing?

Language is the primary means of communication used by humans. It is the
tool we use to express the greater part of our ideas and emotions. It shapes
thought, has a structure, and carries meaning. Learning new concepts and
expressing ideas through them is so natural that we hardly realize how we
process natural language.

Natural language processing (NLP) is concerned with the development of

computational models of aspects of human language processing. There are
two main reasons for such development:

1. To develop automated tools for language processing

2. To gain a better understanding of human communication

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 1

MODULE 1 VI SEM NLP [BAI601]
Building computational models with human language-processing abilities
requires a knowledge of how humans acquire, store, and process language. It
also requires a knowledge of the world and of language.

1.2 Origin of NLP

The origins of Natural Language Processing (NLP) can be traced back to
multiple disciplines, including linguistics, computer science, and artificial
intelligence. Below is a brief history of its development:
Early Foundations (1950s - 1960s)
• Alan Turing's Proposal (1950): The famous Turing Test introduced
the idea of machines understanding and generating human-like text.
• Machine Translation (1954): The Georgetown-IBM experiment
translated 60 Russian sentences into English, marking the first NLP
application.
• Chomsky's Linguistic Theories (1957): Noam Chomsky introduced
Transformational-Generative Grammar, influencing syntactic
analysis in NLP.
• Rule-Based Systems: Early NLP models relied on hand-crafted
grammar rules and lexicons for sentence structuring

Statistical NLP (1970s - 1990s)

• Shift from Rule-Based to Probabilistic Models: Introduction of
Hidden Markov Models (HMMs) and n-gram models improved
language modeling.
• Part-of-Speech (POS) Tagging: Statistical techniques replaced rule-
based tagging, enhancing efficiency in syntax processing.
• Machine Learning in NLP (1990s): Introduction of probabilistic
parsing, word sense disambiguation, and named entity recognition
(NER).
Machine Learning and Deep Learning Era (2000s - Present)
• 2000s: Rise of Support Vector Machines (SVMs) and Conditional
Random Fields (CRFs) for NLP tasks like Named Entity Recognition.
• 2010s: Deep learning revolutionized NLP with Word2Vec (2013) for
word embeddings and Recurrent Neural Networks (RNNs) for
sequence tasks.
• Transformers & Pretrained Models (2017 - Present):
Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 2
MODULE 1 VI SEM NLP [BAI601]
o Google's Attention Is All You Need (2017) introduced the
Transformer architecture, leading to advanced models like
BERT (2018) and GPT series (2019-Present).
o State-of-the-art models like ChatGPT and T5 are now
dominating NLP applications in text generation, summarization,
and translation.
NLP has evolved from simple rule-based methods to complex neural
network-driven models, making human-computer interaction more natural
and effective. With advancements in AI, NLP continues to improve, powering
applications like virtual assistants, machine translation, and conversational
AI.

Figure 1. Timeline of evolution of NLP

1.3 Language and Knowledge

In Natural Language Processing (NLP), understanding and processing
human language require both language and knowledge components. These
two aspects help machines interpret, generate, and manipulate text
meaningfully.
Language in NLP refers to the structural and grammatical aspects of human
communication. It includes the following components:
a) Phonology
• The study of sound patterns in language.
• Example: "cat" and "bat" differ by one phoneme (sound unit), but their
meaning is different.
b) Morphology

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 3

MODULE 1 VI SEM NLP [BAI601]
• The study of word formation and structure.
• Example: "run" → "running" (adding -ing as a morphological change).
c) Syntax
• The arrangement of words to form meaningful sentences (grammar).
• Example: "He plays football" is syntactically correct, while "Plays he
football" is not.
d) Semantics
• The meaning of words and sentences.
• Example: "The bank is closed" (does 'bank' refer to a financial
institution or a riverbank?).
e) Pragmatics
• The context-dependent interpretation of language.
• Example: "Can you pass the salt?" is a request, not a question about
ability.
f) Discourse Analysis
• Understanding language beyond individual sentences (conversation
flow).
• Example: "John bought a car. He is happy." (Understanding that 'He'
refers to John).
Machines need knowledge to understand, reason, and infer meaning from
language. Different types of knowledge are used in NLP:
a) World Knowledge
• General facts about the world that help in understanding meaning.
• Example: "The sun rises in the east" is a common fact that machines
should know.
b) Linguistic Knowledge
• Includes grammar rules, syntax, and morphology of a language.
• Example: Understanding pluralization (cat → cats, child → children).
c) Domain Knowledge
• Specific knowledge required for industry-related applications
(medical, legal, finance).
• Example: "BP is high" in healthcare refers to blood pressure, not a
company.

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 4

MODULE 1 VI SEM NLP [BAI601]
d) Common Sense Knowledge
• Basic everyday reasoning skills that humans possess.
• Example: "If you drop a glass, it will likely break."
e) Ontological Knowledge
• Relationships between concepts in structured forms like knowledge
graphs.
• Example: Google’s Knowledge Graph links entities like people,
places, and events.
f) Contextual Knowledge
• Understanding meaning based on previous conversations or
document history.
• Example: "She said she would come tomorrow." (Understanding who
'she' is requires context).
Both language and knowledge play a crucial role in NLP. While language
components focus on the structure and meaning of words, knowledge sources
help machines make sense of ambiguous, context-dependent information. The
combination of linguistic models and knowledge-based systems enables
modern NLP applications like chatbots, machine translation, and
sentiment analysis.
REVIEW QUESTIONS

1. What is the main goal of NLP?

2. Name the first machine translation system developed in the 1950s.
3. ow did the rise of statistical models influence NLP in the 1990s?
4. What is the difference between syntax and semantics in language processing?

____________________LECTURE 02______________________

1.4 The Challenges of NLP

Natural Language Processing (NLP) faces several challenges due to the
complexity, ambiguity, and variability of human language. These challenges
arise from linguistic, computational, and data-related issues, making it
difficult for machines to understand and generate language as humans do.
Ambiguity in Language: Ambiguity is a major challenge in NLP, as words

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 5

MODULE 1 VI SEM NLP [BAI601]
and sentences can have multiple meanings based on context.
• Lexical Ambiguity: A single word has multiple meanings.
o Example: "The bank is near the river." (Bank = financial
institution or riverbank?)
• Syntactic Ambiguity: A sentence has multiple grammatical
interpretations.
o Example: "I saw the man with the telescope." (Who has the
telescope—'I' or 'the man'?)
• Semantic Ambiguity: The meaning of a sentence is unclear.
o Example: "Visiting relatives can be boring." (Is visiting relatives
boring, or are the relatives boring?)
• Pragmatic Ambiguity: Understanding depends on real-world context.
o Example: "Can you open the window?" (Is this a question about
ability or a request?)
Variability and Diversity in Language: Human language is highly variable
due to different dialects, slang, idioms, and evolving usage.
• Different Ways to Express the Same Idea
o Example: "I'm feeling great today!" vs. "Today, I am in a good
mood!"
o Machines struggle to recognize that these sentences convey the
same meaning.
• Colloquial and Informal Language
o Example: "Gotta go now" vs. "I have to leave now."
o Informal expressions make NLP processing difficult.
• Code-Switching (Mixing Languages)
o Example: "I love eating dosa in Bangalore." (English + Indian
language elements)
o Many people mix languages in speech and text, posing a
challenge for NLP models.
Data Limitations and Bias: The effectiveness of NLP models depends on
high-quality, unbiased data.
• Lack of Sufficient Training Data
o Many languages and dialects lack large, annotated datasets for
NLP training.

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 6

MODULE 1 VI SEM NLP [BAI601]
• Bias in Data
o NLP models may learn biases present in training data.
o Example: Gender bias—"Doctor" is often associated with "he"
in older datasets.
• Domain-Specific Limitations
o General NLP models may perform poorly in specialized fields
like medicine or law due to domain-specific terminology.

1.5 Language and Grammar

Language is the fundamental medium of human communication, comprising
structured syntax, semantics, and pragmatics that enable effective information
exchange. Natural Language Processing (NLP) aims to bridge the gap between
human language and computational understanding by analyzing, processing, and
generating text-based data. Grammar, a crucial component of language, defines
the rules governing sentence structure, word formation, and syntactical
correctness. In NLP, formal grammar frameworks like Context-Free Grammar
(CFG), Dependency Grammar, and Universal Grammar help in constructing
accurate linguistic models for parsing and syntactic analysis. Morphology, which
deals with the formation and structure of words, plays an essential role in tasks like
stemming and lemmatization. Furthermore, Part-of-Speech (POS) tagging is
employed to categorize words into their grammatical classes, enhancing tasks such
as machine translation, sentiment analysis, and information retrieval. By
leveraging computational linguistics and machine learning techniques, NLP
models can process large-scale text corpora, enabling applications such as
chatbots, speech recognition, and text summarization.

1.5.1 Surface Structure and Deep Structure of Sentences

The concepts of surface structure and deep structure were introduced by Noam
Chomsky in Transformational-Generative Grammar (TGG) to explain the
relationship between syntax and meaning in language processing.

A. Deep Structure (D-Structure):

o It represents the underlying meaning or abstract grammatical

relations of a sentence.

o It captures the essential syntactic relationships between words and

phrases before any transformations occur.

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 7

MODULE 1 VI SEM NLP [BAI601]
o It is closely related to semantics (meaning) rather than how the
sentence is phrased.

o Example:

▪ " The teacher explained the concept to the students."

▪ " She will complete the project."

B. Surface Structure (S-Structure):

o It refers to the actual arrangement of words in a sentence as

spoken or written.

o It is derived from the deep structure through transformations such

as passivation, question formation, and word reordering.

o It focuses more on syntax and how the sentence appears rather than
its fundamental meaning.

o Example:

▪ “The concept was explained to the students by the teacher."

(Passive voice)

▪ "Will she complete the project?" (Yes/No question

formation)

Understanding deep and surface structures helps in parsing sentences, machine

translation, and information extraction. NLP models use techniques like
dependency parsing and constituency parsing to analyze deep structure,
enabling more accurate sentence transformation and comprehension.

1.5.2 Phrase Structure Grammar and Parsing with Examples

Phrase Structure Grammar (PSG) is a syntactic formalism used in linguistics and

Natural Language Processing (NLP) to describe the structure of sentences. It
represents how words group together into phrases and how phrases combine
to form sentences. PSG follows a set of rewrite rules known as Phrase Structure
Rules (PSRs) or Context-Free Grammar (CFG) rules. These rules break down
a sentence into smaller grammatical components using hierarchical structures like
parse trees.

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 8

MODULE 1 VI SEM NLP [BAI601]
A context-free grammar consists of:

• Terminal symbols (words): The actual words in a sentence (e.g., "police,"

"catch," "snatcher").

• Non-terminal symbols: Abstract categories (e.g., NP, VP, S) that

represent different syntactic components.

• Production rules: Rules that define how non-terminals expand into

terminals and other non-terminals.

Common Phrase Structure Rules:

1. Sentence Rule:
S→ NP VP
(A sentence consists of a noun phrase and a verb phrase.)

2. Noun Phrase (NP) Rules:

NP→ Det N
(A noun phrase consists of a determiner and a noun.)
NP→ N
(A noun phrase can also be a single noun.)
NP→NPPP
(A noun phrase can include a prepositional phrase.)

3. Verb Phrase (VP) Rules:

VP→V NP
(A verb phrase consists of a verb followed by a noun phrase.)
VP→V NPPP
(A verb phrase can also include a prepositional phrase.)

4. Prepositional Phrase (PP) Rule:

PP→PNP
(A prepositional phrase consists of a preposition followed by a noun
phrase.)

Example 1: Active Voice

Sentence: "The police will catch the snatcher."

Using phrase structure rules, we can break this sentence into components:

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 9

MODULE 1 VI SEM NLP [BAI601]
1. Apply sentence rule:
S→NP VP

2. Expand NP (The police) and VP (will catch the snatcher):

NP→ Det N
VP→ Aux V NP

3. Further expand NP (the snatcher):

NP→ Det N

The subject "The police" is performing the action "catch" on the object "the
snatcher".

The Parse Tree for this Active Voice is given below:

Example 2: Passive Voice

Transformations modify deep structures (active) to form grammatically correct and

meaningful surface structures (passive).

Active to Passive Transformation

• Rule: Changes a sentence from active to passive by rearranging the subject

and object.

• Formula:

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 10

MODULE 1 VI SEM NLP [BAI601]

• Example:

o Deep Structure (Active): The police will catch the snatcher.

o Surface Structure (Passive): The snatcher will be caught by the

police.

The parse tree for the passive sentence is given below:

REVIEW QUESTIONS

1. Why is ambiguity a major challenge in NLP?

2. What are the main components of grammar in NLP?
3. What is a parse tree in NLP?
4. Give an example of a sentence with subject-verb agreement.

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 11

MODULE 1 VI SEM NLP [BAI601]
____________________LECTURE 03______________________
1.6 Processing Indian Languages
Natural Language Processing (NLP) for Indian languages presents unique
challenges and opportunities due to the diversity of scripts, grammar structures,
and linguistic variations. India has 22 official languages and hundreds of dialects,
making it a rich field for NLP research.

Key Characteristics of Indian Languages are:

• Unlike English, Indic scripts follow a non-linear writing system, meaning

characters and diacritics may be placed above, below, or around
consonants.
• Indian languages follow the SOV (Subject-Object-Verb) word order,
unlike English, which follows SVO (Subject-Verb-Object).

English (SVO): "She reads a book."

Hindi (SOV): "वह एक ककताब पढ़ती है।" (Vah ek kitaab padhti hai.)

• Words in Indian languages can be rearranged within a sentence without

altering the meaning significantly.
"राम स्कूल जाता है।" (Ram school jaata hai.)
"स्कूल राम जाता है।" (School Ram jaata hai.)
Both convey the same meaning.
• Unlike English, where spelling is highly standardized, Indian languages
have more flexibility in spelling and phonetic variations.
• Indian languages exhibit complex word formation with inflections based
on gender, tense, and aspect.
"लड़का" (boy) → "लड़की" (girl)
"खेलता है " (plays - masculine) → "खेलती है " (plays - feminine)
• Indian languages frequently use compound verbs and verb sequences for
nuanced meanings.
"बात कर रहा है " (baat kar raha hai) – "is talking"
"खेल रही हूँ " (khel rahi hoon) – "I am playing"
• English uses prepositions (before nouns), while Indian languages use post-
positions (after nouns).
English: "On the table"

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 12

MODULE 1 VI SEM NLP [BAI601]
Hindi: "मेज़ पर" (mez par)
• Indian languages use auxiliary verbs to express tense, aspect, and
modality.
"गाता है" (gaata hai) – "sings"
"गाता रहा" (gaata raha) – "was singing"

1.7 NLP Applications

The applications utilizing NLP include the following:

Machine Translation: This refers to automatic translation of text from one human
language to another. To carry out this translation, it is necessary to understand
words and phrases, grammars of the two languages involved, semantics of the
languages, and world knowledge.

Speech Recognition: This is the process of mapping acoustic speech signals to a

set of words. The difficulties arise due to wide variations in the pronunciation of
words, homonyms (e.g., dear and deer), and acoustic ambiguities (e.g., in the rest
and interest).

Speech Synthesis: Speech synthesis refers to the automatic production of speech

(utterance of natural language sentences). Such systems can read out your mail on
the telephone or even read out a storybook for you. To generate utterances, text
must be processed. So, NLP remains an important component of any speech
synthesis system.

Natural Language Interfaces to Databases: Natural language interfaces allow

querying a structured database using natural language sentences.

Information Retrieval: This is concerned with identifying documents relevant to

a user’s query. NLP techniques have found useful applications in information
retrieval. Indexing (stop word elimination, stemming, phrase extraction, etc.),
word sense disambiguation, query modification, and knowledge bases have been
used in IR systems to enhance performance, e.g., by providing methods for query
expansion. WordNet, LDOCE (Longman Dictionary of Contemporary English)
and Roget’s Thesaurus are some of the useful lexical resources for IR research.

Information Extraction: An information extraction system captures and outputs

information contained within a document. Like an information retrieval system, IR
responds to a user’s information need. However, unlike in an information retrieval

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 13

MODULE 1 VI SEM NLP [BAI601]
system, the information need is not expressed as a keyword query. Instead, it is
specified as pre-defined database schemas or templates. Whereas an IR system
identifies a subset of documents in a large repository of text database—e.g., in a
library scenario, a subset of resources in a search system—an information
extraction system identifies structured data such as names, locations, or events
from unstructured text.

Question Answering: Given a question and a set of documents, a question

answering system attempts to find the precise answer, or at least the precise portion
of text in which the answer appears. This is unlike an IR system, which returns a
whole document that seems relevant to the user’s query. A question answering
system is different from an information extraction system in that the content that
is to be extracted is unknown. In general, a question answering system benefits
from having an information extraction system to identify entities in the text. A
question answering system requires more NLP than an information retrieval
system or an information extraction system. It requires not only precise analysis of
questions and portions of texts but also semantic as well as background knowledge
to answer certain types of questions.

Text Summarization: This deals with the creation of summaries of documents

and involves syntactic, semantic, and discourse-level processing of text.

REVIEW QUESTIONS

1. How does the word order in Indian languages differ from English?
2. What is machine translation in NLP?
3. What is the role of NLP in information retrieval?
4. What is the difference between text summarization and information
retrieval?

Prepared by: Divyesh Divakar, Dept. of AIML, CEC Page 14