Natural Language Processing
What is Natural Language?
• Natural Language: Human languages like English, Hindi, Spanish, etc.
o Governed by rules (syntax (Grammar) , lexicon (words used in a language),
semantics (meaning)).
o Redundant and ever-changing.
• Computer Language: Used to interact with computers (e.g., Python, C++, HTML).
o Requires specific instructions called programs.
Natural Language Processing (NLP)
• NLP enables computers to understand, interpret, and process human language.
• It converts natural language into a digital (binary) format that machines can understand.
• NLP is a subfield of AI.
Applications of NLP
1. Voice Assistants: Interpret and respond to spoken commands (e.g., Alexa, Siri).
2. Autogenerated Captions: Convert speech to text in real time (e.g., YouTube, Google
Meet).
3. Language Translation: Translate text/speech between languages (e.g., Google Translate).
4. Sentiment Analysis: Detect if an opinion is positive, negative, or neutral.
5. Text Classification: Categorize text into predefined categories (e.g., news topics).
6. Keyword Extraction: Identify key terms in a document for insight or analysis.
Stages of NLP
1. Lexical Analysis
o Breaks down large text into paragraphs, sentences, and words.
2. Syntactic Analysis (Parsing)
o Checks grammar and relationships among words.
3. Semantic Analysis
o Analyzes meaning; rejects semantically incorrect phrases (e.g., "hot ice cream").
4. Discourse Integration
o Ensures sentence relationships are logical in context.
5. Pragmatic Analysis
o Interprets intended meanings using real-world context.
Chatbots
• Simulate human-like conversations.
• Types:
o Script Bot: Follows predefined responses.
o Smart Bot: Learns and adapts using NLP and AI.
Human Language vs. Computer Language
• Human language is complex and ambiguous.
• Computers need it converted to numbers through a process called Text Normalization.
Text Normalization Steps
1. Sentence Segmentation: Break corpus into individual sentences.
2. Tokenization: Break sentences into words/tokens.
3. Stopword Removal: Remove common but non-informative words (e.g., "the", "and").
4. Case Conversion: Convert all text to lower case.
5. Stemming: Reduce words to root forms (e.g., "studies" → "studi").
6. Lemmatization: Reduce words to meaningful root forms (e.g., "studies" → "study").
o More accurate than stemming, but slower.
Bag of Words (BoW)
• An NLP model to extract features from text.
• Outputs:
o A vocabulary (unique words).
o Frequency of each word.
• Steps:
1. Preprocess text.
2. Create a dictionary (unique words).
3. Create document vectors for individual document (word count per document).
4. Create document vector for all the documents (word count in all documents).
Example
Consider the following documents :
Document 1 : NLP is a domain of AI.
Document 2 : NLP stands for Natural Language Processing.
Implement all the four steps of Bag of Words (BoW) model to create a document vector table.
Steps:-
Step 1 – Text Normalisation
Document 1: [nlp is a domain of AI]
Document 2: [nlp stands for natural language processing]
Step 2 : Creating Dictionary
nlp, is, a, domain, of, ai, stands, for, natural, language, processing
Step 3 : Create Document vectors
nlp is a domain of ai stands for natural language processing
Step 4 : Create Document vectors for all the documents/Term Frequency
TF-IDF (Term Frequency–Inverse Document Frequency)
• TF-IDF is used to determine the importance of a word in a document relative to a corpus.
• Words that occur frequently in one document but rarely in others are weighted higher.