Scribd
Upload
2 views7 pages

NLP Class Task 3

The document outlines various NLP text processing techniques using the NLTK and SpaCy libraries, including lowercasing, tokenization, stop word removal, stemming, lemmatization, and more. Each technique is accompanied by code examples and expected outputs to demonstrate functionality. Additionally, it covers advanced methods like Named Entity Recognition and different tokenization strategies such as Byte Pair Encoding and WordPiece.

Uploaded by

24f2000672
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
2 views7 pages

NLP Class Task 3

The document outlines various NLP text processing techniques using the NLTK and SpaCy libraries, including lowercasing, tokenization, stop word removal, stemming, lemmatization, and more. Each technique is accompanied by code examples and expected outputs to demonstrate functionality. Additionally, it covers advanced methods like Named Entity Recognition and different tokenization strategies such as Byte Pair Encoding and WordPiece.

Uploaded by

24f2000672
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

NLP Text Processing Techniques

import nltk

[Link]('punkt')

[Link]('stopwords')

[Link]('wordnet')

[Link]('averaged_perceptron_tagger')

[Link]('omw-1.4')

1. Lowercasing

Converts all text to lowercase.

Code

text = "Natural Language Processing"

print([Link]())

Output

natural language processing

2. Tokenization (Word Tokenization)

Splits text into words.

Code

from [Link] import word_tokenize

text = "NLP is very interesting."

tokens = word_tokenize(text)

print(tokens)

Output

['NLP', 'is', 'very', 'interesting', '.']

3. Sentence Tokenization

Splits text into sentences.

Code
from [Link] import sent_tokenize

text = "NLP is interesting. It is used in chatbots."

sentences = sent_tokenize(text)

print(sentences)

Output

['NLP is interesting.', 'It is used in chatbots.']

4. Removing Stop Words

Removes common words such as is, the, and.

Code

from [Link] import stopwords


from [Link] import word_tokenize

text = "This is an example of NLP"

stop_words = set([Link]("english"))

tokens = word_tokenize(text)

filtered = [word for word in tokens if [Link]() not in stop_words]

print(filtered)

Output

['example', 'NLP']

5. Stemming

Reduces words to root form.

Code

from [Link] import PorterStemmer

stemmer = PorterStemmer()

print([Link]("playing"))
print([Link]("played"))
print([Link]("plays"))

Output
play
play
play

6. Lemmatization

Converts words into meaningful dictionary forms.

Code

from [Link] import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

print([Link]("running", pos="v"))
print([Link]("better", pos="a"))

Output

run
good

7. Punctuation Removal

Code

import string

text = "Hello, World!"

result = [Link]([Link]('', '', [Link]))

print(result)

Output

Hello World

8. Number Removal

Code

import re

text = "Python 3.12 released in 2025"

result = [Link](r'\d+', '', text)

print(result)
Output

Python . released in

9. Regular Expressions (Regex)

Extract email addresses.

Code

import re

text = "Contact us at support@[Link]"

emails = [Link](r'\S+@\S+', text)

print(emails)

Output

['support@[Link]']

10. Part-of-Speech (POS) Tagging

Identifies grammatical roles.

Code

import nltk

text = "The cat is sleeping"

tokens = nltk.word_tokenize(text)

print(nltk.pos_tag(tokens))

Output

[('The', 'DT'),
('cat', 'NN'),
('is', 'VBZ'),
('sleeping', 'VBG')]

Meaning

 DT → Determiner

 NN → Noun

 VBZ → Verb

 VBG → Verb (ing form)


11. Named Entity Recognition (NER)

Code

import spacy

nlp = [Link]("en_core_web_sm")

doc = nlp("Sundar Pichai works at Google.")

for ent in [Link]:


print([Link], ent.label_)

Output

Sundar Pichai PERSON


Google ORG

12. Text Normalization

Code

text = "U R AWESOME!!!"

normalized = [Link]()

print(normalized)

Output

u r awesome!!!

13. Whitespace Tokenization

Splits text based on spaces.

Code

text = "I love Natural Language Processing"

tokens = [Link]()

print(tokens)

Output

['I', 'love', 'Natural', 'Language', 'Processing']

14. Punctuation-Based Tokenization


Code

import re

text = "Hello,world!How are you?"

tokens = [Link](r'\w+', text)

print(tokens)

Output

['Hello', 'world', 'How', 'are', 'you']

15. Byte Pair Encoding (BPE) – Simple Illustration

Corpus

words = ["lower", "lowest"]


print(words)

Frequent Pair

lower
lowest

Most frequent pair:

lo

Merge:

lo w e r
lo w e s t

Next merge:

low er
low est

Output Vocabulary

low, er, est

16. WordPiece Tokenization

Example

word = "unhappiness"

tokens = ["un", "##happy", "##ness"]

print(tokens)
Output

['un', '##happy', '##ness']

Meaning

un → prefix
##happy → root word
##ness → suffix

You might also like