Tokenization:
1. Python Program to Perform word_tokenize for tokenization.
import nltk
[Link]('punkt')
[Link]('punkt_tab')
# Word tokenizer
from [Link] import word_tokenize
text = "The quick brown fox jumped over the lazy dog!"
tokens = word_tokenize(text)
print("Word tokenizer")
print(tokens,'\n')
# Sentence tokenizer
from [Link] import sent_tokenize
text2 = "I used to be happy. But now my happiness is resides in you."
tokens2 = sent_tokenize(text2)
print("Sentence tokenizer")
print(tokens2,'\n')
text3 = "Dr A.P.J Abdul Kalam was an amazing person ! He was what i thrive to
become."
tokens3 = sent_tokenize(text3)
print("Sentence tokenizer")
print(tokens3,'\n')
tokens4 = word_tokenize(text3)
print("Word tokenizer")
print(tokens4,'\n')
2. Python Program to Perform wordpunct_tokenize for tokenization.
from [Link] import wordpunct_tokenize
from [Link] import word_tokenize
text = "Welcome to Abhishek's blog. I welcome you all."
tokens = word_tokenize(text)
print(tokens,'\n')
tokens1 = wordpunct_tokenize(text)
print(tokens1, '\n')
3. Python Program to Perform TreebankWordTokenizer for tokenization.
from [Link] import TreebankWordTokenizer
text = "Welcome to Abhishek's blog. I welcome you all."
tokens = TreebankWordTokenizer()
[Link](text)
Stopwords:
1. Python Program to Perform To check the list of stopwords
import nltk
from [Link] import stopwords
[Link]('stopwords')
print([Link]('english'))
2. Python Program to Perform Removing stop words with NLTK
from [Link] import stopwords
from [Link] import word_tokenize
example_sent = """This is a sample sentence, showing off the stop words filtration."""
stop_words = set([Link]('english'))
word_tokens = word_tokenize(example_sent)
# converts the words in word_tokens to lower case and then checks whether
#they are present in stop_words or not
filtered_sentence = [w for w in word_tokens if not [Link]() in stop_words]
#with no lower case conversion
filtered_sentence = []
for w in word_tokens:
if w not in stop_words:
filtered_sentence.append(w)
print(word_tokens)
print(filtered_sentence)
3. Python Program to Perform Removing stop words with SpaCy
import spacy
# Load spaCy English model
nlp = [Link]("en_core_web_sm")
# Sample text
text = "There is a pen on the table"
# Process the text using spaCy
doc = nlp(text)
# Remove stopwords
filtered_words = [[Link] for token in doc if not token.is_stop]
# Join the filtered words to form a clean text
clean_text = ' '.join(filtered_words)
print("Original Text:", text)
print("Text after Stopword Removal:", clean_text)
STEMMING TECHNIQUES:
1. Python Program to perform PorterStemmer class for stemming
from [Link] import PorterStemmer
# Create a PorterStemmer object
stemmer = PorterStemmer()
# Example words to stem
words = ["running", "runner", "ran", "runs", "easily", "fairly"]
# Stem each word
stemmed_words = [[Link](word) for word in words]
print("Original words:", words)
print("Stemmed words:", stemmed_words)
2. Python Program to perform RegexpStemmer class for stemming
from [Link] import RegexpStemmer
reg_stemmer = RegexpStemmer("ing$|s$|e$|able$|",min=4) # If the last word contains ing or
s or e or able, just remove that
print(reg_stemmer.stem("eating"))
print(reg_stemmer.stem("cable"))
3. Python Program to perform Snowball Stemmer class for stemming
from [Link] import SnowballStemmer
snowballstemmer = SnowballStemmer("english")
words = ["fairly","sportingly","loves"]
for word in words:
print(word + "-->"+[Link](word))
Lemmatization:
1. Python Program to Perform Lemmatization
from [Link] import WordNetLemmatizer
[Link]('wordnet')
wl = WordNetLemmatizer()
words = ['stepped', 'goes', 'loving', 'made', "ate"]
lemma_words = []
for i in words:
lemma_words.append([Link](i))
print("Lemmatization")
print(lemma_words)
lemma_words_of_tokens = []
for i in tokens:
lemma_words_of_tokens.append([Link](i))
print(lemma_words_of_tokens)
lemma_words_of_tokens2 = []
for i in tokens2:
for j in word_tokenize(i):
lemma_words_of_tokens2.append([Link](j))
print(lemma_words_of_tokens2'\n')