Scribd
Upload
17 views13 pages

spaCy xx_ent_wiki_sm Model Usage

Uploaded by

Bharat Mishra
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
17 views13 pages

spaCy xx_ent_wiki_sm Model Usage

Uploaded by

Bharat Mishra
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

DELHI TECHNOLOGICAL

UNIVERSITY
SE-316
NATURAL LANGUAGE PROCESSING

Department of Software Engineering


Delhi Technological University
Bawana Road, Delhi-110042

Submitted by
Prashant Tiwari
Roll Number :- 2K20/IT/103
Batch :- IT-B

Submitted to : Dr. Divyashikha Sethia


Department of Software Engineering
Delhi Technological University
INDEX

S. No. Experiment Date

1. Import nltk and download the ‘stopwords’ 13-01-2023


and ‘punkt’ packages

2. Import spacy and load the language model. 13-01-2023

3. WAP in python to tokenize a given text. 20-01-2023

4. WAP in python to get the sentences of a 03-03-2023


text document.

5. WAP in python to tokenize text with 03-02-2023


stopwords as delimiters.

6. WAP in python to add custom stop words in 03-02-2023


spaCy.

7. WAP to remove punctuations, perform 24-02-2023


stemming, lemmatize given text and extract
usernames from emails

8. WAP to do spell correction, extract all 07-03-2023


nouns, pronouns and verbs in a given text

9. WAP to find similarity between two words 31-03-2023


and classify a text as positive/negative
sentiment
EXPERIMENT - 1
AIM : Import nltk and download the ‘stopwords’ and ‘punkt’
packages

CODE :
import nltk

[Link]('stopwords')
[Link]('punkt')

OUTPUT :
EXPERIMENT - 2
AIM : Import spacy and load the language model

CODE :
import spacy
nlp_eng = [Link]('en_core_web_sm')
nlp_multi = [Link]('xx_ent_wiki_sm')

OUTPUT :
EXPERIMENT - 3
AIM : WAP in python to tokenize a given text

CODE :
from nltk import word_tokenize
text = "Last week, the University of Cambridge shared its own research
that shows if everyone wears a mask outside home,dreaded ‘second wave’
of the pandemic can be avoided."
text = word_tokenize(text)
for t in text:
print(t)

OUTPUT :
EXPERIMENT - 4
AIM : WAP in python to get the sentences of a text document.

CODE :
file = open('[Link]')
Input_text = [Link]()
ans = Input_text.split('.')

for an in ans:
print(an,'\n')

OUTPUT :
EXPERIMENT - 5
AIM : WAP in python to tokenize text with stopwords as
delimiters.

CODE :
text = "Walter was feeling anxious. He was diagnosed today. He probably
is the best person I know."

stop_words_and_delims = ['was', 'is', 'the', '.', ',', '-', '!', '?']


for r in stop_words_and_delims:
text = [Link](r, 'DELIM')

words = [[Link]() for t in [Link]('DELIM')]


words_filtered = list(filter(lambda a: a not in [''], words))
for word in words_filtered:
print(word)

OUTPUT :
EXPERIMENT - 6
AIM : WAP in python to add custom stop words in spaCy.

CODE :
import spacy

nlp = [Link]('en_core_web_sm')

custom_stop_words = ['was', 'is','the','JUNK','NIL','of','more' ,'.',


',', '-', '!', '?','a']
for word in custom_stop_words:
[Link][word].is_stop = True

doc = nlp("Jonas was a JUNK great guy NIL Adam was evil NIL Martha JUNK
was more of a fool")
for token in doc:
if not token.is_stop:
print([Link], end=" ")

OUTPUT :
EXPERIMENT - 7
AIM : WAP to remove punctuations, perform stemming,
lemmatize given text and extract usernames from emails

CODE :
punctuations = '''!()-[]{};:'"\,<>./?@#$%^&*_~'''

string = "Jonas!!! great \\guy <> Adam --evil [Martha] ;;fool() ."

ans = ""
for char in string:
if char not in punctuations:
ans+=char

print(ans)

from [Link] import PorterStemmer


from [Link] import word_tokenize
text= "Dancing is an art. Students should be taught dance as a subject
in schools . I danced in many of my school function. Some people are
always hesitating to dance."
ans = ""
stemmer = PorterStemmer()
tokens = word_tokenize(text)
for token in tokens:
ans+=[Link](token)
ans+=" "
print(ans)

from [Link] import wordnet


from [Link] import word_tokenize

from [Link] import WordNetLemmatizer


lemmatizer = WordNetLemmatizer()
text= "Dancing is an art. Students should be taught dance as a subject
in schools . I danced in many of my school function. Some people are
always hesitating to dance."
ans = ""
tokens = word_tokenize(text)
for token in tokens:
ans+=[Link](token, [Link])
ans+=" "
print(ans)
from [Link] import word_tokenize

text= "The new registrations are potter709@[Link] ,


elixir101@[Link]. If you find any disruptions, kindly contact
granger111@[Link] or severus77@[Link] "

text_list = word_tokenize(text)
usernames = []
for i in range(len(text_list)):
if text_list[i] == "@":
[Link](text_list[i-1])
print(usernames)

OUTPUT :
EXPERIMENT - 8
AIM : WAP to do spell correction, extract all nouns, pronouns
and verbs in a given text

CODE :
from textblob import TextBlob
text="He is a gret person. He beleives in bod"
textb = TextBlob(text)
correct_text = [Link]()
print(correct_text)

import nltk
from nltk import word_tokenize, pos_tag
text="James works at Microsoft. She lives in manchester and likes to
play the flute"
tokens = word_tokenize(text)
parts_of_speech = nltk.pos_tag(tokens)
nouns = list(filter(lambda x: x[1] == "NN" or x[1] == "NNP",
parts_of_speech))
for noun in nouns:
print(noun[0])

from nltk import pos_tag, word_tokenize

text = "I may bake a cake for my birthday. The talk will introduce
reader about Use of baking"

words = word_tokenize(text)

verb_phrases = []
for i in range(len(words)):
if i > 0 and pos_tag(words)[i][1] == 'VB':
verb_phrase = words[i-1] + ' ' + words[i]
verb_phrases.append(verb_phrase)

for i in verb_phrases:
print (i)

OUTPUT :
EXPERIMENT - 9
AIM : WAP to find similarity between two words and classify a
text as positive/negative sentiment

CODE :
import spacy

nlp = [Link]('en_core_web_md')
words = "amazing terrible excellent"

tokens = nlp(words)

token1, token2, token3 = tokens[0], tokens[1], tokens[2]

print(f"Similarity between {token1} and {token2} : ",


[Link](token2))
print(f"Similarity between {token1} and {token3} : ",
[Link](token3))

from textblob import TextBlob


text = "It was a very pleasant day"
print(TextBlob(text).sentiment)

OUTPUT :

Common questions

Powered by AI

Stemming and lemmatization are processes used to reduce words to their base or root form. Stemming cuts words to the base form which may not be a valid word, whereas lemmatization transforms words to their base form using a vocabulary and morphological analysis. Stemming is faster but less accurate, while lemmatization is slower but produces more meaningful base forms .

Using stopwords and delimiters in tokenization helps by segmenting text into meaningful elements while removing common, less informative words. This practice reduces noise in text processing, allowing algorithms to focus on more relevant data, improving efficiency and accuracy in tasks like text analysis and natural language processing .

Different tokenization techniques, like using stopwords as delimiters or splitting by sentences, ensure flexibility by allowing the selection of context-appropriate methods. This adaptability meets diverse processing requirements, improving the suitability and effectiveness of analyses across different domains and languages .

Spell correction is vital in textual analysis as it ensures data accuracy and consistency, reducing noise caused by typographical errors. This process enhances the quality of data by enabling more precise matching and analysis of text, resulting in better performance of NLP models and improved outcomes in applications like sentiment analysis and information retrieval .

The method of extracting email usernames involves identifying token patterns typical of email addresses and then parsing tokens around '@' symbols. This process is important for data anonymization, user identification, and communication routing, helping organizations manage customer data and interactions efficiently .

Importing language processing libraries like NLTK and spaCy is crucial because these libraries provide essential tools and resources for text processing, including tokenizers, parsers, and models for various NLP tasks. They streamline the development process, offering built-in functionalities to handle numerous language processing requirements efficiently .

The classification of text sentiment contributes to artificial intelligence by enabling machines to understand and interpret human emotions and opinions within text. This capability aids in applications such as customer feedback analysis, social media monitoring, and recommendation systems, providing valuable insight into consumer behavior and preferences .

Challenges in performing word similarity analysis include polysemy and context variability, where identical words can have different meanings. Potential solutions involve using contextual word embeddings, like those offered by advanced models in spaCy, which consider the word's surroundings to capture more accurate similarity measures .

Adding custom stop words in spaCy enhances text customization by allowing users to tailor the processing pipeline to specific needs, filtering out uninformative or context-specific terms that might not be recognized by default settings. This refinement optimizes NLP model performance by focusing computational resources on relevant data .

Extracting specific parts of speech during data processing involves tagging each word in a text with its appropriate grammatical category, such as nouns or verbs. This process aids in understanding the syntactic structure, enhancing information retrieval, text summarization, and content classification by focusing on meaningful language components .

You might also like