A.

Tokeni zation:
a from nlHe tokenize import sent tokenize, bord_tokenize
text = Helo! 2'm learning tokenizatin in ython
int(Sentence tokens: ", sent tokenize(tert))
print( Wrd token: ", word_tokenize( teE))
|outpuk:
Sentenee tokens: [Hello!',"Tm Leaming tokeni zation in Pythem.]
Word tokens; Hello', i',T,m,'eaming, tokenizatin, iny
Bthons1
B from nltktolkenize import wrdtokenize
text NLP makes maenes understamd human
tokens wrdto kenizeltext)
print okem)

(NLe,ales', maehines' 'undestamd', himan, lamgiage','1

from nlEkitokenize import senttokenize
text ="Natural Lonquage Processing is ammazing Tt helps compukers
understan hunam lange Let's learn NLP together"
sentences senttokeni2e (text)
pint(sentences)

C'Hakural Lamguage Procasing is amazlng,

"gt helps camputtrs undestand
unn derstomd human bngage.",
"Let's Learn NLe toqether!"
 Ste mmingi
)from mitk.
ltb stem impont Porrterstemmer
stemmer Porterstemer
unner", "easily,Fairhess", "studis"]
steoms stemmerstem () tor uw hn woTds]
pant("iginal rds!, oor ds)
pri nt ("stemmed words, ", stems)

Original Wor de: C'rumnng, unner', 'easiy','fairness', studies

steonmed wods:[un','mer, 'easii', Par',studi']
rom [Link] tmpont Porterstemmer
from [Link] inpont Laneasterstemmner
portere Porterstemmer()
Laneaster Lancasterstenme)
print (ulor ter stemmer)
printu cats ",porter stm ( eats")
print (" trouble y ", porter. stem(" troubte))
prnt ("troub ling ’", portemikm("troubling"))
prìntl"Lmoaster stemmer)
print(u Cats =) ", lameas [Link]( cats "))
print(trouble n, [Link] (utrouble )
prnt(troubbng , Lam casteirstern ("troubUngu)
Porter Stemner
eats cat
trowble troubl
troubl
toubling
Lameastar stemnner
Cats => cat
tyowble trowbl
trou bling >troubl
s>Lemmati zation
a) from [Link] mpont Word Nelemmatizer
nltk. douwnloadlwondnet)
nt k. dounlbad (omw-1. 4)
Lemmater: WordNetLemmatizer)
Words "runnng", better", "studies, chinen ", mie eu)
Lemmas [lemmat2erlemmatize(w) for win ord
print(lemmas)
"Dutput:
C'runing', better', (study',eild','muse]
5, fron nltkistem lmprrt WordNetLemmatizr
Lemmatizer WondNetLemmakizer()
Sentenee"He was unning amd eating at samme time. He has
bad habit of siming ater playglong hours
n the Sun,9
punetuatins 9:*, i"
sentemeewrds = nLEk. woordtokenige (sentenee)
Por wornd o senteneends;

senteneewods. emore (wod

sentence _Words
prnt ("fo:20{1120'.format "word, Lemma))
for word nsentence words:
pont ("4o:2061120", frmat (wocd, enmetizerleometiet
wword|))
Jord Lemma
He He
Was wa

Tunning
md
and
eating eahing
at
anne
Eme time
 Word Lemma
He He
has ha
bad bad
habit habit
of
suimming
after
suinming
after
playng plaging
Long
hours Long
hour
in
+he
Sun
4) Remoing Stop wor ds
a) trm nltkicorput onport stop orrde
nltkdoumdoad (utopwords)
stp. wrrdsr set(stopuwords. words ('ang ush')
text is is a snple ampa shauing has how to
to remove stop
wonds S pythm,
words wordtokenieltt)
A\tered_wdswfr huonds if .lowenot
[Link]]
þrint ("After memaingstop wor ds: ", Ltered_wr)
"Bakpuk:
Aflor nemoring stop omds; [simple', 'exom pla', 'shong' omone,
stop','words', Pthen',)
wOrds "hs", is ,a "samp le", "sentenee7
[Link] o set (itopurordi. ords ('eng'sh)
Atered [w tor w h word if [Link]() not b stopord
print (AHerre d)

C'sample',sontemee"
" Pos tag2ng (lart-tf-speeeh tagging)!
nl dron lo od l'averaged-perceptsonbaggereng)
from nlttoke2e imprt word_token e
[Link] import pos-tag
textThe qiek brown umps oves the la2y dog
words 2 Wordtokenlzeltet)

prnt (uWords:", wor dA)

pintpos Tags!", pos-tag)
|buputr
Words:[The','quiek', bran',fx', 'yump<','ver,ha', y sog
Pos Tagsil (Te', DT), ('quiek', 91), ('nn,'NN), ('fx', NN,
('yumps', 'vez'), ('bver', 'TN),(he', T'),
'lazy','3a'), dog', 'NN') , (^', , )
(3)

ss reisub(reazA-Z 0-9\],,s)
words nlek. uoord to keniecs)
return nltkpostag (word)
print (postagging(gn ke to play football. ")
Qutput
I(,p2p') ke', vep), ('to', To'), (ply','ve')
(frotlbalu', NN)]
spaCywagar
import spacy C'emcoe_websm')
S spaey. Load
senterncez sp ("‘ uke to play trotball.‘ hatedi+ h my ehi dhaod
pìnt (sentenee : tet) thngh")
ike to pley Pootball.2 haded ith my casHhrod thrgh
[Link] otyoctitn,tevt tnport tfi4fVectorizer
docum ents -["1 love coks', "T lave doqs', "Doqs ore amatiny]
ve etorriger - 1fiAf Veatod2 erl)
fifmatx = veatori2er. flt-Arameform(Aacument)
yaiat (veetorineri [Link] nonaout ())
print ([Link]. toovroy))
'omnazing'are' eote' doqs' ve]
[Lo. o.19996054 O 6. 60534&51]
[o. 0. O 70710678
[o.6127G601 o. 622166 01 o70710681
o4736296
Word Enbeddings (wov42Vec):
frorn qensim nodele irport word2vee
sentenes -["s"Love, cats ), ["1","Love", 'dogs'J, ['Dogs',"are
mpdele larcd2Vec (uenteness, veetos. Si2e 10o, windwa 5, "Aazig"1
print(no del. v['loq1) wincount 4, wo kego4)
print (nodeleonost simi lar ('love))
L--7274c23e-03 213016 48e-03 -81394426 e-04 -93190&44 e- 03
-9.42%1426e-09 -1i4407480e -03 37040740e- o
-b.4730673e-03 -4o 9994422e-03 -2.286e442e-03
-7.2502 8776e-03 -9 6033118e-03 -27436293e-03 -63628409e-03
-G0322738e-03 -9.6709289 e-03 - 2.34413732-03 -1.7069972 e-o3
-8.95699%6e-03 -73519945e-04 G329 o63e-03 7169 o4297 e-03
-72001199e-03 -3666851 22-03 314829520-63 -9.9107225e-03
44764992¢-03 6.52444 <6 4e-03
57464 495e- 63 -37630 618 e -og
-4.5171444e-0s -8140 J607e-3 4i5963 74e-05
gP1330S6e-03 9.0613 080e-63 5.0610625e-03 9.2436338e-03
959218 36e-03 -73T6 4244e-o3 2"0o3 874e-03 -22653391e-03 -32429111e-03
-1836064e-04 -32161034Q- 03 5.258583-64 "4 e82230e- O3
-69734B8e-04 -1.62 4g4O7 e-03 21443 992 e-02 -&389 1007e-03
7833 8o27e-03 8.9361041e-o3 -9.58 4o 869e-03 24462664 e-63
9. 9o49743e-03 -7,C650 37e-o3 -8.966 1&7 e d1 -173454712-03
3959233e-03 -G8133572e-04 914 44086e -03 - i15822o9e-04
37430846e-03 24330 426e-03 Ti4271322e-b4 2327473 9e -03
-74609392-63 9.35%3735e-03 2.3543765e-03 614 e459 2e- 03
9.735941e -03 -17733 c36e-64:3061643e-03
-94363 1420- 03
3-4o6132ce-og 2667934 3e-04 5857243e~03
 -9.3363142e-03 3.4061326e-03 6675343e-04 3-6572443e-03
7.3257t34 e-03 -6.72546 69e-02 5.3849E07e-03 -9.52 2224&e03
-3.644 5&&6e-o4 -e"6 7367e-03 -5.09 8730e-03 9. 239 226 5e-6
-1.¢5$2619e -03 23144 264 e-03 9.071 2793 e-03 .9381328 e-03
-R.2083 30e-03 -3-0123137e-03 9.6t66o3 7e-03 51044310e-03

-188&0871e-03 - g6920 215e-03 249615164e-03 -6-675G97Le-03

C( cats, b.1613468 11 801195 14), (9, - 6.c4o8391C6 11969411),

(Dogs', -0-027750351985494s 2), (amazing', -0. 65 234673246741295),
(aoge', -0.0598 76 29L669244764) ,(are',-1416705834 865557)1
8 Count Vectorizer Bag of Worde):~
frem [Link] mpont CountVeatoier
text[t oas Hhe best f tmes, "itwas the worst of times,

veetrizer= lountveetorir()
vetorierft(tent)
pr'nt (sortedveetarizer, vo cabulary-)
bukpuki
[age',best', fooshnees', t', for 'of', 4ue',tmes', 'was ',
wisdom', (worst1
4eneode document
veetoraeatorizer. tsamsform tert)
print (eetiee,shape)
printlueetor to arrray())
(410)
[C o 10 1 1 1 1 1 o o]
[O o 1 1 1 14 0 1]

tert2 e [ 4hethe the imes7

veetore veeti2eritsansFrm (tet2)
pint([Link] )
qutputr
 Haslngvecorser
frmm sk [Link] chontet mprt Hasking Vectorher

veetori tera HastngVeatoriter (neatune G)

veetor weetanertam sfm teut)
pntlveetrishapc)
prntlvecton toaay ())
.ohah,6) [Lo.4o124829 -6. &1649458 Oi4y08248291)
0. -0.5 -0.8
[o-7671667 8 0. 0. 0.74710678]
0.707 0678 0.
0.7671068
bag ot Word Model hn lealr
Pen tenenrfloa kerai pnepro ce<hg-text nport tettowrd._cequenae
text heq ek bron fox yumped avar the loey dog.'
mesulte [Link] Ctext)
pht(niult)
. output r
rhei, qie ke, bron',fox', yummped','oor, Hae, 'la'.

fom tonsmflookeras, preproeestng. tert nprt hasng-triek

teato lthe qudck broum fox yumped oer the lary do'
wrds setltexttoord sequene (tert))
vocabsie lenlwrd)
prht (noeab s'e)
kshretna
wesulta hashhgtrek(e, ond( vocehse t3),
int(nesut) md5')
.Oukput
L6, 4, 1, 2, 7, S) 6, 2 6]
trorm denorftosikerasprepo [Link] onport Tokender
fooishuess
t wa the agof sdam" it sas the ageot
tokenr Tokeneer
4ole ener Atonteets (do ee)
tokeniee. word eounts,tokenerdocumentCuont,[Link]
tokenizY (orddo es

COndared bict(e(it',4),
(he', 4),
('best, 4),
(of' 4),
(itmes', 2),
('wort!, 11,
( afe 2),
(wsdm'4),
("fotshnss', 1)1),

as' 2

of 4
4mes' 5,

"worst: 8,
wisd!9,
bolswess; 10},
de taultdiet int
4the'! 4,
best! 1,

Ames'! 2,

wisdom4,
Age'! 2,
«foishmasy';1}))
 eneoded- do esz tokonier.terts_to-matrix ldocs, mode urt}
eneo ded-dses
"Output
arrayeCo., 1. , 1.,4. ,1. , 1.,0., 1., O., D. , O:1,
Co. 1., 1.1,, 1, 4., 0., Oy 1,, 0. O.1,

9>"N-grami
impertnLEk
nltL. doonoad ('pumkt')
nltk,dnload punkttab'
Prom nltk tnpert mgrams
trom nltkto kenie mpot ord_tokenie
semtenee "N-qrom cnhanee languag e pro cessina tacke,
tobens wordtkenize (sentem ee)
blaram Lrt(ngramsl tokens, 2))
tr'qrams Gst(ngrams( to kens, 3)
print("Bigroms;", igrams)
pratliiqrams; ", Arigoom)
Bigroons!CC'Ngrns,'enharee'), (enhonce', 'language),
('lomguge', tprocesshg,(processing','4aals'),(asks!,!
Trgrams:CCN-gans',' onhomei, 'ongage)gg
('enharee','anguge', proceshg)
('enguage', 'proceshg tase),
tasks? . ) ]