📘 ‫فلم حرش‬: Section - NLTK :

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python pip install prettytable

: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python import nltk import string from [Link] import
word_tokenize, sent_tokenize from prettytable import PrettyTable

: ‫ ةيلخلا‬

🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization). okenization (

‫ةقباس‬. ‫ ةيلخلا‬ ‫ ةيلخلا‬

: ‫ ةيلخلا‬1 (‫)دوك‬ ‫ ةيلخلا‬

Text

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python text ="""It's a dangerous business, Frodo, going out your
? door. You step onto the road, and if you don't keep your feet,
there's no knowing where you might be swept off to. The
scientists discover new species every year, Last year, they
discovered an ancient artifact. They are discovering new
techniques with their recent discovery. he likes running. he runs
everyday. she ran last night. IT will be better if they run
together!!. """
‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا ‪:‬‬ ‫ ةيلخلا‬

‫‪. Tokenizing‬ةقباس‬

‫‪okenization (is (the (process (of (splitting (text (into (individual (words (or‬‬
‫‪ (Tokenization‬ةدرفنم تاملك ىلإ ةلمجلا عيطقت 🔹‪sentences‬‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫)'‪python [Link]('punkt_tab‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫‪NLTK‬‬ ‫‪wordnet‬‬
‫‪.‬ةقباس‪punkt‬‬ ‫ ةيلخلا‬ ‫ ةيلخلا‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫))‪python tokens = set(word_tokenize(text‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫‪)Tokenization‬دوك( ‪ 1‬ةيلخلا‪.‬ةقباس‬ ‫دوك‬ ‫ ةيلخلا‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫‪python tokens‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫)‪python len(tokens‬‬
‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا ‪:‬‬ ‫‪0‬‬ ‫ ةيلخلا‬

‫‪. LowerCasing‬ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫] ‪python lc_words = [[Link]() for token in tokens‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫)(‪python lc_words.sort‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫‪python lc_words‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫)‪python len(lc_words‬‬
‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫)] ‪python lc_words = set([[Link]() for token in tokens‬‬
‫‪lc_words‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫)‪python len(lc_words‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫‪python import string [Link]‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا ‪:‬‬ ‫ ةيلخلا‬

‫?‪. Removing Puncatution‬ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫‪python clean_tokens = [token for token in lc_words if token not‬‬
‫)(‪in [Link]] clean_tokens.sort‬‬
: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ 0 ‫ ةيلخلا‬:

python clean_tokens

: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python len(clean_tokens)

: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python (from (nltk🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationtokenize (import (Regexp okenizer

Tokenizer that only keeps words

‫ ةيلخلا‬1 (‫)دوك‬ignores punctuation completely
tokenizer (= (Regexp okenizer).r'\w+'

tokens (= (tokenizer🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationtokenize).text (print)." ( okens (without (punctuation:"
tokens (```
‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫‪python tokens = set([[Link]() for token in tokens ]) tokens‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫)‪python len(tokens‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا ‪:‬‬ ‫ ةيلخلا‬

‫‪: Stop Words‬فلم حرش 📘 ‪. Filtering‬ةقباس‬

‫‪Stop (words (are (common (words (that (are (usually (removed (in (text (processing‬‬
‫‪ (Tokenization‬ةدرفنم تاملك ىلإ ةلمجلا عيطقت 🔹‪tasks‬‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫‪python from [Link] import stopwords‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫)'‪python [Link]('stopwords‬‬
‫ ‪:‬‬ ‫ ةيلخلا‬

‫‪NLTK‬‬ ‫‪wordnet‬‬
‫‪.‬ةقباس‪punkt‬‬ ‫ ةيلخلا‬ ‫ ةيلخلا‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫)'‪python stop_words = [Link]('english‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ‪)the, is, at :‬دوك( ‪ 1‬ةيلخلا ةيلخلا‬ ‫ ةيلخلا‬ ‫دوك‬

‫‪.‬ةقباس‬ ‫ ةيلخلا‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫‪python stop_words‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫‪0‬‬ ‫ ‪ :‬ةيلخلا‬

‫)‪python len(stop_words‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫‪python tokens = set([token for token in tokens if token not in‬‬
‫)‪stop_words]) len(tokens‬‬
: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python tokens

: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

: ‫ ةيلخلا‬1 (‫)دوك‬ ‫ ةيلخلا‬

‫ةقباس‬. 📘 ‫فلم حرش‬: Stemming

Stemming (reduces (words (to (their (base (or (root (form🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python (from (nltk🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationstem (import (PorterStemmer

stemmer (= (PorterStemmer).

stemmed_words (= ([stemmer🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationstem).token (for (token (in (tokens]

table (= (Pretty able). (table🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationfield_names (= (['word', ('stemmed (word']

for (word, (stw (in (zip).tokens, (stemmedwords : (table🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationaddrow).[word, (stw]

print).table (```

: ‫ ةيلخلا‬

🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization'run' ( ('running' ( ، PorterStemmer

‫ةقباس‬. ‫ ةيلخلا‬ ‫ ةيلخلا‬

: ‫ ةيلخلا‬1 (‫)دوك‬ ‫ ةيلخلا‬

‫ةقباس‬. Tagging Parts of 📘 ‫فلم حرش‬: Speech ‫ ةيلخلا‬1 (‫)دوك‬PO📘 ‫فلم حرش‬: S Tagging

POS (tagging (assigns (grammatical (tags (to (each (token, (such (as (noun, (verb,
etc🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization

: ‫ ةيلخلا‬1 (‫)دوك‬ ‫ ةيلخلا‬

Noun (-----> (cat, (book (
Verb (-----> (run, (eat (
Adjective (-----> (big, (smart (
Adverb (-----> (quickly, (always (
Pronoun (-----> (he, (she (
Preposition (-----> (in, (on, (at (
Conjunction (-----> (and, (but (

: ‫ ةيلخلا‬1 (‫)دوك‬ ‫ ةيلخلا‬

D (Determiner ().‫ فيرعت ةادأ‬ ‫"( يز فيرعت ةادأ‬the" (

JJ (Adjective (). ‫ةفص‬
NN (Noun ().‫ا‬
VBZ (Verb, (third (person (singular (). ‫لعف‬
IN (Preposition (). ‫رج فرح‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python from nltk import pos_tag

: ‫ ةيلخلا‬

,noun, verb ‫ ةيلخلا‬ ‫دوك‬ ‫دوك‬

adjective ‫ةقباس‬.

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python [Link]('averaged_perceptron_tagger_eng')
: ‫ ةيلخلا‬

NLTK wordnet
punkt‫ةقباس‬. ‫ ةيلخلا‬ ‫ ةيلخلا‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ 0 ‫ ةيلخلا‬:

python tagged_words = pos_tag(tokens)

: ‫ ةيلخلا‬

,noun, verb ‫ ةيلخلا‬ ‫دوك‬ ‫دوك‬

adjective ‫ةقباس‬.

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python (table (= (Pretty able).field_names=['token', ('pos (tag']

for (word, (pos (in (zip).tokens, (taggedwords : (table🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationaddrow).[word, (pos[1]]
print).table (```

: ‫ ةيلخلا‬

‫ةقباس‬. ‫ ةيلخلا‬ ‫ ةيلخلا‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python (from (nltk🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationcorpus (import (wordnet

‫ةقباس‬. Function to map NLTK PO📘 ‫فلم حرش‬: S to

WordNet PO📘 ‫فلم حرش‬: S
def (getwordnetpos).treebanktag : (if (treebanktag🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationstartswith).'J' : (return
wordnet🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬TokenizationADJ (elif (treebanktag🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationstartswith).'V' : (return (wordnet🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬TokenizationVERB (elif
treebanktag🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationstartswith).'N' : (return (wordnet🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬TokenizationNOUN (elif
treebank_tag🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationstartswith).'R' : (return (wordnet🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬TokenizationADV (else: (return
wordnet🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬TokenizationNOUN (# (Default (to (noun (```
: ‫ ةيلخلا‬

WordNet NLTK
‫ ةيلخلا‬lemmatizer‫ةقباس‬.

: ‫ ةيلخلا‬1 (‫)دوك‬ ‫ ةيلخلا‬

‫ةقباس‬. Lemmatizing

Lemmatization (reduces (words (to (their (base (or (root (form (using (WordNet
Lemmatizer🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python from [Link] import WordNetLemmatizer
[Link]('wordnet')

: ‫ ةيلخلا‬

🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization).lemmatization (‫ فيرعت ةادأ‬ ‫يز فيرعت ةادأةفص‬ ‫( يوغللا اهلصأ ىلإ ةملكلا عاجرإل‬WordNet

NLTK wordnet
punkt‫ةقباس‬. ‫ ةيلخلا‬ ‫ ةيلخلا‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python lemmatizer = WordNetLemmatizer()

: ‫ ةيلخلا‬

‫ ةيلخلا‬ ‫ ةيلخلا‬ ‫ يوغللا اهلصأ ىلإ ةملكلا عاجرإل‬WordNet ‫ ةيلخلا‬ ‫ ةيلخلاةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬lemmatization ‫ةقباس‬.

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python lemma = [[Link](token) for token in tokens]
: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python (table (= (Pretty able).field_names=['token', ('stemmed (word',
'lemma (word']

for (token, (stm, (lm (in (zip).tokens, (stemmedwords, (lemma :

table🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationaddrow).[token, (stm, (lm]

print).table (```

: ‫ ةيلخلا‬

‫ةقباس‬. ‫ ةيلخلا‬ ‫ ةيلخلا‬

: ‫ ةيلخلا‬1 (‫)دوك‬ ‫ ةيلخلا‬

‫ةقباس‬. Lemmatization using PO📘 ‫فلم حرش‬: S

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python tagged_words

: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ 0 ‫ ةيلخلا‬:

```python

Prepare table
table (= (Pretty able).field_names=['word', ('POS (tag', ('stemmed (word', ('lemma
word (without (tagging', ('lemma (word (using (tagging']
for (word, (pos (in (taggedwords: (wnpos (= (getwordnetpos).pos (stemmed (=
stemmer🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationstem).word (lemmatizedwithout (= (lemmatizer🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationlemmatize).word
lemmatized (= (lemmatizer🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationlemmatize).word, (pos=wnpos (table🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationaddrow).[word,
pos, (stemmed, (lemmatizedwithout (,lemmatized] (print).table (```

: ‫ ةيلخلا‬

(NL K (🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization ‫لعف‬

WordNet ( (lemmatizer🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization

‫ ةيلخلا‬ ‫ةقباس‬ ‫ ةيلخلا‬

Porter📘 ‫فلم حرش‬: Stemmer، 'running' ‫صت‬ 'run'‫ةقباس‬.

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python

```

: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

📘 ‫فلم حرش‬: Building Index :

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:
```python (from (whoosh (import (fields, (index (from (PyPDF2 (import (PdfReader
import (pandas (as (pd (import (os (from (whoosh🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationindex (import (createin (from (glob
import (glob (from (whoosh🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationindex (import (opendir (from (whoosh🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationanalysis (import
Regex okenizer, (LowercaseFilter, (StopFilter, (StemFilter, (StemmingAnalyzer

custom_analyzer (= (Regex okenizer). (| (LowercaseFilter). (| (StopFilter). (|

StemFilter).

Define the schema for Whoosh

indexing
schema (= (fields🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬TokenizationSchema). (title=fields🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization EX ).stored= rue,
analyzer=StemmingAnalyzer). , (author=fields🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization EX ).stored= rue,
analyzer=StemmingAnalyzer). ,
content=fields🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenization EX ).analyzer=custom_analyzer, (stored=False (,
phrase= rue , (path=fields🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬TokenizationS ORED (# (Store (file (path (for (retrieval ( (```
: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python

Create index directory

indexdir (= ("pdfindex" (if (not (os🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationpath🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationexists).indexdir : (os🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationmkdir).indexdir

Initialize index with the schema

ix (= (createin).indexdir, (schema

def (extracttextfrompdf).pdfpath : (with (open).pdf_path, ("rb" (as (file: (reader (=

PdfReader).file

text = ""
for page in [Link]:
text += page.extract_text()
metadata = [Link]
title = [Link]("/Title", [Link](pdf_path).replace(
author = [Link]("/Author", "Unknown")
return {
"title": title,
"content": text,
"author": author,
}

```

: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

python pdf_files = glob("pdfs/*.pdf") pdf_files
: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python (with (ix🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationwriter). (as (writer: (for (pdffile (in (pdffiles: (pdfdata (=
extracttextfrompdf).pdf_file

# Generate a title (use filename if no title exists)

title = pdf_data["title"]
content = pdf_data["content"]
author = pdf_data["author"]
# Add document to index

writer.add_document(
title=title,
content=content,
author = author,
path=pdf_file
)

print)." (All (PDFs (have (been (indexed (successfully!" (```

: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python

Extract stored data

indexdata (= ([] (with (ix🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationsearcher). (as (searcher: (for (doc (in
searcher🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationdocuments). : (indexdata🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationappend).doc

Convert to DataFrame
df (= (pd🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬TokenizationDataFrame).index_data (df (```
: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python

index ‫ ةيلخلا‬ ‫ ةيلخلا‬ ‫ ةيلخلا‬

with (ix🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationreader). (as (reader: (# ( ‫لعف‬ ("content" (terms (=
reader🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationfield_terms)."author" (for (term (in (sorted).terms : (print).term (```

: ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫ةقباس‬. ‫ةقباس‬

‫ ةيلخلا‬1 (‫)دوك‬ ‫دوك‬ ‫ ةيلخلا‬:

```python

index ‫ ةيلخلا‬ ‫ ةيلخلا‬ ‫ ةيلخلا‬

termdocmapping (= ([] (with (ix🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationreader). (as (reader: (# ( ‫لعف‬
content" (terms (= (reader🔹 ‫( ةدرفنم تاملك ىلإ ةلمجلا عيطقت‬Tokenizationfield_terms)."author" " (

for term in sorted(terms):

matcher = [Link]("author", term)

term ‫ لكل‬posting list #

while matcher.is_active():
doc_id = [Link]()
freq = matcher.value_as("frequency") # ‫تسملا يف ةملكلا روهظ تارم ددع‬ ‫ لكل‬ ‫ لكل‬
term_doc_mapping.append({
"term": term,
"doc_id": doc_id,
"frequency": freq
})
[Link]()

```
‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫‪python df = [Link](term_doc_mapping) df‬‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬

‫)دوك( ‪ 1‬ةيلخلا‬ ‫دوك‬ ‫ ‪ :‬ةيلخلا‬

‫‪```python‬‬

‫```‬

‫ ‪:‬‬ ‫ ةيلخلا‬

‫ ةيلخلا ةيلخلا‬
‫‪.‬ةقباس‬ ‫ةقباس‬