Natural Language
Processing
Incident Matrix- Inverted Indexer Python Implementation
Incident Matrix
2
Result
3
Display Doc
4
Display Doc
5
What is the use of NLP model?
NLP models are used to understand, process, and generate human
language text. They find applications in sentiment analysis, chatbots,
language translation, speech recognition, and information retrieval,
enabling automation and insights from vast amounts of textual data.
What are the basic steps in
modeling NLP?
Modeling NLP involves data preprocessing, feature extraction, selecting an
NLP model (e.g., LSTM, Transformer), training and evaluation, and often
fine-tuning hyperparameters for optimal performance.
Installing Required Libraries
we will use the Natural Language Toolkit (NLTK) library, which is one of the
most popular Python libraries for NLP. To install it, open a terminal and run
the following command:
pip install nltk
SMS Spam Collection Dataset
The SMS Spam Collection is a set of SMS tagged messages that have been collected for SMS
Spam research. It contains one set of SMS messages in English of 5,574 messages, tagged
according being ham (legitimate) or spam.
The files contain one message per line. Each line is composed by two columns: v1 contains the label
(ham or spam) and v2 contains the raw text.
This corpus has been collected from free or free for research sources at the Internet
9
Loading and Preprocessing the Dataset
Before we create our NLP model, we need to load and preprocess a
dataset. We will use the "SMS Spam Collection" dataset available
on Kaggle. Download the dataset and load it into a Pandas DataFrame:
import pandas as pd
data = pd.read_csv('[Link]', encoding='latin-1')
[Link]()
10
Text Preprocessing
Text preprocessing is an essential step in NLP. It involves cleaning and
transforming raw text data into a format that can be easily understood by
the NLP model. Some common text preprocessing techniques include
tokenization, stopword removal, and stemming. We will use the NLTK
library to perform these tasks:
11
Code
import nltk from [Link]
import stopwords from [Link]
import word_tokenize from [Link]
import PorterStemmer
# Download NLTK Stopwords
[Link]('stopwords')
[Link]('punkt')
def preprocess_text(text):
# Lowercase the text
text = [Link]()
# Tokenize the text
words = word_tokenize(text)
# Remove stopwords
words = [word for word in words if word not in [Link]('english')]
12
Code
# Perform stemming
stemmer = PorterStemmer()
words = [[Link](word) for word in words]
# Join words back into a single string
preprocessed_text = ' '.join(words)
return preprocessed_text
# Apply the preprocessing function to the dataset
data['text'] = data['v2'].apply(preprocess_text)
[Link]()
13
Creating the NLP Model
• Now that our dataset is preprocessed, we can create our NLP
model. We will use the Multinomial Naive Bayes classifier from
the Scikit-learn library:
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from [Link] import accuracy_score, confusion_matrix
20XX presentation title 14
Code
# Vectorize the preprocessed text
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(data['text'])
# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, data['v1'], test_size=0.2, random_state=42)
# Train the Multinomial Naive Bayes classifier
classifier = MultinomialNB()
[Link](X_train, y_train)
# Make predictions on the testing set
y_pred = [Link](X_test)
# Evaluate the model
print("Accuracy:", accuracy_score(y_test, y_pred))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
15
16
17
18
THANK YOU!