Concepts of Natural language Processing

Module 2

NLP Pipeline
NLP Pipeline is a set of steps followed to build an end to end NLP software.

The first step in the process of developing any NLP system is to collect data relevant to the given
task. Even if we were building a rule-based system, we still need some data to design and test our
rules. The data we get is seldom clean, and this is where text cleaning comes into play. After
cleaning, text data often has a lot of variations and needs to be converted into a canonical form.
This is done in the pre-processing step. This is followed by feature engineering, where we carve
out indicators that are most suitable for the task at hand. These indicators are converted into a
format that is understandable by modeling algorithms. Then comes the modeling and evaluation
phase, where we build one or more models and compare and contrast them using a relevant
evaluation metric(s). Once the best model among the ones evaluated is chosen, we move toward
deploying this model in production. Finally, we regularly monitor the performance of the model
and, if need be, update it to keep up its performance.
Exact step-by-step procedures may depend on the specific task at hand.
For example, a text-classification system may require a different feature extraction step
compared to a text-summarization system.

Data Acquisition

● Ideal Situation - we have the required datasets with millions of data points. In such
cases, we don't have to worry about data acquisition.

But that's not the case in the majority of cases.

● When data is less - we can start by looking at patterns in the data. We can use regular
expressions and other heuristics to match these patterns

Eg: Separating support and sales queries in a company

We can start by looking at patterns in the data that indicate if the incoming message is a
sales or support query. We can then use regular expressions and other heuristics to match
these patterns to separate sales queries from support queries. We evaluate this solution by
collecting a set of queries from both categories and calculating what percentage of the
messages were correctly identified by our system.

We need labeled data, a collection of queries where each one is labeled with sales or support.
How can we get such data?

➢ Use a public dataset

We could see if there are any public datasets available that we can leverage.

If you find a suitable dataset that's similar to the task at hand

➢ Scrape data

We could find a source of relevant data on the internet—for example, a consumer or discussion
forum where people have posted queries (sales or support). Scrape the data from there and get it
labeled by human annotators. For many industrial settings, gathering data from external sources
does not suffice because the data doesn't contain nuances like product names or product specific
user behavior and thus might be very different from the data seen in production environments.
This is when we have to start looking for data inside the organization.
 ➢ Product intervention
In most industrial settings, AI models seldom exist by themselves. They are developed mostly to
serve users via a feature or product. In all such cases, the AI team should work with the product
team to collect more and richer data by developing better instrumentation in the product. In the
tech world, this is called product intervention. Product intervention is often the best way to
collect data for building intelligent applications in industrial settings. Tech giants like Google,
Facebook, Microsoft, Netflix, etc., have known this for a long time and have tried to collect as
much data as possible from as many users as possible.

➢ Data augmentation

While instrumenting products is a great way to collect data, it takes time. Even if you instrument
the product today, it can take anywhere between three to six months to collect a decent-sized,
comprehensive dataset. So, can we do something in the meantime? NLP has a bunch of
techniques through which we can take a small dataset and use some tricks to create more data.
These tricks are also called data augmentation, and they try to exploit language properties to
create text that is syntactically similar to source text data.

★ Synonym replacement - Randomly choose “k” words in a sentence that are not
stop words. Replace these words with their synonyms. For synonyms, we can use
Synsets in Wordnet
★ Back translation : Say we have a sentence, S1, in English. We use a
machine-translation library like Google Translate to translate it into some other
language—say, German. Let the corresponding sentence in German be S2. Now,
well use the machine-translation library again to translate back to English. Let the
output sentence be S3. Well find that S1 and S3 are very similar in meaning but
are slight variations of each other. Now we can add S3 to our dataset. This trick
works beautifully for text classification.
★ TF-IDF–based word replacement : Back translation can lose certain words that are crucial
to the sentence. [Refer Below for Detailed Explanation]
★ Bigram flipping: Divide the sentence into bigrams. Take one bigram at random and flip it.
For example: “I am going to the supermarket.” Here, we take the bigram “going to” and
replace it with the flipped one: “to going.”
★ Replacing entities: Replace entities like person name, location, organization, etc., with
other entities in the same category. That is, replace person name with another person
name, city with another city, etc. For example, in “I live in California,” replace “Califor
nia” with “London.”
★ Adding noise to data: In many NLP applications, the incoming data contains spelling
mistakes. This is primarily due to characteristics of the platform where the data is being
generated (for example, Twitter). In such cases, we can add a bit of noise to data to train
robust models. For example, randomly choose a word in a sentence and replace it with
another word thats closer in spelling to the first word.

Another source of noise is the “fat finger” problem on mobile keyboards. Simulate a
QWERTY keyboard error by replacing a few characters with their neighboring characters
on the QWERTY keyboard.

★ Advanced techniques:
○ Snorkel - This is a system for building training data automatically, without
manual labeling. Using Snorkel, a large training dataset can be
“created”—without manual labeling—using heuristics and creating synthetic data
by transforming existing data and creating new data samples. This approach was
shown to work well at Google in the recent past [9].
○ Easy Data Augmentation (EDA) and NLPAug - These two libraries are used to
create synthetic samples for NLP. They provide implementation of various data
 augmentation techniques, including some techniques that we discussed
previously.
○ Active learning - This is a specialized paradigm of ML where the learning
algorithm can interactively query a data point and get its label. It is used in
scenarios where there is an abundance of unlabeled data but manually labeling is
expensive. In such cases, the question becomes: for which data points should we
ask for labels to maximize learning while keeping the labeling cost low? One key
requirement is a clean dataset to start with, even if it's not very big - data
augmentation techniques can work really well. Further, in day-to-day ML
practice, datasets come from heterogeneous sources. A combination of public
datasets, labeled datasets, and augmented datasets are used for building
early-stage production models, as we often may not have large datasets for our
custom scenarios to start with.

Text Extraction and Cleanup

Text extraction and cleanup refers to the process of extracting raw text from the input data by
removing all the other non-textual information, such as markup, metadata, etc., and converting
the text to the required encoding format. Text extraction is a standard data-wrangling step, and
we don't usually employ any NLP-specific techniques during this process.

Some cleanup requirements are:

● HTML Parsing and Cleanup
● Unicode Normalization - To parse such non-textual symbols and special characters, we
use Unicode normalization. This means that the text we see should be converted into
some form of binary representation to store in a computer. This process is known as text

encoding.
● Spelling Correction
● System-Specific Error Correction - Different PDF documents are encoded differently,
and some times, we may not be able to extract the full text, or the structure of the text
may get messed up - PyPDF , PDFMiner,etc - Text extraction from scanned documents is
typically done through optical character recognition (OCR), using libraries such as
Tesseract.
Pre-processing

Sentence segmentation
Breaking up text into sentences at the appearance of full stops and question marks but there may
be abbreviations, forms of addresses (Dr., Mr., etc.), or ellipses (...)

So what to do? We can use the library : Natural Language Tool Kit (NLTK)

Word tokenization
Tokenize a sentence into word - split text into words based on the presence of punctuation marks.
The NLTK library allows us to do that.

for sentence in my_sentences:

print(sentence)
print(word_tokenize(sentence))
But there is an issue with this:

Stop word Removal

Some of the frequently used words in English, such as a, an, the, of, in, etc., are not particu larly
useful for this task, as they don't carry any content on their own to separate between the four
categories. Such words are called stop words and are typically (though not always) removed
from further analysis in such problem scenarios.

Lowercasing : Similarly, in some cases, upper or lowercase may not make a difference for the
prob lem. So, all text is lowercased (or uppercased, although lowercasing is more common).
Removing punctuation and/or numbers is also a common step for many NLP problems, such as
text classification , information retrieval, and social media analytics.

Stemming and lemmatization

Stemming refers to the process of removing suffixes and reducing a word to some base form
such that all different variants of that word can be represented by the same form (e.g., “car” and
“cars” are both reduced to “car”). This is accomplished by applying a fixed set of rules (e.g., if
the word ends in “-es,” remove “-es”). Although such rules may not always end up in a
linguistically correct base form, stemming is commonly used in search engines to match user
queries to relevant documents and in text classification to reduce the fea ture space to train
machine learning models.

Stemming Algorithm => Porter Stemmer (using NLTK)

Lemmatization is the process of mapping all the different forms of a word to its base word, or
lemma. While this seems close to the definition of stemming, they are, in fact, different. For
example, the adjective “better,” when stemmed, remains the same. However, upon
lemmatization, this should become “good,”

Text Normalization
Consider a scenario where were working with a collection of social media posts to detect
news events. Social media text is very different from the language wed see in, say,
newspapers. A word can be spelled in different ways, including in shortened forms, a
phone number can be written in different formats (e.g., with and without hyphens), names
are sometimes in lowercase, and so on. When were working on developing NLP tools to
work with such data, its useful to reach a canonical representation of text that captures all
these variations into one representation. This is known as text normalization. Some
common steps for text normalization are to convert all text to lowercase or uppercase,
convert digits to text (e.g., 9 to nine), expand abbreviations, and so on.

Language detection
A lot of web content is in non-English languages. For example, say we were asked to collect all
reviews about our product on the web. As we navigate different e-commerce websites and start
crawling pages related to our product, we notice several non English reviews showing up. Since
a majority of the pipeline is built with language specific tools, what will happen to our NLP
pipeline, which is expecting English text? In such cases, language detection is performed as the
first step in an NLP pipeline. We can use libraries like Polyglot [36] for language detection. Once
this step is done, the next steps could follow a language-specific pipeline.

Code mixing and transliteration

Scenario where a single piece of content is in more than one language. Many people across the
world speak more than one language in their day-to-day lives. Thus, its not uncommon to see
them using multiple languages in their social media posts, and a single post may contain many
languages
Code mixing refers to this phenomenon of switching between languages. When people use
multiple languages in their write-ups, they often type words from these languages in Roman
script, with English spelling. So, the words of another language are written along with English
text. This is known as transliteration.

Advanced Processing

Imagine were asked to develop a system to identify person and organization names in our
company’s collection of one million documents. The common pre-processing steps we discussed
earlier may not be relevant in this context. Identifying names requires us to be able to do POS
tagging, as identifying proper nouns can be useful in identifying person and organization names.
To identify and link multiple mentions of an entity (e.g., Satya Nadella, Mr. Nadella, he, etc.).
We accomplish this with the pre-processing step known as coreference resolution.

A way of identifying person and organization names - Named Entity Recognition (NER)

Feature Engineering

The goal of feature engineering is to capture the characteristics of the text into a numeric vector
that can be understood by the ML algorithms. Also referred to as Text Representation.

Two approaches : Classical NLP and DL pipeline

Modeling

- building a solution
How?
● Simple Heuristics - filters, regular expressions,

Evaluation

Depending on the NLP task or problem, the evaluation metrics can vary.

Evaluations are of two types: intrinsic and extrinsic. Intrinsic focuses on inter mediary
objectives, while extrinsic focuses on evaluating performance on the final objective. For
example, consider a spam-classification system. The ML metric will be precision and
recall, while the business metric will be “the amount of time users spent on a spam
email.” Intrinsic evaluation will focus on measuring the system performance using
precision and recall. Extrinsic evaluation will focus on measuring the time a user wasted
because a spam email went to their inbox or a genuine email went to their spam folder.

Post-Modeling Phases : Once our model has been tried and tested, we move on to the
post-modeling phase: deploying, monitoring, and updating the model.

Deployment

An NLP module is typically deployed as a web service. Lets say we designed a

web service that takes a text as input and returns the emails category (spam or
non-spam) as output. Now, each time someone gets a new email, it goes to the
microservice, which classifies the email text. This, in turn, can be used to make a
decision about what to do with the email (either show it or send it to the spam
folder). In certain circumstances, like batch processing, the NLP module is
deployed in the larger task queue.
Monitoring and Model Updating

TEXT REPRESENTATIONS IN DETAIL

 ● Basic vectorization approaches
● Distributed representations
● Universal language representation
● Handcrafted features

Vector space model (VSM)

It's a simple algebraic model used extensively for representing any text blob. VSM
is fundamental to many information-retrieval operations, from scoring documents
on a query to document classification and document clustering [2]. Its a
mathematical model that represents text units as vectors. In the simplest form,
these are vectors of identifiers, such as index numbers in a corpus vocabulary. In
this setting, the most common way to calculate similarity between two text blobs is
using cosine similarity: the cosine of the angle between their corresponding
vectors. The cosine of 0° is 1 and the cosine of 180° is –1, with the cosine
monotonically decreasing from 0° to 180°.

● Basic Vectorization Approaches

Map each word in the vocabulary (V) of the text corpus to a unique ID (integer
value), then represent each sentence or document in the corpus as a V-dimensional
vector.

a. One Hot Encoding

Each word w in the corpus vocabulary is given a unique integer ID wid that is
between 1 and |V|, where V is the set of the corpus vocabulary. Each word is then
represented by a V-dimensional binary vector of 0s and 1s. This is done via a |V|
dimension vector filled with all 0s barring the index, where index = wid . At this
index, we simply put a 1. The representation for individual words is then combined
to form a sentence representation.

Issues with One-Hot Encoding

➔ The size of a one-hot vector is directly proportional to size of the

vocabulary, and most real-world corpora have large vocabularies. This
results in a sparse representation where most of the entries in the vectors are
zeroes, making it computationally inefficient to store, compute with, and
learn from (sparsity leads to overfitting).
➔ This representation does not give a fixed-length representation for text, i.e.,
if a text has 10 words, you get a longer representation for it as compared to a
text with 5 words. For most learning algorithms, we need the feature vectors
to be of the same length.
➔ It treats words as atomic units and has no notion of (dis)similarity between
words. For example, consider three words: run, ran, and apple. Run and ran
have similar meanings as opposed to run and apple. But if we take their
respective vectors and compute Euclidean distance between them, they're all
equally apart ( √2). Thus, semantically, they're very poor at capturing the
meaning of the word in relation to other words.
➔ Say we train a model using our toy corpus. At runtime, we get a sentence:
“man eats fruits.” The training data didnt include “fruit” and theres no way
to represent it in our model. This is known as the out of vocabulary (OOV)
problem. A one-hot encoding scheme cannot handle this. The only way is to
retrain the model: start by expanding the vocabulary, give an ID to the new
word, etc.

● Bag of Words
Represent the text under consideration as a bag (collection) of words while
ignoring the order and context.

BoW maps words to unique integer IDs between 1 and |V|. Each document in the
corpus is then converted into a vector of |V| dimensions where in the ith
component of the vector, i = wid , is simply the number of times the word w occurs
in the document, i.e., we simply score each word in V by their occurrence count in
the document. With this representation, documents having the same words will
have their vector representations closer to each other in Euclidean space as
compared to documents with completely different words. The distance between D1
and D2 is 0 as compared to the distance between D1 and D4 , which is 2. Thus, the
vector space resulting from the BoW scheme captures the semantic similarity of
documents. So if two documents have similar vocabulary, theyll be closer to each
other in the vector space and vice versa. We have a fixed-length encoding for any
sentence of arbitrary length.

Issues with BoW

➔ The size of the vector increases with the size of the vocabulary. Thus,
sparsity continues to be a problem. One way to control it is by limiting
the vocabulary to n number of the most frequent words.
➔ It does not capture the similarity between different words that mean
the same thing. Say we have three documents: “I run”, “I ran”, and “I
ate”. BoW vectors of all three documents will be equally apart
➔ This representation does not have any way to handle out of vocabulary
words (i.e., new words that were not seen in the corpus that was used
to build the vec torizer).
➔ As the name indicates, it is a “bag” of words—word order information
is lost in this representation. Both D1 and D2 will have the same
representation in this scheme