Unit-III

Clustering and Dimensionality Reduction

Clustering and dimensionality reduction are both fundamental techniques in

unsupervised learning, but they serve different purposes. Clustering aims to
group similar data points into clusters, while dimensionality reduction focuses
on reducing the number of features or dimensions in a dataset while
preserving essential information. Though they can be used together, they
optimize different criteria: clustering focuses on grouping similar data points,
while dimensionality reduction aims to preserve variance

Introduction to Dimensionality Reduction

When working with machine learning models, datasets with too many
features can cause issues like slow computation and overfitting.
Dimensionality reduction helps to reduce the number of features while
retaining key information. Techniques like principal component analysis
(PCA), singular value decomposition (SVD) and linear discriminant analysis
(LDA) convert data into a lower-dimensional space while preserving
important details.
Before Reduction You can see that Data exist in 3D (X,Y,Z). It has high
redundancy and Z contributes little meaningful information

On the right after reducing the dimensionality the data is represented

in lower-dimensional spaces. The top plot (X-Y) maintains the meaningful
structure while the bottom plot (Z-Y) shows that the Z-dimension contributed
little useful information.

This process makes data analysis more efficient, improving computation

speed and visualization while minimizing redundancy

Dimensionality Reduction Techniques

Dimensionality reduction techniques can be broadly divided into two
categories:

1. Feature Selection

Feature selection chooses the most relevant features from the dataset
without altering them. It helps remove redundant or irrelevant features,
improving model efficiency. Some common methods are:
Filter methods rank the features based on their relevance to the target
variable.

Wrapper methods use the model performance as the criteria for selecting
features.

Embedded methods combine feature selection with the model training

process.

Please refer to Feature Selection Techniques for better in depth understanding

about the techniques.

2. Feature Extraction

Feature extraction involves creating new features by combining or

transforming the original features. These new features retain most of the
dataset’s important information in fewer dimensions. Common feature
extraction methods are:

Principal Component Analysis (PCA): Converts correlated variables into

uncorrelated 'principal components, reducing dimensionality while
maintaining as much variance as possible enabling more efficient analysis.

Missing Value Ratio: Variables with missing data beyond a set threshold are
removed, improving dataset reliability.

Backward Feature Elimination : Starts with all features and removes the least
significant ones in each iteration. The process continues until only the most
impactful features remain, optimizing model performance.

Forward Feature Selection: Forward Feature Selection Begins with one

feature, adds others incrementally and keeps those improving model
performance.

Random Forest: Random forest Uses decision trees to evaluate feature

importance, automatically selecting the most relevant features without the
need for manual coding, enhancing model accuracy.

Factor Analysis: Groups variables by correlation and keeps the most relevant
ones for further analysis.
Independent Component Analysis (ICA) : Identifies statistically independent
components, ideal for applications like ‘blind source separation’ where
traditional correlation-based methods fall short.

K means Clustering
What is K-Means Clustering?
K-means clustering is a popular unsupervised machine learning
algorithm used for partitioning a dataset into a pre-defined number of
clusters. The goal is to group similar data points together and discover
underlying patterns or structures within the data.

Recall the first property of clusters – it states that the points within a
cluster should be similar to each other. So, our aim here is to minimize the
distance between the points within a cluster.

There is an algorithm that tries to minimize the distance of the points in a

cluster with their centroid – the k-means clustering technique.

K-means is a centroid-based algorithm or a distance-based algorithm,

where we calculate the distances to assign a point to a cluster. In K-Means,
each cluster is associated with a centroid.

The main objective of the K-Means algorithm is to minimize the sum of

distances between the points and their respective cluster centroid.

Optimization plays a crucial role in the k-means clustering algorithm. The

goal of the optimization process is to find the best set of centroids that
minimizes the sum of squared distances between each data point and its
closest centroid.

How K-Means Clustering Works?

Here’s how it works:

Initialization: Start by randomly selecting K points from the dataset. These

points will act as the initial cluster centroids.
Assignment: For each data point in the dataset, calculate the distance
between that point and each of the K centroids. Assign the data point to
the cluster whose centroid is closest to it. This step effectively forms K
clusters.

Update centroids: Once all data points have been assigned to clusters,
recalculate the centroids of the clusters by taking the mean of all data
points assigned to each cluster.

Repeat: Repeat steps 2 and 3 until convergence. Convergence occurs when

the centroids no longer change significantly or when a specified number of
iterations is reached.

Final Result: Once convergence is achieved, the algorithm outputs the final
cluster centroids and the assignment of each data point to a cluster.

Hierarchical Clustering in Machine Learning

Hierarchical clustering is used to group similar data points together based on

their similarity creating a hierarchy or tree-like structure. The key idea is to
begin with each data point as its own separate cluster and then progressively
merge or split them based on their similarity.

Dendogram
A dendrogram is like a family tree for clusters. It shows how individual data
points or groups of data merge together. The bottom shows each data point
as its own group, and as you move up, similar groups are combined. The
lower the merge point, the more similar the groups are. It helps you see how
things are grouped step by step. The working of the dendrogram can be
explained using the below diagram:
 Dendogr
am

In the above image on the left side there are five points labeled P, Q, R, S and
T. These represent individual data points that are being clustered. On the
right side there’s a dendrogram which show how these points are grouped
together step by step.

At the bottom of the dendrogram the points P, Q, R, S and T are all separate.

As you move up, the closest points are merged into a single group.

The lines connecting the points show how they are progressively merged
based on similarity.

The height at which they are connected shows how similar the points are to
each other; the shorter the line the more similar they are

Types of Hierarchical Clustering

Now we understand the basics of hierarchical clustering. There are two main
types of hierarchical clustering.

Agglomerative Clustering
Divisive clustering
Hierarchical Agglomerative Clustering
It is also known as the bottom-up approach or hierarchical agglomerative
clustering (HAC). Unlike flat clustering hierarchical clustering provides a
structured way to group data. This clustering algorithm does not require us to
prespecify the number of clusters. Bottom-up algorithms treat each data as a
singleton cluster at the outset and then successively agglomerate pairs of
clusters until all clusters have been merged into a single cluster that contains
all data.

Hierarch
ical Agglomerative Clustering

Hierarchical Divisive clustering

It is also known as a top-down approach. This algorithm also does not require
to prespecify the number of clusters. Top-down clustering requires a method
for splitting a cluster that contains the whole data and proceeds by splitting
clusters recursively until individual data have been split into singleton
clusters.
Principal Component Analysis(PCA)

What is PCA?
PCA (Principal Component Analysis) is a dimensionality reduction technique
used in data analysis and machine learning. It helps you to reduce the
number of features in a dataset while keeping the most important
information. It changes your original features into new features these new
features don’t overlap with each other and the first few keep most of the
important differences found in the original data.
PCA is commonly used for data preprocessing for use with machine learning
algorithms. It helps to remove redundancy, improve computational efficiency
and make data easier to visualize and analyze especially when dealing with
high-dimensional [Link] Component Analysis (PCA): A powerful tool for
data simplif
Principal Component Analysis (PCA) is an unsupervised learning technique
primarily used for dimensionality reduction in large datasets. It simplifies
complex data by transforming a large set of potentially correlated variables
into a smaller set of uncorrelated variables called principal components. The
 goal is to retain as much of the original information as possible in this reduced
representation.

How does PCA work?

PCA operates by identifying the directions (principal components) along which

the data varies the most. It essentially rotates the data to align with these
directions of maximal variance, which are essentially eigenvectors of the data's
covariance matrix.

Here's a breakdown of the process:

1. Standardize the Data: If the features have different scales (e.g., salary vs. age),
PCA first standardizes the data by making each feature have a mean of 0 and a
standard deviation of 1. This ensures that no single feature dominates the
analysis due to its larger scale.

2. Calculate the Covariance Matrix: This matrix reveals the relationships between
variables, showing how they vary together. A positive covariance means they
tend to increase or decrease together, while a negative covariance indicates
they move in opposite directions.

3. Find the Principal Components: PCA identifies new axes, the principal
components, where the data spreads out the most. The first principal
component (PC1) captures the maximum variance, the second (PC2) the next
highest variance while being perpendicular to PC1, and so on. These principal
components are represented by eigenvectors of the covariance matrix.

4. Select the Principal Components: Eigenvalues, which correspond to each

eigenvector (principal component), quantify the amount of variance captured
by that component. Higher eigenvalues indicate more important components.
A scree plot, which shows the eigenvalues in decreasing order, can be used to
select the number of principal components to retain.

5. Transform the Data: The original data is transformed into a new coordinate
system defined by the selected principal components, resulting in a lower-
dimensional representation of the data.

Feature selection and feature extraction are two key techniques used in
machine learning to improve model performance by handling irrelevant or
redundant features. While both works on data preprocessing, feature
selection uses a subset of existing features whereas feature extraction
transforms data into a new feature. In this article we will learn more about
their key differences.
 Feature selection: involves selecting a subset of the most relevant features
that are actually contributing in prediction while discarding the rest
features. This helps improve reducing overfitting and increased accuracy.
Common techniques include filter, wrapper and embedded methods.
 Feature extraction: transforms existing features into a new set of features
that captures better underlying patterns in data. It is useful when raw data
is in high dimension or complex. Techniques like PCA, LDA and
Autoencoders are used for this purpose.
Key Benefits of both
1. Improved Model Performance: Reduces noise and retains only necessary
features leading to higher accuracy and lower error rates.
2. Reduced Overfitting: Prevents model to be overly complex and ensures
that it captures meaningful patterns rather than memorizing dataset.
3. Faster Model Training and Inference: Helps in dimensionality reduction of
dataset hence make faster training and reduced computational cost.
4. Improved Interpretability: Simplifies the model by focusing on significant
features making it easier to understand.
While both has similar benefits their working are totally different.
Text and sentiment analysis
What is Sentiment Analysis?
Sentiment analysis is the process of classifying whether a block of text is
positive, negative, or neutral. The goal that Sentiment mining tries to gain is
to be analysed people’s opinions in a way that can help businesses expand. It
focuses not only on polarity (positive, negative & neutral) but also on
emotions (happy, sad, angry, etc.). It uses various Natural Language
Processing algorithms such as Rule-based, Automatic, and Hybrid.
What is NLP?
Natural Language Processing (NLP) is a field that combines computer science,
artificial intelligence and language studies. It helps computers understand,
process and create human language in a way that makes sense and is useful.
With the growing amount of text data from social media, websites and other
sources, NLP is becoming a key tool to gain insights and automate tasks like
analyzing text or translating languages.

NLP is used by many applications that use language, such as text translation,
voice recognition, text summarization and chatbots. You may have used some
of these applications yourself, such as voice-operated GPS systems, digital
assistants, speech-to-text software and customer service bots. NLP also helps
businesses improve their efficiency, productivity and performance by
simplifying complex tasks that involve language.

NLP Techniques
NLP encompasses a wide array of techniques that aimed at enabling
computers to process and understand human language. These tasks can be
categorized into several broad areas, each addressing different aspects of
language processing. Here are some of the key NLP techniques:
1. Text Processing and Preprocessing
 Tokenization: Dividing text into smaller units, such as words or sentences.
 Stemming and Lemmatization : Reducing words to their base or root forms.
 Stopword Removal: Removing common words (like "and", "the", "is") that
may not carry significant meaning.
 Text Normalization : Standardizing text, including case normalization,
removing punctuation and correcting spelling errors.
2. Syntax and Parsing
 Part-of-Speech (POS) Tagging : Assigning parts of speech to each word in a
sentence (e.g., noun, verb, adjective).
 Dependency Parsing : Analyzing the grammatical structure of a sentence to
identify relationships between words.
 Constituency Parsing : Breaking down a sentence into its constituent parts
or phrases (e.g., noun phrases, verb phrases).
3. Semantic Analysis
 Named Entity Recognition (NER) : Identifying and classifying entities in text,
such as names of people organizations, locations, dates, etc.
 Word Sense Disambiguation (WSD) : Determining which meaning of a word
is used in a given context.
 Coreference Resolution : Identifying when different words refer to the
same entity in a text (e.g., "he" refers to "John").
4. Information Extraction
 Entity Extraction : Identifying specific entities and their relationships within
the text.
 Relation Extraction : Identifying and categorizing the relationships between
entities in a text.
5. Text Classification in NLP
 Sentiment Analysis : Determining the sentiment or emotional tone
expressed in a text (e.g., positive, negative, neutral).
 Topic Modeling: Identifying topics or themes within a large collection of
documents.
 Spam Detection: Classifying text as spam or not spam.
6. Language Generation
 Machine Translation : Translating text from one language to another.
 Text Summarization : Producing a concise summary of a larger text.
 Text Generation : Automatically generating coherent and contextually
relevant text.
7. Speech Processing
 Speech Recognition : Converting spoken language into text.
 Text-to-Speech (TTS) Synthesis : Converting written text into spoken
language.
8. Question Answering
 Retrieval-Based QA : Finding and returning the most relevant text passage
in response to a query.
 Generative QA: Generating an answer based on the information available
in a text corpus.
9. Dialogue Systems
 Chatbots and Virtual Assistants : Enabling systems to engage in
conversations with users, providing responses and performing tasks based
on user input.
10. Sentiment and Emotion Analysis in NLP
 Emotion Detection : Identifying and categorizing emotions expressed in
text.
 Opinion Mining: Analyzing opinions or reviews to understand public
sentiment toward products, services or topics.
How Natural Language Processing (NLP) Works

NLP Working

Working in natural language processing (NLP) typically involves using

computational techniques to analyze and understand human language. This
can include tasks such as language understanding, language generation and
language interaction.

1. Text Input and Data Collection

 Data Collection: Gathering text data from various sources such as

websites, books, social media or proprietary databases.

 Data Storage: Storing the collected text data in a structured format, such
as a database or a collection of documents.

2. Text Preprocessing

Preprocessing is crucial to clean and prepare the raw text data for analysis.
Common preprocessing steps include:

 Tokenization: Splitting text into smaller units like words or sentences.

 Lowercasing: Converting all text to lowercase to ensure uniformity.

 Stopword Removal: Removing common words that do not contribute

significant meaning, such as "and," "the," "is."

 Punctuation Removal: Removing punctuation marks.

 Stemming and Lemmatization: Reducing words to their base or root

forms. Stemming cuts off suffixes, while lemmatization considers the
context and converts words to their meaningful base form.

 Text Normalization: Standardizing text format, including correcting

spelling errors, expanding contractions and handling special characters.

3. Text Representation
  Bag of Words (BoW): Representing text as a collection of words, ignoring
grammar and word order but keeping track of word frequency.

 Term Frequency-Inverse Document Frequency (TF-IDF) : A statistic that

reflects the importance of a word in a document relative to a collection
of documents.

 Word Embeddings: Using dense vector representations of words where

semantically similar words are closer together in the vector space (e.g.,
Word2Vec, GloVe).

4. Feature Extraction

Extracting meaningful features from the text data that can be used for various
NLP tasks.

 N-grams: Capturing sequences of N words to preserve some context and

word order.

 Syntactic Features: Using parts of speech tags, syntactic dependencies

and parse trees.

 Semantic Features: Leveraging word embeddings and other

representations to capture word meaning and context.

5. Model Selection and Training

Selecting and training a machine learning or deep learning model to perform

specific NLP tasks.

 Supervised Learning: Using labeled data to train models like Support

Vector Machines (SVM), Random Forests or deep learning models like
Convolutional Neural Networks (CNNs) and Recurrent Neural Networks
(RNNs).

 Unsupervised Learning: Applying techniques like clustering or topic

modeling (e.g., Latent Dirichlet Allocation) on unlabeled data.

 Pre-trained Models: Utilizing pre-trained language models such

as BERT, GPT or transformer-based models that have been trained on
large corpora.
6. Model Deployment and Inference

Deploying the trained model and using it to make predictions or extract

insights from new text data.

 Text Classification: Categorizing text into predefined classes (e.g., spam

detection, sentiment analysis).

 Named Entity Recognition (NER): Identifying and classifying entities in

the text.

 Machine Translation: Translating text from one language to another.

 Question Answering: Providing answers to questions based on the

context provided by text data.

7. Evaluation and Optimization

Evaluating the performance of the NLP algorithm using metrics such as

accuracy, precision, recall, F1-score and others.

 Hyperparameter Tuning: Adjusting model parameters to improve

performance.

 Error Analysis: Analyzing errors to understand model weaknesses and

improve robustness.

Technologies related to Natural Language Processing

There are a variety of technologies related to natural language processing
(NLP) that are used to analyze and understand human language. Some of the
most common include:

1. Machine learning: NLP relies heavily on machine learning techniques

such as supervised and unsupervised learning, deep learning and
reinforcement learning to train models to understand and generate
human language.

2. Natural Language Toolkits (NLTK) and other libraries: NLTK is a popular

open-source library in Python that provides tools for NLP tasks such as
 tokenization, stemming and part-of-speech tagging. Other popular
libraries include spaCy, OpenNLP and CoreNLP.

3. Parsers: Parsers are used to analyze the syntactic structure of sentences,

such as dependency parsing and constituency parsing.

4. Text-to-Speech (TTS) and Speech-to-Text (STT) systems: TTS systems

convert written text into spoken words, while STT systems convert
spoken words into written text.

5. Named Entity Recognition (NER) systems: NER systems identify and

extract named entities such as people, places and organizations from the
text.

6. Sentiment Analysis: A technique to understand the emotions or opinions

expressed in a piece of text, by using various techniques like Lexicon-
Based, Machine Learning-Based and Deep Learning-based methods

7. Machine Translation: NLP is used for language translation from one

language to another through a computer.

8. Chatbots: NLP is used for chatbots that communicate with other

chatbots or humans through auditory or textual methods.

9. AI Software: NLP is used in question-answering software for knowledge

representation, analytical reasoning as well as information retrieval.

Applications of Natural Language Processing (NLP)

 Spam Filters: One of the most irritating things about email is spam.
Gmail uses natural language processing (NLP) to discern which emails
are legitimate and which are spam. These spam filters look at the text in
all the emails you receive and try to figure out what it means to see if it's
spam or not.

 Algorithmic Trading: Algorithmic trading is used for predicting stock

market conditions. Using NLP, this technology examines news headlines
about companies and stocks and attempts to comprehend their meaning
in order to determine if you should buy, sell or hold certain stocks.
  Questions Answering: NLP can be seen in action by using Google Search
or Siri Services. A major use of NLP is to make search engines understand
the meaning of what we are asking and generate natural language in
return to give us the answers.

 Summarizing Information: On the internet, there is a lot of information

and a lot of it comes in the form of long documents or articles. NLP is
used to decipher the meaning of the data and then provides shorter
summaries of the data so that humans can comprehend it more quickly.

Future Scope

NLP is shaping the future of technology in several ways:

 Chatbots and Virtual Assistants: NLP enables chatbots to quickly

understand and respond to user queries, providing 24/7 assistance
across text or voice interactions.

 Invisible User Interfaces (UI): With NLP, devices like Amazon Echo allow
for seamless communication through voice or text, making technology
more accessible without traditional interfaces.

 Smarter Search: NLP is improving search by allowing users to ask

questions in natural language, as seen with Google Drive's recent
update, making it easier to find documents.

 Multilingual NLP: Expanding NLP to support more languages, including

regional and minority languages, broadens accessibility.

Future Enhancements: NLP is evolving with the use of Deep Neural Networks
(DNNs) to make human-machine interactions more natural. Future
advancements include improved semantics for word understanding and
broader language support, enabling accurate translations and better NLP
models for languages not yet supported.