Part 1

Title
Use the following dataset and classify tweets into positive and negative tweets.
[Link]

Introduction

In the modern era of information technology, Twitter (X) represents one of the most
significant examples of Big Data in action. With over 33,000 entries analyzed in this project
alone, the platform serves as a massive, real-time repository of human thought and
interaction. This project utilizes Data Science methodologies to transform this high-velocity,
unstructured data into structured, actionable insights.

Twitter as a Big Data Resource:

Twitter is a textbook example of Big Data due to its complexity and scale:
High Volume and Variety: The dataset contains 35 distinct columns, including text,
timestamps, user IDs, and engagement metrics like likes and retweets.
Velocity: The data captures chronological trends, allowing for a time-series analysis of how
public opinion shifts over years.
Unstructured Nature: The core of the data—the tweet—is raw text that requires advanced
processing to filter out noise like URLs, mentions, and symbols.

The Role of Data Science:

This mini-project demonstrates the standard Data Science pipeline to handle this Big Data
environment:
Data Processing: We use Python and Pandas to ingest and clean the dataset, converting raw
social media "noise" into a clean format for analysis.
Sentiment Modeling: By applying Natural Language Processing (NLP) via the NLTK library, we
quantify the emotional content of tweets using mathematical "compound scores".
Pattern Extraction: Through custom lexicons, we adapt the model to recognize industry-
specific language or social media trends that standard algorithms might overlook.
Visualization: Finally, using Plotly, we condense thousands of data points into a single visual
narrative, mapping the "emotional pulse" of the data science community over time.
By viewing Twitter through the lens of Data Science, we can move beyond reading individual
posts to understanding the global sentiment and macro-trends hidden within the Big Data
stream.
 Problem Statement:

The core challenge of this project is to develop an automated pipeline to collect, process, and
visualize the sentiment of Twitter data over an extended period. The project must address
the following technical hurdles:
a) Data Handling: Efficiently importing and managing large datasets (over 33,000 entries)
with varied data types and potential inconsistencies.
b) Text Pre-processing: Cleaning raw tweet data by removing non-essential elements such
as URLs, @mentions, and special symbols that interfere with sentiment scoring.
c) Sentiment Classification: Implementing a computational model using the VADER
(Valence Aware Dictionary and sentiment Reasoner) lexicon to categorize tweets into
Positive, Negative, or Neutral classes.
d) Lexicon Customization: Addressing the limitations of standard sentiment dictionaries
by manually updating them with context-specific or "slang" terms (e.g., "teamjames" or
"cancel culture") to improve prediction accuracy.
e) Temporal Visualization: Aggregating the classified results by date and generating a
multi-series line chart to identify how positive and negative sentiments fluctuate over a
decade (2010–2022).

Objectives:
The primary aim of this assignment is to move from raw data to actionable insights through
the following specific goals:
a) Automated Data Acquisition and Storage: To implement a process for scraping and
storing large volumes of Twitter data for analysis.
b) Text Classification: To categorize unstructured tweet text into three distinct sentiment
classes: positive, negative, or neutral.
c) Computational Accuracy through Lexicon Customization: To improve the model's
performance by manually updating the sentiment dictionary with custom words (e.g.,
"teamjames" or "pedophile") that are specific to the dataset's context.
d) Data Refinement and Noise Reduction: To develop a cleaning function that removes
distracting elements like URLs, @mentions, and hashtags while retaining the core
textual meaning.
e) Temporal Trend Identification: To identify and visualize how strong sentiments
(positive and negative) fluctuate over a specific timeline, specifically from 2010 to 2022.
 Technology Used:
The project is built using the Python ecosystem, leveraging specialized libraries for data
science and linguistics:
1. Data Management:
a) Pandas: Utilized for loading the CSV dataset (data_visualization.csv), exploring data
structures via [Link] (), and performing data manipulation such as grouping by date.

2. Natural Language Processing (NLP):

a) NLTK (Natural Language Toolkit): The core library used for text processing.
b) VADER (Sentiment Intensity Analyzer): A specific module within NLTK used to
calculate compound scores to determine the emotional polarity of a tweet.
c) Regular Expressions (re): Employed within a custom cleaner function to identify and
strip out web links and social media handles using pattern matching.

3. Data Visualization:
a) Plotly: Used to create interactive, color-coded line and marker charts. Green lines are
used to represent positive sentiment trends, while red lines represent negative trends.
b) Matplotlib: Integrated alongside Plotly to help identify and build charts for sentiment
trends.

4. Development Environment:
a) IPython / Google Colab: The project execution environment, as evidenced by the file
paths (e.g., /content/data_visualization.csv) and specific kernel warnings.
 Introduction to the Twitter Sentiment Analysis Project:
This project, developed by the Department of Computer Engineering at Paravatibai Genba
Moze College of Engineering, focuses on a mini-project for the DSBDAL subject. The primary
objective is to perform a comprehensive sentiment analysis on Twitter data to identify and
visualize trends over time.

The project is structured into three distinct phases:

1. Data Collection and Exploration:

The initial phase involves scraping and storing Twitter data into a structured format. Using
the Pandas library, a dataset (e.g., data_visualization.csv) containing 33,590 entries and 36
columns is loaded. Key variables identified for this analysis include the date of the tweet and
the tweet text itself. Preliminary exploration is conducted to understand the data's
composition, such as user IDs, conversation IDs, and engagement metrics like likes and
retweets.

2. Sentiment Analysis Methodology:

The core of the project utilizes Natural Language Processing (NLP) techniques to classify
tweets as positive, negative, or neutral.
Tools Used: The process employs the NLTK (Natural Language Toolkit) library and its
Sentiment Intensity Analyzer (SID) module.
The SID Module: This module generates scores across four categories: positive, negative,
neutral, and a normalized compound score.
a) Classification: A compound score closer to 1 signifies a positive tweet, while a score
closer to -1 indicates a negative tweet.
b) Customization: The model is further refined using a custom dictionary (word_dict) to
account for specific social media slang or context-specific terms (e.g., "teamjames") that
standard analyzers might misinterpret.

3. Data Cleaning and Visualization:

Before final analysis, a cleaning function is applied to the tweets to remove noise such as @
symbols, HTTP links, and hashtags. Once the sentiments are predicted and stored in a new
column, the data is grouped by date.
The final stage uses visualization libraries like Plotly and Matplotlib to generate interactive
charts. These visualizations specifically track how strong sentiments (positive vs. negative)
have evolved over a timeline ranging from 2010 to 2022, allowing for the identification of
specific peaks or shifts in public opinion.
 Commands, Input and Output:
We will begin by scraping and storing Twitter data.
We will then classify the Tweets into positive, negative, or neutral sentiment with a simple
algorithm.
Then, we will build charts using Plotly and Matplotlib to identify trends in sentiment.

Let's now take a look at some of the variables present in the data frame:
The data frame has 35 columns. The most main variables we will be using in this analysis are
date and tweet.
Let's take a look at a sample Tweet in this dataset, and see if we can predict whether it is
positive or negative:

Step 2: Sentiment Analysis:

The Tweet above is clearly positive. Let's see if the model is able to pick up on this, and return
a positive prediction. Run the following lines of code to import the NLTK library, along with
the Sentiment Intensity Analyzer (SID) module.
The SID module takes in a string and returns a score in each of these four categories: positive,
negative, neutral, and compound. The compound score is calculated by normalizing the
positive, negative, and neutral scores. If the compound score is closer to 1, then the Tweet
can be classified as positive. If it is closer to -1, then the Tweet can be classified as negative.
Let's now analyze the above sentence with the sentiment intensity analyzer.
The output of the code above is 0.7089, indicating that the sentence is of positive sentiment.
Let's now create a function that predicts the sentiment of every Tweet in the data frame, and
stores it as a separate column called 'sentiment.' First, run the following lines of code to clean
the Tweets in the data frame:

Now that the Tweets are cleaned, run the following lines of code to perform the sentiment
analysis:
The word_dict created above is a dictionary of custom words I wanted to add into the model.
Words like 'teamjames' mean that people's sentiment around James Charles is positive, and
that they support him. The dictionary used to train the sentiment intensity analyzer wouldn't
already have these words in them, so we can update it ourselves with custom words. Now,
we need to convert the compound scores into categories - 'positive', 'negative', and 'neutral'.

Let's take a look at the head of the data frame to ensure everything is working properly:

Notice that the first few Tweets are the combination of positive, negative and neutral
sentiment. For this analysis, we will only be using Tweets with positive and negative
sentiment, since we want to visualize how stronger sentiments have changed over time.
Step 3: Visualization:
Now that we have Tweets classified as positive and negative, let's take a look at changes in
sentiment over time. We first need to group positive and negative sentiment and count them
by date:

Now, we can visualize sentiment by date using Plotly, by running the following lines of code:
Final Output - You should see a chart that looks like this:

The red line represents negative sentiment, and the green line represents positive sentiment.
 Conclusion:

The Twitter Sentiment Analysis mini-project successfully demonstrates the end-to-end

pipeline of a Data Science and Big Data Analytics (DSBDAL) workflow. By leveraging Natural
Language Processing (NLP), the project transitions from raw, unstructured social media data
to a clear, visual representation of public opinion trends over a twelve-year span.

Key Findings and Outcomes.

Effective Data Classification: The implementation of the NLTK Sentiment Intensity Analyzer
proved effective in categorizing 33,590 tweets into positive, negative, and neutral sentiments
using normalized compound scores.
The Importance of Customization: A critical finding of the project was that standard
sentiment lexicons may not capture modern social media nuances. By manually updating the
word_dict with context-specific terms like "teamjames" or "cancel culture," the model's
accuracy was significantly improved for the targeted dataset.
Data Cleaning as a Prerequisite: The use of a custom cleaner function to remove @mentions,
HTTP links, and special characters was essential to ensure the sentiment scores were based
on meaningful text rather than digital noise.
Long-term Trend Visualization: The final output, generated through Plotly, successfully
mapped sentiment fluctuations from 2010 to 2022. The visualization clearly distinguishes
between positive (green) and negative (red) sentiments, allowing for the identification of
specific periods of high emotional engagement.
Final Summary:
Ultimately, this project highlights how Python-based tools like Pandas, NLTK, and Plotly can
be integrated to handle "Big Data" challenges in social media. By filtering out neutral data
and focusing on strong emotional polarities, the analysis provides a focused lens through
which to view how public sentiment has evolved in the digital space over the last decade.
 Future Scope
Based on the methodology and current implementation of the Twitter sentiment analysis
project, the following areas represent a detailed future scope for expansion and refinement:
1. Integration of Real-Time Data Streaming:
While the current project operates on a static dataset stored in a CSV file, the framework
could be evolved to use the Twitter API (v2) for real-time sentiment monitoring.
This would allow for:
Live Dashboards: Creating dynamic visualizations that update as new tweets are posted.
Event-Triggered Alerts: Setting up systems that notify users when negative sentiment spikes
during a specific time window.
2. Implementation of Advanced Machine Learning Models:
The current approach relies on the VADER lexicon and a simple manual dictionary update.
Future iterations could incorporate:
Deep Learning: Utilizing models like BERT (Bidirectional Encoder Representations from
Transformers) or LSTMs to better understand context, sarcasm, and complex sentence
structures that simple algorithms might miss.
Automated Feature Engineering: Moving beyond manual cleaning to automated tokenization
and vectorization techniques for higher precision.
3. Multi-Dimensional Sentiment Analysis:
The project currently classifies tweets into three basic categories: positive, negative, and
neutral. Future scope includes:
Emotion Detection: Expanding the classification to include specific emotions such as anger,
joy, sadness, or surprise.
Sarcasm Detection: Developing specialized algorithms to identify ironic or sarcastic
statements which often carry a "positive" score in simple lexicons but are "negative" in
intent.
4. Geographical and Demographic Mapping:
The dataset includes columns for place, geo, and language that are currently largely
underutilized or null. Future work could involve:
Geospatial Analysis: Visualizing how sentiment toward a topic varies by region or country.
Demographic Insights: Correlating sentiment with user metadata to understand how
different groups perceive specific trends.
5. Enhanced Customization and Domain-Specific Lexicons:
The project demonstrates the value of custom word dictionaries. This can be expanded by:
Dynamic Lexicon Updates: Using machine learning to automatically identify and add new
trending "slang" or "cancel culture" terms to the dictionary without manual intervention.
Industry-Specific Models: Tailoring the sentiment analyzer for specific sectors like finance
(analyzing "cash tags"), healthcare, or politics.