Real-time Hashtag Frequency Analysis

in Tweets using Distributed Processing

Frameworks
Jheel Jain Mahek Jain Neel Jain Tanish Jain
Information Technology Information Technology Information Technology Information Technology
Thakur College of Thakur College of Thakur College of Thakur College of
Engineering and Engineering and Engineering and Engineering and
Technology Technology Technology Technology
jheeljain@[Link] mahekjain@[Link] neeljain@[Link] tanishjain@[Link]

Abstract— The explosive growth of social media human behavior—from political debates and social
platforms such as Twitter (now X) has led to the movements to cultural trends and entertainment.
generation of massive, high-velocity data streams that
reflect real-time public sentiment, global events, and The volume, velocity, and variety of Twitter data pose
emerging trends. One of the most insightful approaches to both opportunities and challenges for researchers,
understanding these dynamics is the analysis of hashtags, businesses, and policymakers. Every second, thousands
which serve as markers for trending topics and collective of tweets are generated, producing streams of
discourse. This research focuses on the implementation of unstructured data that encapsulate user sentiment,
a distributed hashtag frequency analysis system using the trending topics, and collective attention. Effectively
MapReduce paradigm. The proposed methodology analyzing these real-time streams offers unparalleled
efficiently processes large-scale tweet data by dividing it opportunities: governments can monitor crises and
into parallel input splits, mapping hashtags into key– disasters, companies can track brand perception,
value pairs, shuffling and sorting them for grouping, and journalists can identify breaking news, and researchers
finally reducing them into aggregated counts. The system can study large-scale social phenomena.
was developed using a modern full-stack architecture that
integrates a Reactbased user interface, Supabase for A key analytical task in this domain is the real-time
database management, and visualization modules to counting and analysis of hashtags, which serve as
present realtime hashtag analytics. The implementation semantic markers that organize discourse and amplify
addresses the challenges of scalability, fault tolerance, visibility. Hashtags provide immediate insights into what
and high throughput inherent in processing millions of people are talking about right now, functioning as
tweets per hour. Furthermore, the project not only dynamic indicators of public interest. Monitoring hashtag
highlights the effectiveness of MapReduce for stream frequencies in real time allows for trend forecasting, event
data analysis but also situates it in comparison with detection, and targeted decisionmaking across sectors.
modern alternatives such as Apache Kafka, Apache Spark For example, sudden spikes in health-related hashtags
Streaming, and Flink. The findings demonstrate that the may indicate the outbreak of disease awareness, while
MapReducebased approach provides a robust, sustained trends around political slogans often reflect
transparent, and scalable solution for social media trend broader public mobilization.
monitoring, while also opening discussions on ethical and
legal implications of mining user-generated content. However, extracting meaningful knowledge from these
Keywords— Social Media Analytics; Twitter; Hashtag high-velocity data streams is non-trivial. Traditional
Frequency; MapReduce; Big Data; Real-Time batch-processing methods are inadequate for coping with
Processing; Distributed Computing; Stream the scale and speed of Twitter data, where millions of
Processing; Visualization; Supabase; React; Scalability messages per hour demand scalable, distributed, and low-
latency solutions. This has motivated the adoption of
distributed processing frameworks, most notably
I. INTRODUCTION MapReduce, which offers a scalable abstraction for
The proliferation of social media platforms has breaking large tasks into parallelizable subtasks (map
profoundly transformed the dynamics of communication, phase) and aggregating results (reduce phase). Within the
information dissemination, and public discourse in the context of hashtag analysis, the MapReduce paradigm
digital era. Among these platforms, Twitter (recently provides a natural approach: individual map workers can
rebranded as X) occupies a central role as a real-time tokenize and count hashtags from subsets of tweets, while
microblogging service, enabling millions of users reducers aggregate these counts across the system to yield
worldwide to instantly share short messages, opinions, global frequencies.
and updates. With its concise message format, fast-paced
interactions, and global reach, Twitter has become a Beyond classical MapReduce, the rise of real-time stream
digital pulse of society, reflecting diverse aspects of processing technologies (such as Apache Kafka, Apache
Flink, and Apache Spark Streaming) highlights the revealing topical clusters, information diffusion paths,
evolution toward systems designed specifically for low- and influencer roles—leveraging modularity
latency, high-throughput event processing. While these maximization, label propagation, and random-walk
modern systems extend beyond batch-oriented embeddings for community detection and
MapReduce, understanding MapReduce fundamentals recommendation. On the systems side, large-scale
provides essential grounding in distributed data counting and enrichment migrated from batch
processing and the challenges of scale, fault tolerance, MapReduce pipelines (tokenization → emit (tag,1) →
and resource management. shuffle/group → reduce) toward low-latency stream
processors such as Spark Streaming/Structured
This project—Hashtag Flow—embeds these ideas into a Streaming, Flink, and Storm, often fronted by Kafka for
practical, interactive system that simulates and visualizes durable ingestion and relying on exactly-once or
real-time hashtag analytics. Built using React, effectively-once semantics, state backends, windowed
TypeScript, Supabase, and data visualization libraries, the aggregations, and watermarking to handle out-of-order
project offers an educational yet functional framework for events. Complementary strands integrated NLP tasks—
exploring distributed hashtag processing. language identification, normalization, sentiment/
emotion classification, stance detection, and topic
By combining multiple components, Hashtag Flow modeling—first with linear models and later with
bridges the gap between theoretical concepts (e.g., transformer encoders, improving the precision of trend
distributed processing, MapReduce) and hands-on interpretation and the quality of downstream
applications (real-time analytics dashboards). It not only recommendations (e.g., “related hashtags” via PMI/lift,
demonstrates the scalability challenges of analyzing embeddings, or graph proximity). More recent work
millions of tweets but also offers a sandbox for emphasizes real-time observability and human-in-theloop
experimenting with streaming architectures, visualization dashboards that couple aggregation with interactive
strategies, and interactive education tools. visualization, highlighting the pedagogical value of
explicating the map → shuffle → reduce stages for
Finally, while the project emphasizes computational nonexpert stakeholders. Across the literature, recurring
techniques, it also acknowledges the broader ethical and challenges include sampling bias (API rate limits,
legal challenges of social media data analysis. Issues of demographic skew), bot/coordination detection,
privacy, consent, bias, and misinformation are critical multilingual and code-mixed text handling, and the ethics
when dealing with user-generated data, especially at of privacy, consent, and responsible reporting—
scale. Responsible use of such technologies requires a prompting guidelines for data minimization,
balance between innovation and adherence to ethical anonymization, and reproducibility. Within this context,
standards. the present project aligns with established counting and
visualization paradigms while contributing an
In summary, this research and implementation project instructional, end-to-end implementation that bridges
explores the real-time counting and visualization of theoretical MapReduce concepts with practical, UIdriven
Twitter hashtags using distributed processing principles. analytics and a cloud data backend suited to incremental
It situates itself at the intersection of big data analytics, [Link] have become an indispensable element
social media research, and interactive system design, of social media communication, serving as a powerful
contributing both conceptually (through the MapReduce mechanism for content categorization and trend
abstraction) and practically (through the Hashtag Flow identification. Their evolution and impact on digital
platform). The outcome is a system that not only discourse are crucial for understanding the analytical
processes and visualizes hashtags effectively but also requirements of systems designed to process them.
deepens our understanding of real-time analytics in an era
where digital platforms continuously shape the narratives
of society.
A History and Origin of Hashtags on Twitter:
The concept of hashtags on Twitter originated organically
II. LITERATURE SURVEY in August 2007, when American blogger and product
consultant Chris Messina proposed using the '#' (pound)
Existing scholarship on social media analytics has symbol to categorize groups or topics.1
established Twitter/X as a canonical testbed for Messina's vision was for these tags to be a
highvelocity text mining, where hashtags act as communitydriven tool, famously stating that they were
lightweight, user-generated metadata that enable topic "born of the internet, and owned by no one". 2
discovery, community formation, and event tracking; Initially, Twitter did not immediately embrace Messina's
early studies modeled hashtags as evolving folksonomies suggestion. However, the convention gained widespread
and used frequency dynamics to detect bursts and popular acceptance through user initiative during the
exogenous shocks, while subsequent work formalized 2007 San Diego forest fires, as individuals leveraged
trend detection with time-series models (e.g., burstiness, hashtags to tag relevant content and facilitate information
Kleinberg-style states), change-point methods, and sharing.2 The term "hash tag" itself was first formally
survival/hazard analysis to characterize lifecycle and published in a blog post by Stowe Boyd on August 26,
decay. Parallel research framed hashtag analysis as a 2007.2 This organic emergence and userdriven adoption
graph problem—co-occurrence and retweet networks of hashtags underscore their fundamental role as a
decentralized, community-driven content categorization For experimental purposes, we consider two types of
mechanism. This evolution demonstrates that the utility datasets:
of hashtags was recognized and driven by the user base,
subsequently influencing platform design rather than Real-time Data: Using the Twitter Streaming API (or any
being dictated by it. Twitter later formalized their use by live feed simulator), tweets are captured continuously,
hyperlinking hashtags in tweets (July 2009) and focusing on hashtags and metadata like timestamps and
introducing "Trending Topics" in 2010, displaying rapidly user IDs.
popular hashtags.2
Batch/Offline Data: To test the scalability and accuracy
B Evolution of Hashtag Usage in Social Media: of the system before deployment, we also simulate large-
scale input by using archived tweet datasets from Kaggle
From their inception on Twitter, hashtags rapidly and other open repositories.
transcended their original platform, becoming a universal
tool for content categorization, enhancing discoverability, Each tweet record includes:
and fostering engagement across a wide array of social
media platforms, including Instagram, Facebook, Tweet text
LinkedIn, and TikTok.3 In their early stages, the primary Extracted hashtags
benefits of hashtags were their ability to organize content Metadata (time, location, user ID, etc.)
and improve searchability.3 Brands and influencers
quickly recognized their potential, adopting them for The primary focus is on the frequency of hashtags, as
marketing campaigns and expanding their reach. this metric indicates the popularity and virality of specific
Concurrently, hashtags played a pivotal role in mobilizing topics in real-time.
and amplifying social movements, with examples such as
#BlackLivesMatter and #MeToo gaining significant
momentum through widespread 2. Data Preprocessing
usage.3 The raw Twitter data is noisy and requires extensive
Over time, the dynamics of hashtag usage have undergone cleaning before analysis. The preprocessing pipeline
substantial shifts. These changes are primarily includes:
attributable to evolving platform algorithms, instances of Hashtag Extraction: Parsing each tweet to extract
hashtag overload and misuse, and the increasing hashtags, ignoring case sensitivity.
sophistication of Artificial Intelligence (AI) and smart Noise Removal: Filtering out URLs, mentions (@user),
search features.3 Modern social media platforms now emojis, punctuation, and retweet symbols (RT).
prioritize content based on its inherent relevance and user Normalization: Converting all hashtags to lowercase
engagement rather than relying solely on the presence or (e.g., #COVID19 and #covid19 treated as the same).
frequency of hashtags.3 This shift in social media Time Window Partitioning: Tweets are grouped into
algorithms from hashtag-centric discovery to AI-driven time windows (e.g., 1-minute or 5minute batches) to
relevance suggests a move towards more implicit content enable real-time trending analysis.
categorization and a potential reduction in the direct The processed data is then stored in a distributed file
efficacy of manual hashtagging for achieving broad reach. system (e.g., HDFS) for large-scale operations.
This algorithmic evolution encourages content creators to 3. Distributed Processing Framework (MapReduce)
focus on producing highquality, contextually relevant
material that resonates with audiences, rather than merely
relying on keyword stuffing. For instance, Instagram's To handle millions of tweets per hour, we employ the
Explore Page and Twitter's trending topics are now MapReduce paradigm.
heavily influenced by user behavior and engagement
levels, diminishing the impact of simply adding numerous Mapper Phase:
hashtags.3 This algorithmic refinement was partly a Each mapper reads a batch of tweets and extracts
response to users overloading posts with excessive or hashtags. It emits key-value pairs in the form: (#hashtag,
repetitive hashtags, which led to reduced authenticity and 1)
even Combiner Phase:
"shadowbanning" on some platforms.3 Local aggregation occurs to reduce network overhead,
combining counts of hashtags within the same mapper.
Reducer Phase:
III. METHODOLOGY Reducers aggregate hashtag counts across all mappers
and output the final frequencies for each hashtag in the
1. Data Collection time window.

Our system primarily targets Twitter (now X) as the This approach ensures scalability, parallelism, and fault
source of real-time hashtag data. Twitter generates tolerance, making it suitable for large-scale hashtag
millions of tweets every hour, making it an ideal platform frequency analysis.
for studying fast-changing trends and discussions. The 4. System Architecture (from Project Code)
data stream typically consists of short text messages
containing hashtags (#topic) that represent user interests, Based on the provided React frontend code, the
movements, or events. architecture consists of:
 1. Data Ingestion Layer – Streams tweets from Python for backend processing, Hadoop MapReduce for
the Twitter API or simulated dataset. distributed computation, and [Link] for the visualization

2. Processing Layer – Runs MapReduce (or Spark

Streaming/Flink in modern implementations)
for distributed hashtag frequency counting.

3. Storage Layer – Uses HDFS or NoSQL

databases (e.g., Cassandra, MongoDB) to store
intermediate and final results.

4. Visualization Layer – The frontend (developed

in [Link]/React with Tailwind CSS) provides
real-time dashboards, interactive charts, and
trend visualizations of top hashtags.

4. Modern Stream Processing Alternatives

Although the project is built on MapReduce, the

methodology also acknowledges modern
streamprocessing systems like:

• Apache Spark Streaming – For micro-batch

hashtag analysis.

• Apache Flink / Kafka Streams – For

lowlatency, event-driven real-time processing.

These alternatives improve on MapReduce’s limitations

in latency and are better suited for real-time monitoring
of fast-evolving topics.

5. Evaluation Phase
dashboard. This architecture was chosen to ensure
The effectiveness of the system is measured using the
scalability, efficiency, and interactive visualization of
following parameters:
results. Python served as the core programming language
because of its rich ecosystem of libraries for data
• Scalability: Ability to process increasing
preprocessing, text handling, and integration with big data
volumes of tweets per second without
frameworks. Hadoop MapReduce was employed for
performance degradation.
parallel processing of large tweet datasets, leveraging its
mapper–reducer paradigm to count hashtag frequencies at
• Latency: Time taken from tweet ingestion to
scale. To make the outputs more accessible and user-
visualization of updated hashtag frequencies.
friendly, the processed data was visualized using a
[Link]-based dashboard styled with Tailwind CSS and
• Accuracy: Correct counting and normalization enhanced by charting libraries for real-time trend
of hashtags, ensuring that variations of the same representation.
hashtag are aggregated.
The workflow began with data collection, where tweets
• Visualization Effectiveness: Quality of trend were obtained using the Twitter Streaming API and
supplemented with archived datasets to simulate
detection in the real-time dashboard. largescale streaming environments. Each record
contained tweet text, hashtags, and timestamps, which
were subjected to preprocessing before analysis. During
this stage, hashtags were extracted using regular
expressions, normalized to lowercase, and stripped of
Fig 2. Block Diagram URLs, mentions, emojis, and special symbols to ensure
clean input. The cleaned data was then segmented into
time windows, such as one-minute intervals, to enable
IV. IMPLEMENTATION trend tracking in near real time.
The implementation of the real-time hashtag frequency The preprocessed tweets were passed into the Hadoop
analysis system was carried out using a combination of MapReduce pipeline. In the mapper function, each tweet
was scanned, and key-value pairs in the format (#hashtag, Shuffling Gathers and Grouped (key,
1) were generated for every detected hashtag. These
redistributes list(values))
intermediate pairs were locally aggregated by a combiner
to minimize data transfer overhead. The reducer function intermediate data, pairs.
then collected and summed all hashtag occurrences from grouping all values
across the distributed nodes, producing the final hashtag with the same key
frequency counts for the given time window. The results to a single
were stored in HDFS to ensure persistence and scalability, reducer.
after which a Python-based integration layer transformed
the raw counts into structured JSON files that could be Sorting Sorts the Sorted (key,
consumed by the frontend. intermediate list(values))
keyvalue pairs by pairs.
For visualization, a [Link] dashboard was developed to key, preparing
display real-time hashtag trends. The interface provided them for efficient
users with an interactive leaderboard of trending reduction.
hashtags, time-series plots of frequency variations, and
comparative charts such as bar graphs, line graphs, and Reducing Processes Final
pie charts. Additional features such as search and filtering
grouped and aggregated
allowed users to focus on specific hashtags of interest.
Initially, a prototype version of the system was tested sorted data, (key, value)
using Streamlit dashboards to validate outputs quickly, applying an pairs (e.g.,
but the final deployment relied on a modern [Link] aggregation (hashtag,
application for a polished and scalable user experience. function to count)).
produce final
To enhance scalability beyond batch processing, the results.
system was extended with Apache Spark Streaming and
Apache Kafka, which enabled near real-time ingestion Result Stores the final Final output
and processing of continuous tweet streams. This hybrid output from the files (e.g., in
design ensured low latency, fault tolerance, and the ability reducers to a HDFS).
to handle millions of tweets per hour while still providing distributed file
immediate insights into trending topics. Overall, the system.
implementation demonstrates how distributed data
processing frameworks like Hadoop and Spark can be Table 1: MapReduce Workflow Stages
effectively integrated with modern web technologies to
create a powerful real-time analytics platform capable of The results of the system can be best understood by
monitoring global conversations at scale. analyzing the sequential phases of the MapReduce
workflow. The process begins with Input Splits, where
raw tweet data is divided into manageable blocks that
V. RESULT AND DISCUSSION enable parallel execution across distributed nodes. This
division ensures that the system can handle massive
volumes of tweets efficiently without overwhelming a
single machine. In the Mapping phase, each tweet within
Phase Name Description Output/Key a split is parsed to extract hashtags, and every occurrence
Function is converted into a key–value pair such as (#hashtag, 1).
These pairs are then redistributed during the Shuffling
stage, where the framework automatically groups all
identical keys together, ensuring that all instances of a
Input splits Divides raw input Independent hashtag are collected for aggregation. Once grouped, the
data into smaller, data blocks for Sorting step organizes the keys in lexicographical order,
manageable mappers. which streamlines the reduction process and makes
computation more efficient. The Reducing phase then
chunks for parallel
aggregates the grouped values by summing counts to
processing. generate the final frequency for each hashtag, such as
(#AI, 3500). Finally, in the Result stage, the aggregated
outputs are written into the Hadoop Distributed File
Mapping Processes each (key, value) System (HDFS), providing structured data that can be
input split, pairs (e.g., directly consumed by visualization tools. This structured,
transforming raw (hashtag, 1)). multi-phase pipeline demonstrates how the MapReduce
data into framework efficiently transforms high-velocity
intermediate (key, unstructured social media data into meaningful insights
value) pairs. on trending topics.
VI CONCLUSION AND FUTURE WORK VII. ACKNOWLEDGEMENT

Beyond basic hashtag frequency counting, the real-time We would like to thank Mrs. Neha Patvari Ma’am very
processing of tweet data opens avenues for more much for guiding us in this paper. Indeed, the guidance
sophisticated analytics and future research. These support she gave considerably helped make this paper.
advanced techniques leverage the high-velocity nature of Her experience and insights have enriched our
social media streams to derive deeper, more actionable understanding of Prediction using Machine Learning
insights. greatly. We thank her for encouragement and mentorship
without which this effort may not have materialized.
Real-time Sentiment Analysis of Hashtags
REFERENCES
Sentiment analysis, also known as opinion mining, is a
Natural Language Processing (NLP) technique used to [1]How to Twitter: History Hashtags - Liz Covart,
determine the emotional tone (positive, negative, neutral) accessed on August 1, 2025, [Link]
expressed within a text.89 When applied to hashtags in blog/how-to-twitter-history-hashtags
real-time tweet streams, sentiment analysis can provide
immediate insights into public opinion, brand perception, [2]Hashtag - Wikipedia, accessed on August 1, 2025,
and reactions to events.90 [Link]

The process typically involves a multi-step pipeline: [3]Are Hashtags Still Relevant? The Evolution of
Hashtags in Social Media Strategy, accessed on August 1,
1. Data Pre-processing: Raw tweet text undergoes 2025, [Link]
cleaning, including tokenization, removal of the-evolution-of-hashtags-in-social-mediastrategy/
stop-words (common words like "the," "is"), and
[4]The evolving role of hashtags - Talon Media, a c c e s
stemming (reducing words to their root form). 89
se d o nAu gu st1 ,2 02 5, h tt p s:/ /
2. Feature Extraction: Relevant features are
extracted from the cleaned text. [Link]/the-evolving-role-of-hashtags/
3. Sentiment Classification: Machine learning
models, such as Logistic Regression or [5]How Social Media Algorithms Work in 2025
Bidirectional Long Short-Term Memory [Platform Breakdown] - Sprinklr, accessed on August 1,
(BiLSTM) networks, are trained to classify the 2025, [Link]
sentiment polarity of tweets or hashtags.89 mediaalgorithm/

Real-time sentiment analysis of hashtags has diverse [6]Twitter (X) Hashtag Analytics: All You Need to Know
applications, including crisis management (detecting and in 2025 - Twilert, accessed on August 1, 2025,
responding to negative sentiment before it escalates), [Link]
public relations, customer feedback analysis, and tracking
[7]Twitter Trends: Your Guide to Global Conversations -
public opinion on social and political events.90 Integrating
Tweetlio, accessed on August 1, 2025, https://
real-time hashtag counting with sentiment analysis offers
a richer, more nuanced understanding of public discourse [Link]/blog/mastering-twitter-trends-howto-
than mere frequency alone. This transforms raw stay-updated-on-global-conversations
quantitative counts into qualitative, actionable insights for [8]What Are the Best Times to Post on Twitter in 2025? -
diverse applications. For instance, knowing that a hashtag SocialBee, accessed on August 1, 2025, https://
is trending is useful, but understanding [Link]/blog/best-times-to-post-on-twitter/
why it is trending—whether due to positive engagement or [9]Best Times to Post on Twitter (X) in 2025 - Sprout
negative backlash—provides a far more valuable and Social, accessed on August 1, 2025, https://
actionable piece of information for brands, [Link]/insights/best-times-to-post-on-twitter/
policymakers, or event organizers.90
[10]Streaming at Scale - Digital Turbine, accessed on
August 1, 2025, [Link]
streaming-at-scale