Analytics Architecture

Analytics architecture refers to the overall design and structure of an analytical

system or environment, which includes the hardware, software, data, and
processes used to collect, store, analyze, and visualize data. It encompasses
various technologies, tools, and processes that support the end-to-end
analytics workflow.
Key components of Analytics Architecture-

Analytics architecture refers to the infrastructure and systems that are used to support the collection,
storage, and analysis of data. There are several key components that are typically included in an analytics
architecture:
1. Data collection: This refers to the process of gathering data from various sources, such as sensors,
devices, social media, websites, and more.
2. Transformation: When the data is already collected then it should be cleaned and transformed
before storing.
3. Data storage: This refers to the systems and technologies used to store and manage data, such as
databases, data lakes, and data warehouses.
4. Analytics: This refers to the tools and techniques used to analyze and interpret data, such as
statistical analysis, machine learning, and visualization.
Analytics Architecture Components and Design Style
1. Load Leveling with queues:

● Load leveling with queues is a scalable system design pattern used to manage variable
workloads and prevent systems from being overwhelmed. It involves placing a message
queue between the producer (work generator) and the consumer (work processor),
effectively decoupling them and smoothing the load.
● Load leveling aims to:
● Absorb traffic spikes
● Prevent system crashes due to overload
● Ensure consistent performance even during high traffic
Example:
Upload: Users upload images (producer).
Queue: Uploaded images go into RabbitMQ.
Consumer: Worker service resizes/processes images at a steady rate.
Benefits of Load Leveling Queues
● Prevents system overload

● Handles bursty traffic efficiently

● Allows asynchronous processing

● Supports scaling (add more consumers as needed)

● Improves system reliability and fault tolerance

Limitations
● Queue overflow: If producers are much faster, the queue can grow indefinitely.

● Message delay: Processing might be delayed under heavy load.

● Poison messages: Bad data that keeps failing processing must be handled (e.g.,
moved to a dead-letter queue).
2. Load Balancing with Multiple Consumers
Load balancing with multiple consumers is a design pattern used in distributed systems where a set of
consumers (workers or services) share the load of processing tasks from a common source (like a queue or
stream). The goal is to distribute work evenly across all consumers to maximize resource utilization, minimize
latency, and improve system reliability.

1. Producers and Consumers

● Producer: Generates data or tasks and sends them to a queue or broker.

● Consumer: Retrieves and processes tasks from the queue.

2. Load Balancer

● A system component or algorithm that distributes incoming workload (tasks, messages, requests) across
multiple consumers.
● Can be external (e.g., a dedicated load balancer service) or internal (built into message brokers like Kafka,
RabbitMQ).
Load Balancing Strategies
1. Round-Robin

● Tasks are assigned to consumers in a circular order.

● Simple but does not consider consumer workload or processing time.

2. Workload-Based (Dynamic Load Balancing)

● Tasks are assigned based on current load or resource usage of each consumer.
● More efficient but requires monitoring and feedback mechanisms.

3. Queue-Based Load Balancing

● Tasks are pushed to a queue; consumers pull from the queue as they become available.
● Ensures that faster consumers get more tasks, naturally balancing the load.

4. Partitioning (for Stream Processing)

● Data is divided into partitions, and each consumer is assigned a partition.

● Common in Kafka, helps in scaling horizontally.
Benefits
● Scalability: Easily add more consumers to handle increased load.

● Fault Tolerance: If one consumer fails, others can take over its tasks.

● Efficiency: Distributes tasks according to processing capability.

3. Sharding
● Data sharding is a crucial technique employed in Big Data systems to horizontally partition
and distribute large datasets across multiple nodes or servers within a distributed computing
environment.
● This strategy aims to improve data management, scalability, and performance by breaking
down datasets into smaller, more manageable pieces called shards. Each shard is a subset of
the complete dataset, and it contains a specific portion of the data.
● One of the primary reasons for using data sharding is to enhance data distribution and
parallel processing. By distributing shards across different nodes or servers, Big Data systems
can leverage parallelism to process multiple shards simultaneously.
● This is particularly vital in scenarios where the volume of data is so massive that processing it
sequentially on a single machine would be impractical or time-consuming.
Working
1. Shard Key Selection

○ A shard key (e.g., user_id) determines which shard a particular piece of data
goes to.
2. Data Distribution
○ The system maps shard keys to shard servers using:
■ Range-based sharding – Example: IDs 1–1000 go to Shard 1, 1001–2000
to Shard 2.
■ Hash-based sharding – The shard key is hashed to decide shard location.
■ Directory-based sharding – A lookup table maps keys to shards.

3. Query Routing
○ The system routes queries to the correct shard(s) based on the shard key.
Advantages and Disadvantages
Advantages
● Linear scalability – more servers → more capacity.

● Reduces single-node bottlenecks.

● Works well for massive datasets (like those of Facebook, Twitter, YouTube).

Challenges
● Complexity – Application logic must handle routing, re-sharding, and cross-shard queries.

● Rebalancing – Adding or removing shards may require moving a lot of data.

● Joins and transactions – Harder when data spans multiple shards.

4. Lambda Architecture

● Lambda architecture is an excellent architecture for handling massive real-time data

and building fault-tolerant, scalable systems.
● Apart from batch and stream processing, Lambda architecture also includes a data
serving layer for responding to user queries.
● This Architecture is widely used in many big tech companies as it takes advantage
of both real-time data processing as well as batch processing i.e. one can query
both fresh data by real-time data processing technique and historical data using
batch processing data technique.
There are two approaches to Lambda Architecture:
● Hybrid approach:
○ It is designed to harness enormous volumes of rapidly created data, enabling businesses to
make use of data more quickly.
● Specific approach:
○ It attempts to balance latency, throughput, and fault tolerance by using batch processing to
provide accurate views by batch data, while simultaneously using real-time stream
processing to provide views of online data. The outputs from both batch and speed layers
can be merged before the presentation.
Layers in Lambda Architecture

Lambda Architecture has mainly three layers to process big data:

● Batch Layer (Cold process): Batch Layer operates on the complete data and thus allows the system
to produce the most accurate results. However, the results come at the cost of high latency due to
high computation time.
● Stream Layer (Hot process or Speed Layer): Stream Layer operates on the real-time data to
complement the batch views. It receives the arriving data from various clients and performs
incremental updates to the batch layer results and store them in processed data Database.
● Serving layer: Serving Layer is a server or a set of servers which processes output of various queries
from different modules(like analytics module, Notification module) using the results sent from the
batch and speed layers.
Advantages of Lambda Architecture
● It is a good balance of speed, reliability, and scalability.
● The batch layer of Lambda architecture manages historical data with the fault-tolerant, distributed
storage, ensuring a low possibility of errors even if the system crashes.
● The Stream layer of Lambda architecture manages the real time data with immediate response with
somewhat less precision.
● Access to both real-time and offline data results in covering many data analysis scenarios very well.

Disadvantages of Lambda Architecture

● Lambda architecture is complex infrastructure as it has many layers involved.
● Although the offline layer and the real-time stream face different scenarios, their internal processing
logic is the same, so there are many duplicate modules and require different codebase.
● Maintaining the different code base and keeping them in sync so that processed data produces same
results from both paths.
● Computes every batch cycle more then once, which decreases the system performance and requires
more resources.
[Link] Election
● In distributed computing, a process known as "leader election" occurs when nodes, or computers or
devices, select a leader or coordinator from among themselves.
● The leader is in charge of decision-making, action coordination, and making sure the system runs
smoothly. This mechanism helps maintain order and manage resources efficiently.

Leader election holds great importance in system design for several reasons:
● Fault Tolerance
● Consistency
● Scalability
● Load Balancing
Leader Election Algorithms

1. Bully Algorithm: The Bully Algorithm relies on a hierarchy of nodes where each node has a
unique identifier, typically based on some ordering criterion such as IP address or node ID. The
node with the highest identifier is considered the leader.

2. Ring Algorithm: The Ring Algorithm organizes nodes in a logical ring structure, where each
node has knowledge of its successor node in the ring.

3. Paxos: Paxos is a consensus protocol designed to achieve agreement among a group of

nodes, ensuring the selection of a single leader.

4. Raft: Raft is another consensus protocol designed for leader election and log replication in
distributed systems, focusing on simplicity and understandability.
Advantages of Leader Election
Below are the advantages of Leader Election:
● Having a leader means there’s one "boss" making the important decisions, so everyone
knows who to follow. This can help avoid confusion, especially in complex systems.
● The leader can coordinate tasks and ensure everything is working together smoothly.
This is super important when many parts of the system need to work in sync.
● A leader can streamline decision-making and actions, reducing the time it takes to agree
on what to do next. Without a leader, it might take longer to reach a decision because
everyone has to agree on everything.
● If the leader fails, the system can automatically choose a new leader, which helps
maintain stability. So even if one part fails, the system keeps working.
Disadvantages of Leader Election

● If the leader fails and the system cannot quickly choose a new one, things can temporarily fall
apart. The system becomes dependent on the leader, so if the leader breaks, it might cause
delays.
● The process of electing a leader can be tricky, especially in big systems. The system needs a
way to make sure the right leader is chosen, which can take time and resources.
● The leader is responsible for a lot, which can put a heavy load on them. If the leader becomes
overwhelmed or overworked, it might slow down the entire system.
● If the leader makes bad decisions, the whole system suffers. And if the leader's decisions are
not checked or balanced, it can lead to mistakes.
6. Materialised Views
● A Materialized View (MV) is a database object that stores the results of a
query physically, unlike a normal view, which is just a saved query that runs
every time you access it. You can think of it as a snapshot of the data that can
be refreshed periodically or on demand.
● In a distributed system, a materialized view is a precomputed, stored
result of a query that can be kept on one or multiple nodes.
● Instead of computing queries every time, the system stores the results.
● This helps improve performance, consistency, and availability when
data is spread across different machines.
7. Bloom Filter
● A Bloom filter is a space-efficient probabilistic data structure that is used to test whether an element
is a member of a set. For example, checking availability of username is set membership problem,
where the set is the list of all registered username.
Working of Bloom Filter

● Bloom Filters work by using a bit array, typically initialized with all bits set to 0, and a set of hash
functions.
● When an element is added to the Bloom Filter, it undergoes hashing through each of the hash
functions, which produce a set of indexes in the bit array. These indexes are then set to 1.
● To check if an element is present in the Bloom Filter, it undergoes the same hashing process.
● If all the corresponding bits in the array are set to 1, the filter indicates that the element may be
present in the set.
● However, if any of the bits are 0, then the element is definitely not in the set.
Benefits

● Space-efficient (uses very little memory)

● Fast lookups (constant-time membership check)
● No false negatives (guaranteed correctness when saying "not
present")
● Reduces costly operations (avoids unnecessary DB or network
queries)
● Scalable (tunable trade-off between space and false positives)
Drawbacks
● False positives possible (may say an element exists when it doesn’t).
● No deletion (standard Bloom filter can’t remove items once added).
● Fixed size (must know expected dataset size in advance).
● Cannot retrieve items (only tells presence/absence, not the actual data).
● False positive rate grows as more items are inserted beyond the
planned capacity.
[Link] -Agent-Supervisor
The Scheduler-Agent-Supervisor (SAS) pattern is a system design approach for managing distributed tasks in big data
environments. It separates task scheduling from task execution, allowing for efficient resource allocation and improved
system resilience.

1. Scheduler

● Decides what task should run, where, and when.

● Assigns jobs to agents/workers based on policies (e.g., load balancing, priority, availability).
● Ensures efficient resource usage.

2. Agent

● Runs on each worker node (or machine).

● Receives tasks from the scheduler and executes them.
● Reports task status (success/failure, progress) back to the supervisor or scheduler.

3. Supervisor

● Monitors and manages agents.

● Ensures agents are alive, healthy, and performing tasks correctly.
● Restarts or reschedules tasks if an agent fails.
● Scheduler = Brain (decides the
plan)

● Agent = Worker (executes tasks)

● Supervisor = Watchdog (monitors

workers)
Benefits and Drawbacks

Benefits:
● Efficient resource use – Scheduler ensures balanced workload.
● Fault tolerance – Supervisor detects failures and can restart tasks.
● Scalability – Multiple agents can be added easily to handle more load.
● Separation of concerns – Clear roles: scheduling, execution, monitoring.
● Improved reliability – Failures in one agent don’t bring down the system.

Drawbacks:
● Complexity – More components → harder to design and maintain.
● Overhead – Communication between scheduler, agents, and supervisor adds latency.
● Single point of failure – If scheduler/supervisor fails (without redundancy), the system halts.
● Resource cost – Supervisors and schedulers need extra resources beyond workers.
9. Pipes & Filters
● The Pipe and Filter architecture is a design pattern that divides a process into a series of distinct steps,
called filters, linked by channels known as pipes. Each filter is dedicated to a particular processing
function, whether it involves transforming, validating, or aggregating data.
● Data flows through these filters via pipes, which carry the output of one filter to the input of another.
Benefits and drawbacks
Benefits:
● Reusability – Filters are independent and can be reused in different pipelines.
● Modularity – Easy to modify, add, or replace filters without affecting others.
● Scalability – Can parallelize filters or distribute them across systems.
● Clarity – Data processing flow is easy to understand as a sequence of steps.
● Maintainability – Easier debugging and testing of individual filters.

Drawbacks:
● Performance overhead – Data copying between pipes/filters can slow things down.
● Latency – Multiple stages add processing delays.
● Rigid structure – Strict linear flow may not suit all applications.
● State handling is hard – Filters are stateless by design; maintaining state across filters can be complex.
[Link] in Distributed Systems

● In a distributed system, multiple nodes (computers/servers) need to agree on a

single value or decision, even if some nodes fail or send conflicting information.
● Consensus is the process that ensures all correct nodes eventually agree on the same
value.
● A consensus protocol for distributed system is correct if t satisfies the following
properties:
a. Agreement: The nodes in a distributed system should agree on some value.
b. Validity: Only a value that has been proposed by some node must be chosen.
c. Termination: All nodes must eventually agree on some value
Paxos
Paxos is a family of protocols developed by Leslie Lamport for achieving consensus in distributed systems
despite network delays, node failures, and message losses. Paxos ensures that all nodes agree on a single
value even if some nodes fail.
● The protocol involves proposers, acceptors, and learners. Proposers suggest values, acceptors agree
on a value, and learners learn the agreed value.
● The PAXOS consensus algorithm operates in phases:
a. Prepare Phase: A node proposing a value initiates this phase by sending other nodes a
"prepare" message.
b. Promise Phase: Nodes respond with a "promise" message, indicating they won't accept
proposals with a lower number (proposal ID).
c. Accept Phase: The proposer sends an "accept" message with the proposed value to the nodes.
Nodes that were promised in the previous phase accept the value.
Advantages of Consensus in Distributed Systems
● Fault Tolerance – Even if some nodes fail or act maliciously, consensus ensures the system still agrees on a
single result.

● Consistency – Provides a uniform view of data across distributed nodes, avoiding conflicts.

● Reliability – Ensures critical operations (like committing transactions or leader election) are correctly agreed
upon.

● Coordination – Enables synchronization across nodes for distributed tasks.

● Foundation for Services – Consensus protocols (like Paxos, Raft, PBFT) form the basis of reliable
distributed databases, blockchain systems, and coordination services (e.g., Google’s Chubby, Apache
Zookeeper).

● Atomic Commit – Guarantees "all-or-nothing" execution for distributed transactions.

Limitations of Consensus in Distributed Systems
● Complexity – Protocols (like Paxos, Raft) are difficult to design, implement, and maintain.

● Performance Overhead – Requires multiple communication rounds and acknowledgments, increasing latency.

● Scalability Issues – As the number of nodes grows, reaching consensus becomes slower and more
resource-intensive.

● Network Dependence – Consensus relies heavily on reliable communication; network partitions can prevent
progress.

● Impossibility in Asynchronous Systems – According to the FLP impossibility result, in a purely

asynchronous system, deterministic consensus cannot be guaranteed if even one node may fail.

● Energy/Resource Usage – Some consensus mechanisms (like Proof-of-Work in blockchains) consume

significant computational power.
Map Reduce
MapReduce is a programming model (introduced by Google) used for processing and generating large data sets in a distributed and
parallel way.
It breaks down tasks into two main phases:

1. Map Phase → Splits input data into chunks and processes them in parallel to produce intermediate key-value pairs.

2. Reduce Phase → Collects all values for the same key, processes (aggregates/summarizes) them, and produces the final result.
Workflow
1. Input Splitting → Large data split into smaller blocks (e.g., 128MB).

2. Map Function → Applied on each block → produces key-value pairs.

3. Shuffle & Sort → Groups intermediate data by keys.

4. Reduce Function → Processes grouped data → outputs final results.

5. Output → Written back to distributed storage (like HDFS in Hadoop).

Map Reduce
Combiners in MapReduce
● A combiner is a mini-reducer that runs after the Map phase but
before the Reduce phase.
● Its job is to reduce the volume of intermediate data transferred
across the network (called shuffle phase).
● It optimizes performance by doing local aggregation on each
mapper’s output.

Eg:

Input: "cat cat dog"

Map Output: (cat,1), (cat,1), (dog,1)

Combiner Output: (cat,2), (dog,1) ← local aggregation

Reduce Final Output: (cat,total_count), (dog,total_count)

Matrix Multiplication by Mapreduce
Example
Mapreduce Patterns

MapReduce patterns are reusable templates for solving common data

processing problems using the MapReduce framework. They define how the
Map, Shuffle, and Reduce phases are structured to solve specific types of
problems efficiently.
Numeric Summarization
The Numeric Summarization Pattern is a MapReduce design pattern used to compute
aggregates of numeric data such as:
● Sum
● Count
● Minimum
● Maximum
● Average
● Standard deviation, etc.

It is one of the most common Summarization Patterns, and its goal is to take a large volume
of numeric records and generate statistical summaries in a distributed and parallel manner.
Use Cases
This pattern is widely used in big data analytics, such as:

● Finding total sales, average sales per region, etc.

● Counting website visits, click-through rates, etc.

● Computing min, max, average temperature from weather datasets.

● Analyzing sensor data from IoT devices.

● Computing standard deviation or variance for large datasets.

 Top-N pattern in Mapreduce

Find the top N elements (e.g., top 10 customers by revenue).

Approach:

1. Mapper keeps a local priority queue of top N elements.

2. Emit only top N from each mapper.

3. Reducer merges results and selects final top N.

Filtering Pattern in MapReduce
The Filtering Pattern in MapReduce is used to select, reject, or transform data based on some
condition or rule.
It acts like a data sieve, allowing only records of interest to pass through, while discarding
unwanted ones.
Filtering is critical in big data processing, as it reduces the amount of data that needs to be
processed downstream, which:
● Improves performance

● Minimizes network overhead

● Reduces storage and computational costs

Distinct Pattern in MapReduce

● The Distinct Pattern in MapReduce is used to find and output unique

records from a large dataset by eliminating duplicates.
● It is similar to the SQL SELECT DISTINCT operation, but it is implemented in
a distributed and parallel fashion for big data.
● Many real-world datasets contain duplicate records, which must be removed
to:
● Clean and preprocess data before analysis.
● Reduce storage cost by removing redundant records.
● Improve accuracy of downstream analytics like machine learning models
● The goal is to group identical records together and output only one
occurrence of each unique record.
Binning Pattern in MapReduce
The Binning Pattern in MapReduce is a data organization pattern used to categorize, group, or partition
data into multiple "bins" (categories) during the mapper phase, based on a specific rule or attribute.
It is similar to sorting items into different "buckets" where each bucket contains records belonging to a
specific group.

This pattern is especially useful when:

● You want to split data into multiple output files in a single MapReduce job.
● Each bin represents a logical subset of the data (e.g., by region, date, or status).

Think of a mail sorting center:

● Letters are sorted into bins based on postal codes.

● Each bin eventually goes to a different truck or route.

Similarly, in MapReduce, records are sorted into bins (categories) based on some attribute.
Inverted Index Pattern in MapReduce
The Inverted Index Pattern is a fundamental MapReduce pattern used to create searchable structures
where terms (words) are mapped to the documents they appear in.
It is the backbone of search engines like Google, Bing, and other information retrieval systems.

An inverted index is like the index at the back of a textbook:

● In a book, the index tells you which pages a topic appears on.
● Similarly, in big data, an inverted index tells you which documents contain a specific word or
term.

It enables fast keyword searches across massive datasets, forming the core of search engines and
information retrieval systems.

By combining preprocessing (stop-word removal, stemming) and optimizations (combiners, partitioning), it

scales efficiently to handle billions of records.
Sorting Pattern in MapReduce
The Sorting Pattern in MapReduce is one of the most fundamental patterns, used to
arrange large-scale data in a specific order — either ascending or descending, based on
keys or values.
MapReduce is naturally good at sorting, as Hadoop automatically sorts the
intermediate keys during the shuffle and sort phase.
Sorting is a core building block for many complex big data problems, including:
● Ranking
● Deduplication
● Secondary sorting
● Analytics pipelines
Joins in MapReduce
Joining datasets is a core operation in big data processing. Just like SQL joins
in relational databases, MapReduce joins combine two or more datasets based
on a common key, such as customer IDs, order IDs, or product IDs.

However, because MapReduce processes data in parallel and distributed

systems, joins must be carefully designed to minimize data movement and
maximize efficiency.
Types
Distance Measures
Distance Measures
● Distance measures are mathematical methods for quantifying the similarity or dissimilarity between
data points, which is essential for clustering, classification, information retrieval, and many other
data analysis tasks.
● They are fundamental in tasks like clustering, classification (k-NN), recommendation systems,
anomaly detection, and dimensionality reduction.

The choice of distance depends on the nature of the data and the application domain.
● Used to quantify similarity/dissimilarity between objects.
● Smaller distance = higher similarity, larger distance = higher dissimilarity.
● Important for clustering algorithms (e.g., K-means, hierarchical clustering).
● Choice of distance is context-dependent (numerical, categorical or text data).
1. Euclidean Distance

● Euclidean distance is considered the

traditional metric for problems with
geometry. It can be simply explained as
the ordinary distance between two points.
It is one of the most used algorithms in the
cluster analysis.
● Best for: Continuous numerical data
(when features are normalized).
● Example: Distance between two cities on
a 2D map.
 2. Jaccard Index

● The Jaccard distance is set-based

distance that compares dissimilarity by
looking at the ratio of unique to common
elements.
● Best for: Binary or categorical data,
especially sets.
● Example: Comparing similarity of
shopping carts or tag sets.
3. Cosine Similarity / Cosine Distance
● Measures the cosine distance of the angle
between two vectors, focusing on
orientation rather than magnitude.
Commonly converted to distance as
1−similarity.
● Best for: Text mining, NLP,
recommendation systems.
● Example: Measuring similarity between
two documents regardless of their length.
4. Hamming Distance
● The number of positions where two strings (of equal length) differ.
Commonly used for error detection and sequence comparison.
● Best for: Binary strings, DNA sequences, error correction.
● Example: Hamming distance between “karolin” and “kathrin” = 3.
5. Edit Distance
Given two strings s1 and s2 and below operations that can be performed on s1. The task is to find
the minimum number of edits (operations) to convert 's1' into 's2'.
● Insert: Insert any character before or after any index of s1
● Remove: Remove a character of s1
● Replace: Replace a character at any index of s1 with some other character.

Example:
Input: s1 = "gfg", s2 = "gfg"
Output: 0
Explanation: Both strings are same.
Input: s1 = "abcd", s2 = "bcfe"
Output: 3
Explanation: We can convert s1 into s2 by removing 'a', replacing 'd' with 'f' and inserting 'e' at the
end.