Week 8 Assignment
S.M Barkat Ali (20b-047-cs)
Pipeline Architecture Design
This design follows a decoupled, scalable, and fault-tolerant "lambda-style" architecture,
handling both real-time streaming and batch data processing.
Step 1: Data Arrives (The Left Side of the Picture)
What happens: Data comes from many places like phones, websites, or sensors. This is
shown in the "External Sources" box.
Google Pub/Sub's Job: This is our receiving dock. Its only job is to accept massive
amounts of data very quickly and hold it safely. It makes sure no data is lost, even if the
next part of the factory is busy or broken. It's like a giant, never-full waiting room.
Step 2: Data is Sorted and Stored (The Middle Section)
Kubernetes' Job: This is the factory's boss and its workforce. Kubernetes is a system that
runs all our little programs (called containers). It automatically starts them, stops them,
and makes sure they are healthy.
o The Ingestion Service (one of Kubernetes' workers) takes data from the Pub/Sub waiting
room and does a quick check. Then, it puts the raw, unprocessed data into a MinIO
"raw-landing" storage bucket. Think of this as the factory's raw materials warehouse.
MinIO’s Job: This is our simple and reliable storage system (our warehouse). It has two
main rooms:
1. Raw-landing bucket: This holds the original, untouched data forever.
2. Processed-curated bucket: This holds the finished, useful data.
Step 3: Data is Processed (The Factory Floor)
This happens inside the Kubernetes box, using two types of workers:
Stream Processor (e.g., Flink/Spark): This worker is like a fast assembly line. It processes
data the moment it arrives, doing things like counting website clicks in real-time. It gets
data directly from the Ingestion Service or from Pub/Sub.
Batch Processor (e.g., Spark): This worker is like the night shift. It waits until a large
amount of data has piled up in the raw-landing bucket (e.g., overnight) and then
processes it all at once in a big, efficient batch.
Both workers put their finished, useful data into the MinIO processed-curated bucket.
Step 4: Data is Used (The Right Side of the Picture)
The finished data in the processed-curated bucket is now clean, organized, and ready
for people to use. Other systems (like dashboards, reports, or AI models) can easily
access it from here.
Role of Key Cloud Components
1. Google Pub/Sub
Decoupling: It completely decouples data producers from consumers. The apps sending
data don't need to know anything about MinIO, Spark, or Kubernetes. They just send to
a topic. This allows either side to be updated, fail, or scale independently.
Buffering & Resilience: It acts as a shock absorber for traffic spikes. If the processing
layer falls behind or crashes, messages are retained and processed once it recovers,
preventing data loss.
Scalability: It is a fully managed service that can handle from thousands to millions of
messages per second without any operational overhead.
2. MinIO
S3 Compatibility: Its API compatibility with Amazon S3 means a vast ecosystem of tools
(Spark, Flink, and Python scripts) can work with it seamlessly without code changes.
Durability & Cost: It provides highly durable storage for the raw data, which is crucial for
reprocessing scenarios. Separating raw and processed data allows for different
cost/performance profiles (e.g., cheaper storage class for raw data).
Cloud Agnostic: It can run on-premises or on any cloud, preventing vendor lock-in and
providing data sovereignty.
3. Kubernetes
Unified Platform: It provides a single, consistent API to deploy and manage all the
diverse components of the pipeline (stateless services, complex processing frameworks).
Operational Efficiency: It automates deployment, rolling updates, secret management,
and resource allocation (CPU/Memory requests/limits), and drastically reducing
operational toil.
Portability: The entire pipeline, defined as Kubernetes manifests (YAML files) and Docker
images, becomes highly portable. It can be deployed on GKE (GCP), EKS (AWS), AKS
(Azure), or an on-premise cluster with minimal changes.