System Design

Outline
• System Design
• Core Concepts in System Design
• Trade-offs
• System Requirements
• Core building blocks in system design
System Design
• System design is the process of defining the architecture, modules,
interfaces, and data for a system to satisfy specified requirements

• strategically planning how different components will interact and

function together.
Core Concepts in System Design
• Abstraction : Developers focus on what a component does rather than how it
does it.
• Modularity : Modularity involves breaking down a system into smaller,
independent modules, each responsible for a specific function.
• This approach enhances maintainability, testability, and reusability.

• Decomposition: Decomposition is the process of breaking down a complex

problem or system into smaller, more manageable sub-problems or components.
• This allows for easier understanding, development, and maintenance

• Interface : An interface defines the contract between different modules or

components, specifying how they interact with each other.
• It promotes loose coupling and allows for independent development and
evolution.
Trade-offs
• System design is full of trade-offs. Choosing one design approach
often means sacrificing something else. Understanding these trade-
offs is crucial for making informed decisions.
• Common trade-offs include
• Latency vs. Throughput
• Consistency vs. Availability
• Cost vs. Performance
• Scalability vs. Complexity
System Requirements : Scalability
• Scalability refers to a system's ability to handle an increasing amount of
work or load.
• A scalable system can maintain acceptable performance levels even as the
demand on the system grows.
• Type
• Vertical Scalability
• Horizontal Scalability
• Scalability Metrics
• Latency / Response Time
• Throughput
• Resource utilization
• Note: A scalable system should be able to maintain a consistent level of
performance (low latency and high throughput) even as the load increases,
without experiencing a dramatic increase in resource utilization.
System Requirements : Scalability (contd…)
• Type
• Scalability Metrics
• Scalability Strategies
• Load Balancing
• Caching
• Database Shading
• Asynchronous Processing
System Requirements : Reliability
• Reliability refers to a system's ability to operate correctly and consistently over a given period of
time. A reliable system is designed to withstand failures and errors, and to recover quickly from any
disruptions. It should provide consistent and predictable behavior, even in the face of unexpected
events.
• It should consider
 Fault Tolerance: The ability of a system to continue operating correctly even when one or more of its
components fail. Example: Using redundant servers or data centers to provide failover in case of a
hardware failure.
 Error Handling: The ability of a system to detect and handle errors gracefully, without crashing or
corrupting data. Example: Implementing input validation to prevent malicious data from being
injected into the system.
 Data Consistency: Ensuring that data remains consistent and accurate across all parts of the system,
even in the presence of failures. Example: Using transactions to ensure that multiple database
operations are performed atomically, either all succeeding or all failing together.
 Recovery: The ability of a system to restore itself to a working state after a failure. Example: Having a
backup and restore plan for the database, so data can be recovered in case of corruption or loss.
System Requirements : Reliability (contd…)

• Strategies
 Redundancy: Duplicating critical components of the system to provide backup in case of failure.
Example: Using RAID (Redundant Array of Independent Disks) to provide data redundancy at the
disk level.

 Replication: Creating multiple copies of data and distributing them across different locations.
Example: Replicating the database across multiple data centers to provide disaster recovery.

 Monitoring and Alerting: Continuously monitoring the system for errors and anomalies, and
alerting administrators when problems are detected. Example: Using tools like Prometheus and
Grafana to monitor system metrics and alert on high CPU usage or low disk space.

 Testing: Thoroughly testing the system under various conditions to identify and fix potential
problems. Example: Performing load testing to ensure the system can handle peak traffic, and
fault injection testing to simulate failures and verify recovery procedures.
System Requirements : Availability (contd..)
• Availability refers to the percentage of time that a system is operational
and accessible to users.
• Availability = MTBF / (MTBF + MTTR)

• Factors affecting Availability

• Downtime: Any period of time when the system is not operational, whether due to
planned maintenance or unplanned outages.
• Mean Time Between Failures (MTBF): The average time between failures of a
system component.
MTBF=total operational uptime between failures / number of failures
• Mean Time To Recovery (MTTR): The average time it takes to restore a system to a
working state after a failure.
System Requirements : Availability (contd..)
• Availability of n homogeneous servers = 1- (1-Ax)n
Ax = Availability of one server
Availability Example
• A web service running on a single server. On average, the server fails
once every ~1000 hours of operation. When a failure occurs, it takes
about 10 hours to detect, repair, and restore server.
• MTTR = 100 Hrs
• MTBF = 1000 HRs
• Availability = 1000 / (1000+100) = 90%
• If two such servers are deployed in parallel (failover setup) where the
system is available as long as at least one server is running,
• Availability = (1- (1-0.9)2) = 99%
Strategies to improve Availability
• Redundancy: As with reliability, redundancy is crucial for availability.
Duplicating critical components allows the system to continue operating
even if one component fails.
• Monitoring and Alerting: Proactive monitoring and alerting can help
detect and resolve issues before they cause downtime.
• Automated Failover: Implementing automated failover mechanisms allows
the system to automatically switch to a backup component in case of a
failure.
• Rolling Updates: Performing software updates in a rolling fashion, where
only a subset of the servers are updated at a time, minimizes downtime.
• Disaster Recovery Planning: Having a comprehensive disaster recovery
plan ensures that the system can be quickly restored in case of a major
Availability Case study
• Example 1: Google Search: Google Search is designed to be highly
available, with multiple data centers and automated failover
mechanisms. This ensures that users can almost always access search
results, even in the event of a major outage.
• Example 2: Amazon Web Services (AWS): AWS provides a wide range
of services with high availability guarantees. They achieve this
through redundancy, replication, and automated failover.
The core concepts and fundamental building
blocks of system Design
• Client Server • WebSockets • CAP Theorem
Architecture • Webhooks • Monolithic architecture
• Proxy / Reverse Proxy • Load Balancer • Microservices
• Single Sever • Horizontal Scalability • Message Queue
• Multiple Servers • Vertical Scalability • Rate Limiter
• API • Sharding • BLOB store
• API Styles • Caching • index
• REST • CDN
• GraphQL • geoDNS-routing
Client Server Architecture
• Client send request to server using
domain name
• Browser send domain name to DNS
server
• Every Server has unique identifier (IP
Address and Host (domain) Name)
• DNS server maps host name with IP
address
• IP address of any domain name can
be identified using ping command
Proxy Servery
• Proxy server keeps identity private
• Proxy server forward request to
target server by keeping client’s
identity private
• Reverse Proxy intercept the client
request and forward it to target
server on private network
Single Server vs Multiple Server

Single Server Setup Multiple Server separating Web traffic and database

• Separating web/mobile traffic (web tier) and database (data tier) servers allows
them to be scaled independently
HTTP/ HTTPS
• Browser and Server communicates
using set of rules called HTTP
• HTTP request contain request type,
browser type, cookies, request data
• HTTP response contain requested data
with header and code 200 (OK), or
error (404).
• HTTP sends data in plain text
• HTTPS sends data in encrypted data
using SSL/TLS protocol
HTTP/ HTTPS
• HTTP does not define
• How request should be structured ?
• What format responses should be in?
• How different clients should interact with
server?
API (Application Programming Interfaces)
• HTTP does not define
• How request should be structured ?
• What format responses should be in?
• How different clients should interact with
server?
• API
• Client sends request to API without
worrying about low level details i.e. HTTP
header
• API reformat the requests with low level
detail to the database/web/application
server and sends back responses in
structured format (i.e. XML, JSON), which
can be understood easily by client
API Gateway • It is a server that functions as a central entry point
for all client interactions with backend services.
• It is especially valuable in microservices
architectures, where multiple services exist and
client requests need to be managed, secured,
routed, and orchestrated efficiently.
• Most Popular API styles :
• REST
• GraphQL
API Gateway • Key features :
• Single entry point
• Route requests to correct backend service
• Centralized Authentication and authorization to reduce
risk of misconfiguration security at individual service
• Rate limit and throttling to reduce abuse/overuse
• Request and Response Transformation (Add/ remove
header, convert between formats, filter payloads, modify
query parameters / request bodies)
• API Composition and Aggregation : It can Aggregate
responses from multiple services into single response
• Caching data to reduce calls and improve scalability
• Logging, Monitoring, and Analytics
• Translate between different protocols (i.e. HTTP <->
gRPC, WebSocket <-> REST, SOAP <-> REST )
• When to choose API Gateway ?
• when your application exposes multiple APIs or
microservices and you need centralized control over
security, routing, and API lifecycle management.
 REST API
• A set of guiding principles that leverage the existing HTTP protocol to enable
communication between clients and servers
• Standard architectural style for designing APIs.
• Works with fixed endpoints (e.g., /users, /products/{id}).
• Each endpoint returns a fixed data structure (defined by backend)
• Uses HTTP methods:
• GET → Retrieve
• POST → Create
• PUT/PATCH → Update
• DELETE → Remove
• Versioning often required (/api/v1/users).
• When to use ?
• Simple CRUD APIs.
• Standard web services.
• When clients have uniform data needs.
 REST API
• REST APIs typically return data in JSON and use HTTP status codes to
communicate the outcome of the request:
• 200 OK → Success
• 201 Created → Resource successfully created
• 400 Bad Request → Client error (e.g., missing required fields)
• 404 Not Found → Requested resource does not exist
• 500 Internal Server Error → Unexpected server issue
• Benefits of REST
• Simplicity and Intuitive Design
• Statelessness: making REST scalable across distributed system
• Cacheability: HTTP’s caching mechanism to improve performance
• Scalability : It can be scaled easily using load balancer and CDNs
 REST API

• Issues :
• Over-fetching leading to inefficient network usage
• Example: You need just a user’s name, but REST returns the whole user object
• Under-fetching : Multiple requests required to get relevant data
• Example : to get user details and their posts, a client might have to make:
• GET /api/users/123 (fetch user)
• GET /api/users/123/posts (fetch user’s posts)
• Versioning Issue : often require versioned URLs (/v1/users, /v2/users)
GraphQL API
• A single GraphQL endpoint (/graphql)
replaces multiple REST endpoints, allowing
clients to structure their own queries
instead of relying on predefined responses.
• Solves over-fetching and under-fetching
problems
• API responses are loosely structured and
may vary across versions
• Provides three functionalities
• Queries → Fetch Data : equivalent to GET
in REST, but allow to specify desired fields
in output
• Mutations → Modify Data : to POST, PUT,
PATCH, or DELETE in REST.
• Subscriptions → Real-Time Updates
GraphQL API
• Benefits
• Precise Data Fetching
• Single Request for Multiple Resources:
• Strong Typing: using schema definition
• Real-time Data with Subscriptions
• API Evolution Without Versioning: New
fields can be added without breaking
existing queries, avoiding REST-style
/v1, /v2 versioning issues
• Drawbacks
• Complex Setup & Tooling
• Caching challenges
• Increased Server Load
• Security risks with unoptimized queries
• Performance Risks
 API Styles : REST vs GraphQL

Aspect REST GraphQL

Endpoint Multiple (per resource) Single (/graphql)
Cantered Around resources (each identified by a URL) a schema that defines the types of data available
Data fetching Fixed by server Defined by client query
Over-fetching Common issue Avoided
Under-fetching Common (need multiple requests) Avoided (get all in one query)
Versioning Often needed (/v1/, /v2/) Rarely needed (evolves via schema)
Learning curve Easy, widely used Steeper (schema, resolvers, query)
Caching Easy with HTTP (GET) More complex, needs custom caching
Using Subscriptions, supports Real-time Data Update
Notification Client need to polling to know change
notification when data changes on server
Load Balancer (LB)
• A load balancer evenly distributes
incoming traffic among web servers that
are defined in a load-balanced set.
• helps improve the overall performance,
availability, fault tolerance, and scalability
of applications
• Key features :
• Traffic distribution based on various algorithms
like Round Robin, Least Connections, weighted
distribution on more powerful servers
• Continuous Health Check of backend servers
• Session persistence
• SSL Termination
• High Availability and Failover
• Types
• Transport Layer Load Balancer : without
inspecting content
• Application Layer Load Balancer : Based on
content , HTTP methods
Database replication
• Master-slave replication
• Advantages
• Better Performance
• Better Reliability
• Better Availability
Database Scaling :
Horizontal scaling and Vertical Scaling
Database : Sharding
• Sharding separates large databases into
smaller, more easily managed parts called
shards.
• sharding is about distributing data across
multiple servers
• Distributing data in this way makes it
easier to scale and manage more users
• Each shard shares the same schema,
though the actual data on each shard is
unique to the shard.
 Latency and Datacentre
• High latency make application feel slow and unresponsive
• Physical distance between client and Server affects latency of client’s request
• Deploying services across multiple data centres help to improve latency
• Generally, users are geoDNS-routed to closest data centre
• Improve performance
• Challenges
• Traffic redirection
• Data Synchronization
CDN (Content Delivery Network)

• A CDN is a geographically distributed network of servers that work together to deliver web content to
users based on their geographic location
• When a user requests content from a website, the CDN redirects the request to the nearest CDN edge
server in its network, reducing latency and improving load times
• +ve: Faster load time, reduce server load, improve availability, reliability and scalability
• -ve: Increase cost by CDN providers , Increase complexity of DNS configuration, cache rules, content
validation policies
 Message Queue

• Once the consumer processes the message, it may send an acknowledgment back to
the broker, confirming that the message has been successfully handled.
• After acknowledgment, the broker removes the message from the queue to prevent it
from being processed again.
• Decoupling feature allow producers and consumers to operate independently
• Asynchronous processing improve throughput, allow systems to scale horizontally
• Multiple consumer can pull message , allowing work to be distributed
• Fault tolerance : If producer/consumer fails, messages are not loss
• Popular Message Queue Systems : RabbitMQ, Apache Kafka, Amazon SQS, Google
cloud Pub/Sub
CAP Theorem – 3 pilers

•.
CAP Theorem
• It is impossible for a distributed data
store to simultaneously provide all three
guarantees.
• Partition tolerance is must requirement
in distributed system. Trade-off between
consistency and availability.
• Most NoSQLs are distributed database
and highly horizontally scalable. Most
belongs to CP / AP
• Relational Databases are mostly
centralized and ensuring consistency
and availability.
Few important Building blocks of data science
system design
• Data Pipeline
• Data Ingestion
• Batch Processing
• Real time processing
• Lambda processing
 • Processing data in bulk at scheduled intervals such as
daily, weekly etc.

Batch Processing • Since large volumes of data are processed together, it

achieves high throughput.
• Workflow:
• Data collection in DB/ files
• Pre-processing : validation, cleaning, filtering, aggregation
• Batch Execution :
• Computation , ETL, Aggregate/ transforming
• Post-processing : sent to DB / other system
• Job Completion : prepare for next batch
• Challenges :
• High Latency due to long collection time
• High storage spare required to store large data
• Generate Peak resource usage situation , which may slow
other processes
• An error may cause entire batch to be reprocessed
• E.g.
• Generating end-of-day reports.
• Processing payroll at the end of the month.
• Tools/Frameworks : Apache Hadoop, Spark, AWS Batch
 Stream Processing
• Processing data in real time or near real time.
• Popular with the rise of real-time data and the need for immediate insights.
• Key characteristics : Low Latency, Real-time processing, Event-driven, Work on continuous flow
• Workflow:
• Data ingestion of contiguous flow of data from one or more sources i.e. Message brokers, sensors, log
streams
• Processing/ Transformation : Filtering , aggregation, windowing, enrichment
• State Management : Many stream processing tasks require the system to maintain state (e.g., aggregating
transactions per user).
• Output: notification, update real-time dashboards, real-time analytics
• Challenges :
• Maintaining data consistency in real time in distributed system
• Complexity to handle scalability, fault tolerance, error handling
• E.g.
• Monitoring sensor data in IoT systems.
• Detecting fraud in financial transactions in real-time.
• Tools/Frameworks : Apache Hadoop, Flink, AWS Kinesis
Lambda Architecture
Case Study
• Design a Basic Search Engine (Google or Bing) | System Design
Interview Prep