Distributed File Storage System (Case Study)
Definition/Problem: In a distributed file storage system, files are split
and stored across many servers in a network so that users can access
them as if they were local[1]. The goal is to store files across
multiple nodes with replication, ensuring scalability and fault
tolerance. This allows high availability: if one node fails, other nodes
with replica copies serve the data[2][3]. In practice, users should see a
unified namespace even though data is physically distributed[4][1].
Key Concepts – Data Partitioning: Large files or datasets are
divided into smaller chunks or blocks that are placed on different
nodes[5]. Partitioning enables horizontal scalability (adding more
nodes) and parallel access[5]. For example, HDFS (Hadoop Distributed
File System) splits a file into fixed-size blocks (commonly 128 MB) and
stores these blocks on different data nodes[6]. Benefits include faster
data access (reads/writes on smaller blocks) and better load
distribution[5][6].
Key Concepts – Replication: Each data block is kept on multiple
nodes (replicas) to prevent data loss[7]. Replication ensures fault
tolerance: if a node or disk fails, other replicas have the same data[8]
[3]. For instance, HDFS uses a default replication factor of 3 (three
copies of each block)[6]. Replication also improves read throughput by
allowing parallel reads of different replicas. However, it uses extra
storage and network bandwidth.
Key Concepts – Consistency: Consistency determines how updates
propagate to replicas. A consistency model defines this (e.g. strong
vs. eventual consistency)[9]. Strong consistency means all clients
always see the latest data, while eventual consistency may allow
temporary divergence[10]. Many DFS simplify consistency by using a
write-once/read-many model (no in-place edits)[11]. For example,
HDFS writes each file sequentially and then only allows reading or
appending, avoiding complex concurrency. In trade-offs, strict
consistency can reduce availability or throughput in large-scale
systems[9][11].
System Design/Architecture
Master-Worker Model: A common design is a master node
(metadata server) and many storage nodes (data nodes)[12][13].
The master (often called NameNode) maintains the file namespace,
directory hierarchy, and a map of blocks to nodes[12]. Each data node
stores blocks and handles actual read/write of file data[12][14]. Clients
first contact the master to locate blocks, then connect directly to the
appropriate data nodes for I/O.
Data Placement: When storing a file, the master splits it into blocks
and assigns each block to specific data nodes[6]. The master also
instructs data nodes how many replicas to create and on which nodes
(often considering rack-awareness for fault tolerance). Each data node
writes the block to its local disk and replicates it as directed[6][3].
Fault Handling: Data nodes send periodic heartbeat signals to the
master. If a node fails (heartbeat lost), the master marks its blocks as
under-replicated and instructs other nodes to create new replicas to
restore the desired replication factor[15]. This automatic re-replication
maintains durability. For example, HDFS continuously monitors block
health and re-replicates lost blocks upon failures[15][16].
Example Workflow (HDFS): In HDFS, a client writing a file triggers
the master to allocate new blocks and define a pipeline of data nodes.
The client streams data packets through this pipeline: first to one data
node, which forwards to the next, and so on[6][3]. Each block is then
written to multiple data nodes. This pipeline replication ensures that
even if one node fails, other copies exist[3][17].
Figure: Data write pipeline in a distributed file system (e.g., HDFS). The
NameNode splits a large file into blocks and the client writes each block
through a chain of DataNodes (default 3 replicas)[6][3]. Replicated blocks
(shown in blue) on separate machines ensure fault tolerance and high
availability[17].
Data Read: Reading is similar but simpler. The client asks the master
which nodes have the desired blocks, then reads the blocks directly
from a nearby (or least loaded) data node[18]. Blocks may be read
from any of their replicas.
Consistency and Concurrency: To avoid conflicts, many DFS use
simple models: HDFS, for instance, does not allow multiple writers to
the same file concurrently[11]. In systems that allow updates,
protocols (like leases or quorum writes) enforce consistency.
Related Systems / Examples
Google File System (GFS): An early DFS used at Google, designed
for large-scale search data. GFS splits files into large chunks (e.g.
64 MB) and stores 3 copies on chunkservers[19]. It introduced ideas
like a single master for metadata and relaxed consistency, inspiring
many later DFS designs.
HDFS (Hadoop DFS): The open-source Hadoop File System is based
on GFS principles[20]. It uses a single NameNode (master) and many
DataNodes, on commodity Linux servers[21][13]. HDFS is fault-tolerant
via block replication and is optimized for high-throughput batch
processing of large data sets[8][22]. For example, HDFS can scale to
thousands of nodes and petabytes of data on inexpensive
hardware[22].
Ceph: A modern open-source distributed storage platform that
provides object, block, and file interfaces. Ceph uses a decentralized
design (no single metadata master for data) and the CRUSH algorithm
to place and replicate data across OSD nodes[23]. It achieves high
scalability (thousands of nodes, petabyte scale) by intelligently
distributing data and managing replicas via its monitor and OSD
daemons[23].
Other Examples: Distributed file stores also include systems like
Amazon S3 (cloud object storage, not POSIX but for similar use-cases),
Lustre (HPC file system), GlusterFS, and distributed block stores. Each
uses partitioning/replication strategies to meet scale and reliability
goals.
Hardware and Software Requirements
Hardware: Typically built on commodity servers with large-capacity
storage drives and network connectivity. For example, an HDFS cluster
uses one or a few dedicated NameNodes (with high RAM/CPU) and
many DataNodes (commodity hardware)[24][13]. High network
bandwidth (e.g. 10 GbE or more) is important to handle replication
traffic. Servers run common OS like GNU/Linux.
Software: Requires an operating system (commonly Linux), and DFS
software stack. E.g., HDFS requires a Java runtime on each node[24].
The NameNode and DataNode run distributed file system daemons.
Additional software may include tools for cluster management,
monitoring, and client libraries.
Other Needs: A network file system or local file system (e.g. ext4) for
storing block data on each node. Consensus services or metadata
databases might be needed in some designs. Security (Kerberos, TLS)
can be layered for authentication and encryption.
Summary
A distributed file storage system partitions data and replicates it
across multiple nodes to achieve scalability and resilience[5][7].
The problem statement is to store and manage files over many
machines with replication for fault tolerance. This is solved by dividing
files into chunks, distributing them, and keeping multiple copies[5][6].
Data partitioning allows handling large datasets by spreading
workload, while replication provides high availability[7][8].
Consistency mechanisms ensure users see correct data; many
systems trade strict consistency for performance (e.g. using write-once
semantics)[9][11].
Existing systems like HDFS (based on GFS) and Ceph demonstrate
these principles at massive scale[20][23].
References: - Scott Robinson & Ryan Arel, “What is a distributed file system
(DFS)?” TechTarget (2025)[1][2]
- Apache Hadoop, HDFS Architecture Guide[12][24]
- GeeksforGeeks, “Explain the concept of data partitioning…” (2024)[5]
- GeeksforGeeks, “Data Replication Strategies in System Design”[7]
- GeeksforGeeks, “Introduction to HDFS”[6][3][17]
- Databricks Blog, “What is HDFS?” (2024)[25][21]
- IBM, “What is Hadoop Distributed File System (HDFS)?”[22][16]
- Ceph Documentation, Ceph Architecture[23]
- Arteaga & Schleithoff, “File Systems: GFS vs HDFS vs Ceph” (TU Munich,
2020)[19][4].
[1] [2] What is a Distributed File System (DFS)? | Definition from TechTarget
[Link]
DFS
[3] [6] [13] [14] [15] [17] Introduction to Hadoop Distributed File
System(HDFS) - GeeksforGeeks
[Link]
file-systemhdfs/
[4] [19] [Link]
[Link]
[5] Explain the concept of data partitioning and its importance in distributed
systems. - GeeksforGeeks
[Link]
data-partitioning-and-its-importance-in-distributed-systems/
[7] Data Replication Strategies in System Design - GeeksforGeeks
[Link]
system-design/
[8] [16] [22] What is Hadoop Distributed File System (HDFS)? | IBM
[Link]
[9] [10] Consistency Model in Distributed System - GeeksforGeeks
[Link]
distributed-system/
[11] [12] [24] HDFS Architecture Guide
[Link]
[18] Anatomy of File Read and Write in HDFS - GeeksforGeeks
[Link]
write-in-hdfs/
[20] [21] [25] What is a Hadoop Distributed File System (HDFS)? | Databricks
[Link]
[23] Architecture — Ceph Documentation
[Link]