2020 IEEE 6th International Conference on Computer and Communications

A High-Performance Distributed File System for Mass Data

Shuaiyu Jin*, Qing Yan, Yue Zhang, Jie Yang,

Beijing Laboratory of Advanced Information Networks
Beijing University of Posts and Telecommunications
Beijing, China
e-mail: shuaiyujin@[Link]
2020 IEEE 6th International Conference on Computer and Communications (ICCC) | 978-1-7281-8635-1/20/$31.00 ©2020 IEEE | DOI: 10.1109/ICCC51575.2020.9344887

Abstract—With the rapid development of the mobile Internet x We use consistent hashing [4] to ensure load
and cloud computing, storage sub system's pressure increasing balancing, and accelerate system I/O performance
fast as the critical process needs more powerful engines. The through technologies such as memory cache and
paper realizes a high-performance distributed file system pipelines.
based on raft protocol. Our file system guarantees the x FUSE interface [5] is implemented for the users to
reliability of data through replication. Write-Ahead Logging mount and work on files.
(WAL) and raft consensus algorithm provide atomicity,
The remainder of this paper is organized as follows. In
durability and file distribution. In order to solve the problem
of data fragmentation in distributed systems, we design an
section II, we review related work briefly and compare the
automatic fragmentation and load balancing mechanism based characteristics of different distributed storage systems. In
on consistent hashing. Each file goes through the raft state section III, system overview is presented. In section IV, we
machine so that a few machines crash without I/O error. We introduce the key implementation and related performance
also implement the FUSE interface for the users to mount and optimization of each module. Performance measurements
work on files. Performance measurements under some typical under some typical workloads are shown in section V.
workloads show that our file system has excellent I/O Finally, conclusion and future work are presented in section
performance. VI.

Keywords-distributed file system; raft consensus algorithm; II. RELATED WORK

replication; fuse interface Most file systems are stand-alone, providing access and
I. INTRODUCTION management for one or more storage devices within an
operating system. With the rapid development of the internet,
With the rapid development of the mobile Internet, stand-alone file systems are facing many challenges such as
Internet of Things, and cloud computing, the amount of data sharing, capacity, performance and reliability [6]. Some
is also growing by leaps and bounds, to the point where the distributed file systems have emerged to meet these
American IT research company IDC predicts that the total challenges.
global volume of data produced and stored will reach 44 ZB x Google’s GFS [7] is a pioneer and a typical
(zettabytes; ZB = 1021 Bytes) by the year 2020. Recently, representative of distributed file systems. Many later
IDC also reports that the total global volume of data will distributed file systems inspired by its design. GFS is
grow even further to 163 ZB by the year 2025 [1]. This not optimized for large files and is not suitable for
only brings a huge market space for the data storage industry, scenarios with an average file size of less than 1MB.
but also poses an unprecedented challenge to the efficient x The Hadoop Distributed File System (HDFS) [8] is a
storage and management of massive data. distributed file system designed to run on
System designers have long sought to improve the commodity hardware. HDFS is highly fault-tolerant
performance of file systems, which have proved critical to and is designed to be deployed on low-cost hardware.
the overall performance of a wide range of applications. Our HDFS provides high throughput access to
system has many of the same goals as previous distributed application data and is suitable for applications that
file systems [2], such as performance, scalability, reliability, have large data sets. However, applications that
and availability. However, high speed reading and writing of require low-latency data access are not suitable for
files is our primary goal when the system is stable. running on HDFS.
The paper realized a high-performance distributed file x CephFS [9] began with a doctoral thesis study by
system based on raft protocol. The main contributions of this Sage Weil to implement distributed metadata
paper can be summarized as follows. management to support EB-level data scale. CephFS
x We apply the raft consensus algorithm [3] to the file maximizes the separation between data and metadata
system, files are replicated to multiple machines so management by replacing allocation tables with a
that a few machines crash without I/O error. pseudo-random data distribution function (CRUSH)
x We design an efficient file metadata structure that designed for heterogeneous and dynamic clusters of
can quickly establish hierarchical relationships in the unreliable object storage devices (OSDs). The
file system. limitations of CephFS are its high requirements on

978-1-7281-8635-1/20/$31.00 ©2020 IEEE 1804

Authorized licensed use limited to: Addis Ababa University. Downloaded on December 04,2025 at 14:22:42 UTC from IEEE Xplore. Restrictions apply.
 machine configuration and few known large-scale distributed Key-Value database to store metadata and Mnesia
success cases. distributed memory database as cache of metadata. We
Based on the experience of the existing file system, this describe the operation of the client, various node cluster, and
paper applies the raft algorithm to maintain the consistency how they work together. We also describe the status of our
of the file replication. While ensuring the high-availability of prototype and the method used to solve performance
the system, only a cheap general-purpose server problems.
configuration is required to achieve high I/O performance.
IV. KEY MODULES
III. SYSTEM OVERVIEW In this section, the implementation of key modules and
main technical details will be introduced. The use of FUSE
greatly simplified operation by avoiding the kernel and it
introduces its own set of mechanism. Raft algorithm solves
data consistency problems on multiple servers and hotspot is
avoided by consistent hashing. We also introduce the
management of metadata for the distributed file system.
A. FUSE Interface
FUSE (Filesystem in Userspace), as a loadable kernel
module on UNIX-like platforms, allows non-privileged users
to create a fully functional file system without recompiling
the kernel. The FUSE file system is usually implemented as
a standalone application linked with libfuse [10], which
provides functions to mount the file system, unmount it, read
requests from the kernel, and send responses back. We
implement most of the high-level API of FUSE shown in
Table I.
TABLE I. HIGH-LEVEL API AND FUNCTION OF FUSE
Figure 1. System architecture
High-level API Function
getattr get file metadata
The overall system architecture is shown in Fig. 1. Our
statvfs get file system space information
file system has four main components: the client, which
mkdir make directory
exposes a POSIX file system interface based on FUSE; a
rmdir remove directory
control node cluster, which manages the file metadata and
monitors the status of entire system components; and a readdir get all file information in the directory
transfer node cluster, which is responsible for data rename move or rename file
processing and transmission; a storage node cluster, which chmod change file mode
collectively stores all data. The primary goals of the chown change file owner and group
architecture are high-performance, scalability and reliability. utimens update file last access time
The file system separates file metadata management from open make and open file
the storage of file data. We use a distributed database to store link make hard link between file
metadata, avoiding the problem of metadata loss caused by a unlink remove hard link between file
single point of failure and file data is replicated to multiple release release file resources
storage servers consistently based on raft algorithm. truncate shrink or extend the size of file
Consistent hashing can avoid hotspot and ensure that the
load within the cluster is relatively balanced. The increase We also conclude some optimization schemes that can be
and decrease of nodes and hard disks will cause the problem considered when developing FUSE file systems:
of uneven data distribution. In order to make full use of the x Expand pages per write. An application program
I/O performance of physical devices, the system needs to writes to a file system developed based on Fuse, it
perform data migration spontaneously to ensure the data must first pass through the Kernel Fuse module.
distribution is balanced without affecting the normal service Kernel Fuse has authority to decide when to write
transmission. data to the user mode file system. The more
The server-side architecture is mainly constructed based frequently you write, the less efficient it is. Expand
on the idea of Raft algorithm, and the algorithm details are pages per write can improve I/O performance.
adjusted, modified and optimized according to actual needs. x Enable kernel read cache. Linux file system makes
The server uses the erlang-OTP language and framework for full use of memory to cache file data. In this way,
development, uses TCP and Protobuf to realize the applications often only need to copy data from the
communication between the client and the server-side kernel buffer to the user-mode buffer to read the file.
transmission layer and application layer, uses the Riak There is no need to proceed disk I/O at all.

1805
Authorized licensed use limited to: Addis Ababa University. Downloaded on December 04,2025 at 14:22:42 UTC from IEEE Xplore. Restrictions apply.
 x Using DirectIO instead of BufferIO. Fuse provides
us with read and write in DirectIO mode. Compared
with the BufferIO method, the biggest advantage of
DirectIO is that it reduces the overhead of copying
data from the application buffer to the kernel mode.
Performance may be improved if there are a mass of
sequential write requests.
B. Distributed Data Storage
1) Consistent hashing
The file system must distribute mass data among an
evolving cluster of storage devices such that device storage
and bandwidth resources are effectively utilized. Consistent
hashing can guarantee the uniformity and stability of data
(b) breaks down
distribution [11]. The algorithm uses same hash function to
calculate the value of the storage node and file identifier, Figure 2. Map between files and their first storage nodes
then maps these hash values to a same hash ring. The file
will be stored in its nearest three storage nodes (one Raft 2) Data consistency
group) in a clockwise direction. Suppose there are 3 storage Data consistency problem of the distributed system has
nodes ( , , ) and 4 files ( , , , ) now, their hash arisen due to factors such as unreliable network, the
values are defined as: differences of host’s performance and clock. Write-Ahead
Logging (WAL) is a technique for providing atomicity and
ℎ( ) = . . = 1, 2, 3 (1) durability (two of the ACID properties) in the file system.
Raft is a consistent algorithm for managing replicated logs, it
ℎ = . . = 1, 2, 3, 4 (2) explains well how the distributed file system performs node
synchronization and recovery under extreme conditions such
as process crashes, machine power outages, and network
For the convenience of presentation, take the mapping
partitioning.
from files to the first storage node as an example. After
Fig. 3 shows the application of Raft algorithm in
mapping these values to the hash ring, the results are shown distributed storage (three replications). The Leader server
in Fig.2(a). is stored on the node, , are stored on receives the write data request log from the client, and then
the node, and is stored on the node. By calculating synchronizes to other nodes in the cluster. When the log has
the hash value of the file identity, we can quickly find out the been synchronized to more than half of the servers, the
first storage node which the file is stored on. Consistent Leader node then informs all servers in the cluster that logs
hashing does not cause lots of location failure when storage have been successfully replicated and can be committed to
nodes change, but only affects a few files. Fig. 2(b) shows the Raft state machine for execution. Finally, a copy of all
only the location of has changed to when breaks data exists on different physical servers to ensure data
down. distribution and consistency [12].

(a) Normal case

Figure 3. The application of raft algorithm in distributed storage (three

replications)

1806
Authorized licensed use limited to: Addis Ababa University. Downloaded on December 04,2025 at 14:22:42 UTC from IEEE Xplore. Restrictions apply.
 Performance is not enough if simply implemented in Raft become very large. In this way, the reading and modification
Paper's way. We also optimize the Raft algorithm in of a single file record becomes very expensive.
engineering applications to improve the overall performance Finally, we encode the key of each directory and file
of the distributed storage system. For Raft, the Leader can according to {parentid_filename}. Combined with the prefix
collect multiple requests at once and send them to the search of the database we can solve the problem of read and
Follower in one batch. After the Leader sends a batch of logs modification a large number of sub-files in multi-level
to the Follower, it can directly update the NextIndex and directory. The key to this design is that the file id does not
immediately send the subsequent logs without waiting for change with the filename.
the Follower to return. If there is an error in the network, or
the Follower returns some errors, the Leader needs to 2) Multi-version control of files
readjust the NextIndex and then resend the log. When a log Users often overwrite a file and only manipulate a small
is appended by most nodes, we can think that the log is part of the file at a time, so we use a series of segments to
committed, and it will not affect the consistency of the data represent a complete file and segments defined as a two-
when the committed log is applied. So we can use another tuple { , } which mean the offset in file and its
thread to apply this log asynchronously. The benefit of using length. As shown in Fig.4, each segment is represented as a
asynchronous apply is that we are now able to append and rectangle, the height of the rectangle is the version number of
apply log in parallel. Although to a client, its single request the segment, and the width of the rectangle is the length of
still needs to go through the whole Raft process; to multiple the segment in the file. Then all segments are arranged in
clients, the overall concurrency and throughput have ascending order from the offset in the file. The outline
improved. These optimizations have proven useful in our marked by the red line is the final effective segments list of
experimental section. the file.
Our solution, then ( ), in time, is as follows. First,
C. Metadata Management sort the critical points of rectangles. Then scan the critical
1) Metadata storage points from left to right. When we encounter the left edge of
a rectangle, we add the rectangle to the heap and use its
Metadata operations typically account for half of the file height as the key. When we encounter the right edge of a
system workload and are located on critical paths, which rectangle, we will remove the rectangle from the heap. (This
makes MDS clusters critical to overall performance. requires keeping the external pointer in the heap.) Finally,
Metadata must also be stored distributed to avoid single every time a critical point is encountered, after updating the
points of failure and our choice is Riak KV which is a heap, we set the height of the critical point to a value peeking
distributed NoSQL database designed to deliver maximum from the top of the heap. This solution is called skyline.
data availability by distributing data across multiple servers
[13].
The storage of metadata must meet the following two
requirements: operation is fast enough and metadata updates
must be committed to disk for safety. Therefor, we use in-
memory cache to satisfy most requests from client. The
MDS journals allow data to be flushed lazily and provide a
vastly reduced re-write workload. All file system information
is stored in the form of database records, it is distributed
across nodes using consistent hashing and Riak KV makes
scalable consistency and coherence management more
efficient.
In the initial version, the file metadata is a flat structure,
the path is the key, and the file record is the value. Then the
read and write of metadata is very simple. After knowing the
path, a simple read and write can modify or read the file
record. Under this mapping relationship, the file's create,
delete and truncate performances are very high, and each
operation only performs key-value read or write once. Figure 4. Multi-version control of files
However, if the user wants to perform mv operations, the
performance problem is very serious. We also try the parent- V. EXPERIMENTS
child directory nesting method to organize. All directories In this section, we evaluate our file system under a range
and files are stored in a way that the key is file id and the of general servers to demonstrate its performance, reliability,
value is the file record. This nested metadata structure is very and scalability. The experimental settings and measurement
friendly to directory operations such as mv, but as the are also shown.
number of sub-directory entries and sub-file entries in the
directory increases, the hash table in the file record will

1807
Authorized licensed use limited to: Addis Ababa University. Downloaded on December 04,2025 at 14:22:42 UTC from IEEE Xplore. Restrictions apply.
A. Cluster Configuration only ( ) . The performance of the skyline is
We measured performance on a cluster consisting of significantly better than the naive algorithm when the data
three common server machines which are configured with size is large.
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 128GB of 2) Write
memory, 80 TB of disk capacity(12 HDD with capacity of
8T, RAID 50) and a 10000 Mbps full-duplex Ethernet Log appending generally requires writing data to local
connection to a switch. Note that this configuration was set log files. For the disk, the append action is sequential
up for ease of testing. Typical clusters have more machines writing, and the performance is acceptable. If the user has
and higher hardware configuration. extremely high requirements for write performance, even if
it is written to the disk sequentially, the performance loss
B. Experimental Result relative to the write memory will be one to two orders of
magnitude higher, which will greatly affect effectiveness. In
1) Metadata addition, if users need to meet the requirements of
Table II shows the performance results of key metadata concurrent writes for multiple streams, serious random
operations under the three metadata organization structures. writes will be caused, and write throughput will drop very
The test is conducted on directories of different sizes, with seriously. In response to the above problems, we propose
100, 1000, 10000 and 100,000 files respectively. The getattr the following two solutions according to the actual needs of
operation is to read the meta information of a file or users:
directory and the ls operation will initiate getattr requests for x If the user's memory is tight, the Raft log will be
all files in the directory respectively. All operations occur written to the disk. System’s function will not be
under the two-level directory such as {getattr /a/b}, {ls /a/}, affected, but the write performance may be poor.
{mv /a/* to /a1/*}. x If the user's memory is sufficient, the Raft log will
We can find that flat structure is more friendly than be written to the memory. In this case, the write
nested structure to directory operations such as getattr and ls. performance will be greatly improved, especially in
However, if the user wants to perform mv operations, the scenarios where multiple streams are written
performance problem is very serious as the number of sub- concurrently. We can use UPS (Uninterruptible
directory entries and sub-file entries in the directory Power Supply) with SSD to solve the problem of
increases. Our new encode structure combines the memory data loss caused by power failure. The UPS
advantages of the above methods so that our system has high provides extra power when the power is off, and its
performance in metadata operations. The key to this design is capacity is enough to support the backup of the data
that the file id does not change with the filename. in the memory to the SSD. This optimization idea
was partly inspired by FaRM [14].
TABLE II. METADATA OPERATIONS COMPARISON

Structure Operation 100 1000 10000 100000 TABLE III. MULTI-STREAM CONCURRENT WRITE PERFORMANCE
COMPARISON
0.93 1.05 1.11 1.20
getattr
ms ms ms ms
Concurrency 10 50 100 200 300
7.43 32.36 133.04 1.05
flat ls
ms ms ms s 452.21 409.83 334.93 252.17 185.74
0.94 8.95 82.23 788.66 Disk log
mv MB/s MB/s MB/s MB/s MB/s
s s s s
2.03 3.46 10.21 169.57 887.89 864.74 813.16 704.50 560.78
getattr Memory log
ms ms ms ms MB/s MB/s MB/s MB/s MB/s
7.07 53.08 390.45 3.67 Table III shows the performance results of the distributed
nested ls
ms ms ms s file storage system in the multi-stream concurrent writing
26.52 24.95 25.58 25.83 scenario using the disk log or memory log. When the number
mv
ms ms ms ms of concurrently written files increases, the random write
1.28 1.26 1.31 1.49 phenomenon becomes more obvious, resulting in a decrease
getattr
ms ms ms ms
in write throughput. In normal scenarios, the performance of
new 6.10 27.89 107.19 0.91
ls the memory log is significantly better than the disk log.
encode ms ms ms s
24.74 24.41 25.62 25.10 3) Read
mv
ms ms ms ms
In practical applications, file reading is sequential in most
In multi-version control of files, the performance of the cases, so we specifically optimize sequential reading
naive algorithm is better than the skyline algorithm in the scenarios and introduces a pre-reading mechanism. Our file
case of a small number of segments. This is because the system reads more file data than expected for the application
skyline algorithm has to maintain a more complex balanced at a time and put it in the cache if this is a sequential reading
binary tree data structure. However, as the scale of segments scenario, so that some requests can be directly read from the
increases, the time complexity of the naive algorithm cache. The cache stores part of the hot data to reduce the
increases by the scale of ( ^2) and the skyline algorithm is

1808
Authorized licensed use limited to: Addis Ababa University. Downloaded on December 04,2025 at 14:22:42 UTC from IEEE Xplore. Restrictions apply.
number of disk I/O and we use the LRU (Least Recently the National Key Research and Development Program of
Used) to manage the cache. China (2017YFB0802701), in part by the Fundamental
Research Funds for the Central Universities (2018XKJC04),
TABLE IV. MULTI-STREAM CONCURRENT READ PERFORMANCE in part by the 111 Project of China (B08004).

Concurrency 10 50 100 200 300 REFERENCES

[1] Reinsel D, Gantz J, Rydning J. Data age 2025: The evolution of data
682.49 619.64 474.13 317.25 246.81
Read rate to life-critical[J]. Don’t Focus on Big Data, 2017.
MB/s MB/s MB/s MB/s MB/s
[2] Levy E, Silberschatz A. Distributed file systems: Concepts and
Table IV shows the performance results of the distributed examples[J]. ACM Computing Surveys (CSUR), 1990, 22(4): 321-
374.
file storage system in the multi-stream concurrent reading
[3] Ongaro D, Ousterhout J. In search of an understandable consensus
scenario with 5 GB cache. When the number of concurrently algorithm[C]//2014 {USENIX} Annual Technical Conference
read files increases, the random read phenomenon becomes ({USENIX}{ATC} 14). 2014: 305-319.
more obvious, resulting in a decrease in read throughput. Of [4] Karger D, Sherman A, Berkheimer A, et al. Web caching with
course, we can increase the cache capacity to speed up consistent hashing[J]. Computer Networks, 1999, 31(11-16): 1203-
reading. 1213.
[5] Lonczewski F, Schreiber S. The FUSE-System: an Integrated User
VI. CONCLUSION Interface Design Environment[C]//CADUI. 1996, 96: 37-56.
In this work we design a high-performance distributed [6] Kant K. Data center evolution: A tutorial on state of the art, issues,
and challenges[J]. Computer Networks, 2009, 53(17): 2939-2965.
file system for mass data. FUSE interface is implemented for
[7] Ghemawat S, Gobioff H, Leung S T. The Google file
the users to mount and work on files. We apply the raft system[C]//Proceedings of the nineteenth ACM symposium on
consensus algorithm to the system, files are replicated to Operating systems principles. 2003: 29-43.
multiple machines so that a few machines crash without I/O [8] Shvachko K, Kuang H, Radia S, et al. The hadoop distributed file
error. Consistent hashing guarantees the uniformity and system[C]//2010 IEEE 26th symposium on mass storage systems and
stability of data distribution, system I/O performance is also technologies (MSST). Ieee, 2010: 1-10.
accelerated through technologies such as memory cache and [9] Weil S A, Brandt S A, Miller E L, et al. Ceph: A scalable, high-
pipelines. performance distributed file system[C]//Proceedings of the 7th
Performance measurements under some typical symposium on Operating systems design and implementation. 2006:
307-320.
workloads are shown in experiments and the results prove
[10] Developers F. The reference implementation of the Linux FUSE
the file system is high-performance, scalability and reliability. (Filesystem in Userspace) interface[J]. URL: [Link]
We think files are overwrite frequently rather than appending com/libfuse/libfuse.
only, which is different from the traditional distributed file [11] Wong K C, Kumaresan B, Prince Jr H B. Balanced consistent hashing
system like GFS. Fault tolerance is also provided by constant for distributed resource management: U.S. Patent 8,244,846[P]. 2012-
monitoring and automatic recovery. Our distributed file 8-14.
system can successfully meet storage needs and it is a good [12] Ongaro D. Consensus: Bridging theory and practice[D]. Stanford
tool that enables us to process mass data. University, 2014.
[13] Klophaus R. Riak core: Building distributed applications without
ACKNOWLEDGMENT shared state[M]//ACM SIGPLAN Commercial Users of Functional
Programming. 2010: 1-1.
This work was supported in part by the National Natural [14] Dragojević A, Narayanan D, Nightingale E B, et al. No compromises:
Science Foundation of China (61671078), in part by the distributed transactions with consistency, availability, and
Beijing Laboratory of Advanced Information Networks of performance[C]//Proceedings of the 25th symposium on operating
BUPT, in part by the Beijing Key Laboratory of Network systems principles. 2015: 54-70.
System Architecture and Convergence of BUPT, in part by

1809
Authorized licensed use limited to: Addis Ababa University. Downloaded on December 04,2025 at 14:22:42 UTC from IEEE Xplore. Restrictions apply.