X F S i s a fi l e s y s t e m t h at w a s d e -

signed from day one for computer systems

C h r i s t o p h H e ll w i g
with large numbers of CPUs and large disk
arrays. It focuses on supporting large files
XFS: the big and good streaming I/O performance. It also
has some interesting administrative features
storage file not supported by other Linux file systems.

­system for Linux This article gives some background infor-

mation on why XFS was created and how it
Christoph Hellwig is a freelancer providing
differs from the familiar Linux file systems.
consulting, training, and, last but not least, You may discover that XFS is just what your
contract programming for Linux storage and
file systems. He has been working on Linux project needs instead of making do with the
file systems since 2001 and is one of the
most widely known developers in this area.
default Linux file system.
hch@[Link]
Background and History

For years the standard Linux file system was ext2,

a straightforward Berkeley FFS derivative. At the
end of the 1990s, several competitors suddenly ap-
peared to fill the gap for a file system providing
fast crash recovery and transactional integrity for
metadata. The clear winner in mainstream Linux is
ext3, which added journaling on top of ext2 with-
out many additional changes [7].
XFS has been less well known to many average
Linux users but has always been the state of the art
at the very high end. XFS itself did not originate on
Linux but was first released on IRIX, a UNIX vari-
ant for SGI workstations and servers, in December
1994, almost 15 years ago. Starting in 1999, XFS
was ported to Linux as part of SGI’s push to use
Linux and Intel’s Itanium processors as the way
forward for its high-end supercomputing systems.
Designed from the start for large multiprocessor
systems and disk arrays [1] rather than for small,
single-disk consumer workstations, it was for a
long time positioned above the mainstream Linux
market. Today even low-end workstations with a
small number of CPU cores and disks come close
to the limits of ext3 (see Table 1). While there is
another adaption of the FFS concept called ext4
under development to mitigate these limits to a cer-
tain extent, it seems as though basic FFS design is
close to maxed out.
To address these limits, ext3 is evolving into ext4
by incorporating features pioneered by XFS such
as delayed allocations and extents. Even with these
improvements taking the basic FFS design as far
as it can go, it is difficult to match the scalability
limits of XFS, which has been designed for large
storage systems from day one. In a few years, btrfs,
a new file system initiated by Oracle, will mature
from development status to hopefully become the

10 ; L O G I N : VO L . 3 4 , N O. 5
 new standard file system. As a new design that includes advanced manage-
ment and self-healing features, btrfs will compete heavily with XFS on the
lower end of the XFS market, but we will have to see how well it does on the
extreme high end.
Today XFS is used by many well-known institutions, with CERN and Fer-
milab managing petabytes of storage for scientific experiments using XFS,
and [Link] serving the source code to the Linux kernel and many other
projects from XFS file systems.

Limit ext3 ext4 XFS

max file system size 16 TiB 16 TiB 16 EiB
max file size 2 TiB 16 TiB 8 EiB
max extent size 4 kiB 128 MiB 8 GiB
max extended attribute size 4 kiB    4 kiB 64 kiB
max inode number 232 232 264
(All numbers assume the maximum 4 kiB block size on x86 Linux systems.)
T a b l e 1 : F i l e s y s t e m l i m i t s fo r X F S , e x t 3 a n d e x t 4

Space Allocation and Management

Each XFS file system is partitioned into regions called allocation groups
(AGs). Allocation groups are somewhat similar to the block groups in ext3,
but AGs are typically much larger than block groups and are used for scal-
ability and parallelism rather than disk locality. Allocation groups are typi-
cally sized between 0.5 and 4 gigabytes and keep the size of the XFS data
structures in a range where they can operate efficiently and in parallel [2].
Historical UNIX file systems, including ext3, use linear bitmaps to track free
space, which is inefficient especially for larger contiguous allocations. XFS
instead uses a pair of B+ trees in each allocation group. Each entry in the B+
tree nodes consists of a start block and length pair describing a free-space
region. The first B+ tree is indexed by the starting block of the free region,
and the other is indexed by the length of the free region. This double index-
ing allows the allocator to consider two goals for new data placement: local-
ity to existing file data, and best fit into free space.
A similar extent concept is used for tracking the disk blocks assigned to
each file. In addition to the start block on disk and the length of the con-
tiguous range, the extent descriptor also contains two additional fields. The
first one is the logical offset into the file, which allows for efficient sparse file
support by skipping ranges that do not have blocks allocated to them. The
second one is a simple one-bit flag to mark an extent as unwritten, a concept
that will be explained later in this article.
For most files, a simple linear array of extent descriptors is embedded into
the inode, avoiding additional metadata blocks and management overhead.
For very large files or files containing many holes, the number of extents can
be too large to fit directly into the inode.
In this case, extents are tracked by another B+ tree with its root in the
inode. This tree is indexed by the offset into the file, which allows an extent
descriptor for a given file offset to be found quickly, with no linear search
overhead. Figure 1, showing the time needed to remove a large file such
as an HD video or virtual machine image, demonstrates how management
overhead can be reduced by using extents.

; L O G I N : O c to b e r 2 0 0 9 X F S : t h e b ig sto r ag e fil e ­s yst e m fo r L in u x 11

 F i g u r e 1 : T i m e s p e n t r e m ov i n g a v e r y l a r g e f i l e

Inodes and Extended Attributes

The XFS inode consists of three parts: the inode core, the data fork, and
the optional attribute fork. The inode core contains traditional UNIX inode
metadata such as owner and group, number of blocks, timestamps, and a
few XFS-specific additions such as project ID. The data fork contains the
previously mentioned extent descriptors or the root of the extent map. The
optional attribute fork contains the so-called extended attributes. The con-
cept of extended attributes is not part of the Posix file system interface but
is supported by all modern operating systems and file systems with slightly
differing semantics. In Linux, extended attributes are simple name/value
pairs assigned to a file that can be listed and read or written one attribute at
a time. Extended attributes are used internally by Linux to implement access
control lists (ACLs) and labels for SELinux, but they can also be used for
storing arbitrary user metadata [3].
The attribute fork in XFS can either store extended attributes directly in
the inode if the space required for the attributes is small enough, or use the
same scheme of extent descriptors as described for the file data above to
point to additional disk blocks. This allows XFS to support extended attri-
bute sizes up to 64 kilobytes, while most other Linux file systems are lim-
ited to the size of a single disk block.
The size of the inode core is fixed, and the data and attribute forks share the
remaining space in the inode, which is determined by the inode size chosen
at file system creation time, ranging from 256 to 2048 bytes. For file systems
that extensively use ACLs (e.g., for Windows file shares exported by Samba)
or for file systems making extensive use of extended attributes, choosing a
larger inode size can provide performance improvements, because this extra
data can be stored in the inode and does not require reading additional data
blocks.
Inodes in XFS are dynamically allocated, which means that, unlike many
other Linux file systems, their location and number are not determined at
mkfs time. This means that there is no need to predict the expected num-

12 ; L O G I N : VO L . 3 4 , N O. 5
 ber of inodes when creating the file system, with the possibility of under- or
overprovision. Because every block in the file system can now possibly con-
tain inodes, an additional data structure is needed to keep track of inode lo-
cations and allocations. For this, each allocation group contains another B+
tree tracking the inodes allocated within it.
Because of this, XFS uses a sparse inode number scheme where inode num-
bers encode the location of the inode on disk. While this has advantages
when looking up inodes, it also means that for large file systems, inode
numbers can easily exceed the range encodable by a 32-bit integer. Despite
Linux’s having supported 64-bit-wide inode numbers for over 10 years,
many user-space applications on 32-bit systems still cannot accommodate
large inode numbers. Thus by default XFS limits the allocation of inodes to
the first allocation groups, in order to ensure all inode numbers fit into 32
bits. This can have a significant performance impact, however, and can be
disabled with the inode64 mount option.

Directories
XFS supports two major forms of directories. If a directory contains only a
few entries and is small enough to fit into the inode, a simple unsorted lin-
ear format can store all data inside the inode’s data fork. The advantage of
this format is that no external block is used and access to the directory is
extremely fast, since it will already be completely cached in memory once
it is accessed. Linear algorithms, however, do not scale to large directories
with millions of entries. XFS thus again uses B+ trees to manage large di-
rectories. Compared to simple hashing schemes such as the htree option in
ext3 and ext4, a full B+ tree provides better ordering of readdir results and
allows for returning unused blocks to the space allocator when a directory
shrinks. The much improved ordering of readdir results can be seen in Fig-
ure 2, which compares the read rates of files in readdir order in a directory
with 100,000 entries.
Reading 100.000 4kiB files in readdir order
Seagate ST373454SS SATA disk
4000
XFS
ext4
ext3
3500

3000

2500
Rate (Files/s)

2000

1500

1000

500

F i g u r e 2 : Co m p a r i s o n of r e a d i n g a l a r g e ( 1 0 0 , 0 0 0 e n t r y )
­d i r e c t o r y, t h e n r e a d i n g e a c h f i l e

; L O G I N : O c to b e r 2 0 0 9 X F S : t h e b ig sto r ag e fil e ­s yst e m fo r L in u x 13

 I/O Scalability
From day one, XFS has been designed to deal with high-performance disk
subsystems, especially striped disk arrays with large aggregated bandwidth.
When XFS was designed, “high performance” meant a few hundred mega-
bytes per second, but 15 years later XFS still keeps up with aggregated
bandwidth in the tens of gigabytes per second for a single file system in-
stance [4].
To keep a RAID array busy, the file system should submit I/O requests that
are large enough to span all disks in the array. In addition, I/O requests
should be aligned to stripe boundaries where possible, to avoid read-modify-
write cycles for common usage patterns. Because a single I/O can only be as
large as a contiguous range of blocks, it is critical that files are allocated as
contiguously as possible, to allow large I/O requests to be sent to the storage.
The key to achieving large contiguous regions is a method known as “de-
layed allocation.” In delayed allocation, specific disk locations are not chosen
when a buffered write is submitted; only in-memory reservations take place.
Actual disk blocks are not chosen by the allocator until the data is sent to
disk due to memory pressure, periodic write-backs, or an explicit sync re-
quest. With delayed allocation, there is a much better approximation of the
actual size of the file when deciding about the block placement on disk. In
the best case the whole file may be in memory and can be allocated in one
contiguous region. In practice XFS tends to allocate contiguous regions of
50 to 100 GiB when performing large sequential I/O, even when multiple
threads write to the file system at the same time [4].
While delayed allocations help with random write workloads if sufficiently
large contiguous regions are filled before the flush to disk, there are still
many workloads where this is not the case. To avoid fragmentation in patho-
logical cases with random writes filling up a file very slowly (such as some
HPC workloads or BitTorrent clients), XFS allows preallocation of blocks on
disk to a file before actually writing to it. The preallocation just assigns data
blocks to a file without touching the block contents. To avoid security prob-
lems with exposing stale data, preallocated extents are marked as unwritten,
and any read from them will return zeros. Once data is written to unwritten
extents, they are converted to normal extents, which incurs minimal perfor-
mance overhead compared to a write to a normal allocated extent.
Figures 3 and 4 show some performance enhancement when XFS is com-
pared to ext3, the old Linux standard file system, and to ext4, which adds
delayed allocations and extents to it for sequential I/O workloads. On the
other hand, Figure 5 shows that the performance for completely random I/O
is not only really bad but also doesn’t profit much from the XFS features.

Direct I/O
XFS provides a feature, called direct I/O, that provides the semantics of a
UNIX raw device inside the file system namespace. Reads and writes to a
file opened for direct I/O bypass the kernel file cache and go directly from
the user buffer to the underlying I/O hardware. Bypassing the file cache of-
fers the application full control over the I/O request size and caching policy.
Avoiding the copy into the kernel address space reduces the CPU utilization
for large I/O requests significantly. Thus direct I/O allows applications such
as databases, which were traditionally using raw devices, to operate within
the file system hierarchy.

14 ; L O G I N : VO L . 3 4 , N O. 5
 Streaming write performance - 10GB file
RAID 0 of 6 Seagate ST373454SS SATA disks
700
block device
XFS
ext4
600 ext3

500

Throughput (MiB/s)
400

300

200

100

di

bu
re

ffe
ct

re
d
F i g u r e 3 : Co m p a r i n g b lo c k d e v i c e , X F S , e x t 4 , a n d e x t 3 w h e n
w r i t i n g a 1 0 GB f i l e

Tiobench sequential I/O performance - 4GiB working set

Seagate ST373454SS SATA disk
90
XFS
ext4
80 ext3

70

60
Throughput (MiB/s)

50

40

30

20

10

0
re

re

re

w
rit

rit

rit
ad

ad

ad

e,

e,

e,
,4

,8

,1

16
6
th

th

th

th
th

th
re

re

re

re
re

re
ad

ad

ad

ad
ad

ad
s

s
s

F i g u r e 4 : Co m p a r i n g s e q u e n t i a l I / O p e r fo r m a n c e b e t w e e n
X F S , e x t4 , a n d e x t 3

; L O G I N : O c to b e r 2 0 0 9 X F S : t h e b ig sto r ag e fil e ­s yst e m fo r L in u x 15

 Tiobench random I/O performance - 4GiB working set
Seagate ST373454SS SATA disk
3
XFS
ext4
ext3
2.5

Throughput (MiB/s)
1.5

0.5

re

re

re

w
rit

rit

rit
ad

ad

ad

e,

e,

e,
,4

,8

,1

16
6
th

th

th

th
th

th
re

re

re

re
re

re
ad

ad

ad

ad
ad

ad
s

s
s

s
F i g u r e 5 : Co m p a r i n g r a n d o m I / O p e r fo r m a n c e b e t w e e n X F S ,
e x t4 , a n d e x t 3

Direct I/O has been adopted by all major Linux file systems, but the support
outside of XFS is rather limited. While XFS guarantees the uncached I/O
behavior under all circumstances, other file systems fall back to buffered I/O
for many non-trivial cases such as appending writes, hole filling, or writing
into preallocated blocks. A major semantic difference between direct I/O and
buffered I/O in XFS is that XFS allows multiple parallel writers to files using
direct I/O, instead of imposing the single-writer limit specified in Posix for
buffered I/O. Serialization of I/O requests hitting the same region is left to
the application, and thus allows databases to access a table in a single file in
parallel from multiple threads or processes.

Crash Recovery
For today’s large file systems, a full file system check on an unclean shut-
down is not acceptable because it would take too long. To avoid the require-
ment for regular file system checks, XFS uses a write-ahead logging scheme
that enables atomic updates of the file system. XFS only logs structural up-
dates to the file system metadata, but not the actual user data, for which the
Posix file system interface does not provide useful atomicity guarantees.
XFS logs every update to the file system data structures and does not batch
changes from multiple transactions into a single log write, as is done by
ext3. This means that XFS must write significantly more data to the log in
case a single metadata structure gets modified again and again in short se-
quence (e.g., removing a large number of small files). To mitigate the impact
of log writes to the system performance, an external log device can be used.
With an external log the additional seeks on the main device are reduced,
and the log can use the full sequential performance of the log device.
Unfortunately, transaction logging does not help to protect against hard-
ware-induced errors. To deal with these problems, XFS has an offline file
system checking and repair tool called xfs_repair. To deal with the ever
growing disk sizes and worsening seek rates, xfs_repair has undergone a
major overhaul in the past few years to perform efficient read-ahead and
caching and to make use of multiple processors in SMP systems [6].

16 ; L O G I N : VO L . 3 4 , N O. 5
 Disk Quotas
XFS provides an enhanced implementation of the BSD disk quotas. It sup-
ports the normal soft and hard limits for disk space usage and number of
inodes as an integral part of the file system. Both the per-user and per-group
quotas supported in BSD and other Linux file systems are supported. In ad-
dition to group quotas, XFS alternatively can support project quotas, where
a project is an arbitrary integer identifier assigned by the system adminis-
trator. The project quota mechanism in XFS is used to implement directory
tree quota, where a specified directory and all of the files and subdirectories
below it are restricted to using a subset of the available space in the file sys-
tem. For example, the sequence below restricts the size of the log files in
/var/log to 1 gigabyte of space:
# mount -o prjquota /dev/sda6 /var

# echo 42:/var/log >> /etc/projects

# echo logfiles:42 >> /etc/projid
# xfs_quota -x -c ‘project -s logfiles’ /var
# xfs_quota -x -c ‘limit -p bhard=1g logfiles’ /var

Another enhancement in the XFS quota implementation is that the quota

subsystem distinguishes between quota accounting and quota enforcement.
Quota accounting must be turned on during mount time, while quota en-
forcement can be turned on and off at runtime. Using an XFS file system
with quota accounting but no enforcement provides an efficient way to mon-
itor disk usage. For this reason, XFS also accounts (but never enforces) quo-
tas usage for the superuser.
The xfs_quota command [8] seen in the example above offers full access to
all features in the XFS quota implementation. In addition, the standard BSD
quota and edquota tools can be used to administer basic quota functionality.

Day-to-Day Use
A file system in use should be boring and mostly invisible to the system ad-
ministrator and user. But to get to that state the file system must first be cre-
ated. An XFS file system is created with the [Link] command, which is
trivial to use:
# [Link] /dev/vg00/scratch
meta-data =/dev/vg00/scratch isize=256 agcount=4, agsize=1245184 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=4980736, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0

As seen above, the [Link] command returns the geometry information

for the file system to make sure all parameters are set correctly. There are
not many parameters that must be manually set for normal use. For soft-
ware RAID arrays, XFS already extracts the proper stripe size and alignment
parameters from the underlying device, but if that is not possible (e.g., for
hardware RAID controllers), the parameters can be set manually. The follow-
ing example creates a file system aligned correctly for a RAID5 array with
8+1 disks and a stripe unit of 256 kiB:
# [Link] -d su=256k,sw=8 /dev/sdf

; L O G I N : O c to b e r 2 0 0 9 X F S : t h e b ig sto r ag e fil e ­s yst e m fo r L in u x 17

 Other interesting options include using an external log device and changing
the inode size. The example below creates a file system with 2 kiB-sized in-
odes and an external log device:
# [Link] -i size=2048 -l logdev=/dev/vg00/logdev /dev/vg00/home

For more details, see the [Link] man page and the XFS training material
[5].
A command worth note is xfs_fsr. FSR stands for file system reorganizer
and is the XFS equivalent to the Windows defrag tool. It allows defragmen-
tion of the extent lists of all files in a file system and can be run in back-
ground from cron. It may also be used on a single file.
Although all normal backup applications can be used for XFS file systems,
the xfsdump command is specifically designed for XFS backup. Unlike tra-
ditional dump tools such as dumpe2fs for ext2 and ext3, xfsdump uses a
special API to perform I/O based on file handles similar to those used in the
NFS over the wire protocol. That way, xfsdump does not suffer from the in-
consistent device snapshots on the raw block device that plague traditional
dump tools. The xfsdump command can perform backups to regular files
and tapes on local and remote systems, and it supports incremental backups
with a sophisticated inventory management system.
XFS file systems can be grown while mounted using the xfs_growfs com-
mand, but there is not yet the ability to shrink.

Conclusion
This article gave a quick overview of the features of XFS, the Linux file sys-
tem for large storage systems. I hope it clearly explains why Linux needs a
file system that differs from the default and also shows the benefits of a file
system designed for large storage from day one.

acknowledgments

I would like to thank Eric Sandeen for reviewing this article carefully.

references

[1] Adam Sweeney et al., “Scalability in the XFS File System,” Proceedings
of the USENIX 1996 Annual Technical Conference.
[2] Silicon Graphics Inc., XFS Filesystem Structure, 2nd edition,
[Link]
[3] Linux attr(5) man page: [Link]
[4] Dave Chinner and Jeremy Higdon, “Exploring High Bandwidth File-
systems on Large Systems,” Proceedings of the Ottawa Linux Symposium 2006:
[Link]
[5] Silicon Graphics Inc., XFS Overview and Internals: [Link]
projects/xfs/training/[Link].
[6] Dave Chinner and Barry Naujok, Fixing XFS Filesystems Faster: http://
[Link]/pub/[Link]/2008/slides/135-fixing_xfs_faster.pdf.
[7] Dr. Stephen Tweedie, “EXT3, Journaling Filesystem,” transcript of a pre-
sentation at the Ottawa Linux Symposium 2000: [Link]
net/release/OLS2000-ext3/[Link].
[8] xfs_quota(8)—Linux manpage: [Link]

18 ; L O G I N : VO L . 3 4 , N O. 5