V SEM AI & ML Distributed Systems MODULE-2
Chapter-1 Distributed File Systems
1.1. Introduction
➢ A distributed file system enables programs to store and access remote files
exactly as they do local ones, allowing users to access files from any
computer on a network.
➢ The performance and reliability experienced for access to files stored at a
server should be comparable to that for files stored on local disks.
➢ The sharing of stored information is the most important aspect of
distributed resource sharing.
➢ The requirements for sharing within local networks and intranets lead to
a need for a different type of service – one that supports the persistent
storage of data and programs of all types on behalf of clients and the
consistent distribution of up-to-date data.
➢ File systems were originally developed for centralized computer systems
and desktop computers as an operating system facility providing a
convenient programming interface to disk storage.
➢ Distributed file systems support the sharing of information in the form of
files and hardware resources in the form of persistent storage throughout
an intranet.
➢ A well-designed file service provides access to files stored at a server with
performance and reliability similar to, and in some cases better than, files
stored on local disks.
➢ Their design is adapted to the performance and reliability characteristics
of local networks, and hence they are most effective in providing shared
persistent storage for use in intranets.
➢ The first file servers were developed by researchers in the 1970s [Birrell
and Needham 1980, Mitchell and Dion 1982, Leach et al. 1983], and Sun’s
Network File System became available in the early 1980s [Sandberg et al.
1985, Callaghan 1999].
➢ A file service enables programs to store and access remote files exactly as
they do local ones, allowing users to access their files from any computer
in an intranet.
Fig 1.1 Storage Systems and properties
➢ Distributed shared memory provides an emulation of a shared memory
by the replication of memory pages or segments at each host, but it does
not necessarily provide automatic persistence.
➢ Persistent object stores aim to provide persistence for distributed shared
objects.
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML
V SEM AI & ML Distributed Systems MODULE-2
➢ Peer-to-peer storage systems offer scalability to support client loads
much larger than the systems but they incur high performance costs in
providing secure access control and consistency between updatable
replicas.
➢ The consistency column indicates whether mechanisms exist for the
maintenance of consistency between multiple copies of data when updates
occur.
➢ Virtually all storage systems rely on the use of caching to optimize the
performance of programs.
➢ Caching was first applied to main memory and non-distributed file
systems, and for those the consistency is strict (denoted by a ‘1’, for one-
copy consistency in Figure 1.1) – programs cannot observe any
discrepancies between cached copies and stored data after an update.
➢ When distributed replicas are used, strict consistency is more difficult to
achieve.
➢ Distributed file systems such as Sun NFS and the Andrew File System
cache copies of portions of files at client computers, and they adopt specific
consistency mechanisms to maintain an approximation to strict
consistency – this is indicated by a tick in the consistency column.
➢ The Web uses caching extensively both at client computers and at proxy
servers maintained by user organizations.
➢ The consistency between the copies stored at web proxies and client caches
and the original server is only maintained by explicit user actions.
Fig 1.2 File System Modules
1.1.1. Characteristics of File Systems
➢ File systems are responsible for the organization, storage, retrieval,
naming, sharing and protection of files.
➢ They provide a programming interface that characterizes the file
abstraction, freeing programmers from concern with the details of storage
allocation and layout.
➢ Files are stored on disks or other non-volatile storage media.
➢ Files contain both data and attributes. The data consist of a sequence of
data items (typically 8-bit bytes), accessible by operations to read and write
any portion of the sequence.
➢ Some attributes are managed by the file system and are not normally
updatable by user programs.
➢ File systems are designed to store and manage large numbers of files, with
facilities for creating, naming and deleting files.
➢ The naming of files is supported by the use of directories.
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML
V SEM AI & ML Distributed Systems MODULE-2
➢ A directory is a file, often of a special type, that provides a mapping from
text names to internal file identifiers.
➢ Directories may include the names of other directories, forming a
hierarchical file system structure.
➢ Hierarchical structures support multi-part pathnames.
➢ File systems control access to files based on user authorizations.
➢ Access types include reading, updating, and executing files.
➢ The term metadata is often used to refer to all of the extra information
stored by a file system that is needed for the management of files. It
includes file attributes, directories and all the other persistent information
used by the file system.
Fig 1.3 File attribute record structure
➢ A non-distributed file system is typically implemented as a layered
module structure, where each layer depends on the one below.
➢ Distributed file systems require the same components as non-distributed
ones, plus client-server communication mechanisms.
➢ Distributed file systems also need additional support for distributed
naming and file location.
➢ File system operations:
Fig 1.4 UNIX file system operations
➢ These are the system calls implemented by the kernel; application
programmers usually access them through procedure libraries such as the
C Standard Input/Output Library or the Java file classes.
➢ The UNIX operations are based on a programming model in which some
file state information is stored by the file system for each running program.
➢ This consists of a list of currently open files with a read-write pointer for
each, giving the position within the file at which the next read or write
operation will be applied.
➢ The file system is responsible for applying access control for files.
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML
V SEM AI & ML Distributed Systems MODULE-2
➢ In local file systems such as UNIX, it does so when each file is opened,
checking the rights allowed for the user’s identity in the access control list
against the mode of access requested in the open system call. If the rights
match the mode, the file is opened and the mode is recorded in the open
file state information.
1.1.2. Distributed File System Requirements
➢ Initially, distributed file systems offered access transparency and location
transparency; performance, scalability, concurrency control, fault
tolerance and security requirements emerged and were met in subsequent
phases of development.
➢ Requirements
1. Transparency
2. Concurrent file updates
3. File replication
4. Hardware and operating system heterogeneity
5. Fault tolerance
6. Consistency
7. Security
8. Efficiency
1. Transparency: The file service is usually the most heavily loaded service
in an intranet, so its functionality and performance are critical.
➢ The design of the file service should support many of the transparency
requirements for distributed systems.
➢ The design must balance the flexibility and scalability that derive from
transparency against software complexity and performance.
➢ The following forms of transparency are partially or wholly addressed by
current file services:
a. Access transparency: Client programs should be unaware of the
distribution of files.
➢ A single set of operations is provided for access to local and remote files.
➢ Programs written to operate on local files are able to access remote files
without modification.
b. Location transparency: Client programs should see a uniform file name
space.
➢ Files or groups of files may be relocated without changing their pathnames,
and user programs see the same name space wherever they are executed.
c. Mobility transparency: Neither client programs nor system
administration tables in client nodes need to be changed when files are
moved.
➢ This allows file mobility – files or, more commonly, sets or volumes of files
may be moved, either by system administrators or automatically.
d. Performance transparency: Client programs should continue to perform
satisfactorily while the load on the service varies within a specified range.
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML
V SEM AI & ML Distributed Systems MODULE-2
e. Scaling transparency: The service can be expanded by incremental
growth to deal with a wide range of loads and network sizes.
2. Concurrent file updates: Changes to a file by one client should not
interfere with the operation of other clients simultaneously accessing or
changing the same file.
➢ The need for concurrency control for access to shared data in many
applications is widely accepted and techniques are known for its
implementation, but they are costly.
➢ Most current file services follow modern UNIX standards in providing
advisory or mandatory file- or record-level locking.
3. File replication: In a file service that supports replication, a file may be
represented by several copies of its contents at different locations.
➢ This has two benefits – it enables multiple servers to share the load of
providing a service to clients accessing the same set of files, enhancing the
scalability of the service, and it enhances fault tolerance by enabling
clients to locate another server that holds a copy of the file when one has
failed.
➢ Few file services support replication fully, but most support the caching of
files or portions of files locally, a limited form of replication.
4. Hardware and operating system heterogeneity: The service interfaces
should be defined so that client and server software can be implemented
for different operating systems and computers. This requirement is an
important aspect of openness.
5. Fault tolerance: The central role of the file service in distributed systems
makes it essential that the service continue to operate in the face of client
and server failures.
➢ Fortunately, a moderately fault-tolerant design is straightforward for
simple servers.
➢ To cope with transient communication failures, the design can be based
on at-most-once invocation semantics or it can use the simpler at-least-
once semantics with a server protocol designed in terms of idempotent
operations, ensuring that duplicated requests do not result in invalid
updates to files.
➢ The servers can be stateless, so that they can be restarted and the service
restored after a failure without any need to recover previous state.
6. Consistency: Conventional file systems such as that provided in UNIX
offer one-copy update semantics. This refers to a model for concurrent
access to files in which the file contents seen by all of the processes
accessing or updating a given file are those that they would see if only a
single copy of the file contents existed.
➢ When files are replicated or cached at different sites, there is an inevitable
delay in the propagation of modifications made at one site to all of the other
sites that hold copies, and this may result in some deviation from one-copy
semantics.
7. Security: Virtually all file systems provide access-control mechanisms
based on the use of access control lists.
➢ In distributed file systems, there is a need to authenticate client requests
so that access control at the server is based on correct user identities and
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML
V SEM AI & ML Distributed Systems MODULE-2
to protect the contents of request and reply messages with digital
signatures and (optionally) encryption of secret data.
8. Efficiency: A distributed file service should offer facilities that are of at
least the same power and generality as those found in conventional file
systems and should achieve a comparable level of performance.
➢ The techniques used for the implementation of file services are an
important part of the design of distributed systems.
➢ A distributed file system should provide a service that is comparable with,
or better than, local file systems in performance and reliability.
➢ It must be convenient to administer, providing operations and tools that
enable system administrators to install and operate the system
conveniently.
1.1.3. Case Studies
➢ File service architecture: This is an abstract architectural model that
underpins both NFS and AFS.
➢ It is based upon a division of responsibilities between three modules – a
client module that emulates a conventional file system interface for
application programs, and server modules, that perform operations for
clients on directories and on files.
➢ The architecture is designed to enable a stateless implementation of the
server module.
➢ SUN NFS: Sun Microsystems’s Network File System (NFS) has been widely
adopted in industry and in academic environments since its introduction
in 1985.
➢ The design and development of NFS were undertaken by staff at Sun
Microsystems in 1984 [Sandberg et al. 1985, Sandberg 1987, Callaghan
1999].
➢ Although several distributed file services had already been developed and
used in universities and research laboratories, NFS was the first file service
that was designed as a product.
➢ The design and implementation of NFS have achieved success both
technically and commercially.
➢ To encourage its adoption as a standard, the definitions of the key
interfaces were placed in the public domain [Sun 1989], enabling other
vendors to produce implementations, and the source code for a reference
implementation was made available to other computer vendors under
licence.
➢ It is now supported by many vendors, and the NFS protocol (version 3) is
an Internet standard, defined in RFC 1813 [Callaghan et al. 1995].
➢ NFS provides transparent access to remote files for client programs
running on UNIX and other systems.
➢ The client-server relationship is symmetrical: each computer in an NFS
network can act as both a client and a server, and the files at every
machine can be made available for remote access by other machines.
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML
V SEM AI & ML Distributed Systems MODULE-2
➢ Any computer can be a server, exporting some of its files, and a client,
accessing files on other machines.
➢ But it is common practice to configure larger installations with some
machines as dedicated servers and others as workstations.
➢ An important goal of NFS is to achieve a high level of support for hardware
and operating system heterogeneity.
➢ Andrew File System: Andrew is a distributed computing environment
developed at Carnegie Mellon University (CMU) for use as a campus
computing and information system [Morris et al. 1986].
➢ The design of the Andrew File System (henceforth abbreviated AFS) reflects
an intention to support information sharing on a large scale by minimizing
client-server communication.
➢ This is achieved by transferring whole files between server and client
computers and caching them at clients until the server receives a more up-
to-date version.
➢ More recent descriptions can be found in Campbell [1997] and [Linux AFS].
➢ AFS was initially implemented on a network of workstations and servers
running BSD UNIX and the Mach operating system at CMU and was
subsequently made available in commercial and public-domain versions.
➢ AFS was adopted as the basis for the DCE/DFS file system in the Open
Software Foundation’s Distributed Computing Environment (DCE) .
1.2. File Service Architecture
➢ An architecture that offers a clear separation of the main concerns in
providing access to files is obtained by structuring the file service as three
components – a flat file service, a directory service and a client module.
Fig 1.5 File Service Architecture
➢ The flat file service and the directory service each export an interface for
use by client programs, and their RPC interfaces, taken together, provide
a comprehensive set of operations for access to files.
➢ The client module provides a single programming interface with operations
on files similar to those found in conventional file systems.
➢ The design is open in the sense that different client modules can be used
to implement different programming interfaces, simulating the file
operations of a variety of different operating systems and optimizing the
performance for different client and server hardware configurations.
➢ The division of responsibilities between the modules can be defined as
follows:
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML
V SEM AI & ML Distributed Systems MODULE-2
1. Flat file service: The flat file service is concerned with implementing
operations on the contents of files.
➢ Unique file identifiers (UFIDs) are used to refer to files in all requests for
flat file service operations.
➢ UFIDs are long sequences of bits chosen so that each file has a UFID that
is unique among all of the files in a distributed system.
➢ When the flat file service receives a request to create a file, it generates a
new UFID for it and returns the UFID to the requester.
2. Directory service: The directory service provides a mapping between text
names for files and their UFIDs.
➢ Clients may obtain the UFID of a file by quoting its text name to the
directory service.
➢ The directory service provides the functions needed to generate directories,
to add new file names to directories and to obtain UFIDs from directories.
➢ It is a client of the flat file service; its directory files are stored in files of
the flat file service.
3. Client module: A client module runs in each client computer, integrating
and extending the operations of the flat file service and the directory service
under a single application programming interface that is available to user-
level programs in client computers.
➢ The client module also holds information about the network locations of
the flat file server and directory server processes.
➢ Finally, the client module can play an important role in achieving
satisfactory performance through the implementation of a cache of recently
used file blocks at the client.
Flat file service interface: This is the RPC interface used by client modules.
It is not normally used directly by user-level programs.
Fig 1.6 Flat file service operations
➢ A FileId is invalid if the file that it refers to is not present in the server
processing the request or if its access permissions are inappropriate for
the operation requested.
➢ All of the procedures in the interface except Create throw exceptions if the
FileId argument contains an invalid UFID or the user doesn’t have
sufficient access rights.
➢ Both the Read and the Write operation require a parameter i specifying a
position in the file.
➢ The Read operation copies the sequence of n data items beginning at item
i from the specified file into Data, which is then returned to the client.
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML
V SEM AI & ML Distributed Systems MODULE-2
➢ The Write operation copies the sequence of data items in Data into the
specified file beginning at item i, replacing the previous contents of the file
at the corresponding position and extending the file if necessary.
➢ Create creates a new, empty file and returns the UFID that is generated.
➢ Delete removes the specified file.
➢ GetAttributes and SetAttributes enable clients to access the attribute
record.
➢ GetAttributes is normally available to any client that is allowed to read the
file.
➢ Access to the SetAttributes operation would normally be restricted to the
directory service that provides access to the file.
➢ The values of the length and timestamp portions of the attribute record are
not affected by SetAttributes; they are maintained separately by the flat
file service itself.
➢ Comparison with UNIX: Our interface and the UNIX file system primitives
are functionally equivalent.
➢ It is a simple matter to construct a client module that emulates the UNIX
system calls in terms of our flat file service and the directory service
operations.
➢ In comparison with the UNIX interface, our flat file service has no open and
close operations – files can be accessed immediately by quoting the
appropriate UFID.
➢ The Read and Write requests in our interface include a parameter
specifying a starting point within the file for each transfer, whereas the
equivalent UNIX operations do not.
➢ In UNIX, each read or write operation starts at the current position of the
read-write pointer, and the read-write pointer is advanced by the number
of bytes transferred after each read or write.
➢ A seek operation is provided to enable the read-write pointer to be explicitly
repositioned.
➢ The interface to our flat file service differs from the UNIX file system
interface mainly for reasons of fault tolerance:
a. Repeatable operations: With the exception of Create, the operations are
idempotent, allowing the use of at-least-once RPC semantics – clients may
repeat calls to which they receive no reply.
➢ Repeated execution of Create produces a different new file for each call.
➢ Stateless servers: The interface is suitable for implementation by
stateless servers.
➢ Stateless servers can be restarted after a failure and resume operation
without any need for clients or the server to restore any state.
➢ Access control: In the UNIX file system, the user’s access rights are
checked against the access mode (read or write) requested in the open call
and the file is opened only if the user has the necessary rights.
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML
V SEM AI & ML Distributed Systems MODULE-2
➢ The user identity (UID) used in the access rights check is retrieved during
the user’s earlier authenticated login and cannot be tampered with in non-
distributed implementations.
➢ The resulting access rights are retained until the file is closed, and no
further checks are required when subsequent operations on the same file
are requested.
➢ In distributed implementations, access rights checks have to be performed
at the server because the server RPC interface is an otherwise unprotected
point of access to files.
➢ A user identity has to be passed with requests, and the server is vulnerable
to forged identities.
➢ Furthermore, if the results of an access rights check were retained at the
server and used for future accesses, the server would no longer be
stateless.
➢ Two alternative approaches to the latter problem can be adopted:
1. An access check is made whenever a file name is converted to a UFID,
and the results are encoded in the form of a capability which is
returned to the client for submission with subsequent requests.
2. A user identity is submitted with every client request, and access
checks are performed by the server for every file operation.
➢ Directory service interface: The primary purpose of the directory service
is to provide a service for translating text names to UFIDs.
➢ In order to do so, it maintains directory files containing the mappings
between text names for files and UFIDs.
➢ Each directory is stored as a conventional file with a UFID, so the directory
service is a client of the file service.
➢ We define only operations on individual directories. For each operation, a
UFID for the file containing the directory is required (in the Dir parameter).
➢ The Lookup operation in the basic directory service performs a single
Name → UFID translation.
➢ It is a building block for use in other services or in the client module to
perform more complex translations, such as the hierarchic name
interpretation found in UNIX.
Fig 1.7 Directory service operations
➢ UnName removes an entry from a directory and decrements the reference
count. If this causes the reference count to reach zero, the file is removed.
GetNames is provided to enable clients to examine the contents of
directories and to implement pattern-matching operations on file names
such as those found in the UNIX shell.
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML
V SEM AI & ML Distributed Systems MODULE-2
➢ It returns all or a subset of the names stored in a given directory.
➢ The names are selected by pattern matching against a regular expression
supplied by the client.
➢ The provision of pattern matching in the GetNames operation enables
users to determine the names of one or more files by giving an incomplete
specification of the characters in the names.
➢ Hierarchic file system: A hierarchic file system such as the one that UNIX
provides consists of a number of directories arranged in a tree structure.
➢ Each directory holds the names of the files and other directories that are
accessible from it.
➢ Any file or directory can be referenced using a pathname – a multi-part
name that represents a path through the tree.
➢ The root has a distinguished name, and each file or directory has a name
in a directory.
➢ A UNIX-like file-naming system can be implemented by the client module
using the flat file and directory services that we have defined.
➢ A tree-structured network of directories is constructed with files at the
leaves and directories at the other nodes of the tree.
➢ The root of the tree is a directory with a ‘well-known’ UFID.
➢ Multiple names for files can be supported using the AddName operation
and the reference count field in the attribute record.
➢ In a hierarchic directory service, the file attributes associated with files
should include a type field that distinguishes between ordinary files and
directories.
➢ File groups: A file group is a collection of files located on a given server.
➢ A server may hold several file groups, and groups can be moved between
servers, but a file cannot change the group to which it belongs.
➢ A similar construct called a filesystem is used in UNIX and in most other
operating systems.
➢ File groups were originally introduced to support facilities for moving
collections of files stored on removable media between computers. In a
distributed file service, file groups support the allocation of files to file
servers in larger logical units and enable the service to be implemented
with files stored on several servers.
➢ In a distributed file system that supports file groups, the representation of
UFIDs includes a file group identifier component, enabling the client
module in each client computer to take responsibility for dispatching
requests to the server that holds the relevant file group.
➢ File group identifiers must be unique throughout a distributed system.
➢ For example, whenever a new file group is created, a unique identifier can
be generated by concatenating the 32-bit IP address of the host creating
the new group with a 16-bit integer derived from the date, producing a
unique 48-bit integer:
Compiled by: Kavyashree G J
Assistant Professor Dept. of AI&ML