Using HDFS Files
HDFS files are accessed using the HDFS client, which provides an interface between user
applications and the Hadoop Distributed File System. The HDFS client hides the complexity
of HDFS implementation, allowing applications to perform file operations without knowing
the internal storage details. Applications do not need to know where the file blocks are stored
or how many replicas exist.
Hadoop provides an abstract FileSystem class for accessing HDFS. A FileSystem object is
created using a Configuration object, which reads Hadoop configuration files such as core-
[Link] and [Link]. These configuration files should be available in the application's
classpath. This approach simplifies HDFS access and provides transparent, reliable file
operations.
When a client opens a file in HDFS for writing, HDFS grants that client an exclusive write
lease.
A lease is a temporary permission that allows only one client to write to the file.
While one client holds the lease, no other client can modify or write to the same
file.
This prevents data corruption and ensures data consistency.
Example
Suppose Client A opens [Link] for writing.
1. HDFS grants a write lease to Client A.
2. Client B tries to write to the same file.
3. HDFS rejects Client B because Client A owns the lease.
4. If Client A completes writing and closes the file, the lease is released.
5. If Client A crashes without closing the file:
o After the soft limit, the lease can be renewed. After the hard limit, HDFS
automatically closes the file and releases the lease.
Configuration Object
│
▼
Loads Hadoop Configuration
([Link], [Link])
│
▼
[Link](config)
│
▼
Creates FileSystem Object (fsys)
│
▼
Read / Write / Delete / Rename Files
1. Creating a FileSystem Object
Code
Configuration config = new Configuration();
FileSystem fsys = [Link](config);
Step 1: Create a Configuration Object
Configuration config = new Configuration();
Creates an object of the Configuration class.
Loads Hadoop configuration files such as:
o [Link]
o [Link]
Stores all Hadoop settings like:
o NameNode address
o Block size
o Replication factor
Step 2: Create a FileSystem Object
FileSystem fsys = [Link](config);
Creates a FileSystem object.
Connects to the HDFS cluster using the configuration.
The object fsys is used for all file operations.
2. Path Class
A Path object represents the location of a file or directory in HDFS.
Syntax
Path fp = new Path(fileName);
Example
Path fp = new Path("/user/hadoop/[Link]");
The Path object stores only the file path, not the file data.
3. Manipulating HDFS Objects
Create a Path Object
Path fp = new Path(fileName);
Creates an object representing the file or directory.
Check Whether File Exists
if ([Link](fp))
Checks whether the file is present in HDFS.
Returns:
true → File exists
false → File does not exist
Check Whether It Is a File
if ([Link](fp))
Checks whether the given path refers to a file.
Returns:
true → It is a file
false → It is a directory
Create a New File
boolean result = [Link](fp);
Creates an empty file.
Returns:
true → File created successfully
false → File already exists
Delete a File
boolean result = [Link](fp);
Deletes the specified file.
Returns:
true → Successfully deleted
false → Deletion failed
Read a File
FSDataInputStream fin = [Link](fp);
Opens the file for reading.
FSDataInputStream is the input stream used to read data from HDFS files.
Write to a File
FSDataOutputStream fout = [Link](fp);
Creates a file (or overwrites an existing one) and opens it for writing.
FSDataOutputStream is the output stream used to write data to HDFS.
Advantages
Easy access to HDFS.
Hides implementation details.
Automatic handling of replication and failures.
Simple API for developers.
Supports multiple file systems using the same interface.