New Implementation of a Log-structured File System (NILFS)


Jarret W. Buse

New Implementation of a Log-structured File System (NILFS)

The NILFS file system was created by Nippon Telegraph and Telephone Corporation (NTT) at the Research and Development department of CyberSpace Laboratories. NTT was established in 1952 as a Japanese telecommunications company owned by the government. NILFS was released under the General Public License (GPL).

NILFS was introduced in the Linux kernel 2.6.30. The current version of NILFS is version 2, often referred to as NILFS2. It is a loadable kernel module that does not require the kernel to be recompiled.

The inode addresses are 64-bit in size and can manage volume and file sizes up to 8 EB.

File and inode management are maintained in B-Tree indexes. The B-Tree indexes allow for faster file and inode searches.

NILFS uses a continuous checkpoint (snapshot) system which occurs after so many writes or a certain period of time has elapsed. These checkpoints can be mounted as Snapshots. Files can usually be recovered which have been deleted moments before. Whether the file has been deleted, modified or corrupted, file recovery is easy. The number of snapshots is limitless in the number created, until the volume becomes too full. The log file used to store the checkpoints is not circular. That is, as the file reaches a certain size, the older entries are deleted and overwritten. So metadata and data are written sequentially into the log. With NILFS, the Journal is appended, or added to as the checkpoints are processed. When a system crash occurs, the file system can perform a very quick recovery. The recovery time is quick, yet the size of the storage unit does not change the recovery time.

You can execute the command lscp to get a listing of the checkpoints and snapshots. A list is generated of the Checkpoint Number (CNO), the date and time the checkpoint was created, a Mode (‘cp’ for checkpoint and ‘ss’ for snapshot). Checkpoints (cp) can be made into snapshots (ss) by using the mkcp –s command. Use the chcp command to convert a checkpoint to a snapshot or vice versa. To create an immediate checkpoint, use the mkcp command. Snapshots can be deleted by using the rmcp command.

Because of the Snapshots, backups are easy to manage without unmounting the volume and suffering a loss of performance. Since the checkpoint (snapshot) ability is built-in to the file system, there is no need to suspend drive activity while a snapshot is being performed.

A background Garbage Collection system is used to clear deleted data from the log. Without the Garbage Collection, the log would eventually fill the disk. Files can be retrieved until the checkpoint has been deleted during Garbage Collection. Checkpoints will eventually be deleted unless the checkpoint has been made into a Snapshot.

Data written to the log is also protected with CRC32 values on the metadata and data values. This can assure the metadata and data being recovered, especially after a crash, is valid.

For redundancy, the Super Blocks are copied to provide extra copies. If one Super Block should become corrupted, then another can be used to replace the corrupted one.

When a volume becomes full, it is possible to perform Online Resizing to allow the volume size to be increased. The increase can be up to the 8 EB limit.

Where other file systems have issues with storing large dates, the NILFS stores the dates as 64-bits. By using 64-bits, the problem with storing dates past the year 2038 is not a problem. Timestamps are calculated as the number of seconds since 00:00:00 Coordinated Universal Time (UTC) on January 1, 1970 (a Thursday) until present. A 32-bit timestamp allows for the maximum date to be January 29, 2038 at 3:14:08 UTC. Using a 64-bit digit to store the date and time would allow for a maximum date and time of 15:30:08 UTC on December 4, 292,277,026,596 (Sunday).

NOTE: Scientists claim that the sun should become a Red Giant by this date and destroy the Earth in a fiery blaze - :).

NILFS is optimized for performance. An older benchmark test from 2008 shows it compared to other comparable file systems. These results were from an e-mail server. With small files, NILFS was compared to BTRFS, and performed 25-38% faster. For larger files, NILFS was compared to REISERFS/EXT4, with a performance increase of 15-25%.

NOTE: The performance boost is based on Solid-State Disk (SSD) hard disk for which NILFS is optimized. SSD’s have no moving parts in them, such as a drive head or spinning disk. SSD’s are electronic hard disks.

NILFS has many capabilities, and some future enhancements include:


$100 Digital Ocean Credit
Get a free VM to test out Linux!

Staff online