Hands on with lsof

[Rob]

lsof stands for "List Open Files," and as the name implies, it's a command-line tool in Linux that provides a detailed list of all open files on the system. Open files in this context refer to any type of file, including regular files, directories, sockets, pipes, and more. The information provided by lsof is extremely useful for system administrators, as it provides insight into which files and sockets are being used by which processes, and can help with troubleshooting, performance optimization, and security assessments.

Basic Usage

The basic syntax for the lsof command is straightforward: simply run "lsof" followed by any options you want to specify. By default, lsof will display a list of all open files for all processes running on the system, with columns for the process ID (PID), process name, file descriptor, file type, and file name. Here's an example:

rob@buddy:~$ sudo lsof|head
COMMAND    PID  TID TASKCMD               USER   FD      TYPE             DEVICE SIZE/OFF       NODE NAME
systemd      1                            root  cwd       DIR              253,0     4096          2 /
systemd      1                            root  rtd       DIR              253,0     4096          2 /
systemd      1                            root  txt       REG              253,0  1849992    2103000 /usr/lib/systemd/systemd
systemd      1                            root  mem       REG              253,0   149760    2103465 /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.32.1
systemd      1                            root  mem       REG              253,0    27072    2103407 /usr/lib/x86_64-linux-gnu/libcap-ng.so.0.0.0
systemd      1                            root  mem       REG              253,0   613064    2098237 /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.10.4
systemd      1                            root  mem       REG              253,0   170456    2103513 /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5
systemd      1                            root  mem       REG              253,0   841808    2103663 /usr/lib/x86_64-linux-gnu/libzstd.so.1.4.8
systemd      1                            root  mem       REG              253,0  4451632    2097734 /usr/lib/x86_64-linux-gnu/libcrypto.so.3

This output shows the process ID (PID), thread id (TID), task command name (TASKCMD), user running the process (USER), file descriptor (FD), type of file (TYPE), device (DEVICE), size (SIZE/OFF), and file name (NAME) for each open file.

Options

lsof provides a wide range of options that allow you to customize the information it provides. Here are some of the most useful options:

-c: This option allows you to display only the open files for processes with a specific command name. For example, to see all open files for the "nginx" process, you would run:

$ lsof -c nginx

-u: This option allows you to display only the open files for processes running under a specific user. For example, to see all open files for the "root" user, you would run:

$ lsof -u root

-p: This option allows you to display only the open files for a specific process ID. For example, to see all open files for the process with PID 123, you would run:

$ lsof -p 123

-i: This option allows you to display only the open network files. For example, to see all open network files, you would run:

$ lsof -i

-r: This option allows you to repeat the display of open files at a specified interval. For example, to repeat the display every 5 seconds, you would run:

$ lsof -r 5

Interpreting the Output

The information provided by lsof can be a bit overwhelming, but with a little understanding of what each column represents, you can gain valuable insight into the open files on your system. Here are the most important columns:

- PID: The process ID of the process that has the file open
- TID: The thread identifier within a process
- TASKCMD: The name of the command that's running the process.
- USER: The username of the user who is running the process.
- FD: The file descriptor, which is a unique identifier for the file within the process. The first three file descriptors (0, 1, and 2) are reserved for standard input, output, and error, respectively.
- TYPE: The type of file, which can be one of the following:

• REG: A regular file.
• DIR: A directory.
• CHR: A character special file.
• BLK: A block special file.
• FIFO: A named pipe.
• LNK: A symbolic link.
• SOCK: A socket.
• UNIX: A Unix domain socket.

- DEVICE: The device on which the file resides, represented as a major and minor device number.
- SIZE/OFF: The size of the file in bytes, or the offset of the file in a device if the file type is a block or character special file.
- NODE: The inode number of the file, which is a unique identifier for the file on the file system.
- NAME: The name of the file.

Situations where you might use lsof

There are several common Linux issues that can be resolved with the help of information obtained from running the lsof command:

• Disk space usage: If you are running out of disk space, you can use lsof to see which processes are using the most disk space. By using the -d option and specifying a file descriptor, you can see all the files that are open by a particular process.
• Process hanging: If a process appears to be hanging, you can use lsof to see which files it has open. If the process has a large number of open files, it may be a sign that it's stuck waiting for I/O.
• Debugging network issues: If you're having trouble with network connectivity, you can use lsof to see which sockets are open on the system. By using the -i option, you can see all the Internet sockets that are open, including both incoming and outgoing connections.
• Troubleshooting file locks: If you're having trouble with file locks, you can use lsof to see which processes have a particular file open. By using the -t option, you can see the process IDs of all the processes that have a particular file open.
• Understanding the file system layout: If you're trying to understand the layout of a file system, you can use lsof to see which files are open on the system. By using the -a option, you can see all the open files, including those that are not associated with a process.
• Monitoring system activity: If you're trying to monitor system activity, you can use lsof to see which files are open on the system. By using the -p option, you can see the open files for a specific process.
• Detecting rogue processes: If you suspect that there is a rogue process running on your system, you can use lsof to see which files it has open. By using the -u option, you can see all the open files for a specific user.

These are just a few examples of how the lsof command can be used to resolve common Linux issues. With its wide range of options and powerful features, lsof is a versatile tool that can help you diagnose and resolve many different types of problems on your Linux system.

Conclusion

The lsof command is a powerful and versatile tool for Linux administrators, providing a wealth of information about open files on the system. Whether you're troubleshooting a problem, optimizing performance, or assessing security, lsof can help you get the information you need. Whether you're new to Linux or a seasoned administrator, it's well worth taking the time to learn the ins and outs of this essential tool.

 

[ks4ji]

Very helpful and informative. Thank you.

 

[osprey]

Great heads up Rob.
Recently in debian bookworm there was a transition in sound server happening from pulseaudio to pipewire which had a few intermediate states before one took over from the other. It wasn't always clear which sound server had control of the sound devices in the system, so as I was moving through the process, it was lsof that kept me most informed. Today it's clear:

[flip@flop ~]$ lsof /dev/snd/*
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
pipewire  1069  flip   46u   CHR  116,1      0t0  342 /dev/snd/seq
pipewire  1069  flip   47u   CHR  116,1      0t0  342 /dev/snd/seq
wireplumb 1070  flip   25u   CHR 116,16      0t0  552 /dev/snd/controlC0
wireplumb 1070  flip   27u   CHR 116,10      0t0  531 /dev/snd/controlC1