How to find the largest open files?

Question

Looking for the largests among the open files by all processes. lsof already has the open files with their sizes. It may be passing the right parameters to lsof and processing the output.

Answer 1

You can use the -F option of lsof to get almost unambiguous output which is machine-parseable with only moderate pain. The output is ambiguous because lsof rewrites newlines in file names to \n.

The lsof output consists of one field per line. The first character of each name indicates the field type and the rest of the line is the field value. The fields are: p=PID (only for the first descriptor in a given process), f=descriptor, t=type (REG for regular files, the only type that has a size), s=size (only if available), n=name. The awk code below collects entries that have a size and prints the size and the file name. The rest of the pipelines sorts the output and retains the entry with the largest size.

lsof -Fnst | awk '
    { field = substr($0,1,1); sub(/^./,""); }
    field == "p" { pid = $0; }
    field == "t" { if ($0 == "REG") size = 0; else next; }
    field == "s" { size = $0; }
    field == "n" && size != 0 { print size, $0; }
' | sort -k1n -u | tail -n42 | sed 's/^[0-9]* //'

Answer 2

One convoluted way looks like this:

lsof \
| grep REG \
| grep -v "stat: No such file or directory" \
| grep -v DEL \
| awk '{if ($NF=="(deleted)") {x=3;y=1} else {x=2;y=0}; {print $(NF-x) "  " $(NF-y) } }'  \
| sort -n -u  \
| numfmt  --field=1 --to=iec

Output

...
....
129M  /var/log/maillog
166M  /var/log/nginx/access_log
172M  /var/log/metrics/kubernetes/kubelet.log
185M  /var/log/metrics/kubernetes/etcd.log
257M  /var/log/metrics/kubernetes/etcd.log.1
335M  /var/log/metrics/kubernetes/kubelet.log.1

I know this is not perfect. For example if the file name contains "DEL" this would purge that file from the output list. lsof also has a -F option described in the OUTPUT FOR OTHER PROGRAMS section. Using that may be simpler.

Details

lsof prints something like this:

COMMAND      PID    TID           USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
systemd        1                  root  cwd       DIR              253,0      4096        128 /
tuned       2975                  root    7u      REG              253,0      4096  805307770 /tmp/ffiKkVeXD (deleted)
python2    49888  49890           root  DEL       REG               0,18            196039884 /dev/shm/sem.NXPFow
systemd        1                  root  mem       REG              253,0     90664      10063 /usr/lib64/libz.so.1.2.7
java      149435 175229            box   69r      REG              253,0 350872273  808108999 /box/var/log/metrics/kubernetes/kubelet.log.1
java      149435 149580            box  107w     FIFO                0,8       0t0  272526226 pipe
prometheu 147867 148211           root  mem       REG              253,6             31457463 /lib64/ld-2.12.so (stat: No such file or directory)

---

grep REG keep regular files

---

grep -v "stat: No such file or directory" Remove the files that have stat error. (I do not know why this happens)

---

grep -v DEL Discard the Linux map files that have been deleted;

From lsof documentation:

''DEL'' for a Linux map file that have been deleted;

---

After this processing we are left with something like this:

tuned       2975                  root    7u      REG              253,0      4096  805307770 /tmp/ffiKkVeXD (deleted)
systemd        1                  root  mem       REG              253,0     90664      10063 /usr/lib64/libz.so.1.2.7
java      149435 175229            box   69r      REG              253,0 350872273  808108999 /box/var/log/metrics/kubernetes/kubelet.log.1

The size is either the 3rd or the 2nd column from last depending on the value of the last column. If the last column is (deleted) pick the 3rd from last otherwise 2nd.

awk '{if ($NF=="(deleted)") {x=3;y=1} else {x=2;y=0}; {print $(NF-x) " " $(NF-y) } }'

---

sort -n -u | numfmt --field=1 --to=iec

Sort, uniquify and make the bytes count human readable

Answer 3

You can do the following

lsof | grep REG | awk '{ print $1,$7,$9 }' | sort -t ' ' -k 2 -V

Using the awk you filter the output to include command, size and filename and sort it based on the 2nd column, which is size. -t specifies the delimiter, -V sort 'naturally' - so 1, 2, 10 will be sorted this way instead of 1, 10, 2. -k is the key for the sort (the column you want to sort by)