25

Is there a straightforward way to find all the sparse files on my system, or in a particular directory tree?

If it's relevant, I'm using zsh on Ubuntu 12.04, although a more generic Unix-y answer for bash/sh, for example, would be fine.

Edit: to clarify, I'm looking to search for sparse files, not check the sparseness status of a single one.

2

5 Answers 5

14

On systems (and file systems) supporting the SEEK_HOLE lseek flag (like your Ubuntu 12.04 on ext4 would) and assuming the value for SEEK_HOLE is 4 as it is on Linux:

if perl -le 'seek STDIN,0,4;$p=tell STDIN;
   seek STDIN,0,2; exit 1 if $p == tell STDIN'< the-file; then
  echo the-file is sparse
else
  echo the-file is not sparse
fi

That shell syntax is POSIX. The non-portable stuff in it are perl and that SEEK_HOLE.

lseek(SEEK_HOLE) seeks to the start of the first hole in the file, or the end of the file if no hole is found. Above we know the file is not sparse when the lseek(SEEK_HOLE) takes us to the end of the file (to the same place as lseek(SEEK_END)).

If you want to list the sparse files:

find . -type f ! -size 0 -exec perl -le 'for(@ARGV){open(A,"<",$_)or
  next;seek A,0,4;$p=tell A;seek A,0,2;print if$p!=tell A;close A}' {} +

The GNU find (since version 4.3.3) has -printf %S to report the sparseness of a file. It takes the same approach as frostschutz' answer in that it takes the ratio of disk usage vs file size, so is not guaranteed to report all sparse files (like when there's compression at filesystem level or where the space saved by the holes doesn't compensate for the filesystem infrastructure overhead or large extended attributes), but would work on systems that don't have SEEK_HOLE or file systems where SEEK_HOLE is not implemented. Here with GNU tools:

LC_ALL=C find . -type f ! -size 0 -printf '%S:%p\0' |
  LC_ALL=C awk -v RS='\0' -F : '$1 < 1 {sub(/^[^:]*:/, ""); print}'

(note that an earlier version of this answer didn't work properly when find expressed the sparseness as for instance 3.2e-05. Thanks to @flashydave's answer for bringing it to my attention. LC_ALL=C is need for the decimal radix to be . instead of the locale's one (not all awk implementations honour the locale's setting)

12
  • Same comment as above; I'm looking for a way to find all sparse files, not check a particular file. Commented Aug 12, 2013 at 19:12
  • 1
    Maybe find should also exclude 0-byte-files outright? Commented Aug 12, 2013 at 19:56
  • @frostschutz, good point, answer updated. Commented Aug 12, 2013 at 20:04
  • 1
    @Brian, replace the tr command with xargs -r0 rm -f Commented Jan 30, 2017 at 7:29
  • 1
    @MartinvonWittich, with gawk, you'd need to call it with the POSIXLY_CORRECT env var set for it to recognise the locale's decimal radix. Nevertheless, using LC_ALL=C for both find and awk would make it more portable, I'll edit that in. Commented Oct 26, 2022 at 14:37
9

A file is usually sparse when the number of allocated blocks is smaller than the file size (here using GNU stat as found on Ubuntu, but beware other systems may have incompatible implementations of stat).

if [ "$((`stat -c '%b*%B-%s' -- "$file"`))" -lt 0 ]
then
    echo "$file" is sparse
else
    echo "$file" is not sparse
fi

Variant with find: (stolen from Stephane)

find . -type f ! -size 0 -exec bash -c '
    for f do
        [ "$((`stat -c "%b*%B-%s" -- "$f"`))" -lt 0 ] && printf "%s\n" "$f";
    done' {} +

You'd usually put this in a shell script instead, then exec the shell script.

find . -type f ! -size 0 -exec ./sparsetest.sh {} +
8
  • That may not work if the sparse blocks are not enough to cover for the overhead of indirect blocks in traditional file systems for instance, of if compression instead of sparseness is reducing the amount of allocated space. Commented Aug 12, 2013 at 17:28
  • Sure; SEEK_HOLE is just as problematic though, as it's not supported by many platforms/filesystems. In Linux you could also use FIEMAP/FIBMAP, but FIBMAP in particular is horribly slow... there just doesn't seem to be a good way. Commented Aug 12, 2013 at 17:51
  • Also a lot of these methods require the file to be synced first. Commented Aug 12, 2013 at 17:57
  • Thanks. That doesn't really answer the question, though. I'm not looking to check if a particular file is sparse, but to find all sparse files on the system. Commented Aug 12, 2013 at 19:11
  • 2
    @AndrewFerrier sorry, I guess I thought it was trivial enough to wrap this in a for file in * or find. If you can test a single file, you can test all files... although you do have to exclude directories with this method. Commented Aug 12, 2013 at 19:57
7

You can find sparse files with the %S format in find:

# find / -type f -printf "%S\t%p\n" | gawk '$1 < 1.0 {print}'
0.0139994       /var/log/lastlog
0.959592        /usr/lib/locale/locale-archive
...

Found it in this article: https://www.thegeekdiary.com/how-to-find-all-the-sparse-file-in-linux/

3
  • 1
    Changing this to the accepted answer, since it is more straightforward than the others shown so far. Commented Mar 19, 2020 at 11:59
  • 1
    Note that it can give false negatives. See for instance on a ext4 FS with 4KiB block size (the usual default on Ubuntu). fallocate -l 64M file1; fallocate -pl 4K file1 and truncate -s4K file2; setfattr -n user.foo -v "$(seq 1000)" file2. Both file1 and file2 are sparse files with a 4KiB hole, but %S returns 1 for both. IOW, if $1 is >= 1, you can't tell whether the file is sparse or not. If it's < 1, it should be sparse unless the FS supports compression. Commented Oct 5, 2020 at 12:17
  • @StéphaneChazelas, it is a good point and even sparseness > 1 is described in man find. I guess it depends on why exactly one is looking for sparse files. e.g. if you're looking at your VM images to see if they can possibly grow or some other reason. Commented Oct 5, 2020 at 15:37
4

Stephane Chazelas answer above doesn't take into account the fact that some sparse files with the find %S parameter report the ratio as floating point numbers like

9.31323e-09:./somedir/sparsefile.bin

These can be found in addition with

find . -type f ! -size 0 -printf '%S:%p\0' |
   sed -zn '/^\(0[^:]*:\)\|\([0-9.]\+e-.*:\)/p' |
   tr '\0' '\n'
1

A short script I wrote while trying to find out what are the locations of holes in a file:

#!/usr/bin/python3
import os
import sys
import errno

def report(fname):
    fd = os.open(fname, os.O_RDONLY)
    len = os.lseek(fd, 0, os.SEEK_END)
    offset = 0
    while offset < len:
        start = os.lseek(fd, offset, os.SEEK_HOLE)
        if start == len:
            break
        try:
            offset = os.lseek(fd, start, os.SEEK_DATA)
        except OSError as e:
            if e.errno == errno.ENXIO:
                offset = len
            else:
                raise
        print(f'found hole between 0x{start:08X} and 0x{offset:08X} ({offset - start} bytes)')

if __name__ == '__main__':
    for name in sys.argv[1:]:
        report(name)

This prints stuff like:

$ echo -n 'a' >zeros; truncate -s $((4096*4)) zeros; test/report-holes.py zeros
found hole between 0x00001000 and 0x00004000 (12288 bytes)
1
  • Doesn't answer my question as I was looking for sparse files, not the holes in a specific file, but still a useful/relevant script. Thanks. Upvoted. Commented Feb 13, 2018 at 10:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .