How can find support large files?

Question

I would like to find all the files of my system, which sizes are more than a couple gigabits. I thought I would use find -size, but the man page states:

The size does not count indirect blocks

If I understand this correctly, the search is only relevant when the size is below (number of direct blocks)×(size of a block) = 10*4096 = 40MiB. So how can I list files larger than that?

EDIT: I must be wrong somewhere as the man page supports Gigabits as a size unit. Anyone can see where I would be wrong? Changed the title to reflect this.

Answer 1

I think this link might clear things up, The Open Group Base Specifications Issue 7, IEEE Std 1003.1, 2013 Edition. Here's an excerpt of the section form the specification for find:

excerpt from the find spec

The -size operand refers to the size of a file, rather than the number of blocks it may occupy in the file system. The intent is that the st_size field defined in the System Interfaces volume of POSIX.1-2008 should be used, not the st_blocks found in historical implementations. There are at least two reasons for this:

In both System V and BSD, find only uses st_size in size calculations for the operands specified by this volume of POSIX.1-2008. (BSD uses st_blocks only when processing the -ls primary.)

Users usually think of file size in terms of bytes, which is also the unit used by the ls utility for the output from the -l option. (In both System V and BSD, ls uses st_size for the -l option size field and uses st_blocks for the ls -s calculations. This volume of POSIX.1-2008 does not specify ls -s.)

If I understand this section correctly, the first section says it all "The -size operand refers to the size of a file". So the size is what's reported when evaluating st_size NOT st_blocks.

Examples

So you should be able to use commands like:

# find files over 1G in size
$ find / -type f -size +1G

# find files smaller than 1G in size
$ find / -type f -not -size +1G

References

• File System Implementation
• Tuning the Linux file system Ext3

Answer 2

That sentence is severely misleading. If you don't want to get into gory details, forget you read it and just assume that -size looks at the size of the file.

The size of a file is the number of bytes that you can read from it. A file of size N is an array of N bytes.

When you pass a unit other than bytes (c) to -size, the find command rounds the file size up to a multiple of the unit. For example, -size 11k matches files of 10,240 to 11,263 bytes; -size 12345M maches files of 12,943,622,145 to 12,944,670,720 bytes. The default unit, for historical reasons, is called blocks and its value is 512 bytes.

That was the easy part. Now on typical filesystems, the data of a file is stored in blocks. For a filesystem whose block size is 512 bytes¹, a 5123-byte file would occupy 11 data blocks (the last one being only partially used). So find -size 11 normally matches files that consist of 11 blocks.

In fact there are possible complications. The OS needs to have a place where to store the location of all the blocks that make up the file. If there are too many blocks, it needs to allocate some more blocks just to contain the addresses of other blocks. Such blocks are called indirect blocks. The find manual tells you that these blocks are not taken into account — which is unsurprising since find isn't counting blocks, it's looking at the file size.

Conversely, it's possible to have a file that uses fewer blocks than you'd expect from its size, because of compression. Classical unix filesystems only implement a crude form of compression: blocks that consist exclusively of null bytes may be omitted. This is known as sparse files.

A program can know how many blocks have been allocated to a file; this is the st_blocks field of the stat structure, as opposed to st_size. GNU find only uses this in its display code for -ls and -printf, never for any predicate. The st_blocks value gives a crude indication of how much space the file occupies on the disk, which can be less than the file size if the file is sparse, but it does not take indirect blocks into account.

¹ _{Ext2, ext3 and ext4 have blocks of 1kB, 2kB or 4kB.}