14

How do I output how much of file nominal size is actually filled with data? Like vmtouch shows how much of file is currently in memory...

I expect the workflow to be like this:

$ fallocate -l 1000000 data 
$ measure_sparseness data
100%
$ fallocate -p -o 250000 -l 500000  data
$ measure_sparseness
50%

Workaround: use du -bsh and du -sh and compare them.

  • 1
    related: filefrag for any filesystem and xfs_bmap -vpl for XFS are key tools for showing where the data is (and where the pre-allocated unwritten extents are) when playing around with sparse files and hole-punching. – Peter Cordes Aug 7 '16 at 1:53
  • filefrag data -> multiple FIBMAP: Invalid argument -> data: 1 extent found... – Vi. Aug 7 '16 at 11:53
  • on what filesystem? filefrag -e works perfectly on XFS and ext4 at least. I haven't tested on others. It uses FIEMAP (extent-map), with a fallback to FIBMAP. If those ioctls don't work, then it won't be useful. – Peter Cordes Aug 7 '16 at 15:07
  • On tmpfs. My filefrag doesn't have -e option. – Vi. Aug 7 '16 at 23:45
  • How old is your e2fsprogs? I'm pretty sure it's not a recent feature. There's also a -v option which prints the same verbose info (plus some extra header lines). Maybe your filefrag will have that. Unlike xfs_bmap, though, it doesn't explicitly indicate holes with separate lines, it just has discontinuities in file position. Anyway, I'm not surprised that tmpfs doesn't support FIEMAP, because there is no block device as a backing store, so there's no sensible value for the location of the extents. – Peter Cordes Aug 8 '16 at 0:03
18

find has %S format specifier which is even named "sparseness"

         %S     File's  sparseness.   This  is  calculated as (BLOCKSIZE*st_blocks / st_size).  The exact value you will get for an ordinary file of a certain
                 length is system-dependent.  However, normally sparse files will have values less than 1.0, and files which use indirect  blocks  may  have  a
                 value which is greater than 1.0.   The value used for BLOCKSIZE is system-dependent, but is usually 512 bytes.   If the file size is zero, the
                 value printed is undefined.  On systems which lack support for st_blocks, a file's sparseness is assumed to be 1.0.
$ fallocate -l 1000000 data
$ find data -printf '%S\n'
1.00352
$ fallocate -p -o 250000 -l 500000  data
$ find data -printf '%S\n'
0.507904
  • Interesting. Most regular files on a system will have sparseness above 1.0, directories, softlinks and sockets will always have exactly 1.0. – grochmal Aug 6 '16 at 22:36
  • Didn't some systems save (short) symlink directly in the inode, without using data blocks at all? Wonder what the sparseness of that should be. Besides, isn't that definition the wrong way around, surely a normal (i.e. non-sparse) file should have sparseness zero? :) – ilkkachu Aug 6 '16 at 23:24
  • @grochmal, on ext4 (Linux): ln -s foo link, "sparseness" of link: 0. Sockets and FIFOs have length zero, so find shows sparseness 1. – ilkkachu Aug 7 '16 at 10:55
1

If your find doesn't have that option, a method that has work on UNIX since the 70's is:

ls -ls file

Which will print the actual number of blocks used and the highest byte ever written. From that you can easily compute how many blocks actually have not been allocated.

0

While find's %S will print a brief output, for more detail you might want to look at sparsetest which I wrote - open source, and on github here. Feel free to modify it if you want to print out (e.g.) every hole.

Blog article showing problems with sparse allocations here using sparsetest to debug the issue.

  • Can it print a "map" of extents in a file, like vmtouch -v prints map of cached areas in the file? – Vi. Aug 7 '16 at 11:54
  • @Vi. I wrote it a good while ago and forgot some details - what it's actually doing is creating a sparse file, writing data to it, then printing statistics. You just want the statistic creating bit. To print holes you will need lseek with SEEK_HOLE and SEEK_DATA. Easy to do. – abligh Aug 7 '16 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.