4

My shell engine is either Busybox 1.31.0 or bash 3.2

I need to get the size of the files retrieved from find command.

I've been trying to find only files, which have been modified more than 60 days ago and at the same time get the size of all those files SUMARIZED preferably in a one-liner in MB notation. Here's what I've tried.

find -type f -mtime +60 -print0 | xargs -0 du -smc

and

find -type f -mtime +60 -exec du -smc {} \;

The former retrieves line-by-line all files older than 60 days (no problem at all until here) but it weirdly calculates the size several times between all those lines and at the final line I get a "total" size that does not correspond to the actual total size of the output. Here's what it looks like.

.....
.....
0       ./FOLDER 2018/Copy #183 of ~$DATABASE OTHERS - NOV.18N.xlsx
42      ./FOLDER 2018/F9C8A618.tmp
0       ./FOLDER 2018/Copy #166 of ~$DATABASE PORTFOLIO NOV.18.xlsx
3275    total
10      ./FOLDER 2018/CFDC6981.tmp
2       ./FOLDER 2018/D5AAF4EB.tmp
0       ./LIFE INSURANCE/Copy #15 of ~$Copy of LIFE INSURANCE CLIENTS.xlsx
12      total

The latter's output calculate the size of every coinciding file line-by-line with no total.

What I'm expecting is:

    0       ./FOLDER 2018/Copy #183 of ~$DATABASE OTHERS - NOV.18N.xlsx
    42      ./FOLDER 2018/F9C8A618.tmp
    0       ./FOLDER 2018/Copy #166 of ~$DATABASE PORTFOLIO NOV.18.xlsx
    10      ./FOLDER 2018/CFDC6981.tmp
    2       ./FOLDER 2018/D5AAF4EB.tmp
    0       ./LIFE INSURANCE/Copy #15 of ~$Copy of LIFE INSURANCE CLIENTS.xlsx
    54      total

Or simply just the real size result without all the lines

54      total

Any help would be well received.

  • Do you want the size of the output of find, or the size of the files returned by find? Your commands are doing the second, but your question asks for the first. Also, what find is this? Do you have GNU tools? You mention busybox, so I guess this isn't a full GNU/Linux system? – terdon Sep 25 at 16:18
  • With GNU find you can use -printf %b and get rid of du entirely: find . -mtime +60 -type f -printf '%b\n' | awk '{s+=$1}END{print s/2048"M"}'. With busybox find and du, you should use du -smc $(find ...) and hope for the best ;-) Whether your shell is bash or busybox matters very little. – mosvy Sep 25 at 17:40
  • Do you want the sum of the sizes of those files, the sum of the disk usage of those files or the cumulative disk usage of those files. All 3 are different things. The last 2 are different if some of the files are hard linked together or a copy-on-write copies of each other. – Stéphane Chazelas Sep 27 at 8:23
  • Corrected version of the above which only sums hard links once: find . -type f -mtime +60 -printf '%D:%i %b\n' | awk '!a[$1]++{s+=$2}END{print s/2048"M"}'. Just like du, this is not smart enough to do anything sensible about file referenced via hard links from somewhere else. – mosvy Sep 27 at 19:48
3

Try pipe the output of find to du and specify the --files0-from - flag:

find -type f -mtime +60 -print0 | du -shc --files0-from -

This should give you a grand total at the end

To get just the total, pipe that output to tail -n1:

find -type f -mtime +60 -print0 | du -shc --files0-from - | tail -n1

I should mention that I actually just tested this with gnu linux, not busybox. Looking at the busybox page, it does not look like du supports the --files0-from option.

You can change the above command to this to have it work on busybox:

find -type f -mtime +60 -print0 | xargs -0 du -ch | tail -n1

The above also works with files with spaces and newlines in their names, but may not work well if there are too many files found by find command. See the below comment. If you feel that there may be too many files, you can try the other answer on this page.

  • 1
    Great contribution @smac89 Your first two commands worked just fine. They piped correctly the output and executed what I needed. You're right, in the Busybox page there is not support for --files0-from in du but they got the job done. Contrary to this, the latter command didn't work, it printed some du:invalid option exceptions. – aldehc99 Sep 25 at 20:08
  • @aldehc99 glad to hear it. What was the error you get with the last command? – smac89 Sep 25 at 20:16
  • I have tested it with several files and realized it cannot handle filenames containing spaces. It seems to split said filenames, making the other part of them look like other commands: du: cannot access `./TESTINGFILE': No such file or directory When the filename does not contain spaces it works as expected. – aldehc99 Sep 25 at 20:49
  • 1
    There's no guarantee that xargs won't run the du multiple times, and unlike a dumb du -sch $(find ..) that will not give the user any indication that there are too many files. Notice that xargs will run its command with argument batches much smaller than the maximum of the OS. On busybox: yes badger | xargs sh -c 'echo $#' sh | sed 1q => 4678. – mosvy Sep 27 at 5:21
3

In principle this is easy: just tell find to run du on a bunch of files at once.
find . -type f -mtime +60 -exec du -smc {} +
Unfortunately this doesn't work reliably, because -exec … {} + can execute the command multiple times, it only tries to group the arguments, and it cannot possibly group all the arguments if their total length would go over the system's command line length limit. And in fact BusyBox find (at least the version I tested just now) doesn't try grouping at all: -exec … {} + processes one argument at a time, line -exec … {} \;. There's no way to be sure to get a single total line.

GNU du can be told to read an arbitrarily long list of file names with --files0-from, but other versions of du, in particular the one in BusyBox, can only take file names from the command line.

So if you can't assume that you have GNU du, there's no way to avoid running du multiple times, and this means you need another tool to do the summing, which in turn requires that du doesn't round the sizes. The summing is simple with awk if the output of du is parseable.

If you can assume that there are no newlines in file names, or you're ok with excluding paths that contain newlines, the output of du is easy to parse: just one file per line.

newline='
'
find . ! -path "*${newline}*" -type f -mtime +60 -exec du -k {} + |
awk '{kB += $1} END {printf "%d MB\n", (kB + 512) / 1024}'
  • Note that would give different results with busybox and GNU du if there were hard links between those old files (and some potentially nondeterministic result with GNU du if several instances are run). – Stéphane Chazelas Sep 27 at 16:08
  • The newline issue can be addressed by using find .//. – Stéphane Chazelas Sep 27 at 16:09
  • Whether -exec cmd {} + runs one cmd per file or more in busybox depends on whether the FEATURE_FIND_EXEC_PLUS option was enabled at build time or not. – Stéphane Chazelas Sep 27 at 21:39
2

If you wanted the cumulative disk usage (as your usage of du suggests) of the regular files that are over 60 days old and need only to be portable to GNU and busybox systems (though note that which commands are included in busybox and what feature they support is configurable at build time, so you can never know if what works with one instance of busybox will work with the next), you can do:

find . -type f -mtime +59 -print0 |
  xargs -r0 stat -c '%D:%i %b' | awk '
    !seen[$1]++ {sum += $2}
    END {print sum * 512}'

(and yes, you need -mtime +59 for files older than 60 x 24 hours. -mtime +60 would not match on a file that is 60.9 days old as that's rounded down to 60 days and 60 is not greater than 60)

That reports the total in number of bytes. Hard links (or other cases such as bind-mounts where there may be several paths to the same file) are counted only once (like GNU du does; busybox du doesn't do it if the hardlinks are passed as separate arguments as opposed to found in the traversal of a single directory argument). However, like du, it won't detect the cases where some data is shared between non-hardlinked files, like when files have been copied with cp --reflink=always on filesystems like btrfs or when deduplication is performed by the file system.

That should be equivalent to the GNU-specific:

find . -type f -mtime +59 -print0 |
  du -cB1 --files0-from=- |
  awk 'END{print $1}'

POSIXly, and assuming all files are on the same file system, you could do:

LC_ALL=C LS_BLOCK_SIZE=512 BLOCKSIZE=512 POSIXLY_CORRECT=1 \
  find . -type f -mtime +59 -exec ls -nisqd {} + | awk '
    !seen[$1]++ {sum += $2}
    END {print sum * 512}'

(with LS_BLOCK_SIZE=512 BLOCKSIZE=512 POSIXLY_CORRECT=1 to work around the fact that some ls implementations like GNU ls are not POSIX compliant by default. It won't work with busybox ls which doesn't support -q. However since it always renders newline characters in file paths as ? (which is also not POSIX compliant), -q is not needed there).

After (here on a GNU system):

$ seq 10000 > a
$ truncate -s14T a
$ ln a b
$ touch -d '-60 days' a
$ BLOCKSIZE=1 ls -lis --full-time
total 98304
59944369 49152 -rw-rw-r-- 2 me me 15393162788864 2019-07-29 09:49:25.933 +0100 a
59944369 49152 -rw-rw-r-- 2 me me 15393162788864 2019-07-29 09:49:25.933 +0100 b
$ date --iso-8601=s
2019-09-27T09:50:03+01:00
$ du -h
52K     .

All give me 49152, which is the cumulative disk usage of both a and b but is different from the sum of their size (28 TiB) or the size of their disk usage (49152 x 2).

(note that the 52K above also includes the disk usage of the current directory file (., 4KiB in my case)).

For the sum of the apparent sizes.

find . -type f -mtime +59 -print0 |
  xargs -r0 stat -c %s | awk -v sum=0 '
    {sum += $0}; END{print sum}'

Or with GNU du:

find . -type f -mtime +59 -print0 |
  du -cbl --files0-from=- |
  awk 'END{print $1}'

Or POSIXly (here without the restriction about single file system):

LC_ALL=C find . -type f -mtime +59 -exec ls -nqd {} + |
  awk -v sum=0 '{sum += $5}; END {print sum}'

On the above example, they all give: 30786325577728 (28 TiB).

  • There's no stat in popular busybox-based systems like OpenWRT/LEDE. – mosvy Sep 27 at 20:40
  • @mosvy, thanks. I've added a note that YMMV, all commands and features are optional in busybox so one can never know. In any case, since they're using 1.31.0, the OP is probably not using OpenWRT (openwrt.org/packages/pkgdata/busybox shows 1.28.4 atm) – Stéphane Chazelas Sep 27 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.