Assume there's an image storage directory, say, ./photos/john_doe, within which there are multiple subdirectories, where many certain files reside (say, *.jpg). How can I calculate a summary size of those files below the john_doe branch?

I tried du -hs ./photos/john_doe/*/*.jpg, but this shows individual files only. Also, this tracks only the first nest level of the john_doe directory, like john_doe/june/, but skips john_doe/june/outrageous/.

So, how could I traverse the entire branch, summing up the size of the certain files?

up vote 135 down vote accepted
find ./photos/john_doe -type f -name '*.jpg' -exec du -ch {} + | grep total$

If more than one invocation of du is required because the file list is very long, multiple totals will be reported and need to be summed.

  • 4
    find -iname 'file*' -exec du -cb {} + | grep total$ | cut -f1 | paste -sd+ - | bc # summed byte size – Michal Čizmazia Jul 15 '15 at 13:55
  • 2
    If your system works under other language then you need to change total$ to other word like razem$ in Polish. – Zbyszek Jul 26 '15 at 12:49
  • 1
    You can add LC_ALL=POSIX as prefix to always grep for total like this: LC_ALL=POSIX find ./photos/john_doe -type f -name '*.jpg' -exec du -ch {} + | grep total$ – Sven Jun 27 '16 at 5:48
  • 1
    If you're not using -name, then change the grep to grep -P "\ttotal$" or else it will capture all files ending with "total" as well. – thdoan Mar 30 '17 at 7:43
  • 2
    @MichalČizmazia some shells (e.g., Git Bash for Windows) don't come with bc, so here is a more portable solution: find -name '*.jpg' -type f -exec du -bc {} + | grep total$ | cut -f1 | awk '{ total += $1 }; END { print total }' – thdoan Mar 30 '17 at 7:55
du -ch public_html/images/*.jpg | grep total
20M total

gives me the total usage of my .jpg files in this directory.

To deal with multiple directories you'd probably have to combine this with find somehow.

You might find du command examples useful (it also includes find)

  • 1
    This doesn't traverse the underlying directories? – mbaitoff Jun 26 '12 at 5:48

Primarily, you need two things:

  • the -c option to du, to tell it to produce a grand total;
  • either find or ** to traverse subdirectories.
du -ch -- **/*.jpg | tail -n 1
  • very good reply. Simpler than using find (as long * or ** matches the directory structure) – Andre de Miranda Apr 21 '16 at 5:13
  • It can also handle very long lists of files whereas using find can return erroneous results. – Eric Fournie Oct 19 '16 at 8:50

The ultimate answer is:

{ find <DIR> -type f -name "*.<EXT>" -printf "%s+"; echo 0; } | bc

and even faster version, not limited by RAM, but that requires GNU AWK with bignum support:

find <DIR> -type f -name "*.<EXT>" -printf "%s\n" | gawk -M '{t+=$1}END{print t}'

This version has the following features:

  • all capabilities of find to specify the files you're looking for
  • supports millions of files
    • other answers here are limited by the maximum length of the argument list
  • spawns only 3 simple processes with a minimal pipe throughput
    • many answers here spawn C+N processes, where C is some constant and N is the number of files
  • doesn't bother with string manipulation
    • this version doesn't do any grepping, or regexing
    • well, find does a simple wildcard matching of filenames
  • optionally formats the sum into a human-readable form (eg. 5.5K, 176.7M, ...)
    • to do that append | numfmt --to=si
  • I like the simplicity of this answer, although it only worked for me when I introduced spaces after the opening brace and before the closing brace. I do wonder if it will really support an 'infiinte' number of files though :) – andyb Feb 7 '17 at 0:29
  • @andyb thanks for the feedback, the spaces around braces are indeed required in BASH, I'm using ZSH so I didn't notice that. And the number of files is limited by the available RAM on your system as bc's memory usage grows slowly as the numbers flow in. – rindeal Feb 7 '17 at 17:31

The answers given until now do not take into account that the file list passed from find to du may be so long that find automatically splits the list into chunks, resulting in multiple occurences of total.

You can either grep total (locale!) and sum up manually, or use a different command. AFAIK there are only two ways to get a grand total (in kilobytes) of all files found by find:
find . -type f -iname '*.jpg' -print0 | xargs -r0 du -a| awk '{sum+=$1} END {print sum}'

Explanation
find . -type f -iname '*.jpg' -print0: Find all files with the extension jpg regardless of case (i.e. *.jpg, *.JPG, *.Jpg...) and output them (null-terminated).
xargs -r0 du -a: -r: Xargs would call the command even with no arguments passed, which -r prevents. -0 means null-terminated strings (not newline terminated).
awk '{sum+=$1} END {print sum}': Sum up the file sizes output by the previous command

And for reference, the other way would be
find . -type f -iname '*.jpg' -print0 | du -c --files0-from=-

  • Additional hint: On my HDD with 23428 files (22323 being images) the first method runs 1 sec while the second one runs 3.8 secs. – Jan Aug 5 '14 at 10:12
  • Note that both assume a GNU system. The first one assumes file names don't contain newline characters. – Stéphane Chazelas Aug 6 '14 at 13:06
  • I'd bet the du --file0-from took longer because you ran it first (caching effect). – Stéphane Chazelas Aug 6 '14 at 13:07
  • With xargs, several du -a may be run, so you may have discrepancies if there are hard links. – Stéphane Chazelas Aug 6 '14 at 13:09

If the list of files is too big that it can't be passed to a single invocation of du -c, on a GNU system, you can do:

find . -iname '*.jpg' -type f -printf '%b\t%D:%i\n' |
  sort -u | cut -f1 | paste -sd+ - | bc

(size expressed in number of 512 byte blocks). Like du it tries to count hard links only once. If you don't care about hardlinks, you can simplify it to:

(printf 0; find . -iname '*.jpg' -type f -printf +%b) | bc

If you want the size instead of disk usage, replace %b with %s. The size will then be expressed in bytes.

  • -bash: bc: command not found Centos - Linux 2.6.32-431.el6.x86_64 – yeya Jan 10 at 11:32
  • @yeya, sounds like your CentOS deployment is broken. bc is a non-optional POSIX command. – Stéphane Chazelas Jan 10 at 11:35

The solutions mentioned so far are inefficient (exec is expensive) and require additional manual work to sum if the file list is long or they don't work on Mac OS X. The following solution is very fast, should work on any system, and yields the total answer in GB (remove a /1024 if you want to see the total in MB): find . -iname "*.jpg" -ls |perl -lane '$t += $F[6]; print $t/1024/1024/1024 . " GB"'

  • Neither -iname nor -ls are standard/portable, so it won't work on any system either. It will also not work properly if there are filenames or symlink targets that contain newline characters. – Stéphane Chazelas Jun 22 '16 at 8:28
  • Also note that it gives the sum of the file sizes, not their disk usage. For symlinks, it gives the size of the symlinks, not the files they point to. – Stéphane Chazelas Jun 22 '16 at 8:31

du naturally traverses the directory hierarchy and awk can perform the filtering so something like this may be sufficient:

du -ak | awk 'BEGIN {sum=0} /\.jpg$/ {sum+=$1} END {print sum}'

This works without GNU.

  • 1
    This is more expensive since it entails a stat call for files that do not correspond to the searched-for pattern. – Law29 Jul 1 '16 at 9:03
  • Only this solution works on my mac. – Matthias M May 26 '17 at 10:04

Improving SHW's great answer to make it work with any locale, like Zbyszek already pointed out in his comment:

LC_ALL=C find ./photos/john_doe -type f -name '*.jpg' -exec du -ch {} + | grep total$

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.