1

I want to loop over all files in directory on macOS and perform some operations on each file, and do this based on their size, going from big to small. To create a list I can loop over I do this:

files=$(find "$directory" -type f \( -name "*.opus" \) -exec du -h {} + 2>/dev/null | sort -rh)

but in a directory that looks like this:

% ls -l
-rw-------@ 1 db  wheel   1784 11 Jul  2023 1.opus
-rw-------@ 1 db  wheel  30182 20 Jun  2023 2.opus
-rw-------@ 1 db  wheel     62 19 Apr 11:38 3.opus
-rw-------@ 1 db  wheel   1216 25 Jul  2023 4.opus

They get sorted like this:

 32K    ./2.opus
4,0K    ./4.opus
4,0K    ./3.opus
4,0K    ./1.opus

As you can see 3 and 4 are sorted before 1 despite being smaller. It seems like they are sorted on block size, and if that is identical, they are sorted on file name. I want them to be sorted by byte size.

How do I create a list sorted by bytesize I can use for my loop?

7
  • Using stat -c "%s %n"` will give you the file size in bytes in a shell-independent manner, although I like the zsh solution from Stéphane.
    – doneal24
    Commented Apr 30 at 15:03
  • On what operating system? Do you have stat, for instance as doneal24 suggested? And do you have full control over the file names? Can you be sure the names don't have whitespace?
    – terdon
    Commented Apr 30 at 17:33
  • @terdon MacOS .
    – d-b
    Commented Apr 30 at 18:22
  • Ah, so you probably don't have stat then. Or, at least, not one with -c.
    – terdon
    Commented Apr 30 at 18:25
  • @doneal24 OpenBSD has a very different stat(1) than GNU. stat(1) is sadly not portable. Commented May 1 at 6:24

3 Answers 3

8

If you want to sort in byte-sizes, you should ask du for the byte size (-b) instead of a human readable form (-h).

find "$directory" -type f \( -name "*.opus" \) -exec du -b {} \; | sort -n

will do that.

However, find will also report the size of the file in bytes; no need to use du for every file:

 find "$directory" -type f \( -name "*.opus" \) -printf '%s %p\n' | sort -n
2
  • I think a ) got dropped in the second command, although I suspect bracketing the single -name option is not useful anyway. Commented Apr 30 at 22:08
  • 2
    Worth noting that -b and -printf are non-standard GNU extensions, not available in the stock du/find of FreeBSD / macos. You'd also need the -l and -0 options of GNU du and the -z option of GNU (or FreeBSD as FreeBSD's sort used to be GNU sort) sort to get the size of all files and in a post-processable format as indicated in my answer. Commented May 1 at 7:45
8

If switching to zsh is an option, then you can just do:

for file in $directory/**/*.opus(ND.OL); do
  something with $file
done

Or just:

for file ($directory/**/*.opus(ND.OL)) something with $file

Where OL sorts in reverse by Length (size, not disk usage though the two are generally correlated). . is the equivalent of -type f, N for nullglob to not complain if there's no matching file, D for dotglob to include hidden files like find does by default.

With bash 5.3 (not released yet) or newer, you can do:

(
  shopt -s nullglob globstar dotglob
  typeset +x GLOBSORT=-size
  for file in "$directory"/**/*.opus; do
    [ -f "$file" ] || continue
    [ -L "$file" ] && continue
    something with "$file"
  done
)

Where **/ to match any level of directories needs globstar enabled like in ksh93, GLOBSORT=-size (which we unexport to avoid affecting something¹) does the equivalent of OL², [ -f (like -xtype f) with [ ! -L does the equivalent of -type f.

As those options and GLOBSORT can only be enabled globally and not on a per-glob basis like in zsh, that means that if something is a function or shell code that makes use of globbing, it will be affected. It may be preferable to do it as:

get_matching_files_sorted_by_size() {
  local saved_opts="$(shopt -p nullglob globstar dotglob)"
  local GLOBSORT=-size
  shopt -s nullglob globstar dotglob
  files=( "$directory"/**/*.opus )
  eval "$saved_opts"
}
get_matching_files_sorted_by_size
for file in "${files[@]}"; do
  [ -f "$file" ] || continue
  [ -L "$file" ] && continue
  something with "$file"
done

Where the changes to the globbing configuration are only done locally in a function for the expansion of that one glob to emulate zsh's glob qualifiers.

With the GNU implementations of find and sort and zsh or bash, you can also do:

while IFS=/ read -rd '' -u3 size file; do
  {
    something with "$file"
  } 3<&-
done 3< <(
  find -H "$directory" -name '*.opus' -type f -printf '%s/%p\0' |
    sort -zrn
)

Where find prints the size and path of the files NUL-delimited and /-separated³, sort sorts that numerically in reverse (with ties sorted lexically in reverse as well⁴).

The output of sort is retrieved via process substitution (essentially using a named pipe or something acting like one), opened on a fd above 2 (here 3) in the while loop so that stdin/stdout/stderr remain unaffected.

read reads those NUL-delimited records, stores the first /-delimited(bash)/separated(zsh) field in $size and the rest in $file.

In the body of the loop, we close that fd 3 so it's not exposed to something though that's likely unnecessary precaution as there's no reason something would be using that fd.

Replace %s with %b for disk disk usage in number of 512-byte blocks instead of size.

In any case, files=$(...) is a scalar variable assignment, not a list assignment. After that $files will contain only one value: the whole output of ... stripped of trailing newline characters.

To create a list (array) based on find's result (it's NUL-delimited output split on NULs), you'd need:

readarray -td '' files < <(find ... -print0)

in bash 4.4+ or

files=( ${(0)"$(find ... -print0)"} )

In zsh.

More on that at Why is looping over find's output bad practice?.

Using -exec du -h {} + | sort -rh is wrong for several reasons:

  • du reports disk usage, the amount of disk space needed to store the data of the list of files passed as argument, not their size5
  • because it reports disk usage, if two of more of the arguments are the same file for instance because they are hard links to each other, their size will be reported once6 (the other ones will be listed with a disk usage of 0 if at all).
  • By default standard du reports disk usage in number of 512-byte blocks7, but if you add the -h option (a non-standard extension), the disk usage is even further rounded to a number with a minimal number of digits for human consumption, so you're losing a lot of precision.
  • beware that for files of type directory, du doesn't report the disk usage of the directory file itself but also of every file and dir referenced within recursively. Not a problem here with -type f unless the type of the file changes from regular to directory between find finding it and du processing it, but that's a reminder that du can't be used to retrieve file size in the general case even if you use the -bl0 extensions of GNU du.
  • newline and other whitespace are as valid characters as any in a file path, so processing a list of files with line-based utilities is wrong in the general case8

¹ the way bash seems to interact with a GLOBSTAR variable found in the environment seems a bit bogus at the moment and will likely change, though it's hard to guess in which way; there are similar problems with the BASHOPTS and SHELLOPTS variables.

² Well, rather -OL in that the size is retrieved after symlink resolution and there doesn't appear to be a way to change that (not that it's a concern here as we exclude symlinks with [ -L "$file" ] && continue to mimic find's -type f).

³ NULL is the only character that can't occur in a file path, and / can't occur in a file name. Using another character as separator would not work reliably in bash as it could cause that character to be trimmed from the end of the file path by read as $IFS is treated as Internal Field delimiter in bash instead of Separator like in zsh.

⁴ Change to sort -zk1rn, that is with reverse applied to a key specification (still the whole record) instead of globally for the numbers to be sorted in descending order but ties to be sorted lexically in ascending order.

5 though the GNU implementation of du has a --apparent-size option to switch to size instead of disk usage.

6 the GNU implementation of du has a -l/--count-links option to disable that deduplication.

7 generally the minimum unit of disk allocation, beware some du implementation (including GNU du when POSIXLY_CORRECT is not in the environment) use kibibytes instead, and some can change the default block size with environment variables (BLOCKSIZE, BLOCK_SIZE, DU_BLOCKSIZE...) and see the -B/--block-size=SIZE of GNU du, and -b which is short for --apparent-size --block-size=1.

8 See the -0 option of GNU du to report the list NUL-delimited instead. TAB still used to separate the size and file paths

2
  • Thank you! In any case, files=$(...) is a scalar variable assignment, not a list assignment. I don't understand this, can you elaborate?
    – d-b
    Commented May 1 at 5:51
  • @d-b, a scalar variable is a variable that stores one string (or possibly numerical if given the integer attribute) value. Even if files has been declared as an array before, files=something will assign something to ${files[0]} in bash. Array assignment in bash (copied from zsh actually) is with the files=( elements ) syntax. But if you want to assign several elements from the output of a command, you need to split that output. That's where the 0 parameter expansion flag of zsh (to split on NULs) comes in, but bash has no direct equivalent and your need readarray there. Commented May 1 at 6:01
0

The following finds ".opus" files in and under the target directory, sorts them based on descending byte size, and performs operations on them as byte size decreases.

Works on macOS and Linux, using either bash or zsh.

My answer uses wc which requires read permissions for the files. Normally to obtain a file's byte size, like with a stat command, you don't require read permissions but only execute/search permissions to ancestor directories. See also Stéphane Chazelas's helpful comments below. On macOS you can use BSD stat -f %z to obtain a file's byte size, and on systems using the GNU coreutils toolset, you can use GNU stat -c %s. If you have zsh you can use zstat with zmodload -F zsh/stat b:zstat; zstat -H struct_stat_hash <your-file>; echo "${struct_stat_hash[size]}" to get a file byte size, without needing read permissions.

In my wc -c code below, I added a check against find not finding any files using a print0 function provided by Stéphane Chazelas from his helpful comments down below.

print0() { [ "$#" -eq 0 ] || printf '%s\0' "$@"; }                       #1
bytes_pathname_arr=()
while IFS= read -r -d $'\0' pathname; do                                 #3
  bytes_pathname_arr+=( "$(wc -c <"$pathname" | tr -d ' ') $pathname" )  #4
done < <(find "$directory" -type f -name "*.opus" -print0)               #2
while IFS= read -r -d $'\0' bytes_pathname; do                           #6
  bytes=${bytes_pathname%% *} pathname=${bytes_pathname#* }              #7
  ######## Perform file operations based on byte size here ########
  printf "%s %q\n" "$bytes" "$pathname"                                  #8
  #################################################################
done < <(print0 "${bytes_pathname_arr[@]}" | sort -znr)                  #5

Explanation

  1. print0() { [ "$#" -eq 0 ] || printf '%s\0' "$@"; }
  • A function which outputs arguments delimited by NUL bytes, or outputs nothing if there are no arguments.
  1. done < <(find "$directory" -type f -name "*.opus" -print0)
  • Find regular file -type f basenames matching glob -name "*.opus" and output them delimited by NUL bytes -print0, and redirect < the result of this process substitution <( ... ) into stdin of the read command.
  • Note that the safest way in Unix systems to delimit pathnames is with the NUL byte, because pathnames in Unix systems are not allowed to contain a NUL byte.
  1. while IFS= read -r -d $'\0' pathname; do
  • This is a general way to read in NUL-delimited input for processing.
  • IFS= Setting IFS to empty prevents word splitting from occurring, and in doing so, in this case with a single variable pathname to read in input, prevents trimming of trailing and leading whitespace of the value to be read into variable pathname. Leading whitespace is not a worry since find always outputs pathnames beginning with either ./ or /, but trailing whitespace is a big worry, because if a pathname ends in any series of whitespace characters (IFS default whitespace characters), these characters will get removed and not be assigned to pathname, leading to an incorrect pathname.
  • -r prevents interpretation of backslashes.
  • -d $'\0' or -d '' means read in input delimited by the NUL byte.
  • read stores the resulting value into variable pathname, per iteration of the while loop.
  1. bytes_pathname_arr+=( "$(wc -c <"$pathname" | tr -d ' ') $pathname" )
  • Store bytes-pathname pairs, separated by a space, into bytes_pathname_arr array.
  • wc -c <"$pathname" gets the number of bytes of the file without the filename. If wc sees no file operands, stdin is used and no filename is written.
  • tr -d ' ' removes space characters. macOS BSD wc outputs extra leading space characters, unlike the wc implementations commonly found on Linux-based operating systems, such as that of busybox, toybox or GNU coreutils (thank you Stéphane Chazelas), so we want to get rid of these.
  1. done < <(print0 "${bytes_pathname_arr[@]}" | sort -znr)
  • print0 "${bytes_pathname_arr[@]}" outputs bytes-pathname pairs delimited by NUL bytes. If the array is empty, print0 outputs nothing.
  • sort -znr reads in the NUL byte-delimited input -z, and sorts the bytes-pathname pairs numerically -n and descending -r, and then also lexicographically-descending if byte size compares equal, and then outputs this as NUL byte-delimited output to a second read command similar to the first.
  1. while IFS= read -r -d $'\0' bytes_pathname; do
  • Read in the NUL byte-delimited "bytes-pathname" pairs one at a time into variable bytes_pathname.
  1. bytes=${bytes_pathname%% *} pathname=${bytes_pathname#* }
  • Safely break apart the bytes-pathname pair into its original constituents, bytes and pathname, using parameter expansion.
    • %% deletes longest match of substring matching pattern * (space and zero or more characters) from back of string.
    • # deletes shortest match of substring matching pattern * (zero or more characters and space) from front of string.
  1. printf "%s %q\n" "$bytes" "$pathname"
  • This is where you can perform operations based on the file's byte size. The example I give simply prints the bytes and pathname out, but you can do something like check for the byte size, and if it reaches under a certain amount, say 1 MB (1000000 bytes), then stop processing the files.

wc source code comments about obtaining byte size

I want to list some comments in the source codes for various implementations of wc related to obtaining byte size. I cannot read and fully understand the C source codes, but I think the comments can be of interest to readers:

macOS BSD wc:

  /*
   * If all we need is the number of characters and it's a regular file,
   * just stat it.
   */

GNU coreutils wc:

  /* When counting only bytes, save some line- and word-counting
     overhead.  If FD is a 'regular' Unix file, using lseek is enough
     to get its 'size' in bytes.  Otherwise, read blocks of IO_BUFSIZE
     bytes at a time until EOF.  Note that the 'size' (number of bytes)
     that wc reports is smaller than stats.st_size when the file is not
     positioned at its beginning.  That's why the lseek calls below are
     necessary.  For example the command
     '(dd ibs=99k skip=1 count=0; ./wc -c) < /etc/group'
     should make wc report '0' bytes.  */

busybox wc:

 * The previous busybox wc attempted an optimization using stat for the
 * case of counting chars only.  I omitted that because it was broken.
 * It didn't take into account the possibility of input coming from a
 * pipe, or input from a file with file pointer not at the beginning.
 *
 * To implement such a speed optimization correctly, not only do you
 * need the size, but also the file position.  Note also that the
 * file position may be past the end of file.  Consider the example
 * (adapted from example in gnu wc.c)
 *
 *      echo hello > /tmp/testfile &&
 *      (dd ibs=1k skip=1 count=0 &> /dev/null; wc -c) < /tmp/testfile
 *
 * for which 'wc -c' should output '0'.
5
  • Worth noting that wc -c <"$pathname" needs read permission to the files (while getting a file size normally only requires search permission to the directory it is found in). Besides zsh's stat builtin, macos should also have BSD stat that can get you a file size without needing read permission. Since you're already running two commands per file anyway, you could also get it relatively reliably by parsing the output of LC_ALL=C ls -lnd -- "$pathname". Commented May 6 at 9:06
  • "printf "%s\0" "${bytes_pathname_arr[@]}" outputs bytes-pathname pairs delimited by NUL bytes"... or a single NUL byte if the array is empty. In zsh, you'd used print -rNC1 -- $list; in bash/sh, a common approach is to define a print0() { [ "$#" -eq 0 ] || printf '%s\0' "$@"; } function. Commented May 6 at 9:09
  • "unlike Linux wc" note that Linux has no wc being just a kernel. More accurately: unlike the wc implementations commonly found on Linux-based operating systems, such as that of busybox, toybox or GNU coreutils. Commented May 6 at 9:11
  • Thank you for your comments, I updated my answer. In your LC_ALL=C ls -lnd -- "$pathname" command, can you explain why the LC_ALL=C is needed? Thanks!
    – Aeronautix
    Commented May 11 at 1:22
  • The idea is that the C/POSIX locale is the only one where the format of the output of ls -n is specified. I've now clarified it at How can I get the size of a file in a bash script? Commented May 11 at 7:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .