6

The output would include the directory name, file name and file size. One (largest file) for each directory from where the command is run.

If possible the average size of the files in that directory as well.

The purpose is to can the directories looking for files that are much larger than the others in the directory so they can be replaced

1
  • 1
    I routinely use ncdu to see what is taking up space on my disk. See if it works for you as well. Apr 18 '17 at 12:21
11

With GNU find, sort and sed (4.2.2 or above), sort once on the file sizes and again on directory paths:

find /some/dir -type f -printf '%s %f%h\0' | 
  sort -zrn |
  sort -zut/ -k2 |
  sed -zre 's: ([^/]*)(/.*): \2/\1:'

Explanation:

  • The file size, name and path are printed (the first separated by a space and the next two separated by /), and each entry is terminated by the ASCII NUL character.
  • Then we sort numerically using the size, assuming NUL-delimited output (and in reverse order, so largest files first).
  • Then we use sort to print only the first unique entries using everything from the second /-separated field, which would be the path to the directory containing the file.
  • Then we use sed to swap the directory and filenames, so that we get a normal path.

For readable output, replace the ASCII NUL with newlines:

find /some/dir -type f -printf '%s %f%h\0' | 
  sort -zrn |
  sort -zut/ -k2 |
  sed -zre 's: ([^/]*)(/.*): \2/\1:' |
  tr '\0' '\n'

Example output:

$ find /var/log -type f -printf '%s %f%h\0' | sort -zrn | sort -zt/ -uk2 | sed -zre 's: ([^/]*)(/.*): \2/\1:' | tr '\0' '\n'
3090885 /var/log/syslog.1
39789 /var/log/apt/term.log
3968 /var/log/cups/access_log.1
31 /var/log/fsck/checkroot
467020 /var/log/installer/initial-status.gz
44636 /var/log/lightdm/seat0-greeter.log
15149 /var/log/lxd/lxd.log
4932 /var/log/snort/snort.log
3232 /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2
  • Hi Moro, on my ubuntu system, this produces a continuous line, so its a bit hard to read and check the results
    – Derek
    Apr 18 '17 at 8:03
  • Yes, because the output is NUL-separated. Use the command from the example output for more readable output and the first command when you want to pipe it to another command
    – muru
    Apr 18 '17 at 8:04
8

Combining find and awk allows the averages to be calculated too:

find . -type f -printf '%s %h/%f\0'|awk 'BEGIN { RS="\0" } { SIZE=$1; for (i = 1; i <= NF - 1; i++) $i = $(i + 1); NF = NF - 1; DIR=$0; gsub("/[^/]+$", "", DIR); FILE=substr($0, length(DIR) + 2); SUMSIZES[DIR] += SIZE; NBFILES[DIR]++; if (SIZE > MAXSIZE[DIR] || !BIGGESTFILE[DIR]) { MAXSIZE[DIR] = SIZE; BIGGESTFILE[DIR] = FILE } }; END { for (DIR in SUMSIZES) { printf "%s: average %f, biggest file %s %d\n", DIR, SUMSIZES[DIR] / NBFILES[DIR], BIGGESTFILE[DIR], MAXSIZE[DIR] } }'

Laid out in a more readable manner, the AWK script is

BEGIN { RS="\0" }

{
  SIZE=$1
  for (i = 1; i <= NF - 1; i++) $i = $(i + 1)
  NF = NF - 1
  DIR=$0
  gsub("/[^/]+$", "", DIR)
  FILE=substr($0, length(DIR) + 2)
  SUMSIZES[DIR] += SIZE
  NBFILES[DIR]++
  if (SIZE > MAXSIZE[DIR] || !BIGGESTFILE[DIR]) {
    MAXSIZE[DIR] = SIZE
    BIGGESTFILE[DIR] = FILE
  }
}

END {
  for (DIR in SUMSIZES) {
    printf "%s: average %f, biggest file %s %d\n", DIR, SUMSIZES[DIR] / NBFILES[DIR], BIGGESTFILE[DIR], MAXSIZE[DIR]
  }
}

This expects null-separated input records (I stole this from muru’s answer); for each input record, it

  • stores the size (for later use),
  • removes everything before the first character in the path (so we at least handle filenames with spaces correctly),
  • extracts the directory,
  • extracts the filename,
  • adds the size we stored earlier to the sum of sizes in the directory,
  • increments the number of files in the directory (so we can calculate the average),
  • if the size is larger than the stored maximum size for the directory, or if we haven’t seen a file in the directory yet, updates the information for the biggest file.

Once all that’s done, the script loops over the keys in SUMSIZES and outputs the directory, average size, largest file’s name and size.

You can pipe the output into sort to sort by directory name. If you want to additionally format the sizes in human-friendly form, you can change the printf line to

printf "%.2f %d %s: %s\n", SUMSIZES[DIR] / NBFILES[DIR], MAXSIZE[DIR], DIR, BIGGESTFILE[DIR]

and then pipe the output into numfmt --field=1,2 --to=iec. You can still sort the result by directory name, you just need to sort starting with the third field: sort -k3.

5
  • @Derek sudo apt-get install gawk should get you GNU awk
    – muru
    Apr 18 '17 at 8:35
  • Thanks Stephen. That is really great. Are you able to remove the \ that appears when there is a space in the name? also I would love the sizes to appear in human readable format (X.XM Y.YG), and have the output sorted by directory name? Thanks.
    – Derek
    Apr 18 '17 at 9:15
  • Weird, I don’t get backslashes in the output when there’s a space in the name! You can sort on directory names by piping through sort, and format the numbers using numfmt — I’ll update my answer in a little while. Apr 18 '17 at 10:02
  • Would it be possible to restrict this list to only where the largest file is say 50% larger than average for the directory? Also is the largest file used for the average, or just the other files? I realize I am pushing this :) its a great solution and much appreciated, but anything can be made slightly better :)
    – Derek
    Nov 12 '17 at 1:33
  • All files are taken into account when calculating the average. To restrict the list, you can add an if before the printf with the appropriate condition — go on, give it a shot ;-). Nov 16 '17 at 8:05
3

Zsh's wildcard patterns would be very useful for the sort of things you're doing. Specifically, zsh can match files by attributes such as type, size, etc. through glob qualifiers. Glob qualifiers also allow sorting the matches.

For example, in zsh, *(.DOLN[1]) expands to the name of the largest file in the current directory. * is the pattern for the file name (match everything, except possibly dot files depending on shell options). The qualifier . restricts the matches to regular files, D causes * to include dot files, OL sorts by decreasing size (“length”), N causes the expansion to be empty if there is no matching file at all, and [1] selects only the first match.

You can enumerate directories recursively with **/. For example the following loop iterates over all the subdirectories of the current directory and their subdirectories recursively:

for d in **/*(/); do … done

You can use zstat to access a file's size and other metadata without having to rely on other tools for parsing.

zmodload -F zsh/stat b:zstat
files=(*(DNoL))
zstat -A sizes +size -- $files
total=0; for s in $sizes; do total+=$s; done
if ((#sizes > 0)); then
  max=$sizes[-1]
  average=$((total/#sizes))
  median=$sizes[$((#sizes/2))]
fi
1

This is a quick try

find . -type d | 
  while read dir; do 
    find "$dir" -maxdepth 1 -type f -printf '%12s %-30h %f\n' | 
      sort -nr | 
      head -1
  done |
  sort -k2
2
  • Hi enzotib. That produces duplicate lines. Also, can it sort by dir/name?
    – Derek
    Apr 18 '17 at 7:51
  • @Derek: yes, I missed the -maxdepth 1 option.
    – enzotib
    Apr 18 '17 at 8:11
0

Look into using baobab or some of the similar software, and one or more of them are likely included with your distribution. They visualize the problem directories very well.

  • Baobab
  • JDiskReport
  • ncdu
  • K4DirStat
  • QDirStat
  • GD Map

http://alternativeto.net/software/baobab/?platform=linux

And the man page for baobab tells it how it is.

$> man baobab
BAOBAB(1)

NAME
       Baobab - A graphical tool to analyse disk usage

SYNOPSIS
       baobab  [directory]

DESCRIPTION
       baobab is able to scan either specific folders or the whole filesys-
       tem (local and remote), in order to give the user a graphical tree
       representation including each directory size or percentage in the
       branch.  It also auto-detects in real-time any change made to your
       home directory as far as any mounted/unmounted device. A graphical
       treemap window is also provided for any selected folder.

       A detailed documentation on the program could be read at:
       http://www.gnome.org/projects/baobab

AUTHOR
       Fabio MARZOCCA <thesaltydog@gmail.com>



                                                                     BAOBAB(1)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.