7

This is my working code, but I believe it's not optimized - there must be a way to complete the job much faster than this:

find . -type f -iname '*.py' -printf '%h\0' |
  sort -z -u |
  xargs -r -0 -I{} sh -c '
    find "{}" -maxdepth 1 -type f -iname "*.py" -print0 |
      xargs -r -0 du -sch |
      tail -1 |
      cut -f1 |
      tr "\n" " "
    echo -e "{}"' |
  sort -k1 -hr |
  head -50

The goal is to search for all directories recursively that contain *.py then print the total size of all *.py files by the name of each directory, sort them in reverse order by size and show only first 50.

Any ideas how to improve this code (performance-wise) but keeping the same output?

EDIT:

I tested your proposals on the following sample: 47GB total: 5805 files Unfortunately, I couldn't compare it toe-to-toe, since not all proposals follow the same guidelines: the total size should be disk usage and delimiter should be only a single space. Formatting should be as follows: numfmt --to=iec-i --suffix=B

The following 4 are sorted outputs, but David displays accumulative size of files, not real disk usage. However, his improvement is significant: more than 9.5x faster. Stéphane's and Isaac's code are very tight winners, since their code is approximately 32x faster than the reference code.

$ time madjoe.sh
real    0m2,752s
user    0m3,022s
sys     0m0,785s

$ time david.sh 
real    0m0,289s
user    0m0,206s
sys     0m0,131s

$ time isaac.sh 
real    0m0,087s
user    0m0,032s
sys     0m0,032s

$ time stephane.sh 
real    0m0,086s
user    0m0,013s
sys     0m0,047s

The following code unfortunately doesn't sort nor display largest 50 results (besides, during previous comparison to Isaac's code, the following code is approx 6x slower than Isaac's improvement):

$ time hauke.sh 
real    0m0,567s
user    0m0,609s
sys     0m0,122s
10
  • If there are .py files in both a directory and one of its subdirectories then you count the file sizes in the subdirectory twice. Is that intended? – Hauke Laging Jun 29 '20 at 2:23
  • What matters here? The cumulative apparent size of the python source code file or their disk usage. If the latter, what if some of those files have several hard links, in the same dir or in separate dirs? Do you want to count them all, or only one of them (like du does)? – Stéphane Chazelas Jun 29 '20 at 8:27
  • @HaukeLaging Hm... if there are 9MB of .py in current folder only and e.g. 10MB of .py in its subdirectory "xyz" then it should be listed as 9MB . and 10MB ./xyz – madjoe Jun 29 '20 at 8:29
  • @StéphaneChazelas I'd like to display their cumulative disk usage. Reg. separate hard disks, the goal is to replicate exactly what du does. – madjoe Jun 29 '20 at 8:37
  • OK, so if say ./dir1 and ./dir2 have 2 directory entries each (A.py and B.py), which all refer to the same file that occupies 10KiB of disk space, are you expecting to see something like 10KiB ./dir1 followed by 0 ./dir2 for instance? – Stéphane Chazelas Jun 29 '20 at 8:41
6

To count the disk usage as opposed to the sum of the apparent size, you'd need to use %b¹ instead of %s and make sure each file is counted only once, so something like:

LC_ALL=C find . -iname '*.py' -type f -printf '%D:%i\0%b\0%h\0' |
  gawk -v 'RS=\0' -v OFS='\t' -v max=50 '
    {
      inum = $0
      getline du
      getline dir
    }
    ! seen[inum]++ {
      gsub(/\\/, "&&", dir)
      gsub(/\n/, "\\n", dir)
      sum[dir] += du
    }
    END {
      n = 0
      PROCINFO["sorted_in"] = "@val_num_desc"
      for (dir in sum) {
        print sum[dir] * 512, dir
        if (++n >= max) break
      }
    }' | numfmt --to=iec-i --suffix=B --delimiter=$'\t'

Newlines in the dir names are rendered as \n, and backslashes (at least those decoded as such in the current locale²) as \\.

If a file is found in more than one directory, it is counted against the first one it is found in (order is not deterministic).

It assumes there's no POSIXLY_CORRECT variable in the environment (if there is, setting PROCINFO["sorted_in"] has no effect in gawk so the list would not be sorted). If you can't guarantee it³, you can always start gawk as env -u POSIXLY_CORRECT gawk ... (assuming GNU env or compatible; or (unset -v POSIXLT_CORRECT; gawk ...)).

A few other problems with your approach:

  • without LC_ALL=C, GNU find wouldn't report the files whose name doesn't form valid characters in the locale, so you could miss some files.
  • Embedding {} in the code of sh constituted an arbitrary code injection vulnerability. Think for instance of a file called $(reboot).py. You should never do that, the paths to the files should be passed as extra arguments and referenced within the code using positional parameters.
  • echo can't be used to display arbitrary data (especially with -e which doesn't make sense here). Use printf instead.
  • With xargs -r0 du -sch, du may be invoked several times if the list of files is big, and in that case, the last line will only include the total for the last run.

¹ %b reports disk usage in number of 512-byte units. 512 bytes is the minimum granularity for disk allocation as that's the size of a traditional sector. There's also %k which is int(%b / 2), but that would give incorrect results on filesystems that have 512 byte blocks (file system blocks are generally a power of 2 and at least 512 byte large)

² Using LC_ALL=C for gawk as well would make it a bit more efficient, but would possibly mangle the output in locales using BIG5 or GB18030 charsets (and the file names are also encoded in that charset) as the encoding of backslash is also found in the encoding of some other characters there.

³ Beware that if your sh is bash, POSIXLY_CORRECT is set to y in sh scripts, and it is exported to the environment if sh is started with -a or -o allexport, so that variable can also creep in unintentionally.

1
  • I've completely rewritten my previous script that used multiple find and implemented AWK. The final result could be found here: github.com/madjoe/wii – madjoe Jul 3 '20 at 1:30
7

Simplifying the solution from @HaukeLaging by collecting all directory sums in one array and printing it all at the end (using GNU awk). Also, only one call to numfmt is needed (at the end).

#!/bin/sh

find . -type f -iname '*.py' -printf '%s %h\0' |
    awk 'BEGIN { RS="\0"; };

         { gsub(/\\/,"&&"); gsub(/\n/,"\\n");
           size=$1; sub("[^ ]* ",""); dirsize[$0]+=size }

         END {   PROCINFO["sorted_in"] = "@val_num_desc";
                 i=0;
                 for ( dir in dirsize ) { if(++i<=50) 
                     { print dirsize[dir], dir; }else{ exit } 
                 }
             }        ' | numfmt --to=iec-i --suffix=B

This generates the cumulative apparent size of the py files (not their disk usage), and avoids summing files in sub-directories of a directory.

1
3

I suspect you need to write your own du.

Currently, you are triple recursing into the hierarchy, using two finds and a du.

I would suggest starting with perl's File::Find package.

Alternatively, your first find could output with something like -printf '%k %h\n' and then you could sort by directory, use perl or awk (or even bash) to total the directories and convert to "human" readable, and finally sort & head.

Either way, you should A) walk the directory tree only once, and B) create as few processes as possible.

Edit: A sample implementation

#!/bin/bash

find . -type f -iname '*.py' -printf '%k %h\n' | sort -k2 | (
    at=
    bt=
    output() {
        if [[ -n "$at" ]]
        then
            printf '%s\t%s\n' "$at" "$bt"
        fi
    }
    while read a b
    do
        if [[ "$b" != "$bt" ]]
        then
            output
            bt="$b"
            at=0
        fi
        at=$(( $at + $a ))
    done
    output
) | sort -hr | head -50 | numfmt -d'   ' --field=1 --from-unit=Ki --to=iec-i

Note: %k is important. %s reports apparent size, while %k (and du) report disk size. They differ for sparse files and large files. (If you want du --apparent-size, so be it.)

Note: numfmt should go at the end, so it is run once. Using '%k', the from-unit needs to be specified.

Note: numfmt's -d parameter should contain a single tab. I can't type that here, and numfmt won't accept -d'\t'. If the separator isn't a tab, the spacing gets messed up. I thus used printf instead of echo in the main body. (An alternative would be to use echo, and a final sed to change the first space into a tab.

Note: I initially missed the first sort, and got repeated entries for some directories in my re-testing.

Note: numfmt is fairly recent.

4
  • Yes, you are absolutely right reg. starting to resolve this task by rewriting my own du. My idea was to prove this feature would be useful as a new parameter for du. – madjoe Jun 29 '20 at 8:43
  • @madjoe right... based on Alternative for filtering du command by file types only I would recommend the extra option be "--include=" , that the rule be that they are processed in order, and the default be "--include=*" except if only --includes are specified, in which case the default is "--exclude=*". – David G. Jun 29 '20 at 11:21
  • 1
    @madjoe You might also want to look at other tool's options. rsync's filtering is impressive... – David G. Jun 29 '20 at 15:12
  • Thanks, I use rsync excessively for some other purposes and it's an awesome project, so I'll take a look at its filtering capabilities. – madjoe Jun 29 '20 at 15:15
2

This may be a lot faster but is it not completely equivalent to your approach. It does not count the subdirectory files twice:

find . -type f -iname '*.py' -printf '%s %h\0' |
    awk 'BEGIN { RS="\0"; }; '\
'{ pos=index($0," "); size=substr($0,1,(pos-1)); dir=substr($0,pos+1); gsub("\n","\\n",dir); '\
'if(dir!=lastdir) { if(NR>1) { "numfmt --to=iec-i --suffix=B " sizesum " | tr -d \"\n\"" | getline fsize; print fsize " " lastdir; } '\
'sizesum=size; lastdir=dir; } '\
'else sizesum=sizesum+size; }; '\
'END { "numfmt --to=iec-i --suffix=B " sizesum " | tr -d \"\n\"" | getline fsize; print fsize " " lastdir; }'

3,2KiB ./dir1
1,1MiB ./dir2

In addition to being faster it replaces newlines with literal \n. If you expect directory names with newlines then you have to handle them until the end of the pipeline what your code does not do.

2
  • I believe that I have simplified your script in my answer by using an array and only one call to numfmt. Thanks and please take a look. – Isaac Jun 29 '20 at 8:27
  • 1
    If you replace LF with \n, you may want to also replace backslashes with \\ to make the output unambiguous (as otherwise you couldn't tell whether \new.py refers to a \new.py file or a <LF>ew.py one. – Stéphane Chazelas Jun 29 '20 at 8:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.