How to search for large directory containing thousands of files?

Question

Under the folder

/grid/sdh/hadoop/yarn/local/usercache/hdfs/appcache

we have more than 100 recursive folders.

One of the folders contain thousands of files. Is it possible to identify this folder?

I am asking because this one folder that contains the thousands of files and we may have a problem that we can't remove the files there because of the thousands of files.

Answer 1

The number of items in a directory may be counted using

set -- *

This sets the positional parameters ($1, $2, etc.) to the names in the current directory. The number of names that * expands to is found in $#. If you use the bash shell and set the dotglob shell option, this will additionally count hidden names.

Using this to find directories under /grid/sdh/hadoop/yarn/local/usercache/hdfs/appcache that contain more than 1000 names:

find /grid/sdh/hadoop/yarn/local/usercache/hdfs/appcache \
    -type d -exec bash -O dotglob -c '
        for pathname do
            set -- "$pathname"/*
            if [ "$#" -gt 1000 ]; then
                printf "%d\t%s\n" "$#" "$pathname"
            fi
        done' bash {} +

This expands the * shell glob in each found directory and outputs the pathname of the directory if there are more than 1000 names in it, along with the number of names. It does this by executing a short bash script for batches of directories. The script will loop over each batch of directories and for each, it will expand the * glob inside it to count the number of entries. An if statement then triggers printf if appropriate.

Note that if a directory contains millions of names, then it may take a bit of time to actually expand the * glob in that directory.

Answer 2

On a GNU system

(export LC_ALL=C
  find /grid/sdh/hadoop/yarn/local/usercache/hdfs/appcache -print0 |
    tr '\n\0' '\0\n' |
    sed 's|/[^/]*$||' |
    sort |
    uniq -c |
    sort -rn |
    head |
    tr '\z' '\n')

Would list the 10 directories with the most entries.

If the directories have so many files than even listing them would be too expensive, you can try and guess which they are without entering them by looking at their size.

 find /grid/sdh/hadoop/yarn/local/usercache/hdfs/appcache -type d \
   -size +10000000c -print -prune

Would list the directories that are more than 10MB large and not enter them.