1

I have a directory with a lot of folders and I want to move the ones who have more than 100 files inside.

I was thinking of doing:

find . -type d | while read d; do if 

and now it's the tricky part for me.

Do I perform a for to go inside every directory and check if it contains more than 100 files? If so, how do I do this?

for f in *; do cd $f; ll | wc; ?

I'm a little bit confused how I can get the total count of files inside a directory and then move that directory it it contains more than 100 files.

2
  • Do the subdirectories contain further subdirectories whose files you need to count, or do the 100 files occur in one and the same flat directory? That is, you have top-dir/subdir/files or top-dir/subdir/subdir/subdir/files? – Kusalananda Aug 14 '20 at 16:35
  • top-dir/subdir/files – user12225952 Aug 14 '20 at 16:38
2

To count the number of non-hidden names in a directory dir, you can use

set -- dir/*

This expands the * glob in the directory and sets the positional parameters to the resulting names. If the pattern matched anything, the count is then in $#.

To iterate over all directories in some top-level directory top-dir, count the number of names in each, and do something to the directories that contains more than 100 names:

for subdir in top-dir/*/; do
    set -- "$subdir"/*
    if [ -e "$1" ] && [ "$#" -gt 100 ]; then
        # do something to "$subdir"
    fi
done

In the bash shell, setting the nullglob shell option gets rid of the need to check whether the set command managed to match any names at all (since the pattern would be removed completely if there were no matches instead of remaining unexpanded).

shopt -s nullglob

for subdir in top-dir/*/; do
    set -- "$subdir"/*
    if [[ $# -gt 100 ]]; then
        # do something to "$subdir"
    fi
done

Additionally setting the dotglob shell option would make each pattern in the code also match hidden names.

In either piece of code above, the "do something to "$subdir"" comment could be replaced by with whatever you need to do to those subdirectories. To e.g. move them away, use

mv "$subdir" some/other/dir

Which would move them to the directory some/other/dir.

1
  • Beware the expansion of top-dir/*/ (contrary to that of zsh's top-dir/*(/)) includes symlinks to directories in addition to directories. The expansion of "$subdir"/* includes entries of any type including directories, symlinks, regular, fifo... – Stéphane Chazelas Aug 14 '20 at 17:37
1

You can loop across each directory in turn, count the number of files it contains, and then move it ... somewhere. For example,

for dir in ./*/
do
    count=$(find "$dir" -maxdepth 1 -type f -printf "x\n" | wc -l)    # Count the number of files in this subdirectory
    [ $count -gt 100 ] && echo mv "$dir"                              # Output a message if we have enough
done

You can type this straight in at the prompt (you'll get a secondary prompt, >, after the first line until the last) or save it in a script file and run that.

4
  • Note that find "$dir" -maxdepth 1 -type f | wc -l counts the number lines in the filenames, not the number of directory entries. Here you could replace wc -l with grep -c / to count the number of entries. – Stéphane Chazelas Aug 14 '20 at 17:21
  • Note that */ (contrary to zsh's *(/)) includes directories and symlinks to directories. – Stéphane Chazelas Aug 14 '20 at 17:22
  • It should be for dir in ./*/, or you'd need to use find -f "$dir" (assuming BSD find) and mv -- "$dir" ... to guard against dir names start with -. – Stéphane Chazelas Aug 14 '20 at 17:24
  • Note that -maxdepth and -printf are GNU extensions. While -maxdepth has been picked up by a few other find implementations, -printf is still GNU-specific. find "$dir/." ! -name . -prune -type f -print | LC_ALL=C grep -c / would be POSIX (and the equivalent of -mindepth 1 -maxdepth 1 (or -depth 1 with some find implementations) which would be more correct). – Stéphane Chazelas Aug 14 '20 at 17:32
1

With zsh:

mv -- *(/Fe['()(($# > 100)) $REPLY/*(N^-/)']) /dest/

Would move into /dest/ the non-hidden subdirectories of the current working directory that contain over 100 entries¹ that are neither hidden nor of type directory (which I assuming you mean by file).

That makes use of zsh glob qualifiers ((/Fe...) and (N...) above) that further select matching files based on other criteria than their name.

  • /: select files of type directory only. Here (contrary to the */ glob) the type is determined before symlink resolution which is probably preferable here as moving symlinks often breaks them).
  • F: selects full files as an optimisation (for directories, that means non empty directories)
  • e[code]: select based on the result of the interpretation of code where $REPLY contains the file currently being considered.

That code here is ()(($# > 100)) $REPLY/*(N^-/).

() <body> <args> is an inline function. Here the body ((($# > 100))) checks that the number of arguments is greater than 100. The arguments are the expansion of the $REPLY/*(N^-/) glob again using glob qualifiers:

  • N: nullglob: that glob will expand to no argument at all instead of an error when there's no matching file.
  • ^: negates the following qualifiers.
  • -/ is like / above except the - causes the following qualifiers (here /) to apply after symlink resolution. So here we're counting the files that are not of type directory after symlink resolution. You could replace ^-/ with . to count the regular files only (to the exclusion of all other types of files like sockets, fifos, directories, symlinks...), or -. for regular files and symlinks to regular files.

To also consider hidden dirs/files, add the D qualifier (to either or both the outer and inner globs).

To also count the files in subdirectories recursively, replace the second * with **/* (or ***/* to traverse symlinks when descending the directory tree).

You can optimise it further by changing the code to:

()(($#)) $REPLY/*(NoN^-/[101])

That is using oN to disable the sorting of files which as we don't care about the order, and the glob expanding to only the 101st matching file which we just test for presence with (($#)) (number of arguments non-zero).


¹ beware several entries in there could be referring to the same file, like when they are hardlinked or symlinked together. Counting the number of unique files would be a different exercise

0

We form a chain of checks to determine a valid file in a non symlinked dir. the dirs can be hidden or not, files can be hidden or not.

Change the files count in threshold variable and destination path in dest_dir variable. Remove the echo once everything is OK.

Fixed the issue of symlinked directory test failing for a name with a trailing slash and mv command as pointed out by Stephen.

$ threshold=100
$ dest_dir=final/resting/place
$ for d in ./*/ ./..?*/ ./.[!.]*/; do
    [ ! -L "${d%/}" ] && [ -d "$d" ] && \
    [ "$(cd "$d" && find .//. ! -name . -prune -type f | grep -cF .//.)" -gt "$threshold" ] \
     && echo mv -- "$d" "$dest_dir";
  done

We can minimize the number of moves if we use GNU xargs which delimit its arguments on NUL \0 char and recent versions of mv that support the -t option ( target dir):

$ threshold=100
$ dest_dir=final/resting/place
$ for d in ./*/ ./..?*/ ./.[!.]*/; do
    [ ! -L "${d%/}" ] && [ -d "$d" ] && \
    [ "$(cd "$d" && find .//. ! -name . -prune -type f | grep -cF .//.)" -gt "$threshold" ] \
     && printf '%s\0' "$d"
  done | xargs -r0 -t mv -t "$dest_dir"
6
  • It shouldn't be a problem if $dest_dir starts with -, as $dest_dir here is only used as an argument to GNU mv's -t option (with the first one, you'd need mv -- "$dir" "$destdir" though in case mv is GNU mv or another one of those non-POSIX mv implementations that accept options after non-options) – Stéphane Chazelas Aug 15 '20 at 6:36
  • You should not run find if cd failed. – Stéphane Chazelas Aug 15 '20 at 6:38
  • We wouldn't need -- in the first one. Y? Coz the names come safely prepended with a ./. – Rakesh Sharma Aug 15 '20 at 7:02
  • Fixed the issue of symlink test condition. Thanks. – Rakesh Sharma Aug 15 '20 at 7:05
  • 1
    With GNU mv and if POSIXLY_CORRECT is not in the environment, mv ./dir -dest-dir- would fail. More generally, with most GNU tools (or utilities using the GNU getopt API), you need -- if any argument (not just the first) is variable/unknown. – Stéphane Chazelas Aug 15 '20 at 7:13
0

Besides shell approaches you can also use a pipeline made of a find command crafted to select the folders/files to examine, feeding them to an Awk script that does the filtering for a final xargs that bundles the actual mv commands in as few runs as possible. It could also be a shell script, but Awk is usually better and quicker at handling text.

The below uses GNU tools instructed to work with nul-delimited I/O so as to support filenames having newlines embedded:

find . -maxdepth 2 \( -regex '^./[^/]+' -o -type f \) ! -name '.*' -print0 \
    | LC_ALL=C gawk -F/ -v RS='\0' -v ORS='\0' \
        '{if (NF==2) {d=1; n=0} else if (d && ++n>100) {d=0; print $2}}' \
    | xargs -r0 mv -t dest/

Compared to pure shell solutions this pipeline should be less resource hungry because it doesn't do any buffering along the processing hence should be essentially unaffected by any number of folders and files.

Note the n>100 comparison in the awk script: that is where you would adjust the threshold at will.

This pipeline expects to be run from the directory containing the folders to examine because it uses a "naked" find .. However, you can easily make it general by just prepending the find . piece with a cd -- "${topdir:-.}" && piece, so that you would provide the starting directory via a custom $topdir shell variable defaulting to . i.e. the current directory.

A fair equivalent of such pipeline using BSD tools, except for support of newlines in filenames due to inherent limitations of BSD tools, could be as follows:

find -E . -maxdepth 2 \( -regex '^./[^/]+' -o -type f \) ! -name '.*' \
    | LC_ALL=C awk -F/ -v q=\' \
        '{if (NF==2) {d=1; n=0} else if (d && ++n>100) {d=0; gsub(q, q"\\"q q, $2); print q$2q}}' \
    | xargs sh -c '${1:+mv -- "$@" dest/}' --

which is essentially the same as the GNU tools version except the various -print0, -z and -0 options plus an additional gsub() operation by the awk script in order to quote separation characters (the " ' <space> chars) possibly present in filenames, as is needed by POSIX xargs to consume.

This latter pipeline should work correctly on any BSD system, assuming no examined path (folders and files) contains newlines.

Speaking of POSIX compliancy, the BSD tools pipeline should also work on any POSIX system except for the find command because POSIX doesn't have -maxdepth and -regex clauses. A POSIX equivalent for that find could be like:

# replace the find command of the BSD tools version, up to and including the trailing backslash character
find . \( -path '*/*/*' ! -path '*/*/*/*' -type f \) -o \( -path '*/*' ! -path '*/*/*' -type d \) ! -name '.*' \

The find expression is crafted to ease the awk script's job too, and selects regular files at the third level of the directory hierarchy (where level 1 is .) plus only directories at the second level. Lacking the more powerful clauses available in BSD and GNU find, I'm obtaining the same result with a game of -path clauses.

Note finally that these pipelines explicitly ignore hidden files via the ! -name '.*' clause to find, and they consider only regular files via the -type f clause (hence excluding e.g. symlinks) because that seemed the most sensible selection according to your question, but if you do want to account for hidden files and/or sub-folders, symlinks, and maybe special files (named pipes, named sockets, etc.) possibly present inside the folders, you can just take the respective clause away or perhaps fine tune them with additional clauses to the find command. In this latter case pay attention to make find always produce the lone folders' names too1, because those lone names are the "signal" used by the awk script to detect that the following names are in a different folder than the previous names2.


1. just the ones at the second level

2. for better performance I've used a true/false test instead of a string comparison, see the d variable in the awk script

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.