How to move all files matching a certain name to a new folder if the number of matching files is greater than 10?

Question

I'm looking to create a script that when executed it will look at a directory and search for all files and then automatically discover the filename patterns to then move them based on additional logic stated below.

Say I have the following files in a folder:

• aaa.txt
• temp-203981.log
• temp-098723.log
• temp-123197.log
• temp-734692.log
• test1.sh
• test2.sh
• test3.sh

The script should automatically be able to search the directory and it should find that there are 4 files (temp-XXX.log) and 3 files (testXXX.sh) that have a matching prefix in their name. Then once having found the number of files it should compare it to a defined limit, say 3.

If the number of files matching the specified name is greater than the limit it should then move the found files into a folder named after the part of the file names that matched.

So the parent folder from above should now look like:

• aaa.txt
• temp.log (This would be the folder that contains temp-734692.log, temp-123197.log, temp-098723.log, temp-203981.log)
• test.sh (This would be the folder that contains test1.sh, test2.sh, test3.sh)

Hope this makes sense.

P.S. I am using ASH for this script so it will need to be able to run without many of the fancy bash abilities, otherwise this would be easier.

Thanks!

EDIT: Clarity changes in the beginning. Also, it might be easier if I supply a predetermined delimiter, say "&", which all the file names will have. The script will still need to create variable folder names based on the file names before the delimiter, but I think this will make things more clear and easier.

Answer 1

Check, does it work and I will add explanation, how it works. I tested it in the dash.

Note: file names should not contain spaces, newlines.

#!/bin/dash

limit=1

printf "%s\n" * |
sed 's/[-0-9]*\..*$//' |
uniq -c |
awk -v lim=${limit} '$1 >= lim {print $2}' |
sort -r |
while read -r i; do
    for j in "${i}"*; do
        [ -f "$j" ] || continue

        dir=${i}.${j#*.}

        [ -d "$dir" ] || mkdir "$dir"
        mv -v "$j" "$dir"
    done
done

There is one problem here - the case, when the file name equals to the future directory name, like the aaa.txt. In the aaa.txt case, the file name doesn't have any extra characters, so will nothing removed from it, therefore, the new directory name will be the same, that causes error:

mkdir: cannot create directory ‘aaa.txt’: File exists
mv: 'aaa.txt' and 'aaa.txt' are the same file

One workaround of this problem is checking, does the supposed directory name equals to the file name and, then adding some number to the future directory name, like aaa1.txt.

Demonstration

Before the script execution.

$ tree
.
├── aaa.txt
├── temp-098723.log
├── temp-123197.log
├── temp-203981.log
├── temp-734692.log
├── temp-new-file123.log
├── temp-new-file-2323-12.log
├── temp-new-file-342.log
├── test1.sh
├── test2.sh
└── test3.sh

0 directories, 11 files

After the script execution: script.sh

$ tree
.
├── aaa.txt
├── temp.log
│   ├── temp-098723.log
│   ├── temp-123197.log
│   ├── temp-203981.log
│   └── temp-734692.log
├── temp-new-file.log
│   ├── temp-new-file123.log
│   ├── temp-new-file-2323-12.log
│   └── temp-new-file-342.log
└── test.sh
    ├── test1.sh
    ├── test2.sh
    └── test3.sh

3 directories, 11 files

Answer 2

It's possible that I'm misunderstanding what you're asking here, but as stated I think that this question has some subtleties to it and requires a relatively sophisticated solution, i.e. I don't know how simple a script can be that will do what you want. For example, let's look carefully at your example list of files:

aaa.txt
temp-203981.log
temp-098723.log
temp-123197.log
temp-734692.log
test1.sh
test2.sh
test3.sh

According to your question, you want the prefixes extracted from this list to be temp and test, where aaa is excluded since there is only one file with aaa as a prefix and your example threshold is three. But why isn't te a prefix, since there are 7 files that start with te? Or, since it seems that you want to first group the files based on their filename suffixes, why is one of the new subdirectories not t.log or temp-.log instead of temp.log? I hope this discussion makes clear that if you really want your program to determine the potential prefixes on its own without taking a list of prefixes as an argument, then there are some ambiguities in your question statement that need to be resolved (and some corresponding choices that need to be made).

Here is a Python script that uses a simple trie data structure to search for the longest matching prefixes satisfying a few constraints (which can be supplied as arguments):

#!/usr/bin/env python2
# -*- coding: ascii -*-
"""
trieganize.py

Use the trie data structure to look for prefixes of filenames in a given
directory and then reorganiz those files into subdirectories based on
those prefixes.

In this script the trie data structure is just a dictionary of the
following form:

    trie = {
        "count":    integer,
        "children": dictionary,
        "leaf":     boolean
    }

Where the dictionary keys have the following semantics.

count:
    stores the number of total descendents of the given trie node

children:
    stores the child trie nodes of the given node

leaf:
    denotes whether this trie corresponds to the final character in a word
"""

import sys
import os
import string

def add_word_to_trie(trie, word):
    """Add a new word to the trie."""
    if word:
        trie["count"] += 1
        if word[0] not in trie["children"]:
            trie["children"][word[0]] = \
                {"count": 0, "children": {}, "leaf": False}
        add_word_to_trie(trie=trie["children"][word[0]], word=word[1:])
    else:
        trie["leaf"] = True
    return(trie)

def expand_trie(trie, prefix='', words=None):
    """Given a trie, return the list of words it encodes."""
    if words is None:
        words = list()
    if trie["leaf"]:
        words.append(prefix)
    for character, child in trie["children"].iteritems():
        if trie["children"]:
            expand_trie(trie=child, prefix=prefix+character, words=words)
    return(words)

def extract_groups_from_trie(
    trie, threshold=0, prefix='', groups=None,
    minimum_prefix_length=0,
    maximum_prefix_length=float("inf"),
    prefix_charset=string.ascii_letters,
):
    """Given a trie and some prefix constraints, return a dictionary which
    groups together the words in the trie based on shared prefixes which
    satisfy the specified constraints.
    """
    if groups is None:
        groups = dict()
    if trie["count"] >= threshold:
        children = {
            character: child
            for character, child in trie["children"].iteritems()
            if (
                child["count"] >= threshold and
                len(prefix) + 1 >= minimum_prefix_length and
                len(prefix) + 1 <= maximum_prefix_length and
                character in prefix_charset
            )
        }
        if not children:
            groups[prefix] = expand_trie(trie, prefix)
        else:
            for character, child in children.iteritems():
                extract_groups_from_trie(
                    trie=child, threshold=threshold,
                    prefix=prefix+character, groups=groups
                )
    return(groups)

def reorganize_files(basedir, suffix_separator='.', threshold=3):
    """Takes a path to a directory and reorganizes the files in that
    directory into subdirectories based on the prefixes of their
    filenames."""

    # Get the list of file names
    filenames = os.listdir(basedir)

    # Group the filenames by suffix
    suffixes = {}
    for filename in filenames:
        basename, separator, suffix = filename.rpartition(suffix_separator)
        if suffix not in suffixes:
            suffixes[suffix] = []
        suffixes[suffix].append(basename)

    # For each suffix, search for prefixes
    for suffix, basenames in suffixes.iteritems():

        # Initialize a trie object
        trie = {"count":0, "children": {}, "leaf": False}

        # Add the filenames to the trie
        for basename in basenames:
            add_word_to_trie(trie, basename)

        # Break the filenames up into groups based on their prefixes
        groups = extract_groups_from_trie(trie, threshold)

        # Organize the groups of files into subdirectories
        for prefix, group in groups.iteritems():
            targetdir = os.path.join(basedir, prefix + suffix_separator + suffix)
            os.mkdir(targetdir)
            for basename in group:
                filename = basename + suffix_separator + suffix
                sourcefile = os.path.join(basedir, filename) 
                targetfile = os.path.join(targetdir, filename)
                os.rename(sourcefile, targetfile)

if __name__=="__main__":
    reorganize_files(basedir=sys.argv[1])

In order to demonstrate this Python script, I wrote a small shell script to create and populate a test directory:

#!/usr/bin/bash

# create-test-dir.sh

rm -rf /tmp/testdir
mkdir -p /tmp/testdir

files=(
aaa.txt
temp-203981.log
temp-098723.log
temp-123197.log
temp-734692.log
test1.sh
test2.sh
test3.sh
)

for file in ${files[@]}; do touch "/tmp/testdir/${file}"; done

We can run the script:

bash create-test-dir.sh

Afterwards, our test directory looks like this (run tree /tmp/testdir):

/tmp/testdir/
|-- aaa.txt
|-- temp-098723.log
|-- temp-123197.log
|-- temp-203981.log
|-- temp-734692.log
|-- test1.sh
|-- test2.sh
`-- test3.sh

0 directories, 8 files

Now we can run the Python script:

python trieganize.py /tmp/testdir

And afterwards the files are organized as follows:

/tmp/testdir/
|-- aaa.txt
|-- temp.log
|   |-- temp-098723.log
|   |-- temp-123197.log
|   |-- temp-203981.log
|   `-- temp-734692.log
`-- test.sh
    |-- test1.sh
    |-- test2.sh
    `-- test3.sh

2 directories, 8 files

Answer 3

Yes, bash would make this easier, but here's a POSIX solution:

#!/bin/sh

for pattern in "$@"; do
   set -- "$pattern"*

   if [ $# -gt 2 ]; then

      for f in "$@"; do
         [ -f "$f" ] || continue
         ext="${f##*.}"
         dest="${pattern}.${ext}"
         [ -d "$dest" ] || mkdir "$dest"
         mv "$f" "$dest"
      done

   fi
done

exit

This takes any number of patterns, e.g. ./script temp test. For each pattern, set the positional parameters to the files matching the pattern, and move them to a folder named <pattern>.<file_extension> if there are 3 or more files matching the pattern. I used your example files and got the intended results.

EDIT: test that $f is a regular file to avoid moving directories and such.