2

I'm looking to create a script that when executed it will look at a directory and search for all files and then automatically discover the filename patterns to then move them based on additional logic stated below.

Say I have the following files in a folder:

  • aaa.txt
  • temp-203981.log
  • temp-098723.log
  • temp-123197.log
  • temp-734692.log
  • test1.sh
  • test2.sh
  • test3.sh

The script should automatically be able to search the directory and it should find that there are 4 files (temp-XXX.log) and 3 files (testXXX.sh) that have a matching prefix in their name. Then once having found the number of files it should compare it to a defined limit, say 3.

If the number of files matching the specified name is greater than the limit it should then move the found files into a folder named after the part of the file names that matched.

So the parent folder from above should now look like:

  • aaa.txt
  • temp.log (This would be the folder that contains temp-734692.log, temp-123197.log, temp-098723.log, temp-203981.log)
  • test.sh (This would be the folder that contains test1.sh, test2.sh, test3.sh)

Hope this makes sense.

P.S. I am using ASH for this script so it will need to be able to run without many of the fancy bash abilities, otherwise this would be easier.

Thanks!

EDIT: Clarity changes in the beginning. Also, it might be easier if I supply a predetermined delimiter, say "&", which all the file names will have. The script will still need to create variable folder names based on the file names before the delimiter, but I think this will make things more clear and easier.

  • 2
    Is it the shortest prefix or the longest one? In other words, if there are two temp-aaa-* and three temp-bbb-* files, should it move the three temp-bbb-* files and leave alone the two temp-aaa-* or should it move the five temp-* files? Or is it just a matter of dropping the final digits (which is likely a lot simpler)? – xenoid Nov 15 '17 at 21:21
  • 1
    If I'm understanding you correctly then I'd like it to just drop the final parts that are in disagreement. For example if I had files named grc-wget-errors-20171115-004503.log and grc-wget-errors-20171115-005503.log it would then create a directory grc-wget-errors and move both of those files in. Does that make sense? – lukemk1 Nov 15 '17 at 21:37
  • 2
    Still ambiguous. If I have log-09, log-10, log-11, log-12, do I move the log-1* files only or do I move all the log-* ones? – xenoid Nov 15 '17 at 21:45
  • Are you allowing the user to supply the prefixes as a list of arguments, or is the program supposed to determine the prefixes itself based on the filenames in the given directory? – igal Nov 16 '17 at 7:01
  • @xenoid, it should move all the log-* ones. – lukemk1 Nov 16 '17 at 15:02
1

Check, does it work and I will add explanation, how it works. I tested it in the dash.

Note: file names should not contain spaces, newlines.

#!/bin/dash

limit=1

printf "%s\n" * |
sed 's/[-0-9]*\..*$//' |
uniq -c |
awk -v lim=${limit} '$1 >= lim {print $2}' |
sort -r |
while read -r i; do
    for j in "${i}"*; do
        [ -f "$j" ] || continue

        dir=${i}.${j#*.}

        [ -d "$dir" ] || mkdir "$dir"
        mv -v "$j" "$dir"
    done
done

There is one problem here - the case, when the file name equals to the future directory name, like the aaa.txt. In the aaa.txt case, the file name doesn't have any extra characters, so will nothing removed from it, therefore, the new directory name will be the same, that causes error:

mkdir: cannot create directory ‘aaa.txt’: File exists
mv: 'aaa.txt' and 'aaa.txt' are the same file

One workaround of this problem is checking, does the supposed directory name equals to the file name and, then adding some number to the future directory name, like aaa1.txt.

Demonstration

Before the script execution.

$ tree
.
├── aaa.txt
├── temp-098723.log
├── temp-123197.log
├── temp-203981.log
├── temp-734692.log
├── temp-new-file123.log
├── temp-new-file-2323-12.log
├── temp-new-file-342.log
├── test1.sh
├── test2.sh
└── test3.sh

0 directories, 11 files

After the script execution: script.sh

$ tree
.
├── aaa.txt
├── temp.log
│   ├── temp-098723.log
│   ├── temp-123197.log
│   ├── temp-203981.log
│   └── temp-734692.log
├── temp-new-file.log
│   ├── temp-new-file123.log
│   ├── temp-new-file-2323-12.log
│   └── temp-new-file-342.log
└── test.sh
    ├── test1.sh
    ├── test2.sh
    └── test3.sh

3 directories, 11 files
  • Is this the same approach as the one taken by @m0dular? I don't think that this is what's being asked. I think that the OP wanted a script that would figure out the prefixes on its own, not one that takes the list of prefixes as an argument. – igal Nov 16 '17 at 14:28
  • @igal The script determines prefixes automatically. Takes all filenames and extracts common prefixes from them. Try it yourself on your files set. – MiniMax Nov 16 '17 at 14:31
  • Ah, I see. I apologize. This is different. I didn't look at your code carefully. But I think this is still assuming much more than what's stated in the OP. You're using numerical characters to delimit the prefix from the rest of the filename. This solves a much simpler problem than the one I thought was being given. This works for the examples given in the OP, but it wouldn't work in general, right? But maybe this is all that the OP wanted. I think the question needs to be clarified. – igal Nov 16 '17 at 14:39
  • @igal OP was wrote in the comment section: "For example if I had files named grc-wget-errors-20171115-004503.log and grc-wget-errors-20171115-005503.log it would then create a directory grc-wget-errors and move both of those files in". So, I was understand, that the numerical part is not taken into account. – MiniMax Nov 16 '17 at 14:45
  • Yeah, you might be right. Like I said, I think that the question needs to be clarified and rewritten. Since they didn't explicitly say that the prefix would be nonnumerical and that tail of the filename would be numberical, I had assumed that the OP would also want to handle cases with files like grc-wget-errors-abcdefgh-ijklmn.log or grc1-wget2-errors3-abcd1234-efg567.log. – igal Nov 16 '17 at 14:48
1

It's possible that I'm misunderstanding what you're asking here, but as stated I think that this question has some subtleties to it and requires a relatively sophisticated solution, i.e. I don't know how simple a script can be that will do what you want. For example, let's look carefully at your example list of files:

aaa.txt
temp-203981.log
temp-098723.log
temp-123197.log
temp-734692.log
test1.sh
test2.sh
test3.sh

According to your question, you want the prefixes extracted from this list to be temp and test, where aaa is excluded since there is only one file with aaa as a prefix and your example threshold is three. But why isn't te a prefix, since there are 7 files that start with te? Or, since it seems that you want to first group the files based on their filename suffixes, why is one of the new subdirectories not t.log or temp-.log instead of temp.log? I hope this discussion makes clear that if you really want your program to determine the potential prefixes on its own without taking a list of prefixes as an argument, then there are some ambiguities in your question statement that need to be resolved (and some corresponding choices that need to be made).

Here is a Python script that uses a simple trie data structure to search for the longest matching prefixes satisfying a few constraints (which can be supplied as arguments):

#!/usr/bin/env python2
# -*- coding: ascii -*-
"""
trieganize.py

Use the trie data structure to look for prefixes of filenames in a given
directory and then reorganiz those files into subdirectories based on
those prefixes.

In this script the trie data structure is just a dictionary of the
following form:

    trie = {
        "count":    integer,
        "children": dictionary,
        "leaf":     boolean
    }

Where the dictionary keys have the following semantics.

count:
    stores the number of total descendents of the given trie node

children:
    stores the child trie nodes of the given node

leaf:
    denotes whether this trie corresponds to the final character in a word
"""

import sys
import os
import string

def add_word_to_trie(trie, word):
    """Add a new word to the trie."""
    if word:
        trie["count"] += 1
        if word[0] not in trie["children"]:
            trie["children"][word[0]] = \
                {"count": 0, "children": {}, "leaf": False}
        add_word_to_trie(trie=trie["children"][word[0]], word=word[1:])
    else:
        trie["leaf"] = True
    return(trie)

def expand_trie(trie, prefix='', words=None):
    """Given a trie, return the list of words it encodes."""
    if words is None:
        words = list()
    if trie["leaf"]:
        words.append(prefix)
    for character, child in trie["children"].iteritems():
        if trie["children"]:
            expand_trie(trie=child, prefix=prefix+character, words=words)
    return(words)

def extract_groups_from_trie(
    trie, threshold=0, prefix='', groups=None,
    minimum_prefix_length=0,
    maximum_prefix_length=float("inf"),
    prefix_charset=string.ascii_letters,
):
    """Given a trie and some prefix constraints, return a dictionary which
    groups together the words in the trie based on shared prefixes which
    satisfy the specified constraints.
    """
    if groups is None:
        groups = dict()
    if trie["count"] >= threshold:
        children = {
            character: child
            for character, child in trie["children"].iteritems()
            if (
                child["count"] >= threshold and
                len(prefix) + 1 >= minimum_prefix_length and
                len(prefix) + 1 <= maximum_prefix_length and
                character in prefix_charset
            )
        }
        if not children:
            groups[prefix] = expand_trie(trie, prefix)
        else:
            for character, child in children.iteritems():
                extract_groups_from_trie(
                    trie=child, threshold=threshold,
                    prefix=prefix+character, groups=groups
                )
    return(groups)

def reorganize_files(basedir, suffix_separator='.', threshold=3):
    """Takes a path to a directory and reorganizes the files in that
    directory into subdirectories based on the prefixes of their
    filenames."""

    # Get the list of file names
    filenames = os.listdir(basedir)

    # Group the filenames by suffix
    suffixes = {}
    for filename in filenames:
        basename, separator, suffix = filename.rpartition(suffix_separator)
        if suffix not in suffixes:
            suffixes[suffix] = []
        suffixes[suffix].append(basename)

    # For each suffix, search for prefixes
    for suffix, basenames in suffixes.iteritems():

        # Initialize a trie object
        trie = {"count":0, "children": {}, "leaf": False}

        # Add the filenames to the trie
        for basename in basenames:
            add_word_to_trie(trie, basename)

        # Break the filenames up into groups based on their prefixes
        groups = extract_groups_from_trie(trie, threshold)

        # Organize the groups of files into subdirectories
        for prefix, group in groups.iteritems():
            targetdir = os.path.join(basedir, prefix + suffix_separator + suffix)
            os.mkdir(targetdir)
            for basename in group:
                filename = basename + suffix_separator + suffix
                sourcefile = os.path.join(basedir, filename) 
                targetfile = os.path.join(targetdir, filename)
                os.rename(sourcefile, targetfile)

if __name__=="__main__":
    reorganize_files(basedir=sys.argv[1])

In order to demonstrate this Python script, I wrote a small shell script to create and populate a test directory:

#!/usr/bin/bash

# create-test-dir.sh

rm -rf /tmp/testdir
mkdir -p /tmp/testdir

files=(
aaa.txt
temp-203981.log
temp-098723.log
temp-123197.log
temp-734692.log
test1.sh
test2.sh
test3.sh
)

for file in ${files[@]}; do touch "/tmp/testdir/${file}"; done

We can run the script:

bash create-test-dir.sh

Afterwards, our test directory looks like this (run tree /tmp/testdir):

/tmp/testdir/
|-- aaa.txt
|-- temp-098723.log
|-- temp-123197.log
|-- temp-203981.log
|-- temp-734692.log
|-- test1.sh
|-- test2.sh
`-- test3.sh

0 directories, 8 files

Now we can run the Python script:

python trieganize.py /tmp/testdir

And afterwards the files are organized as follows:

/tmp/testdir/
|-- aaa.txt
|-- temp.log
|   |-- temp-098723.log
|   |-- temp-123197.log
|   |-- temp-203981.log
|   `-- temp-734692.log
`-- test.sh
    |-- test1.sh
    |-- test2.sh
    `-- test3.sh

2 directories, 8 files
0

Yes, bash would make this easier, but here's a POSIX solution:

#!/bin/sh

for pattern in "$@"; do
   set -- "$pattern"*

   if [ $# -gt 2 ]; then

      for f in "$@"; do
         [ -f "$f" ] || continue
         ext="${f##*.}"
         dest="${pattern}.${ext}"
         [ -d "$dest" ] || mkdir "$dest"
         mv "$f" "$dest"
      done

   fi
done

exit

This takes any number of patterns, e.g. ./script temp test. For each pattern, set the positional parameters to the files matching the pattern, and move them to a folder named <pattern>.<file_extension> if there are 3 or more files matching the pattern. I used your example files and got the intended results.

EDIT: test that $f is a regular file to avoid moving directories and such.

  • The question is how to "automatically" discover the patterns... If you give them in as paramaters, the whole thing is trivial. – xenoid Nov 16 '17 at 9:21
  • @xenoid In m0dular's defense, the question wasn't stated in a way that made it unambiguously clear what was being asked. The OP asked for "a simple script that [...] will look at a directory and search for all files matching a certain name." This seemed to contradict (or at least be inconsistent with) the rest of the question, but I personally found it pretty confusing. – igal Nov 16 '17 at 13:58
  • @igal, xenoid is correct in that it should automatically discover the patterns. I can see how my question was not exactly clear in that intent though. I will edit it to remove confusion. – lukemk1 Nov 16 '17 at 15:07
  • @user3529430 Well, I'm glad to hear that since I had the same interpretation. There is also some ambiguity as whether or not there is some assumed pattern in the filenames. See my discussion in the comments on the answer posted by MiniMax. – igal Nov 16 '17 at 15:12
  • Hmm, I suppose I could have read the question more closely. I'll leave this here in case someone finds it useful. – m0dular Nov 17 '17 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.