How to delete all directories in a directory older than 2 weeks except the latest one that match a file pattern?

Question

I have the following path:

/dir1/dir2/

In this path I have the following directories containing various (not relevant) application detrius:

follower1234  1-Dec-2018
follower3456  2-Dec-2018
follower4567  3-Dec-2018
follower7890  9-Jan-2019
follower8901 10-Jan-2019
leader8765    4-Dec-2018
bystander6789 5-Dec-2018

Assume that today is 10 Jan 2019.

Assume that there can be any number of followerXXXX, leaderXXXX and bystanderXXXX directories.

What I want is to delete all followerXXXX directories but the latest followerXXX directory, that are older than two weeks old.

Now I can delete all directories older than a particular date. But that isn't my question. I'm adding two additional parameters.

In this case I want to delete:

follower1234  1-Dec-2018
follower3456  2-Dec-2018
follower4567  3-Dec-2018

But not

follower7890  9-Jan-2019
follower8901 10-Jan-2019
leader8765    4-Dec-2018
bystander6789 5-Dec-2018

ie I want to delete files

(a) matching a pattern

(b) older than two weeks

(c) not the latest directory matching the pattern (ie keep the last one)

My question is: How to delete all directories in a directory older than 2 weeks except the latest one that match a file pattern?

Answer 1

Introduction

The question has been modified.

• My first alternative (the oneliner) does not match the new specification, but saves the newest directory among the directories that are old enough to be deleted (older than 14 days).

• I made a second alternative, (the shellscript) that uses

  @ seconds since Jan. 1, 1970, 00:00 GMT, with fractional part.

  and subtracting the seconds corresponding to 14 days to get a timestamp for the 'limit-in-seconds' at seclim in the sorted list of directories.

1. Oneliner

The previous answers are clean and nice, but they do not preserve the newest follower directory. The following command line will do it (and can manage names with spaces but names with newlines create problems),

find . -type d -name "follower*" -printf "%T+ %p\n"|sort|head -n -1 | cut -d ' ' -f 2- | sed -e 's/^/"/' -e 's/$/"/' | xargs echo rm -r

tested on this directory structure,

$ find -printf "%T+ %p\n"|sort
2019-01-10+13:11:40.6279621810 ./follower1
2019-01-10+13:11:40.6279621810 ./follower1/2/3
2019-01-10+13:11:40.6279621810 ./follower1/2/dirnam with spaces
2019-01-10+13:11:40.6279621810 ./follower1/2/name with spaces
2019-01-10+13:11:56.5968732640 ./follower1/2/file
2019-01-10+13:13:18.3975675510 ./follower2
2019-01-10+13:13:19.4016254340 ./follower3
2019-01-10+13:13:20.4056833250 ./follower4
2019-01-10+13:13:21.4097412230 ./follower5
2019-01-10+13:13:22.4137991260 ./follower6
2019-01-10+13:13:23.4138568040 ./follower7
2019-01-10+13:13:24.4219149500 ./follower8
2019-01-10+13:13:25.4259728780 ./follower9
2019-01-10+13:15:34.4094596830 ./leader1
2019-01-10+13:15:36.8336011960 .
2019-01-10+13:15:36.8336011960 ./leader2
2019-01-10+13:25:03.0751878450 ./follower1/2

like so,

$ find . -type d -name "follower*" -printf "%T+ %p\n"|sort|head -n -1 | cut -d ' ' -f 2- | sed -e 's/^/"/' -e 's/$/"/' | xargs echo rm -r
rm -r ./follower1 ./follower2 ./follower3 ./follower4 ./follower5 ./follower6 ./follower7 ./follower8

So follower9 is excluded because it is the newest followerdirectory (directories with names, that do not start with follower (leader1, leader2 and 2 are not in the game).

Now we add the time criterion, -mtime +14 and do another 'dry run' to check that it works like it should, when we have changed directory to where there are real follower directories,

find . -type d -name "follower*" -mtime +14 -printf "%T+ %p\n"|sort|head -n -1 | cut -d ' ' -f 2- | sed -e 's/^/"/' -e 's/$/"/' | xargs echo rm -r

Finally we remove echo and have a command line that can do what we want,

find . -type d -name "follower*" -mtime +14 -printf "%T+ %p\n"|sort|head -n -1 | cut -d ' ' -f 2- | sed -e 's/^/"/' -e 's/$/"/' | xargs rm -r

---

• find in the current directory, directories with names beginning with follower, that are not modified since 14 days ago.
• After printing and sorting head -n -1 will exclude the newest follower directory.
• The time stamps are cut away, and double quotes are added at the head end and tail end of each directory name.
• Finally the result is piped via xargs as parameters to rm -r in order to remove the directories, that we want to remove.

2. Shellscript

I made a second alternative, (the shellscript) that uses

@      seconds since Jan. 1, 1970, 00:00 GMT, with fractional part.

It has also two options,

• -n dry run

• -v verbose

• I modified the shellscript according to what the OP wants: enter the pattern as a parameter within single quotes e.g. 'follower*'.

• I suggest that the name of the shellscript is prune-dirs because it is more general now (no longer only prune-followers to prune directories follower*).

You are recommended to run the shellscript with both options the first time in order to 'see' what you will do, and when it looks correct, remove the -n to make the shellscript remove the directories that are old enough to be removed. So let us call it prune-dirs and make it executable.

#!/bin/bash

# date        sign     comment
# 2019-01-11  sudodus  version 1.1
# 2019-01-11  sudodus  enter the pattern as a parameter
# 2019-01-11  sudodus  add usage
# 2019-01-14  sudodus  version 1.2
# 2019-01-14  sudodus  check if any parameter to the command to be performed

# Usage

usage () {
 echo "Remove directories found via the pattern (older than 'datint')

 Usage:    $0 [options] <pattern>
Examples: $0 'follower*'
          $0 -v -n 'follower*'  # 'verbose' and 'dry run'
The 'single quotes' around the pattern are important to avoid that the shell expands
the wild card (for example the star, '*') before it reaches the shellscript"
 exit
}

# Manage options and parameters

verbose=false
dryrun=false
for i in in "$@"
do
 if [ "$1" == "-v" ]
 then
  verbose=true
  shift
 elif [ "$1" == "-n" ]
 then
  dryrun=true
  shift
 fi
done
if [ $# -eq 1 ]
then
 pattern="$1"
else
 usage
fi

# Command to be performed on the selected directories

cmd () {
 echo rm -r "$@"
}

# Pattern to search for and limit between directories to remove and keep

#pattern='follower*'
datint=14  # days

tmpdir=$(mktemp -d)
tmpfil1="$tmpdir"/fil1
tmpfil2="$tmpdir"/fil2

secint=$((60*60*24*datint))
seclim=$(date '+%s')
seclim=$((seclim - secint))
printf "%s limit-in-seconds\n" $seclim > "$tmpfil1"

if $verbose
then
 echo "----------------- excluding newest match:"
 find . -type d -name "$pattern" -printf "%T@ %p\n" | sort |tail -n1 | cut -d ' ' -f 2- | sed -e 's/^/"/' -e 's/$/"/'
fi

# exclude the newest match with 'head -n -1'

find . -type d -name "$pattern" -printf "%T@ %p\n" | sort |head -n -1 >> "$tmpfil1"

# put 'limit-in-seconds' in the correct place in the sorted list and remove the timestamps

sort "$tmpfil1" | cut -d ' ' -f 2- | sed -e 's/^/"/' -e 's/$/"/' > "$tmpfil2"

if $verbose
then
 echo "----------------- listing matches with 'limit-in-seconds' in the sorted list:"
 cat "$tmpfil2"
 echo "-----------------"
fi

# create 'remove task' for the directories older than 'limit-in-seconds'

params=
while read filnam
do
 if [ "${filnam/limit-in-seconds}" != "$filnam" ]
 then
  break
 else
  params="$params $filnam"
 fi
done < "$tmpfil2"
cmd $params > "$tmpfil1"
cat  "$tmpfil1"

if ! $dryrun && ! test -z "$params"
then
 bash "$tmpfil1"
fi
rm -r $tmpdir

• Change current directory to the directory with the follower subdirectories
• create the file prune-dirs
• make it executable

• and run with the two options -v -n

  cd directory-with-subdirectories-to-be-pruned/
  nano prune-dirs  # copy and paste into the editor and save the file
  chmod +x prune-dirs
  ./prune-dirs -v -n
  

Test

I tested prune-dirs in a directory with the following sub-directories, as seen with find

$ find . -type d -printf "%T+ %p\n"|sort
2018-12-01+02:03:04.0000000000 ./follower1234
2018-12-02+03:04:05.0000000000 ./follower3456
2018-12-03+04:05:06.0000000000 ./follower4567
2018-12-04+05:06:07.0000000000 ./leader8765
2018-12-05+06:07:08.0000000000 ./bystander6789
2018-12-06+07:08:09.0000000000 ./follower with spaces old
2019-01-09+10:11:12.0000000000 ./follower7890
2019-01-10+11:12:13.0000000000 ./follower8901
2019-01-10+13:15:34.4094596830 ./leader1
2019-01-10+13:15:36.8336011960 ./leader2
2019-01-10+14:08:36.2606738580 ./2
2019-01-10+14:08:36.2606738580 ./2/follower with spaces
2019-01-10+17:33:01.7615641290 ./follower with spaces new
2019-01-10+19:47:19.6519169270 .

Usage

$ ./prune-dirs
Remove directories found via the pattern (older than 'datint')

 Usage:    ./prune-dirs [options] <pattern>
Examples: ./prune-dirs 'follower*'
          ./prune-dirs -v -n 'follower*'  # 'verbose' and 'dry run'
The 'single quotes' around the pattern are important to avoid that the shell expands
the wild card (for example the star, '*') before it reaches the shellscript

Run with -v -n (a verbose dry run)

$ ./prune-dirs -v -n 'follower*'
----------------- excluding newest match:
"./follower with spaces new"
----------------- listing matches with 'limit-in-seconds' in the sorted list:
"./follower1234"
"./follower3456"
"./follower4567"
"./follower with spaces old"
"limit-in-seconds"
"./follower7890"
"./follower8901"
"./2/follower with spaces"
-----------------
rm -r "./follower1234" "./follower3456" "./follower4567" "./follower with spaces old"

A verbose dry run with a more general pattern

$ LANG=C ./prune-dirs -v -n '*er*'
----------------- excluding newest match:
"./follower with spaces new"
----------------- listing matches with 'limit-in-seconds' in the sorted list:
"./follower1234"
"./follower3456"
"./follower4567"
"./leader8765"
"./bystander6789"
"./follower with spaces old"
"limit-in-seconds"
"./follower7890"
"./follower8901"
"./leader1"
"./leader2"
"./2/follower with spaces"
-----------------
rm -r "./follower1234" "./follower3456" "./follower4567" "./leader8765" "./bystander6789" "./follower with spaces old"

Run without any options (a real case removing directories)

$ ./prune-dirs 'follower*'
rm -r "./follower1234" "./follower3456" "./follower4567" "./follower with spaces old"

Run with -v 'trying again'

$ LANG=C ./prune-dirs -v 'follower*'
----------------- excluding newest match:
"./follower with spaces new"
----------------- listing matches with 'limit-in-seconds' in the sorted list:
"limit-in-seconds"
"./follower7890"
"./follower8901"
"./2/follower with spaces"
-----------------
rm -r

The shellscript lists no directory 'above' "limit-in-seconds", and there are no files listed for the rm -r command line, so the work was done already (which is the correct result). But if you run the shellscript again several days later, some new directory may be found 'above' "limit-in-seconds" and be removed.

Answer 2

Complementing the rowan's answer. You can change the dot by the path to directories

find . -type d -name follower* -mtime +14 -exec rm -rf {} +;

Answer 3

With zsh:

(){ n=$#; } follower<->(/)       # count the number of follower<n> dirs

to_remove=(follower<->(/m+13om)) # assumes the dir list is not changed
                                 # since the previous command

(($#to_remove < n)) || to_remove[1]=() # keep the youngest if they're
                                       # all over 2 weeks old



echo rm -rf $to_remove

(remove echo when happy)

• <-> any sequence of decimal digits (a short form of <1-20> be without bound).
• (){code} args: anonymous function which here stores its number of arguments in $n.
• (/omm+13): glob qualifier
• /: only select files of type directory (equivalent of find's -type d)
• m+13: files whose age in whole days is strictly greater than 13 days, so files that are 14 days old or older (equivalent of find's -mtime +13).
• om: order by modification time (like ls -t younger files first)

Note that it's dangerous to rely on directory modification time. directories are modified when files are added, removed or renamed in them (or when they're touched). Since those directories are numbered, you may want to rely on that numbering instead, so replace om with nOn (numerically Order in reverse (capital O) by name).

To have the pattern in a variable, replace follower<-> with $~pattern and set pattern='follower<->' or any other value.

Answer 4

The moment I need to delete files or directories related to time, I would use find.

Before deleting anything, you can run the command a few times to see if it finds everything you desire.

find . -type d -mtime +14
# -type c, File is of type c: d  directory
# -mtime n, File's data was last modified n*24 hours ago.

If it matches all your criteria, you can add -exec rm -r {} + behind it:

find . -type d -mtime +14 -exec rm -r {} +

The reason we are using -exec here, is because -delete will not work if the directory is not empty.

Check out man find for more guidance.

Answer 5

A couple of solutions:

1. Based on GNU find:

#!/bin/bash

# The pattern has to be available to subshells
export patt="$1"

find . -maxdepth 1 -type d -name "${patt}*" -mtime +14 \
  -exec sh -c '[ "$(find . -maxdepth 1 -type d -name "${patt}*" -print0 |
    sort -z -V |
    tail -z -n 1 |
    tr -d "\0")" != "$1" ]' sh {} \; \
  -exec sh -c 'echo rm -r "$1"' sh {} \;

The script is meant to be invoked as:

./script name_pattern

As-is, it will give you a dry run. Remove echo in the last -exec action to let it actually delete directories.

It will:

• Find all the directories in the current directory that have been modified more than 14 days ago (but see the note about -mtime below) and have a name that starts with the value of ${patt}; for each:
• Ensure (the first -exec) that the found directory is not the last one matching the name pattern, sorting in ascending version order (-V) (to have, for instance, follower100 placed after follower2); if the test ([) fails, find skips to the next cycle and does not perform the actions that follow;
• Remove the found directory (the second -exec).

Here I am assuming an equivalence between sorting your directories in lexicographical order by name and sorting them by modification date. This is ok if your latest directory is defined in terms of its name.
If, instead, your latest directory is the one with the most recent modification time, we have to replace the first -exec ... in the above code with this one:

  -exec sh -c '[ "$(find . -maxdepth 1 -type d -name "${patt}*" -printf "%T@\n" |
    sed "s/\..*$//" |
    sort -n |
    tail -n 1)" != "$(stat -c "%Y" "$1")" ]' sh {} \; \

Where with an inner find we find all the directories matching the name pattern, print the list of their modification times in seconds since Epoch, cut the fractional part away, sort, take the last one and check that it is not equal to that of the current result of the outer find.

Note that, using this filter, if all the matching directories are older than 14 days and have all exactly the same modification time none of them is deleted.

---

Notes:

Limiting the search to the content of the current directory (-maxdepth 1) is not strictly required.

You may want to tell sort how to order things, e.g. adding export LC_ALL=C at the beginning of the script (refer to this answer to 'What does "LC_ALL=C" do?' about the issues you may have when sorting, depending on your localization settings).

Note that, using -mtime +14, files that have been modified between 14 and 15 days ago are skipped even if their modification time is technically older than 14*24 hours from now (refer to man find for details; specifically, the description of -atime n).

It will work even when names contain spaces, newlines, uncommon and non-printable characters.

Compatibility: the flip side is that it is not portable: some features used here, notably find's -maxdepth, -print0 and -printf, the stat command, the -V option to sort and the -z option to sort and tail (and I am possibly forgetting some more), are not specified in POSIX.

2. Based on shell features

#!/bin/sh

patt="$1"                 # The name pattern
test -z "$patt" && exit   # Caution: pattern must not be empty

days=14     # How old has to be a directory to get deleted, in days?
last=       # The youngest directory

dirs=( "$patt"* )     # Array of files matched by name (note, here we
                      # have everything that matches, not just dirs)
now="$(date +'%s')"   # Now in seconds since Epoch

threshold="$(( "$now" - ( "$days" * 24 * 60 *60 ) ))"
                      # Dirs older than this date (in seconds since
                      # Epoch) are candidates for deletion

# We find the youngest directory in the array
#
for i in "${!dirs[@]}"; do
  if  [ -z "$last" ] ||
    ( [ -d "${dirs[$i]}" ] &&
      [ "$(stat -c '%Y' -- "${dirs[$i]}")" -gt "$(stat -c '%Y' -- "$last")" ] ); then
    last="${dirs[$i]}"
  fi
done

# We delete all the directories in the array that are
# not the youngest one AND are older that the thrashold
#
for i in "${!dirs[@]}"; do
  if  [ -d "${dirs[$i]}" ] &&
      [ "${dirs[$i]}" != "$last" ] &&
      [ "$(stat -c '%Y' -- "${dirs[$i]}")" -lt "$threshold" ]; then
    echo rm -rf -- "${dirs[$i]}"
  fi
done

This script, too, is meant to be invoked as

./script name_pattern

Again, it will give you a dry run until you remove echo from echo rm -rf -- "${dirs[$i]}".

It will:

• Populate an array with the names of all files, in the current directory, that match the name pattern;
• Determine the youngest directory in the array;
• Delete all the directories in the array that 1) are older than 14 days AND 2) are not the youngest one.

---

Notes:

It will target directories older then 14 days from now (unlike find). Thus, these two solutions are not strictly equivalent.
Also, if all the matching directories are older than the threshold and have all the same modification time, it will delete all but one of them - randomly chosen.

Names with uncommon characters are ok, including newlines and non printable ones.

Compatibility: even this solution relies on some non POSIX features: namely, stat and the %s date format. Ah, and arrays, apparently...