2

What I Need

I'm having trouble creating a script (bash?) that is going to be used in cron, run nightly, unattended, that deletes all backup DIRS except ALL first-of-month backups AND the 14 latest, even if older. If bash won't do, then shell and POSIX as I need this to be portable.

Script must be safe, elegant, and Where I'm Stuck: recognize that no backups have happened since May and STILL keep the most recent (May) 14 backups, as those are the MOST RECENT, even if script runs in November. In all cases, script must keep all backups dated with a 01 in the day (-YYYYMMDD-) portion of the name.

What I Have

  • I have DIRS containing backups
  • The backup date is in the DIR name
  • Script must read contents of /path/to/backups/example.com/ and decide which DIRS inside to remove
  • DIRS are not empty. They contain the backups for that day.
/path/to/backups/example.com/example.com-20210101-backup/ // Keep (first of month)
/path/to/backups/example.com/example.com-20210201-backup/ // Keep (first of month)
/path/to/backups/example.com/example.com-20210301-backup/ // Keep (first of month)
/path/to/backups/example.com/example.com-20210401-backup/ // Keep (first of month)
/path/to/backups/example.com/example.com-20210501-backup/ // Keep (first of month)
/path/to/backups/example.com/example.com-20210502-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210503-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210504-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210505-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210506-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210507-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210508-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210509-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210510-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210511-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210512-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210513-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210514-backup/ // <-- Script to remove
/path/to/backups/example.com/example.com-20210515-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210516-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210517-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210518-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210519-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210520-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210521-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210522-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210523-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210524-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210525-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210526-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210527-backup/ // Keep (Most recent 14 days even if old)
/path/to/backups/example.com/example.com-20210528-backup/ // Keep (Most recent 14 days even if old)

Why a Script for This

Because, one day backups might resume and have new DIRS to remove unattended.

What I've Found

Everything I find either deletes all but the most recent 14 OR only keeps the first-of-months, not both.

For example, as per: https://unix.stackexchange.com/a/379041

find /path/to/backups/example.com/ -type d -mtime +14 -exec rm -rf {} +

would only show the 14 most recent from the date the script was run, which if run in November, won't show or remove anything.

My Trouble

Trouble creating a safe script that BOTH keeps all first-of-month backups AND the 14 latest, even if old.

Any help pointing me in the right direction is appreciated,

--Thank You!

3
  • Not exactly what you want, but you seem to be re-inventing something like rdiff-backup, which can backup a directory and keep an incremental history of previous history using rdiff, and has various options for expiry of old backups. I don't think it does the "keep the first of the month" backup you want, but that's easily solved by running rdiff-backup twice (once for daily, once for monthly) with different expiry options. I used to use it a lot until I switched to ZFS snapshots for my backups.
    – cas
    Nov 19, 2021 at 2:19
  • @cas thank you for the idea. Although the script would not be portable, If I don't come up with a solution, i will probably have to implement something like that.
    – Scott
    Nov 19, 2021 at 16:59
  • I'm pretty sure you could do this with some fancy zsh globbing. Something like set -- (fancy glob here, including sorting modifiers) followed by shift 14 and then rm -rf "$@" or your favorite deletion command.
    – Wildcard
    Nov 20, 2021 at 0:56

4 Answers 4

2

Assumptions:

  • your backup directories, at least at the topmost levels, are sanely constructed without whitespace/newlines/regex or shell glob metachars, etc.
  • you have a collection of directories within a path base_path
  • each directory begins with a prefix base_prefx
  • each directory ends with a suffix base_suffx
  • once stripped of the path, prefix and suffix, each directory name is a date YYYYMMDD
  • directories not meeting those criteria are to be ignored

With those givens, we can plan our strategy accordingly.

The crux of the task at hand is to remove zero or more directories, based on the YYYYMMDD portion of the directory name. To determine the specific directories (if any) to remove, we:

  • exclude all directories where the DD portion of the date is 01, or where any non-numeric character appears in the field where YYYYMMDD is expected
  • of the remaining directories, exclude the N most recent dates
  • all the remaining directories (if any) are to be removed

You have chosen N=14.

#!/usr/bin/env bash

retain=14

base_path='./path/to/backups/example.com/'
base_prefx='example.com-'
base_suffx='-backup'

find "$base_path" -maxdepth 1 -mindepth 1 \
    -type d \
    -name "${base_prefx}????????${base_suffx}" |

while IFS= read dir
do
        base="$(basename "$dir" "$base_suffx")"
        printf '%s\n' "${base#$base_prefx}"
done |
grep -Ev '([^[:digit:]]|01$)' |
sort -r |
tail +$(($retain+1)) |
while IFS= read base
do
        printf 'rm -rf "%q%q%q%q"\n' \
                "$base_path" "$base_prefx" "$base" "$base_suffx"
done

The find command looks in the base_path directory for sub-directory names which match the template of our assumed directory structure, and which reside exactly one level below the base_path directory.

find's output is fed to the while loop, which reads each line of input, strips off the base_path, base_prefx and base_suffx and writes the base part of the directory name (ostensibly the date) to stdout.

That stdout is then passed to grep which removes any entries which contain any non-numeric characters or which end in 01. Removing entries ending in 01 is important so that first-of-the-month backups are retained indefinitely.

grep's output is then sorted in descending order so that the most recent entries (excluding any ??????01 entries) are at the top of the output, and the less recent entries are later.

Now that we have excluded all ??????01 backup directory dates, and sorted the dates in descending order with the most recent dates first, the only remaining task is to skip the first N entries, and then delete any entries N+1 and higher.

The code uses the variable retain to represent N. tail reads the sorted output and begins outputting lines beginning at line retain+1, and that stdout stream is passed to a while loop.

The loop reads each line as variable base and re-constructs an rm -rf command which references the base_path followed by the base_prefx followed by the base itself, followed by the base_suffx. That command is then written to stdout.

Note that since the rm command is merely written to stdout, this script does not remove anything. The output is intended to be inspected for accuracy before acting upon it. If the commands appear correct, the output can be piped to sh and the rm commands will execute. Once you have tested this script to your satisfaction, the printf line could be revised to actually invoke the proper rm -rf command so that this script could be automated via cron.

Let's create some directories to test with:

mkdir -p path/to/backups/example.com/example.com-20210101-backup
mkdir -p path/to/backups/example.com/example.com-20210201-backup
mkdir -p path/to/backups/example.com/example.com-20210301-backup
mkdir -p path/to/backups/example.com/example.com-20210401-backup
mkdir -p path/to/backups/example.com/example.com-20210501-backup
mkdir -p path/to/backups/example.com/example.com-20210502-backup
mkdir -p path/to/backups/example.com/example.com-20210503-backup
mkdir -p path/to/backups/example.com/example.com-20210504-backup
mkdir -p path/to/backups/example.com/example.com-20210505-backup
mkdir -p path/to/backups/example.com/example.com-20210506-backup
mkdir -p path/to/backups/example.com/example.com-20210507-backup
mkdir -p path/to/backups/example.com/example.com-20210508-backup
mkdir -p path/to/backups/example.com/example.com-20210509-backup
mkdir -p path/to/backups/example.com/example.com-20210510-backup
mkdir -p path/to/backups/example.com/example.com-20210511-backup
mkdir -p path/to/backups/example.com/example.com-20210512-backup
mkdir -p path/to/backups/example.com/example.com-20210513-backup
mkdir -p path/to/backups/example.com/example.com-20210514-backup
mkdir -p path/to/backups/example.com/example.com-20210515-backup
mkdir -p path/to/backups/example.com/example.com-20210516-backup
mkdir -p path/to/backups/example.com/example.com-20210517-backup
mkdir -p path/to/backups/example.com/example.com-20210518-backup
mkdir -p path/to/backups/example.com/example.com-20210519-backup
mkdir -p path/to/backups/example.com/example.com-20210520-backup
mkdir -p path/to/backups/example.com/example.com-20210521-backup
mkdir -p path/to/backups/example.com/example.com-20210522-backup
mkdir -p path/to/backups/example.com/example.com-20210523-backup
mkdir -p path/to/backups/example.com/example.com-20210524-backup
mkdir -p path/to/backups/example.com/example.com-20210525-backup
mkdir -p path/to/backups/example.com/example.com-20210526-backup
mkdir -p path/to/backups/example.com/example.com-20210527-backup
mkdir -p path/to/backups/example.com/example.com-20210528-backup
mkdir -p path/to/backups/example.com/example.com-20210228-backup/example.com-20210101-backup
mkdir -p path/to/backups/example.com/example.com-messedup-backup/example.com-20210227-backup
mkdir -p path/to/backups/example.com/example.com-20210428-backup/example.com-20210601-backup

And then run the script:

$ ./test.sh 
rm -rf "./path/to/backups/example.com/example.com-20210514-backup"
rm -rf "./path/to/backups/example.com/example.com-20210513-backup"
rm -rf "./path/to/backups/example.com/example.com-20210512-backup"
rm -rf "./path/to/backups/example.com/example.com-20210511-backup"
rm -rf "./path/to/backups/example.com/example.com-20210510-backup"
rm -rf "./path/to/backups/example.com/example.com-20210509-backup"
rm -rf "./path/to/backups/example.com/example.com-20210508-backup"
rm -rf "./path/to/backups/example.com/example.com-20210507-backup"
rm -rf "./path/to/backups/example.com/example.com-20210506-backup"
rm -rf "./path/to/backups/example.com/example.com-20210505-backup"
rm -rf "./path/to/backups/example.com/example.com-20210504-backup"
rm -rf "./path/to/backups/example.com/example.com-20210503-backup"
rm -rf "./path/to/backups/example.com/example.com-20210502-backup"
rm -rf "./path/to/backups/example.com/example.com-20210428-backup"
rm -rf "./path/to/backups/example.com/example.com-20210228-backup"

Looks good, let's run it:

$ ./test.sh | sh

UPDATE

Mixing shell globs (like ????????) with regexes (like [0-9]{6}Z) in filenames can get unruly. The script can certainly be adjusted to uses regexes throughout, with a little added complexity.

#!/usr/bin/env bash
    
retain=15

# This is a shell glob (with no wildcards); must end in slash
base_path='./path/to/backups/example.com/'

# This is an extended regex pattern:
base_regex='\./path/to/backups/example\.com/example\.com-([0-9]{8}-[0-9]{6}Z)-backup'

# This is a printf spec to printf a base_path and a date-time to a full directory name:
printf_spec='%qexample.com-%q-backup'


find -E "$base_path" -maxdepth 1 -mindepth 1 \
    -type d \
    -regex "${base_regex}" |

sed -Ee "s~^${base_regex}$~\1~" |
grep -Ev '^[0-9]{6}01-' |

sort -r |
tail -n +$(($retain+1)) |
while IFS= read line
do
    printf "rm -rf ${printf_spec}\n" "${base_path}" "$line"
done

At the top, comments have been added to indicate clearly which variables are shell globs, which are regexes, and which is a printf specification. These are needed because:

  • base_path needs to be a shell glob to tell find where to look.
  • base_regex needs to be a full-line regex, because find ... -regex expects a regex that matches the entire line (of the directory name). Note that the regex character . is escaped wherever it appears.
  • printf_spec needs to be a printf-compatible specification that will format a string YYYYMMDD-HHMMSSZ into a valid directory name.

Now we can point find -E at $base_path and tell it to find directories exactly one level below that with names that will form an entire-line match (ala grep -Ex) with the extended regex $base_regex.

Notice that the portion of the regex that is designed to match YYYYMMDD-HHMMSSZ is parenthesized. This creates a "back-reference" for sed which becomes handy in the next step. We pass the entire output of find to sed and tell it to replace each line of input with just that part of the line matching the parenthesized part of the regexp, which is the YYYYMMDD-HHMMSSZ part that we need for chronological sorting. The earlier script used a bash-ism to parse out the timestamp, but that bash-ism relies on globs, so to achieve a regex-based solution, we use sed.

The rest of the script is largely the same: sed's output is passed to grep to remove any backup jobs from the first of any month. That output in turn goes to a reverse-order sort, tail then skips over the $retain largest values at the top of the list, outputting every line after that to a while loop that passes each line to printf.

Caveats:

More experienced U&L users will likely point out others, but some caveats are:

  • Be sure to escape any regex characters you use in base_regex that are expected to match literally to directory names
  • The sed command uses a ~ as the search-and-replace delimiter. Thus, we must avoid using tildes in directory names. So long as you don't put a tilde in the base_regex string, find should eliminate such directories for you, even if they do somehow get created in the filesystem.
  • Because this algorithm processes each date/time combination as a unique backup, "keeping the last 14 backups" might keep only yesterday's backups, if 14 backup jobs were run yesterday.
4
  • Thank you. To use your solution I simply needed to replace: tail +$(($retain+1)) | with tail -n +$(($retain+1)) |.
    – Scott
    Dec 13, 2021 at 23:00
  • ^ to be able to use it on a Red Hat Linux Distro.
    – Scott
    Dec 13, 2021 at 23:23
  • Jim, thank you!!! That is beautiful! I just had to adjust find so it works on a GNU distro (it seems like your solution is from a BSD distro). I have learned SO MUCH from looking at the two different versions of this process. You've helped me level up my scripting and I will study this model in my future designs. "You've lifted others!" Thanks again!!!!! P.S. - I'm going to post a "GNU Version" answer that works for me below, but I've marked yours as the solution.
    – Scott
    Dec 14, 2021 at 16:51
  • @Scott Thanks for the upvote. I changed the printf_spec and printf slightly to reduce the amount of redundant info in the constants at the top. That'll make it easier when you edit this to go into production use. Made the same change to your answer below.
    – Jim L.
    Dec 14, 2021 at 21:09
0

Thank you @Jim. That worked until I needed to add a Zulu time to the dir names making them:

./path/to/backups/example.com/example.com-20210101-040538Z-backup

NOTE: Zulu time will change day-to-day.

Which broke it.

So I am trying to follow your scripting with regex added to try to work around that time string like this. But the following is not working as I'm sure I have an error in my Regex formatting:

#!/usr/bin/env bash

retain=15

base_path='./path/to/backups/example.com/'
base_prefx='example.com-'
base_time="-040538Z"            # <--- Works but is not Regex.
# base_time="-[[:digit:]]{6}Z"  # <--- Regex (I think) but not working.
base_suffx="-backup"

find "$base_path" -maxdepth 1 -mindepth 1 \
    -type d \
    -name "${base_prefx}????????${base_time}${base_suffx}" |

while IFS= read dir
do
        base="$(basename "$dir" "$base_time$base_suffx")"
        printf '%s\n' "${base#$base_prefx}"
done |
grep -Ev '([^[:digit:]]|01$)' |
sort -r |
tail -n +$(($retain+1)) |
while IFS= read base
do
        printf 'success-safety-rm -rf "%q%q%q%q%q"\n' \
                "$base_path" "$base_prefx" "$base" "$base_time" "$base_suffx"
done


Maybe I am going about this wrong?

How would I better frame this script?

1
  • I have updated my answer to accommodate your change request. Mixing shell globs and regex patterns can be tricky, so I've changed the approach to use regex patterns. Also, instead of discarding the time field, I have kept it, since you have formatted it into a sortable field. However, see the caveats in the answer.
    – Jim L.
    Dec 14, 2021 at 1:29
0

GNU Distro Solution

Just for reference to future generations: the Accepted Answer's UPDATE section above by @Jim L is the SOLUTION and is what this is based from.

Why a Separate Answer?

Because find -E in the Accepted Answer UPDATE section above results in an error in a GNU environment: find: unknown predicate '-E'.

Here is the working script for GNU based Linux:

#!/usr/bin/env bash
    
retain=15

# CHANGEME This is a shell glob (with no wildcards); must end in slash; example:
base_path='./path/to/backups/example.com/'

# CHANGEME This is an extended regex pattern example:
base_regex='\./path/to/backups/example\.com/example\.com-([0-9]{8}-[0-9]{6}Z)-backup'

# CHANGEME This is a printf spec to printf a base_path and a date-time to a full directory name example:
printf_spec='%qexample.com-%q-backup'

find "$base_path" -maxdepth 1 -mindepth 1 \
    -type d \
    -regextype posix-extended \
    -regex "${base_regex}" |

sed -Ee "s~^${base_regex}$~\1~" |
grep -Ev '^[0-9]{6}01-' |

sort -r |
tail -n +$(($retain+1)) |
while IFS= read line
do
    printf "REMOVEME-SAFETY-rm -rf ${printf_spec}\n" "$base_path" "$line"
done

Running with:

./test.sh

Results in:

REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20211105-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20211104-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20211103-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20211102-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210303-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210302-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210203-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210202-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210110-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210109-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210108-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210107-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210106-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210105-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210104-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210103-040538Z-backup
REMOVEME-SAFETY-rm -rf /path/to/backups/example.com/example.com-20210102-040538Z-backup

Which are the backup dirs to be removed from my example.com test setup.

Now the printf "REMOVEME-SAFETY-rm... line can be adjusted to be run un-attended and actually remove the dirs.

NOTE: Jim's Caveats: from the Accepted Answer above do stand for this version as well.

Thank you @Jim L again.

0

The other solutions didn't work reliably in my scenario, as I wanted to remove a lot of different backups with different file naming patterns. So I wrote the following script that is based on file creation (or modification) dates rather than file names:

#!/usr/bin/env bash

dryRun=true # Set to `false` or remove this line to move from logging to deletion
rootFolder="/var/lib/psa/dumps/" # Files within this folder will be checked recursively
fileGlob="*.*" # Limit to specific file type if required
fileAgeLimit="30 days ago" # All files up to this age (date only and inclusive) will be kept
regularExpressionForDatesToBeKept='-[0-9][0-9]-01$' # This default regular expression will keep all files from the 1st of each month

checkForDeletion() {
    filePath=$1
    fileName=$(basename "$filePath")
    fileDateTimeString=$(stat -c '%w' "$filePath")   # Creation date; only available on newer file systems
    if [[ "$fileDateTimeString" = "-" ]]; then
        fileDateTimeString=$(stat -c '%y' "$filePath") # Use modification date instead
    fi
    fileDateString="$(date +"%Y-%m-%d" -d "$fileDateTimeString")"
    fileAgeLimitDateString="$(date +"%Y-%m-%d" -d "$fileAgeLimit")"
    if [[ "$fileDateString" < "$fileAgeLimitDateString" ]]; then
        if [[ "$fileDateString" =~ $regularExpressionForDatesToBeKept ]]; then
            [[ $dryRun = true ]] && echo -e "To be kept\tFile date: $fileDateString (matches '$regularExpressionForDatesToBeKept')\t$fileName"
        else
            if [[ $dryRun = true ]]; then
                echo -e "To be DELETED\tFile date: $fileDateString\t\t\t\t$fileName"
            else
                rm -f "$filePath"
            fi
        fi
    else
        [[ $dryRun = true ]] && echo -e "To be kept\tFile date: $fileDateString ($fileAgeLimit or younger)\t$fileName"
    fi
}

# Safely loop through all find matches
while read -r -d ''; do
    checkForDeletion "$REPLY"
done < <(find "$rootFolder" -type f -name "$fileGlob" -print0)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .