Assumptions:
- your backup directories, at least at the topmost levels, are sanely constructed without whitespace/newlines/regex or shell glob metachars, etc.
- you have a collection of directories within a path
base_path
- each directory begins with a prefix
base_prefx
- each directory ends with a suffix
base_suffx
- once stripped of the path, prefix and suffix, each directory name is a date
YYYYMMDD
- directories not meeting those criteria are to be ignored
With those givens, we can plan our strategy accordingly.
The crux of the task at hand is to remove zero or more directories, based on the YYYYMMDD portion of the directory name. To determine the specific directories (if any) to remove, we:
- exclude all directories where the
DD portion of the date is 01, or where any non-numeric character appears in the field where YYYYMMDD is expected
- of the remaining directories, exclude the
N most recent dates
- all the remaining directories (if any) are to be removed
You have chosen N=14.
#!/usr/bin/env bash
retain=14
base_path='./path/to/backups/example.com/'
base_prefx='example.com-'
base_suffx='-backup'
find "$base_path" -maxdepth 1 -mindepth 1 \
-type d \
-name "${base_prefx}????????${base_suffx}" |
while IFS= read dir
do
base="$(basename "$dir" "$base_suffx")"
printf '%s\n' "${base#$base_prefx}"
done |
grep -Ev '([^[:digit:]]|01$)' |
sort -r |
tail +$(($retain+1)) |
while IFS= read base
do
printf 'rm -rf "%q%q%q%q"\n' \
"$base_path" "$base_prefx" "$base" "$base_suffx"
done
The find command looks in the base_path directory for sub-directory names which match the template of our assumed directory structure, and which reside exactly one level below the base_path directory.
find's output is fed to the while loop, which reads each line of input, strips off the base_path, base_prefx and base_suffx and writes the base part of the directory name (ostensibly the date) to stdout.
That stdout is then passed to grep which removes any entries which contain any non-numeric characters or which end in 01. Removing entries ending in 01 is important so that first-of-the-month backups are retained indefinitely.
grep's output is then sorted in descending order so that the most recent entries (excluding any ??????01 entries) are at the top of the output, and the less recent entries are later.
Now that we have excluded all ??????01 backup directory dates, and sorted the dates in descending order with the most recent dates first, the only remaining task is to skip the first N entries, and then delete any entries N+1 and higher.
The code uses the variable retain to represent N. tail reads the sorted output and begins outputting lines beginning at line retain+1, and that stdout stream is passed to a while loop.
The loop reads each line as variable base and re-constructs an rm -rf command which references the base_path followed by the base_prefx followed by the base itself, followed by the base_suffx. That command is then written to stdout.
Note that since the rm command is merely written to stdout, this script does not remove anything. The output is intended to be inspected for accuracy before acting upon it. If the commands appear correct, the output can be piped to sh and the rm commands will execute. Once you have tested this script to your satisfaction, the printf line could be revised to actually invoke the proper rm -rf command so that this script could be automated via cron.
Let's create some directories to test with:
mkdir -p path/to/backups/example.com/example.com-20210101-backup
mkdir -p path/to/backups/example.com/example.com-20210201-backup
mkdir -p path/to/backups/example.com/example.com-20210301-backup
mkdir -p path/to/backups/example.com/example.com-20210401-backup
mkdir -p path/to/backups/example.com/example.com-20210501-backup
mkdir -p path/to/backups/example.com/example.com-20210502-backup
mkdir -p path/to/backups/example.com/example.com-20210503-backup
mkdir -p path/to/backups/example.com/example.com-20210504-backup
mkdir -p path/to/backups/example.com/example.com-20210505-backup
mkdir -p path/to/backups/example.com/example.com-20210506-backup
mkdir -p path/to/backups/example.com/example.com-20210507-backup
mkdir -p path/to/backups/example.com/example.com-20210508-backup
mkdir -p path/to/backups/example.com/example.com-20210509-backup
mkdir -p path/to/backups/example.com/example.com-20210510-backup
mkdir -p path/to/backups/example.com/example.com-20210511-backup
mkdir -p path/to/backups/example.com/example.com-20210512-backup
mkdir -p path/to/backups/example.com/example.com-20210513-backup
mkdir -p path/to/backups/example.com/example.com-20210514-backup
mkdir -p path/to/backups/example.com/example.com-20210515-backup
mkdir -p path/to/backups/example.com/example.com-20210516-backup
mkdir -p path/to/backups/example.com/example.com-20210517-backup
mkdir -p path/to/backups/example.com/example.com-20210518-backup
mkdir -p path/to/backups/example.com/example.com-20210519-backup
mkdir -p path/to/backups/example.com/example.com-20210520-backup
mkdir -p path/to/backups/example.com/example.com-20210521-backup
mkdir -p path/to/backups/example.com/example.com-20210522-backup
mkdir -p path/to/backups/example.com/example.com-20210523-backup
mkdir -p path/to/backups/example.com/example.com-20210524-backup
mkdir -p path/to/backups/example.com/example.com-20210525-backup
mkdir -p path/to/backups/example.com/example.com-20210526-backup
mkdir -p path/to/backups/example.com/example.com-20210527-backup
mkdir -p path/to/backups/example.com/example.com-20210528-backup
mkdir -p path/to/backups/example.com/example.com-20210228-backup/example.com-20210101-backup
mkdir -p path/to/backups/example.com/example.com-messedup-backup/example.com-20210227-backup
mkdir -p path/to/backups/example.com/example.com-20210428-backup/example.com-20210601-backup
And then run the script:
$ ./test.sh
rm -rf "./path/to/backups/example.com/example.com-20210514-backup"
rm -rf "./path/to/backups/example.com/example.com-20210513-backup"
rm -rf "./path/to/backups/example.com/example.com-20210512-backup"
rm -rf "./path/to/backups/example.com/example.com-20210511-backup"
rm -rf "./path/to/backups/example.com/example.com-20210510-backup"
rm -rf "./path/to/backups/example.com/example.com-20210509-backup"
rm -rf "./path/to/backups/example.com/example.com-20210508-backup"
rm -rf "./path/to/backups/example.com/example.com-20210507-backup"
rm -rf "./path/to/backups/example.com/example.com-20210506-backup"
rm -rf "./path/to/backups/example.com/example.com-20210505-backup"
rm -rf "./path/to/backups/example.com/example.com-20210504-backup"
rm -rf "./path/to/backups/example.com/example.com-20210503-backup"
rm -rf "./path/to/backups/example.com/example.com-20210502-backup"
rm -rf "./path/to/backups/example.com/example.com-20210428-backup"
rm -rf "./path/to/backups/example.com/example.com-20210228-backup"
Looks good, let's run it:
$ ./test.sh | sh
UPDATE
Mixing shell globs (like ????????) with regexes (like [0-9]{6}Z) in filenames can get unruly. The script can certainly be adjusted to uses regexes throughout, with a little added complexity.
#!/usr/bin/env bash
retain=15
# This is a shell glob (with no wildcards); must end in slash
base_path='./path/to/backups/example.com/'
# This is an extended regex pattern:
base_regex='\./path/to/backups/example\.com/example\.com-([0-9]{8}-[0-9]{6}Z)-backup'
# This is a printf spec to printf a base_path and a date-time to a full directory name:
printf_spec='%qexample.com-%q-backup'
find -E "$base_path" -maxdepth 1 -mindepth 1 \
-type d \
-regex "${base_regex}" |
sed -Ee "s~^${base_regex}$~\1~" |
grep -Ev '^[0-9]{6}01-' |
sort -r |
tail -n +$(($retain+1)) |
while IFS= read line
do
printf "rm -rf ${printf_spec}\n" "${base_path}" "$line"
done
At the top, comments have been added to indicate clearly which variables are shell globs, which are regexes, and which is a printf specification. These are needed because:
base_path needs to be a shell glob to tell find where to look.base_regex needs to be a full-line regex, because find ... -regex expects a regex that matches the entire line (of the directory name). Note that the regex character . is escaped wherever it appears.printf_spec needs to be a printf-compatible specification that will format a string YYYYMMDD-HHMMSSZ into a valid directory name.
Now we can point find -E at $base_path and tell it to find directories exactly one level below that with names that will form an entire-line match (ala grep -Ex) with the extended regex $base_regex.
Notice that the portion of the regex that is designed to match YYYYMMDD-HHMMSSZ is parenthesized. This creates a "back-reference" for sed which becomes handy in the next step. We pass the entire output of find to sed and tell it to replace each line of input with just that part of the line matching the parenthesized part of the regexp, which is the YYYYMMDD-HHMMSSZ part that we need for chronological sorting. The earlier script used a bash-ism to parse out the timestamp, but that bash-ism relies on globs, so to achieve a regex-based solution, we use sed.
The rest of the script is largely the same: sed's output is passed to grep to remove any backup jobs from the first of any month. That output in turn goes to a reverse-order sort, tail then skips over the $retain largest values at the top of the list, outputting every line after that to a while loop that passes each line to printf.
Caveats:
More experienced U&L users will likely point out others, but some caveats are:
- Be sure to escape any regex characters you use in
base_regex that are expected to match literally to directory names
- The
sed command uses a ~ as the search-and-replace delimiter. Thus, we must avoid using tildes in directory names. So long as you don't put a tilde in the base_regex string, find should eliminate such directories for you, even if they do somehow get created in the filesystem.
- Because this algorithm processes each date/time combination as a unique backup, "keeping the last 14 backups" might keep only yesterday's backups, if 14 backup jobs were run yesterday.
rdiff, and has various options for expiry of old backups. I don't think it does the "keep the first of the month" backup you want, but that's easily solved by running rdiff-backup twice (once for daily, once for monthly) with different expiry options. I used to use it a lot until I switched to ZFS snapshots for my backups.zshglobbing. Something likeset -- (fancy glob here, including sorting modifiers)followed byshift 14and thenrm -rf "$@"or your favorite deletion command.