Assumptions:
- your backup directories, at least at the topmost levels, are sanely constructed without whitespace/newlines/regex or shell glob metachars, etc.
- you have a collection of directories within a path
base_path
- each directory begins with a prefix
base_prefx
- each directory ends with a suffix
base_suffx
- once stripped of the path, prefix and suffix, each directory name is a date
YYYYMMDD
- directories not meeting those criteria are to be ignored
With those givens, we can plan our strategy accordingly.
The crux of the task at hand is to remove zero or more directories, based on the YYYYMMDD
portion of the directory name. To determine the specific directories (if any) to remove, we:
- exclude all directories where the
DD
portion of the date is 01
, or where any non-numeric character appears in the field where YYYYMMDD
is expected
- of the remaining directories, exclude the
N
most recent dates
- all the remaining directories (if any) are to be removed
You have chosen N=14
.
#!/usr/bin/env bash
retain=14
base_path='./path/to/backups/example.com/'
base_prefx='example.com-'
base_suffx='-backup'
find "$base_path" -maxdepth 1 -mindepth 1 \
-type d \
-name "${base_prefx}????????${base_suffx}" |
while IFS= read dir
do
base="$(basename "$dir" "$base_suffx")"
printf '%s\n' "${base#$base_prefx}"
done |
grep -Ev '([^[:digit:]]|01$)' |
sort -r |
tail +$(($retain+1)) |
while IFS= read base
do
printf 'rm -rf "%q%q%q%q"\n' \
"$base_path" "$base_prefx" "$base" "$base_suffx"
done
The find
command looks in the base_path
directory for sub-directory names which match the template of our assumed directory structure, and which reside exactly one level below the base_path
directory.
find
's output is fed to the while loop, which reads each line of input, strips off the base_path
, base_prefx
and base_suffx
and writes the base
part of the directory name (ostensibly the date) to stdout
.
That stdout
is then passed to grep
which removes any entries which contain any non-numeric characters or which end in 01
. Removing entries ending in 01
is important so that first-of-the-month backups are retained indefinitely.
grep
's output is then sort
ed in descending order so that the most recent entries (excluding any ??????01
entries) are at the top of the output, and the less recent entries are later.
Now that we have excluded all ??????01
backup directory dates, and sorted the dates in descending order with the most recent dates first, the only remaining task is to skip the first N
entries, and then delete any entries N+1
and higher.
The code uses the variable retain
to represent N
. tail
reads the sort
ed output and begins outputting lines beginning at line retain+1
, and that stdout
stream is passed to a while
loop.
The loop reads each line as variable base
and re-constructs an rm -rf
command which references the base_path
followed by the base_prefx
followed by the base
itself, followed by the base_suffx
. That command is then written to stdout
.
Note that since the rm
command is merely written to stdout
, this script does not remove anything. The output is intended to be inspected for accuracy before acting upon it. If the commands appear correct, the output can be piped to sh
and the rm
commands will execute. Once you have tested this script to your satisfaction, the printf
line could be revised to actually invoke the proper rm -rf
command so that this script could be automated via cron
.
Let's create some directories to test with:
mkdir -p path/to/backups/example.com/example.com-20210101-backup
mkdir -p path/to/backups/example.com/example.com-20210201-backup
mkdir -p path/to/backups/example.com/example.com-20210301-backup
mkdir -p path/to/backups/example.com/example.com-20210401-backup
mkdir -p path/to/backups/example.com/example.com-20210501-backup
mkdir -p path/to/backups/example.com/example.com-20210502-backup
mkdir -p path/to/backups/example.com/example.com-20210503-backup
mkdir -p path/to/backups/example.com/example.com-20210504-backup
mkdir -p path/to/backups/example.com/example.com-20210505-backup
mkdir -p path/to/backups/example.com/example.com-20210506-backup
mkdir -p path/to/backups/example.com/example.com-20210507-backup
mkdir -p path/to/backups/example.com/example.com-20210508-backup
mkdir -p path/to/backups/example.com/example.com-20210509-backup
mkdir -p path/to/backups/example.com/example.com-20210510-backup
mkdir -p path/to/backups/example.com/example.com-20210511-backup
mkdir -p path/to/backups/example.com/example.com-20210512-backup
mkdir -p path/to/backups/example.com/example.com-20210513-backup
mkdir -p path/to/backups/example.com/example.com-20210514-backup
mkdir -p path/to/backups/example.com/example.com-20210515-backup
mkdir -p path/to/backups/example.com/example.com-20210516-backup
mkdir -p path/to/backups/example.com/example.com-20210517-backup
mkdir -p path/to/backups/example.com/example.com-20210518-backup
mkdir -p path/to/backups/example.com/example.com-20210519-backup
mkdir -p path/to/backups/example.com/example.com-20210520-backup
mkdir -p path/to/backups/example.com/example.com-20210521-backup
mkdir -p path/to/backups/example.com/example.com-20210522-backup
mkdir -p path/to/backups/example.com/example.com-20210523-backup
mkdir -p path/to/backups/example.com/example.com-20210524-backup
mkdir -p path/to/backups/example.com/example.com-20210525-backup
mkdir -p path/to/backups/example.com/example.com-20210526-backup
mkdir -p path/to/backups/example.com/example.com-20210527-backup
mkdir -p path/to/backups/example.com/example.com-20210528-backup
mkdir -p path/to/backups/example.com/example.com-20210228-backup/example.com-20210101-backup
mkdir -p path/to/backups/example.com/example.com-messedup-backup/example.com-20210227-backup
mkdir -p path/to/backups/example.com/example.com-20210428-backup/example.com-20210601-backup
And then run the script:
$ ./test.sh
rm -rf "./path/to/backups/example.com/example.com-20210514-backup"
rm -rf "./path/to/backups/example.com/example.com-20210513-backup"
rm -rf "./path/to/backups/example.com/example.com-20210512-backup"
rm -rf "./path/to/backups/example.com/example.com-20210511-backup"
rm -rf "./path/to/backups/example.com/example.com-20210510-backup"
rm -rf "./path/to/backups/example.com/example.com-20210509-backup"
rm -rf "./path/to/backups/example.com/example.com-20210508-backup"
rm -rf "./path/to/backups/example.com/example.com-20210507-backup"
rm -rf "./path/to/backups/example.com/example.com-20210506-backup"
rm -rf "./path/to/backups/example.com/example.com-20210505-backup"
rm -rf "./path/to/backups/example.com/example.com-20210504-backup"
rm -rf "./path/to/backups/example.com/example.com-20210503-backup"
rm -rf "./path/to/backups/example.com/example.com-20210502-backup"
rm -rf "./path/to/backups/example.com/example.com-20210428-backup"
rm -rf "./path/to/backups/example.com/example.com-20210228-backup"
Looks good, let's run it:
$ ./test.sh | sh
UPDATE
Mixing shell globs (like ????????
) with regexes (like [0-9]{6}Z
) in filenames can get unruly. The script can certainly be adjusted to uses regexes throughout, with a little added complexity.
#!/usr/bin/env bash
retain=15
# This is a shell glob (with no wildcards); must end in slash
base_path='./path/to/backups/example.com/'
# This is an extended regex pattern:
base_regex='\./path/to/backups/example\.com/example\.com-([0-9]{8}-[0-9]{6}Z)-backup'
# This is a printf spec to printf a base_path and a date-time to a full directory name:
printf_spec='%qexample.com-%q-backup'
find -E "$base_path" -maxdepth 1 -mindepth 1 \
-type d \
-regex "${base_regex}" |
sed -Ee "s~^${base_regex}$~\1~" |
grep -Ev '^[0-9]{6}01-' |
sort -r |
tail -n +$(($retain+1)) |
while IFS= read line
do
printf "rm -rf ${printf_spec}\n" "${base_path}" "$line"
done
At the top, comments have been added to indicate clearly which variables are shell globs, which are regexes, and which is a printf
specification. These are needed because:
base_path
needs to be a shell glob to tell find
where to look.
base_regex
needs to be a full-line regex, because find ... -regex
expects a regex that matches the entire line (of the directory name). Note that the regex character .
is escaped wherever it appears.
printf_spec
needs to be a printf
-compatible specification that will format a string YYYYMMDD-HHMMSSZ
into a valid directory name.
Now we can point find -E
at $base_path
and tell it to find directories exactly one level below that with names that will form an entire-line match (ala grep -Ex
) with the extended regex $base_regex
.
Notice that the portion of the regex that is designed to match YYYYMMDD-HHMMSSZ
is parenthesized. This creates a "back-reference" for sed
which becomes handy in the next step. We pass the entire output of find
to sed
and tell it to replace each line of input with just that part of the line matching the parenthesized part of the regexp, which is the YYYYMMDD-HHMMSSZ
part that we need for chronological sorting. The earlier script used a bash-ism to parse out the timestamp, but that bash-ism relies on globs, so to achieve a regex-based solution, we use sed
.
The rest of the script is largely the same: sed
's output is passed to grep
to remove any backup jobs from the first of any month. That output in turn goes to a reverse-order sort
, tail
then skips over the $retain
largest values at the top of the list, outputting every line after that to a while loop that passes each line to printf
.
Caveats:
More experienced U&L users will likely point out others, but some caveats are:
- Be sure to escape any regex characters you use in
base_regex
that are expected to match literally to directory names
- The
sed
command uses a ~
as the search-and-replace delimiter. Thus, we must avoid using tildes in directory names. So long as you don't put a tilde in the base_regex
string, find
should eliminate such directories for you, even if they do somehow get created in the filesystem.
- Because this algorithm processes each date/time combination as a unique backup, "keeping the last 14 backups" might keep only yesterday's backups, if 14 backup jobs were run yesterday.
rdiff
, and has various options for expiry of old backups. I don't think it does the "keep the first of the month" backup you want, but that's easily solved by running rdiff-backup twice (once for daily, once for monthly) with different expiry options. I used to use it a lot until I switched to ZFS snapshots for my backups.zsh
globbing. Something likeset -- (fancy glob here, including sorting modifiers)
followed byshift 14
and thenrm -rf "$@"
or your favorite deletion command.