2

every our backup files of a database are created. The files are named like this:

prod20210528_1200.sql.gz 
pattern: prod`date +\%Y%m%d_%H%M`

The pattern could be adjusted if needed.

I would like to have a script that:

  • keeps all backups for the last x (e.g. 3) days
  • for backups older than x (e.g. 3) days only the backup from time 00:00 shall be kept
  • for backups older than y (e.g. 14) days only one file per week (monday) shall be kept
  • for backups older than z days (e.g. 90) only one file per month (1st of each month) shall be kept
  • the script should rather use the filename instead of the date (created) information of the file, if that it possible
  • the script should run every day

Unfortunately, I have very little knowledge of the shell-/bash-script language. I would do something like this:

if (file < today - x AND date > today - (x + 1))
{
  if (%H_of_file != 00 AND %M_of_file != 00)
  {  
    delete file
  }
}

if (file < today - y AND date > today - (y + 1))
{
  if (file != Monday)
  {  
    delete file
  }
}

if (file < today - z AND date > today - (z + 1))
{
  if (%m_of_file != 01)
  {  
    delete file
  }
}

Does this makes any sense for you?

Thank you very much! 

All the best,
Phantom


1
  • find is very good for checking a files modification date [= creation date] so just for example, something like find pathname/ -type f -mtime +99 -exec ls {} \; could list all files which are at least 99 days old. May 28, 2021 at 20:18

2 Answers 2

0

As a committer of pyExpireBackups i can point you to the ExpirationRule implementation of my solution (source below and in the github repo)

see https://wiki.bitplan.com/index.php/PyExpireBackups for the doku.

The installation script:

pip install PyExpireBackups

will install the shell commmand "expireBackups". The usage is:

expireBackups -h
usage: expireBackups [-h] [-d] [--days DAYS] [--weeks WEEKS] [--months MONTHS] [--years YEARS]
                     [--minFileSize MINFILESIZE] [--rootPath ROOTPATH] [--baseName BASENAME] [--ext EXT]
                     [--createTestFiles CREATETESTFILES] [-f] [-V]

Backup expiration based on rules (yearly,monthly,weekly,daily ...)

  Created by Wolfgang Fahl on 2022-04-01.
  Copyright 2008-2022 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           show debug info
  --days DAYS           number of consecutive days to keep a daily backup (default: 7)
  --weeks WEEKS         number of consecutive weeks to keep a weekly backup (default: 6)
  --months MONTHS       number of consecutive month to keep a monthly backup (default: 8)
  --years YEARS         number of consecutive years to keep a yearly backup (default: 4)
  --minFileSize MINFILESIZE
                        minimum File size in bytes to filter for (default: 1)
  --rootPath ROOTPATH
  --baseName BASENAME   the basename to filter for (default: None)
  --ext EXT             the extension to filter for (default: None)
  --createTestFiles CREATETESTFILES
                        create the given number of temporary test files (default: None)
  -f, --force
  -V, --version         show program's version number and exit

An example run would lead to:

expireBackups --ext .tgz
keeping 7 files for dayly backup
keeping 6 files for weekly backup
keeping 8 files for monthly backup
keeping 4 files for yearly backup
expiring 269 files dry run
#   1✅:   0.0 days(    5 GB/    5 GB)→./sql_backup.2022-04-02.tgz
#   2✅:   3.0 days(    5 GB/    9 GB)→./sql_backup.2022-03-30.tgz
#   3✅:   4.0 days(    5 GB/   14 GB)→./sql_backup.2022-03-29.tgz
#   4✅:   5.0 days(    5 GB/   18 GB)→./sql_backup.2022-03-28.tgz
#   5✅:   7.0 days(    5 GB/   23 GB)→./sql_backup.2022-03-26.tgz
#   6✅:   9.0 days(    5 GB/   27 GB)→./sql_backup.2022-03-24.tgz
#   7✅:  11.0 days(    5 GB/   32 GB)→./sql_backup.2022-03-22.tgz
#   8❌:  15.0 days(    5 GB/   37 GB)→./sql_backup.2022-03-18.tgz
#   9❌:  17.0 days(    5 GB/   41 GB)→./sql_backup.2022-03-16.tgz
#  10✅:  18.0 days(    5 GB/   46 GB)→./sql_backup.2022-03-15.tgz
#  11❌:  19.0 days(    5 GB/   50 GB)→./sql_backup.2022-03-14.tgz
#  12❌:  20.0 days(    5 GB/   55 GB)→./sql_backup.2022-03-13.tgz
#  13❌:  22.0 days(    5 GB/   59 GB)→./sql_backup.2022-03-11.tgz
#  14❌:  23.0 days(    5 GB/   64 GB)→./sql_backup.2022-03-10.tgz
#  15✅:  35.0 days(    4 GB/   68 GB)→./sql_backup.2022-02-26.tgz
#  16❌:  37.0 days(    4 GB/   73 GB)→./sql_backup.2022-02-24.tgz
#  17❌:  39.0 days(    4 GB/   77 GB)→./sql_backup.2022-02-22.tgz
#  18❌:  40.0 days(    5 GB/   82 GB)→./sql_backup.2022-02-21.tgz
#  19✅:  43.0 days(    4 GB/   86 GB)→./sql_backup.2022-02-18.tgz
...
...
# 264✅:1426.0 days(    4 GB/    1 TB)→./sql_backup.2018-05-07.tgz
# 265❌:1433.0 days(    4 GB/    1 TB)→./sql_backup.2018-04-30.tgz
# 266❌:1447.0 days(    4 GB/    1 TB)→./sql_backup.2018-04-16.tgz
# 267❌:1482.0 days(    4 GB/    1 TB)→./sql_backup.2018-03-12.tgz
# 268❌:1489.0 days(    4 GB/    1 TB)→./sql_backup.2018-03-05.tgz
# 269❌:1509.0 days(    4 GB/    1 TB)→./sql_backup.2018-02-12.tgz
kept 24 files   105 GB
0

This works for me (BSD date, not GNU date)

#!/usr/local/bin/bash
DIR_BACKUPS='/backups'
KEEP_DAILY=3
KEEP_WEEKLY=14
KEEP_MONTHLY=90

date_keep_daily=$(date -j -v-${KEEP_DAILY}d +"%Y%m%d")
date_keep_weekly=$(date -j -v-${KEEP_WEEKLY}d +"%Y%m%d")
date_keep_monthly=$(date -j -v-${KEEP_MONTHLY}d +"%Y%m%d")

for file in $DIR_BACKUPS/prod*.sql.gz; do
    timestamp=${file#*onwalt}
    timestamp=${timestamp%%.*}
    date_of_file=${timestamp%%_*}
    hour_of_file=${timestamp:(-4):2}
    
    if [[ $date_of_file < $date_keep_monthly ]]; then
        if [[ $(date -jf "%Y%m%d_%H%M" $timestamp +"%d") != 01 ]]; then
            rm $file
        fi
    elif [[ $date_of_file < $date_keep_weekly ]]; then
        if [[ $(date -jf "%Y%m%d_%H%M" $timestamp +"%u") != 1 ]]; then
            rm $file
        fi
    elif [[ $date_of_file < $date_keep_daily ]]; then
        if [[ $(date -jf "%Y%m%d_%H%M" $timestamp +"%H") != 00 ]]; then
            rm $file
        fi
    fi
done
3
  • date -j doesn't work for me. I'd suggest simplifying the whole thing by using UTC dates in a unix epoch form [seconds since 1970] since this makes date comparisons trivial. Jun 2, 2021 at 21:45
  • The script works for BSD date, not GNU date. GNU date should be something like $(date +%s --date="-30 days")
    – Phantom
    Jun 3, 2021 at 12:03
  • I don't see how this can work. If you drop in to the last elif, for example, that will likely delete anything you might have wanted to preserve according to keep_weekly or keep_monthly dates.
    – Wallace
    Aug 21, 2023 at 14:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .