14

I've created a script to run automated backups on my CentOS 7 server. The backups get stored to the /home/backup directory. The script works, but now I would like to incorporate a way to count the files after the backup happens and if the number is more than 5, delete the oldest backup.

Below is what I have for my backup script.

#!/bin/bash

#mysqldump variables

FILE=/home/backup/databasebk_!`date +"Y-%m-%d_%H:%M"`.sql
DATABASE=database
USER=root
PASS=my password

#backup command process

mysqldump --opt --user=${USER} --password=${PASS} ${DATABASE} > ${FILE}

#zipping the backup file

gzip $FILE

#send message to the user with the results

echo "${FILE}.gz was created:"
ls -l ${FILE}.gz

# This is where I would like to count the number of files 
# in the directory and if there are more than 5 I would like
# to delete the oldest file. Any help is greatly appreciated

Thanks -Mike

10

You could look at set -- /home/backup/databasebk_* and while $# is greater than five, delete a file.

So the code would look similar to

set -- /home/backup/databasebk_*
while [ $# -gt 5 ]
do
  echo "Removing old backup $1"
  rm "$1"
  shift
done

This works because the filenames you picked are automatically in "oldest first" order.

For consistency I would set a variable (I normally call it BASE but you can call it whatever you like)

So

BASE=/home/backup/databasebk_
FILE=${BASE}!`date +"%Y-%m-%d_%H:%M"`.sql
....
set -- ${BASE}*
while [ $# -gt 5 ]
do
  echo "Removing old backup $1"
  rm "$1"
  shift
done
| improve this answer | |
  • 2
    should have quotes on "$1" everywhere. – Jasen Feb 4 at 1:54
  • This worked perfectly once I added the quotes on the "$1" like @Jasen added. Thank you so much for the help. Now I will try and understand what you had me do so I can learn it. Thanks again. – Mike Feb 4 at 2:25
  • 1
    Quotes added to the examples. set -- ${BASE}* sets the positional parameters $1 $2 $3... etc to the filenames. $# is the number of parameters set. Each time round the loop, if there are more than 5 parameters, delete the first one, then "shift" the list. So if you start with filenames a b c d e f g then $1==a, $2==b, $3==c and $#=7. After the first loop we'll have $1==b $2==c $3==d and $#=6. And we stop when there are only 5 files left. – Stephen Harris Feb 4 at 3:22
19

In pure Bash without loops:

ls -t | tail -n +6 | xargs -d '\n' rm

Explanation:

  • ls -t prints all files in the current dir, sorted by modification time, newest first.
  • tail -n +6 ignores the first 5 lines and prints lines 6 onwards.
  • xargs -d '\n' rm removes the files passed to it, one per line. If you are absolutely sure there are no spaces or quotes in the filenames, you can use just xargs rm.

Note this deletes as many files as needed to leave only 5 files in the dir. If you want to delete only the one oldest file, replace +6 with 1.

| improve this answer | |
  • 2
    This is the easy way for sorting by the actual timestamp. It works fine for "nice" filenames (namely ones without newlines in them), but there is the caveat that some unnice filenames will break it. Plain xargs without options would also interpret quotes. – ilkkachu Feb 4 at 10:26
  • 3
    Since the filenames are created by the same script, I wouldn't worry too much about newlines in them. Good tip on the quotes, I'm editing the answer. – BoppreH Feb 4 at 11:13
11

I think a better approach would be to use logrotate, as this application already performs the actions you are trying to achieve with your script. I tested the following scripts on my test server.

  1. Manually backup the database, this is needed because logrotate will not run if the file doesn't exist.
    $ mysqldump -uroot -pmy5trongpass database > mydatabase.sql
    $ ls
    $ mydatabase.sql

Example of error if the first database was not created.

    #**logrotate -f** will force our new rule to run now and backup the database
    #this is for testing purposes.

    $ logrotate -f /etc/logrotate.d/mysql_backup
    error: stat of /home/backups/mydatabase.sql failed: No such file or directory
  1. Create a logrotate file/rule. Here is the one that I created based on other rules locates in /etc/logrotate.d/
    $ cat /etc/logrotate.d/mysql_backup
    /home/backups/mydatabase.sql{
            create 640 root mysql
           daily
            rotate 2
            nocompress
        postrotate
            mysqldump -uroot -pmy5trongpass test > mydatabase.sql;
            echo "done"
        endscript
    }
  1. Test you configuration using
$ logrotate -f /etc/logrotate.d/mysql_backup
$ logrotate -f /etc/logrotate.d/mysql_backup
done
$ ll
total 16
-rw-r-----. 1 root mysql 1261 Feb  3 21:46 mydatabase.sql
-rw-r-----. 1 root mysql 1261 Feb  3 21:44 mydatabase.sql.1
-rw-r-----. 1 root mysql 1261 Feb  3 21:44 mydatabase.sql.2

you will see that a new file will be created followed by a number, this can be changed to show dates by using the following options

https://linux.die.net/man/8/logrotate

dateext
Archive old versions of log files adding a daily extension like YYYYMMDD instead of simply adding a number. The extension may be configured using the dateformat option.

dateformat format_string
Specify the extension for dateext using the notation similar to strftime(3) function. Only %Y %m %d and %s specifiers are allowed. The default value is -%Y%m%d. Note that also the character separating log name from the extension is part of the dateformat string. The system clock must be set past Sep 9th 2001 for %s to work correctly. Note that the datestamps generated by this format must be lexically sortable (i.e., first the year, then the month then the day. e.g., 2001/12/01 is ok, but 01/12/2001 is not, since 01/11/2002 would sort lower while it is later). This is because when using the rotate option, logrotate sorts all rotated filenames to find out which logfiles are older and should be removed.
| improve this answer | |
  • +1 for thinking outside of the box. The name logrotate doesn't have to limit itself to rotating actual logs. – Monty Harder Feb 6 at 22:49
6

Using the zsh shell:

rm -f /home/backup/databasebk_*.sql.gz(.om[6,-1])

This would create a list of the pathnames of regular files matching /home/backup/databasebk_*.sql.gz, sort the list on the modification timestamps of the files (most recently modified first), and extract the elements 6 through to the end. These files would then be passed to rm -f for deletion.

It's the glob qualifier, (.om[6,-1]), at the end of the filename globbing pattern that specifies regular files (.), ordering by modification time (om), and that the 6th through to the last match should be returned ([6,-1]).

| improve this answer | |
2
files=( /home/backup/* )
# files is now an array containing file names in the directory

oldest=""
if (( ${#files[@]} > 5 ))  ## are there more than five?
then 
  for name in "${files[@]}" ## loop to find oldest
  do
    a=$( stat --printf=%Y "$name" )
    if  [ -z "$oldest" ] || (( a < age ))
    then 
       age="$a"
       oldest="$name"
    fi
  done
  rm "$oldest"
fi

Some comments:

I say /home/backup/* which finds all the files in that directory I could say /home/backup/databasebk_*.sql.gz instead to just find compressed dumps

I'm using 'stat' to find the oldest file according to the filesystem as your dates are in YMDhm format order I could compare filenames instead

 if  [ -z "$oldest" ] || [ "$name" -lt "$oldest" ]

the presence of subdirectories in /home/backup/ will break this script

| improve this answer | |
  • Please forgive me as I am new to this. I am getting a syntax error... operand expected (error token is "#"). I am sure that this is because I took the # and the @ as literally those symbols should be in the code....was I incorrect making that assumption, were they plpace holders for data I was supposed to insert?? – Mike Feb 4 at 2:14
  • 3
    It's because ${files[#]} should be ${#files[@]} I think.. – steeldriver Feb 4 at 2:33
1

Before I start, I would like to say that I hold the other answers in the highest esteem and they reflect a good understanding of the process.

I will now put myself in your shoes and look for something simple to get started, then build upon it. Let me first restate your problem:

  1. Determine if the number of files is more than 5.
  2. If under or equal to 5, do nothing.
  3. If over 5, find and delete the file which is older than 5 (only 1 file).

Here is my approach:

  1. Use ls -1 |wc -l to count the number of files.
  2. Check if the number of listings is greater than 5, using an if statement.
  3. If it is greater than 5, use the find command with suitable options to delete the ones which are older than 5 days.

Here is the code:

if [[ $(ls -1|wc -l) -gt 5 ]]; then 

  ls -1t |tail -n1 |xargs -n1 rm -f  

fi

Explanation of the first line: It checks if the number of files in that directory exceeds 5.

Explanation of the second line: It will run only of the number of files exceeds 5 (if there are 4 files and out of them one has a date of 100 days ago, it will not run). It will date-sort them and run the tail command to get the oldest one (the last one in the list). It will then pass on that file name to the xargs command which will use rm -f (force) to delete the file.

Thanks to all others who chipped in with a solution ! I know that there are cases where this won't work (like more than 6 files, 2 files of the same date, etc), but I have tried to keep it simple as Mike is new :)

| improve this answer | |
1

Create rm-except-last.sh:

#!/bin/bash -

set -o nounset

if ! (($#)); then
  echo Usage: "$BASH_SOURCE" N_NOT_TO_DELETE FILES_TO_DELETE...
  exit 1
fi

declare N="${1:?Missing N_NOT_TO_DELETE.}"
if ! [[ "$N" =~ ^[0-9]+$ ]]; then
  echo 1>&2 -E Not a number: "$N"
  exit 1
fi
shift

declare -a roll=()
while ((${#roll[@]} < N)); do
  roll+=('')
done

if ! ((N)); then
  rm -rfv -- "$@"
  exit $?
fi

declare n=0 file=''
for file in "$@"; do
  declare rmfile="${roll[n]}"
  [ -n "$rmfile" ] && rm -rfv -- "$rmfile"
  roll[n]="$file"
  ((n = (n + 1) % N))
done

Then:

$ rm-except-last.sh 5 /home/backup/databasebk_*.sql

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.