How can I count the number of files in a directory and delete the oldest if the number exceeds 5

Question

I've created a script to run automated backups on my CentOS 7 server. The backups get stored to the /home/backup directory. The script works, but now I would like to incorporate a way to count the files after the backup happens and if the number is more than 5, delete the oldest backup.

Below is what I have for my backup script.

#!/bin/bash

#mysqldump variables

FILE=/home/backup/databasebk_!`date +"Y-%m-%d_%H:%M"`.sql
DATABASE=database
USER=root
PASS=my password

#backup command process

mysqldump --opt --user=${USER} --password=${PASS} ${DATABASE} > ${FILE}

#zipping the backup file

gzip $FILE

#send message to the user with the results

echo "${FILE}.gz was created:"
ls -l ${FILE}.gz

# This is where I would like to count the number of files 
# in the directory and if there are more than 5 I would like
# to delete the oldest file. Any help is greatly appreciated

Thanks -Mike

Answer 1

In pure Bash without loops:

ls -t | tail -n +6 | xargs -d '\n' rm

Explanation:

• ls -t prints all files in the current dir, sorted by modification time, newest first.
• tail -n +6 ignores the first 5 lines and prints lines 6 onwards.
• xargs -d '\n' rm removes the files passed to it, one per line. If you are absolutely sure there are no spaces or quotes in the filenames, you can use just xargs rm.

Note this deletes as many files as needed to leave only 5 files in the dir. If you want to delete only the one oldest file, replace +6 with 1.

Answer 2

You could look at set -- /home/backup/databasebk_* and while $# is greater than five, delete a file.

So the code would look similar to

set -- /home/backup/databasebk_*
while [ $# -gt 5 ]
do
  echo "Removing old backup $1"
  rm "$1"
  shift
done

This works because the filenames you picked are automatically in "oldest first" order.

For consistency I would set a variable (I normally call it BASE but you can call it whatever you like)

So

BASE=/home/backup/databasebk_
FILE=${BASE}!`date +"%Y-%m-%d_%H:%M"`.sql
....
set -- ${BASE}*
while [ $# -gt 5 ]
do
  echo "Removing old backup $1"
  rm "$1"
  shift
done

Answer 3

I think a better approach would be to use logrotate, as this application already performs the actions you are trying to achieve with your script. I tested the following scripts on my test server.

1. Manually backup the database, this is needed because logrotate will not run if the file doesn't exist.

    $ mysqldump -uroot -pmy5trongpass database > mydatabase.sql
    $ ls
    $ mydatabase.sql

Example of error if the first database was not created.

    #**logrotate -f** will force our new rule to run now and backup the database
    #this is for testing purposes.

    $ logrotate -f /etc/logrotate.d/mysql_backup
    error: stat of /home/backups/mydatabase.sql failed: No such file or directory

2. Create a logrotate file/rule. Here is the one that I created based on other rules locates in /etc/logrotate.d/

    $ cat /etc/logrotate.d/mysql_backup
    /home/backups/mydatabase.sql{
            create 640 root mysql
           daily
            rotate 2
            nocompress
        postrotate
            mysqldump -uroot -pmy5trongpass test > mydatabase.sql;
            echo "done"
        endscript
    }

3. Test you configuration using

$ logrotate -f /etc/logrotate.d/mysql_backup

$ logrotate -f /etc/logrotate.d/mysql_backup
done
$ ll
total 16
-rw-r-----. 1 root mysql 1261 Feb  3 21:46 mydatabase.sql
-rw-r-----. 1 root mysql 1261 Feb  3 21:44 mydatabase.sql.1
-rw-r-----. 1 root mysql 1261 Feb  3 21:44 mydatabase.sql.2

you will see that a new file will be created followed by a number, this can be changed to show dates by using the following options

https://linux.die.net/man/8/logrotate

dateext
Archive old versions of log files adding a daily extension like YYYYMMDD instead of simply adding a number. The extension may be configured using the dateformat option.

dateformat format_string
Specify the extension for dateext using the notation similar to strftime(3) function. Only %Y %m %d and %s specifiers are allowed. The default value is -%Y%m%d. Note that also the character separating log name from the extension is part of the dateformat string. The system clock must be set past Sep 9th 2001 for %s to work correctly. Note that the datestamps generated by this format must be lexically sortable (i.e., first the year, then the month then the day. e.g., 2001/12/01 is ok, but 01/12/2001 is not, since 01/11/2002 would sort lower while it is later). This is because when using the rotate option, logrotate sorts all rotated filenames to find out which logfiles are older and should be removed.

Answer 4

Using the zsh shell:

rm -f /home/backup/databasebk_*.sql.gz(.om[6,-1])

This would create a list of the pathnames of regular files matching /home/backup/databasebk_*.sql.gz, sort the list on the modification timestamps of the files (most recently modified first), and extract the elements 6 through to the end. These files would then be passed to rm -f for deletion.

It's the glob qualifier, (.om[6,-1]), at the end of the filename globbing pattern that specifies regular files (.), ordering by modification time (om), and that the 6th through to the last match should be returned ([6,-1]).

Answer 5

files=( /home/backup/* )
# files is now an array containing file names in the directory

oldest=""
if (( ${#files[@]} > 5 ))  ## are there more than five?
then 
  for name in "${files[@]}" ## loop to find oldest
  do
    a=$( stat --printf=%Y "$name" )
    if  [ -z "$oldest" ] || (( a < age ))
    then 
       age="$a"
       oldest="$name"
    fi
  done
  rm "$oldest"
fi

Some comments:

I say /home/backup/* which finds all the files in that directory I could say /home/backup/databasebk_*.sql.gz instead to just find compressed dumps

I'm using 'stat' to find the oldest file according to the filesystem as your dates are in YMDhm format order I could compare filenames instead

 if  [ -z "$oldest" ] || [ "$name" -lt "$oldest" ]

the presence of subdirectories in /home/backup/ will break this script

Answer 6

Before I start, I would like to say that I hold the other answers in the highest esteem and they reflect a good understanding of the process.

I will now put myself in your shoes and look for something simple to get started, then build upon it. Let me first restate your problem:

1. Determine if the number of files is more than 5.
2. If under or equal to 5, do nothing.
3. If over 5, find and delete the file which is older than 5 (only 1 file).

Here is my approach:

1. Use ls -1 |wc -l to count the number of files.
2. Check if the number of listings is greater than 5, using an if statement.
3. If it is greater than 5, use the find command with suitable options to delete the ones which are older than 5 days.

Here is the code:

if [[ $(ls -1|wc -l) -gt 5 ]]; then 

  ls -1t |tail -n1 |xargs -n1 rm -f  

fi

Explanation of the first line: It checks if the number of files in that directory exceeds 5.

Explanation of the second line: It will run only of the number of files exceeds 5 (if there are 4 files and out of them one has a date of 100 days ago, it will not run). It will date-sort them and run the tail command to get the oldest one (the last one in the list). It will then pass on that file name to the xargs command which will use rm -f (force) to delete the file.

Thanks to all others who chipped in with a solution ! I know that there are cases where this won't work (like more than 6 files, 2 files of the same date, etc), but I have tried to keep it simple as Mike is new :)

Answer 7

Create rm-except-last.sh:

#!/bin/bash -

set -o nounset

if ! (($#)); then
  echo Usage: "$BASH_SOURCE" N_NOT_TO_DELETE FILES_TO_DELETE...
  exit 1
fi

declare N="${1:?Missing N_NOT_TO_DELETE.}"
if ! [[ "$N" =~ ^[0-9]+$ ]]; then
  echo 1>&2 -E Not a number: "$N"
  exit 1
fi
shift

declare -a roll=()
while ((${#roll[@]} < N)); do
  roll+=('')
done

if ! ((N)); then
  rm -rfv -- "$@"
  exit $?
fi

declare n=0 file=''
for file in "$@"; do
  declare rmfile="${roll[n]}"
  [ -n "$rmfile" ] && rm -rfv -- "$rmfile"
  roll[n]="$file"
  ((n = (n + 1) % N))
done

Then:

$ rm-except-last.sh 5 /home/backup/databasebk_*.sql