How to delete the files in one folder which are more than 60 days old in UNIX?

Question

I know how to delete the files which are more than 60 days old by modification time, but I want to delete files based on a timestamp in the file name.

For example, I have the below files for each day on monthly basis, and I have these files for last 3 years.

vtm_data_12month_20140301.txt
vtm_data_12month_20140301.control
vtm_mtd_20130622.txt
vtm_mtd_20130622.control
vtm_ytd_20131031.txt
vtm_ytd_20131031.control

I'd like to write a script find the all files which are more than 60 days old (based on file name) and delete them all except for last file of each month. For example in January I want to delete all but vtm_data_12month_20140131.txt. The issue here is there is a chance that I might have files received for January 30th so in that case I should not delete the latest file but I have to delete the rest.

Please advise me how can I achieve this via shell script?

Answer 1

OK, I have remade this script, and by sorting it backwards it looks like it should work. It compares the year and month to the previous one, and if it is lower it should be the last entry for that month.

#!/bin/bash

#the tac reverses the listing, so we go from newest to oldest, vital for our logic below
FILES=`ls | tac`

#create a cutoff date by taking current date minus our 60 day limit
CUTOFFDATE=`date --date="60 days ago" +%Y%m%d`

#we are setting this value to month 13 of the current year 
LASTYEARMONTH=`date +%Y`13

for file in $FILES; do

    #get datestamp
    FILEDATE=`expr match "$file" '.*\(20[0-9][0-9][0-9][0-9][0-9][0-9]\)'`

    #get year and month, ie 201410
    THISYEARMONTH=${FILEDATE:0:6}

    if [ ! -z $FILEDATE ] && [ $THISYEARMONTH -lt $LASTYEARMONTH ]; then

            echo "$file IS THE LAST FILE OF THE MONTH. NOT DELETING"

    else

            #now we see if the file is older than 60 days (ie, has a LOWER unix timestamp than our cutoff)
            if [ ! -z $FILEDATE ] && [ $FILEDATE -lt $CUTOFFDATE ]; then

                    echo "$file has a file date of $FILEDATE which is older than 60 days."

                    #uncomment this next line to remove
                    #rm $file
            fi
    fi
    LASTYEARMONTH=$THISYEARMONTH
done

Answer 2

The following code does not preserve the last file of each month.

#! /bin/bash

cmp_timestamp=$(date --date="60 days ago" +%Y%m%d)

while read filename; do
        [[ $filename =~ _(20[0-9][0-9][01][0-9][0123][0-9])\. ]]
        timestamp=${BASH_REMATCH[1]}
        printf "%-40s : %s\n" "$filename" "${timestamp}" 
        if [ "$timestamp" -lt "$cmp_timestamp" ]; then
                echo "   delete this file"
                : rm "$filename"
        else
                echo "   DO NOT delete this file"
        fi
        echo
done <file

From man bash:

BASH_REMATCH
An array variable whose members are assigned by the =~ binary operator to
the [[ conditional command.  The element with index 0 is the portion of the
string  matching  the entire regular expression.  The element with index n
is the portion of the string matching the nth parenthesized subexpression.
This variable is read-only.

Answer 3

You can try this in while loop:

#!/bin/bash
A=vtm_data_12month_20140301.txt
B=`ls vtm_data_12month_20140301.txt  | awk -F "_" '{print $4}' | awk -F "." '{print $1}'`
C=`date --date="60 days ago" +%Y%m%d`
if [ "$B" <  "$C" ]
then
   rm -fr $A
else
   echo "$A is not older"
fi

Answer 4

Below python script does the job, days from on which the files should be deleted can be configured with days variable.

#!/usr/bin/env python3

import os
import re
import datetime

days=60

delta = datetime.date.today() - datetime.timedelta(days=days)
files = [ x for x in os.listdir() if re.search('_\d{8}\.', x)]

for file in files:
    date = re.search('_(\d{8})\.', file).group(1)
    if datetime.datetime.strptime(date, '%Y%m%d').date() <= delta:
        print('Removing file: ',file)
        os.remove(file)

Output:

$ ./remove.py 
Removing file:  vtm_data_12month_20140301.txt
Removing file:  vtm_mtd_20130622.control
Removing file:  vtm_data_12month_20140301.control
Removing file:  vtm_mtd_20130622.txt
Removing file:  vtm_ytd_20131031.txt
Removing file:  vtm_ytd_20131031.control