2

I know how to delete the files which are more than 60 days old by modification time, but I want to delete files based on a timestamp in the file name.

For example, I have the below files for each day on monthly basis, and I have these files for last 3 years.

vtm_data_12month_20140301.txt
vtm_data_12month_20140301.control
vtm_mtd_20130622.txt
vtm_mtd_20130622.control
vtm_ytd_20131031.txt
vtm_ytd_20131031.control

I'd like to write a script find the all files which are more than 60 days old (based on file name) and delete them all except for last file of each month. For example in January I want to delete all but vtm_data_12month_20140131.txt. The issue here is there is a chance that I might have files received for January 30th so in that case I should not delete the latest file but I have to delete the rest.

Please advise me how can I achieve this via shell script?

  • I did some significant cleanup on your question to what I think you were trying to ask. However the only uncertainty I had was in regards to: "The issue here is there is a chance that I might have files received for January 30th so in that case I should not delete the latest file but I have to delete the rest". Are you meaning to say you might have a file for January 30th, but not January 31st, and so in that case you should preserve the January 30th file? – Patrick Nov 27 '14 at 4:57
  • yes patrick exactly. there is also chance that we might have file for 28th or 29th in thatcase we should keep latest file and delete the rest. thanks in advance. – Vamsi Krishna Nov 27 '14 at 5:10
  • Is the list of file names sorted by date? I.e. are all files for a month a non-interrupted block and is the last entry the oldest? – Hauke Laging Nov 27 '14 at 8:16
  • Try logrotate, it's been created to do that for you without the need of scripts. – YoMismo Nov 27 '14 at 8:22
  • vtm_data_12month_20130301.txt, vtm_data_12month_20130302.txt,vtm_data_12month_20130303.txt.....vtm_data_12month_20130331.txt...in case march month it shouldn not delete last file. – Vamsi Krishna Nov 27 '14 at 9:18
2

OK, I have remade this script, and by sorting it backwards it looks like it should work. It compares the year and month to the previous one, and if it is lower it should be the last entry for that month.

#!/bin/bash

#the tac reverses the listing, so we go from newest to oldest, vital for our logic below
FILES=`ls | tac`

#create a cutoff date by taking current date minus our 60 day limit
CUTOFFDATE=`date --date="60 days ago" +%Y%m%d`

#we are setting this value to month 13 of the current year 
LASTYEARMONTH=`date +%Y`13

for file in $FILES; do

    #get datestamp
    FILEDATE=`expr match "$file" '.*\(20[0-9][0-9][0-9][0-9][0-9][0-9]\)'`

    #get year and month, ie 201410
    THISYEARMONTH=${FILEDATE:0:6}

    if [ ! -z $FILEDATE ] && [ $THISYEARMONTH -lt $LASTYEARMONTH ]; then

            echo "$file IS THE LAST FILE OF THE MONTH. NOT DELETING"

    else

            #now we see if the file is older than 60 days (ie, has a LOWER unix timestamp than our cutoff)
            if [ ! -z $FILEDATE ] && [ $FILEDATE -lt $CUTOFFDATE ]; then

                    echo "$file has a file date of $FILEDATE which is older than 60 days."

                    #uncomment this next line to remove
                    #rm $file
            fi
    fi
    LASTYEARMONTH=$THISYEARMONTH
done
  • Hi paulburkeland, this script is also listing the latest file of every month. It shouldn't return it. can you please modify accordingly. script should keep last file of each month and delete the rest. – Vamsi Krishna Nov 27 '14 at 9:08
  • vtm_data_12month_20130301.txt, vtm_data_12month_20130302.txt,vtm_data_12month_20130303.txt.....vtm_data_12month_20130331.txt...in case march month it shouldn not delete last file. – Vamsi Krishna Nov 27 '14 at 9:20
  • I believe we need to add one more if condition to find latest file for each month to keep saved. please advice. – Vamsi Krishna Nov 27 '14 at 9:46
  • Hi paulburkeland, are you working on my requirement? – Vamsi Krishna Nov 27 '14 at 9:57
  • Try it out now. I have remade the script for you. – nyxgeek Nov 27 '14 at 10:27
0

The following code does not preserve the last file of each month.

#! /bin/bash

cmp_timestamp=$(date --date="60 days ago" +%Y%m%d)

while read filename; do
        [[ $filename =~ _(20[0-9][0-9][01][0-9][0123][0-9])\. ]]
        timestamp=${BASH_REMATCH[1]}
        printf "%-40s : %s\n" "$filename" "${timestamp}" 
        if [ "$timestamp" -lt "$cmp_timestamp" ]; then
                echo "   delete this file"
                : rm "$filename"
        else
                echo "   DO NOT delete this file"
        fi
        echo
done <file

From man bash:

BASH_REMATCH
An array variable whose members are assigned by the =~ binary operator to
the [[ conditional command.  The element with index 0 is the portion of the
string  matching  the entire regular expression.  The element with index n
is the portion of the string matching the nth parenthesized subexpression.
This variable is read-only.
  • What is BASH_REMATCH? i have never seen it before. – user78605 Nov 27 '14 at 6:25
  • Hi it is saying below error(line 16: file: No such file or directory)/..do we need to add filename instead? – Vamsi Krishna Nov 27 '14 at 7:46
  • @VamsiKrishna Of course, you must use your file's name there. – Hauke Laging Nov 27 '14 at 7:54
  • Sorry Hauke, i dont follow you. can you please let me know what to put exactly over there. Also i need to list them files before i give rm command. so please tell me the correct 16th line in script.is below correct? – Vamsi Krishna Nov 27 '14 at 8:01
  • 1
    @VamsiKrishna you do understand that the people who answer you are volunteers doing it to help you right? Please don't badger them for updates like this. Also, please don't use comments for clarification, edit your answer instead. – terdon Nov 27 '14 at 15:56
0

You can try this in while loop:

#!/bin/bash
A=vtm_data_12month_20140301.txt
B=`ls vtm_data_12month_20140301.txt  | awk -F "_" '{print $4}' | awk -F "." '{print $1}'`
C=`date --date="60 days ago" +%Y%m%d`
if [ "$B" <  "$C" ]
then
   rm -fr $A
else
   echo "$A is not older"
fi
0

Below python script does the job, days from on which the files should be deleted can be configured with days variable.

#!/usr/bin/env python3

import os
import re
import datetime

days=60

delta = datetime.date.today() - datetime.timedelta(days=days)
files = [ x for x in os.listdir() if re.search('_\d{8}\.', x)]

for file in files:
    date = re.search('_(\d{8})\.', file).group(1)
    if datetime.datetime.strptime(date, '%Y%m%d').date() <= delta:
        print('Removing file: ',file)
        os.remove(file)

Output:

$ ./remove.py 
Removing file:  vtm_data_12month_20140301.txt
Removing file:  vtm_mtd_20130622.control
Removing file:  vtm_data_12month_20140301.control
Removing file:  vtm_mtd_20130622.txt
Removing file:  vtm_ytd_20131031.txt
Removing file:  vtm_ytd_20131031.control

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.