2

I know how to delete the files which are more than 60 days old by modification time, but I want to delete files based on a timestamp in the file name.

For example, I have the below files for each day on monthly basis, and I have these files for last 3 years.

vtm_data_12month_20140301.txt
vtm_data_12month_20140301.control
vtm_mtd_20130622.txt
vtm_mtd_20130622.control
vtm_ytd_20131031.txt
vtm_ytd_20131031.control

I'd like to write a script find the all files which are more than 60 days old (based on file name) and delete them all except for last file of each month. For example in January I want to delete all but vtm_data_12month_20140131.txt. The issue here is there is a chance that I might have files received for January 30th so in that case I should not delete the latest file but I have to delete the rest.

Please advise me how can I achieve this via shell script?

7
  • I did some significant cleanup on your question to what I think you were trying to ask. However the only uncertainty I had was in regards to: "The issue here is there is a chance that I might have files received for January 30th so in that case I should not delete the latest file but I have to delete the rest". Are you meaning to say you might have a file for January 30th, but not January 31st, and so in that case you should preserve the January 30th file?
    – phemmer
    Nov 27, 2014 at 4:57
  • yes patrick exactly. there is also chance that we might have file for 28th or 29th in thatcase we should keep latest file and delete the rest. thanks in advance. Nov 27, 2014 at 5:10
  • Is the list of file names sorted by date? I.e. are all files for a month a non-interrupted block and is the last entry the oldest? Nov 27, 2014 at 8:16
  • Try logrotate, it's been created to do that for you without the need of scripts.
    – YoMismo
    Nov 27, 2014 at 8:22
  • vtm_data_12month_20130301.txt, vtm_data_12month_20130302.txt,vtm_data_12month_20130303.txt.....vtm_data_12month_20130331.txt...in case march month it shouldn not delete last file. Nov 27, 2014 at 9:18

4 Answers 4

2

OK, I have remade this script, and by sorting it backwards it looks like it should work. It compares the year and month to the previous one, and if it is lower it should be the last entry for that month.

#!/bin/bash

#the tac reverses the listing, so we go from newest to oldest, vital for our logic below
FILES=`ls | tac`

#create a cutoff date by taking current date minus our 60 day limit
CUTOFFDATE=`date --date="60 days ago" +%Y%m%d`

#we are setting this value to month 13 of the current year 
LASTYEARMONTH=`date +%Y`13

for file in $FILES; do

    #get datestamp
    FILEDATE=`expr match "$file" '.*\(20[0-9][0-9][0-9][0-9][0-9][0-9]\)'`

    #get year and month, ie 201410
    THISYEARMONTH=${FILEDATE:0:6}

    if [ ! -z $FILEDATE ] && [ $THISYEARMONTH -lt $LASTYEARMONTH ]; then

            echo "$file IS THE LAST FILE OF THE MONTH. NOT DELETING"

    else

            #now we see if the file is older than 60 days (ie, has a LOWER unix timestamp than our cutoff)
            if [ ! -z $FILEDATE ] && [ $FILEDATE -lt $CUTOFFDATE ]; then

                    echo "$file has a file date of $FILEDATE which is older than 60 days."

                    #uncomment this next line to remove
                    #rm $file
            fi
    fi
    LASTYEARMONTH=$THISYEARMONTH
done
9
  • Hi paulburkeland, this script is also listing the latest file of every month. It shouldn't return it. can you please modify accordingly. script should keep last file of each month and delete the rest. Nov 27, 2014 at 9:08
  • vtm_data_12month_20130301.txt, vtm_data_12month_20130302.txt,vtm_data_12month_20130303.txt.....vtm_data_12month_20130331.txt...in case march month it shouldn not delete last file. Nov 27, 2014 at 9:20
  • I believe we need to add one more if condition to find latest file for each month to keep saved. please advice. Nov 27, 2014 at 9:46
  • Hi paulburkeland, are you working on my requirement? Nov 27, 2014 at 9:57
  • Try it out now. I have remade the script for you. Nov 27, 2014 at 10:27
0

The following code does not preserve the last file of each month.

#! /bin/bash

cmp_timestamp=$(date --date="60 days ago" +%Y%m%d)

while read filename; do
        [[ $filename =~ _(20[0-9][0-9][01][0-9][0123][0-9])\. ]]
        timestamp=${BASH_REMATCH[1]}
        printf "%-40s : %s\n" "$filename" "${timestamp}" 
        if [ "$timestamp" -lt "$cmp_timestamp" ]; then
                echo "   delete this file"
                : rm "$filename"
        else
                echo "   DO NOT delete this file"
        fi
        echo
done <file

From man bash:

BASH_REMATCH
An array variable whose members are assigned by the =~ binary operator to
the [[ conditional command.  The element with index 0 is the portion of the
string  matching  the entire regular expression.  The element with index n
is the portion of the string matching the nth parenthesized subexpression.
This variable is read-only.
12
  • What is BASH_REMATCH? i have never seen it before.
    – user78605
    Nov 27, 2014 at 6:25
  • Hi it is saying below error(line 16: file: No such file or directory)/..do we need to add filename instead? Nov 27, 2014 at 7:46
  • @VamsiKrishna Of course, you must use your file's name there. Nov 27, 2014 at 7:54
  • Sorry Hauke, i dont follow you. can you please let me know what to put exactly over there. Also i need to list them files before i give rm command. so please tell me the correct 16th line in script.is below correct? Nov 27, 2014 at 8:01
  • 1
    @VamsiKrishna you do understand that the people who answer you are volunteers doing it to help you right? Please don't badger them for updates like this. Also, please don't use comments for clarification, edit your answer instead.
    – terdon
    Nov 27, 2014 at 15:56
0

You can try this in while loop:

#!/bin/bash
A=vtm_data_12month_20140301.txt
B=`ls vtm_data_12month_20140301.txt  | awk -F "_" '{print $4}' | awk -F "." '{print $1}'`
C=`date --date="60 days ago" +%Y%m%d`
if [ "$B" <  "$C" ]
then
   rm -fr $A
else
   echo "$A is not older"
fi
0

Below python script does the job, days from on which the files should be deleted can be configured with days variable.

#!/usr/bin/env python3

import os
import re
import datetime

days=60

delta = datetime.date.today() - datetime.timedelta(days=days)
files = [ x for x in os.listdir() if re.search('_\d{8}\.', x)]

for file in files:
    date = re.search('_(\d{8})\.', file).group(1)
    if datetime.datetime.strptime(date, '%Y%m%d').date() <= delta:
        print('Removing file: ',file)
        os.remove(file)

Output:

$ ./remove.py 
Removing file:  vtm_data_12month_20140301.txt
Removing file:  vtm_mtd_20130622.control
Removing file:  vtm_data_12month_20140301.control
Removing file:  vtm_mtd_20130622.txt
Removing file:  vtm_ytd_20131031.txt
Removing file:  vtm_ytd_20131031.control

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .