keep the last file of every month and delete the rest

Question

I have files like below. I want to keep SEP 30, OCT 30....files and delete the rest.

-rw-r--r-- 1 nbswmcm advboprd 13463761 Sep  2 01:19 vtm_data_12month_20140902.txt
-rw-r--r-- 1 nbswmcm advboprd 13474125 Sep  3 01:51 vtm_data_12month_20140903.txt
-rw-r--r-- 1 nbswmcm advboprd 13492755 Sep  4 01:57 vtm_data_12month_20140904.txt
-rw-r--r-- 1 nbswmcm advboprd 13499981 Sep  5 01:59 vtm_data_12month_20140905.txt
-rw-r--r-- 1 nbswmcm advboprd 13507296 Sep  7 08:39 vtm_data_12month_20140907.txt
-rw-r--r-- 1 nbswmcm advboprd 13508099 Sep  9 04:02 vtm_data_12month_20140909.txt
-rw-r--r-- 1 nbswmcm advboprd 13508886 Sep 10 01:33 vtm_data_12month_20140910.txt
-rw-r--r-- 1 nbswmcm advboprd 13513056 Sep 11 02:25 vtm_data_12month_20140911.txt
-rw-r--r-- 1 nbswmcm advboprd 13512334 Sep 12 02:36 vtm_data_12month_20140912.txt
-rw-r--r-- 1 nbswmcm advboprd 13512391 Sep 14 08:41 vtm_data_12month_20140914.txt
-rw-r--r-- 1 nbswmcm advboprd 13515984 Sep 16 02:35 vtm_data_12month_20140916.txt
-rw-r--r-- 1 nbswmcm advboprd 13516946 Sep 17 02:27 vtm_data_12month_20140917.txt
-rw-r--r-- 1 nbswmcm advboprd 13523528 Sep 18 02:39 vtm_data_12month_20140918.txt
-rw-r--r-- 1 nbswmcm advboprd 13520200 Sep 19 02:28 vtm_data_12month_20140919.txt
-rw-r--r-- 1 nbswmcm advboprd 13514677 Sep 21 09:33 vtm_data_12month_20140921.txt
-rw-r--r-- 1 nbswmcm advboprd 13518239 Sep 23 02:29 vtm_data_12month_20140923.txt
-rw-r--r-- 1 nbswmcm advboprd 13488002 Sep 24 02:51 vtm_data_12month_20140924.txt
-rw-r--r-- 1 nbswmcm advboprd 13491370 Sep 25 02:38 vtm_data_12month_20140925.txt
-rw-r--r-- 1 nbswmcm advboprd 13414606 Sep 26 02:42 vtm_data_12month_20140926.txt
-rw-r--r-- 1 nbswmcm advboprd 13411621 Sep 28 08:59 vtm_data_12month_20140928.txt
-rw-r--r-- 1 nbswmcm advboprd 13529594 Sep 30 02:52 vtm_data_12month_20140930.txt
-rw-r--r-- 1 nbswmcm advboprd 13520560 Oct  1 02:54 vtm_data_12month_20141001.txt
-rw-r--r-- 1 nbswmcm advboprd 13519613 Oct  2 02:54 vtm_data_12month_20141002.txt
-rw-r--r-- 1 nbswmcm advboprd 13534704 Oct  3 02:19 vtm_data_12month_20141003.txt
-rw-r--r-- 1 nbswmcm advboprd 13545015 Oct  5 08:47 vtm_data_12month_20141005.txt
-rw-r--r-- 1 nbswmcm advboprd 13541506 Oct  7 02:51 vtm_data_12month_20141007.txt
-rw-r--r-- 1 nbswmcm advboprd 13556650 Oct  8 02:31 vtm_data_12month_20141008.txt
-rw-r--r-- 1 nbswmcm advboprd 13551903 Oct  9 02:33 vtm_data_12month_20141009.txt
-rw-r--r-- 1 nbswmcm advboprd 13567484 Oct 10 02:33 vtm_data_12month_20141010.txt
-rw-r--r-- 1 nbswmcm advboprd 13569503 Oct 12 08:40 vtm_data_12month_20141012.txt
-rw-r--r-- 1 nbswmcm advboprd 13567657 Oct 14 02:25 vtm_data_12month_20141014.txt
-rw-r--r-- 1 nbswmcm advboprd 13574132 Oct 15 02:40 vtm_data_12month_20141015.txt
-rw-r--r-- 1 nbswmcm advboprd 13581260 Oct 16 02:50 vtm_data_12month_20141016.txt
-rw-r--r-- 1 nbswmcm advboprd 13585758 Oct 17 02:27 vtm_data_12month_20141017.txt
-rw-r--r-- 1 nbswmcm advboprd 13587851 Oct 19 10:02 vtm_data_12month_20141019.txt
-rw-r--r-- 1 nbswmcm advboprd 13591515 Oct 21 02:43 vtm_data_12month_20141021.txt
-rw-r--r-- 1 nbswmcm advboprd 13602271 Oct 22 02:59 vtm_data_12month_20141022.txt
-rw-r--r-- 1 nbswmcm advboprd 13604358 Oct 23 03:22 vtm_data_12month_20141023.txt
-rw-r--r-- 1 nbswmcm advboprd 13607622 Oct 24 02:52 vtm_data_12month_20141024.txt
-rw-r--r-- 1 nbswmcm advboprd 13605666 Oct 26 10:25 vtm_data_12month_20141026.txt
-rw-r--r-- 1 nbswmcm advboprd 13612303 Oct 28 02:32 vtm_data_12month_20141028.txt
-rw-r--r-- 1 nbswmcm advboprd 13617526 Oct 29 02:49 vtm_data_12month_20141029.txt
-rw-r--r-- 1 nbswmcm advboprd 13627963 Oct 30 02:38 vtm_data_12month_20141030.txt

Answer 1

On a GNU system, with that particular file name pattern,

ls -r | uniq -w23

would give you the ones to keep. So you could move them away:

ls -r | uniq -w23 | xargs mv -t ../to-keep/

And remove all the remaining files.

23 is the length of vtm_data_12month_201409. So uniq would return only the first file (in the reversely sorted list of files) among the list of files that have an identical 23 character long prefix.

To do it in one go:

ls -r | awk -F_ 'a[substr($NF,1,6)]++'

That lists the files to remove.

ls -r | awk -F_ 'a[substr($NF,1,6)]++' | xargs rm -f

(that assumes the file names don't contain, blank, quote or backslash characters).

Here, instead of the 23 first characters, we look at the first 6 characters after the last _ character.

Answer 2

Use the logrotate command. That's the command that handles your log files in /var/log. It can rotate logs based on date or size.

Look at /etc/cron.daily/logrotate and /etc/logrotate.conf. You should just be able to put an entry in /etc/logrotate.conf to handle the file your want to rotate.

Also see this website: http://www.rackspace.com/knowledge_center/article/understanding-logrotate-utility

Let me know if you need anymore help. I will try my best.

OK, so on my system /etc/cron.daily/logrotate calls logrotate once every day. /etc/logrotate.conf tells logrotate how to handle each file. So you only need to worry about modifying /etc/logrotate.conf

You should just have to add an entry like this to your /etc/logrotate.conf file.

/var/log/LOGFILENAME {
    monthly
    create 0664 root root
    rotate 1
}

Then restart your system or cron or however you make the changes take effect. You can also test your config file by calling logrotate directly, thusly:

/usr/sbin/logrotate /etc/logrotate.conf

The line rotate 1 tells logrotate to keep one old copy so you will end up with two months worth of logs: the previous month and this month. Change rotate 1 to rotate 0 if you want to keep only the current month's log.

Answer 3

Do you need to solve this problem generally or just once? If you just need to solve it once, sometimes a simple solution is best. We can use brace expansion in Bash:

rm vtm_data_12month_2014{09,10}{01..29}.txt

This might work in your case since:

• The number of arguments that we will send to rm is unlikely to send us over ARG_MAX, and
• The day you want to stop at is the same in both October and September.

If you need to solve this problem more generally, you would need to account for the fact that different months end on different days (October, for instance, ends on the 31st, but your example only has data up to the 30th).

If the file naming scheme in your example can be relied on, then the following will delete all but the latest file available for every month (based on the file name, not the actual timestamps related to the file's creation or modification):

get_years() {
  find ./ -type f | cut -d'_' -f4 | cut -c1-4 | sort | uniq
}

get_months_for_year() {
  year=$1
  find ./ -iname "vtm_data_12month_${year}*.txt" -type f | cut -d'_' -f4 | cut -c5-6 | sort | uniq
}

get_latest_for_year_month() {
  year=$1
  month=$2
  find ./ -iname "vtm_data_12month_${year}${month}*.txt" -type f | cut -d'_' -f4 | cut -c7-8 | sort | tail -1
}

for year in $(get_years); do
  for month in $(get_months_for_year $year); do
     latest=$(get_latest_for_year_month $year $month)
     end=$(($latest - 1))
     for i in $(seq 1 $end); do 
      day=$(printf '%02d' $i)
      rm vtm_data_12month_${year}${month}${day}.txt
     done
  done
done

A language other than bash that has a DateTime library will likely give you a more parsimonious and reliable solution. Both solutions presented here will emit errors if files for certain days are missing.

Answer 4

Checking validity of date and generating date range is difficult and error prone using bash script. The best choice would be to use python or perl or any other higher level scripting languages which can do indepth checks. I have updated the script so that it can delete files within a range of date and excluding the range of date.

The options will be like

To delete files within the range

./dfile.py --from 20140929 --to 20141001  --range

To delete files out side of the range

./dfile.py --from 20140929 --to 20141001  --out-range

This program also generates a useful help message is the arguments are given wrongly.

#!/usr/bin/env python3


import os
import sys
import argparse
import datetime

parser = argparse.ArgumentParser()
parser.add_argument('--from', dest="fromd", type=str, required=True, help="From date")
parser.add_argument('--to', dest="tod", type=str, required=True, help="To date")
parser.add_argument('--range', dest='drange', action='store_true', help="Delete files between the given date")
parser.add_argument('--out-range', dest="orange", action='store_true', help="Delete files apart form the given date")

args = parser.parse_args()

if args.drange and args.orange:
    print('Enter either --range or --out-range option')
    sys.exit(1)
elif not (args.drange or args.orange):
    print('You must select either --range or --out-range option')
    sys.exit(1)

from_date = args.fromd
to_date = args.tod
path='/tmp/tmp.54JUy4ZP6x/'


try:
    from_date = datetime.datetime.strptime(from_date, '%Y%m%d').date()
    to_date   = datetime.datetime.strptime(to_date, '%Y%m%d').date()
except ValueError:
    print('Check if the date has been given in YYYYMMDD format')
    sys.exit(1)

files = [ x for x in os.listdir(path) if x.endswith('.txt') ]


for file in files:

    file_date = datetime.datetime.strptime(file[17:25], '%Y%m%d').date()

    if args.orange:
        # Delete files out of date range
        if not from_date < file_date < to_date:
            print('Deleting file {}'.format(file))
            os.remove(path + '/' + file)
    else:
        # Delete files within date range
        if from_date < file_date < to_date:
            print('Deleting file {}'.format(file))
            os.remove(path + '/' + file)

Here all parameters are made configurable except that the path needs to be changed in the program.

Hope it helps you.