0

I have files like below. I want to keep SEP 30, OCT 30....files and delete the rest.

-rw-r--r-- 1 nbswmcm advboprd 13463761 Sep  2 01:19 vtm_data_12month_20140902.txt
-rw-r--r-- 1 nbswmcm advboprd 13474125 Sep  3 01:51 vtm_data_12month_20140903.txt
-rw-r--r-- 1 nbswmcm advboprd 13492755 Sep  4 01:57 vtm_data_12month_20140904.txt
-rw-r--r-- 1 nbswmcm advboprd 13499981 Sep  5 01:59 vtm_data_12month_20140905.txt
-rw-r--r-- 1 nbswmcm advboprd 13507296 Sep  7 08:39 vtm_data_12month_20140907.txt
-rw-r--r-- 1 nbswmcm advboprd 13508099 Sep  9 04:02 vtm_data_12month_20140909.txt
-rw-r--r-- 1 nbswmcm advboprd 13508886 Sep 10 01:33 vtm_data_12month_20140910.txt
-rw-r--r-- 1 nbswmcm advboprd 13513056 Sep 11 02:25 vtm_data_12month_20140911.txt
-rw-r--r-- 1 nbswmcm advboprd 13512334 Sep 12 02:36 vtm_data_12month_20140912.txt
-rw-r--r-- 1 nbswmcm advboprd 13512391 Sep 14 08:41 vtm_data_12month_20140914.txt
-rw-r--r-- 1 nbswmcm advboprd 13515984 Sep 16 02:35 vtm_data_12month_20140916.txt
-rw-r--r-- 1 nbswmcm advboprd 13516946 Sep 17 02:27 vtm_data_12month_20140917.txt
-rw-r--r-- 1 nbswmcm advboprd 13523528 Sep 18 02:39 vtm_data_12month_20140918.txt
-rw-r--r-- 1 nbswmcm advboprd 13520200 Sep 19 02:28 vtm_data_12month_20140919.txt
-rw-r--r-- 1 nbswmcm advboprd 13514677 Sep 21 09:33 vtm_data_12month_20140921.txt
-rw-r--r-- 1 nbswmcm advboprd 13518239 Sep 23 02:29 vtm_data_12month_20140923.txt
-rw-r--r-- 1 nbswmcm advboprd 13488002 Sep 24 02:51 vtm_data_12month_20140924.txt
-rw-r--r-- 1 nbswmcm advboprd 13491370 Sep 25 02:38 vtm_data_12month_20140925.txt
-rw-r--r-- 1 nbswmcm advboprd 13414606 Sep 26 02:42 vtm_data_12month_20140926.txt
-rw-r--r-- 1 nbswmcm advboprd 13411621 Sep 28 08:59 vtm_data_12month_20140928.txt
-rw-r--r-- 1 nbswmcm advboprd 13529594 Sep 30 02:52 vtm_data_12month_20140930.txt
-rw-r--r-- 1 nbswmcm advboprd 13520560 Oct  1 02:54 vtm_data_12month_20141001.txt
-rw-r--r-- 1 nbswmcm advboprd 13519613 Oct  2 02:54 vtm_data_12month_20141002.txt
-rw-r--r-- 1 nbswmcm advboprd 13534704 Oct  3 02:19 vtm_data_12month_20141003.txt
-rw-r--r-- 1 nbswmcm advboprd 13545015 Oct  5 08:47 vtm_data_12month_20141005.txt
-rw-r--r-- 1 nbswmcm advboprd 13541506 Oct  7 02:51 vtm_data_12month_20141007.txt
-rw-r--r-- 1 nbswmcm advboprd 13556650 Oct  8 02:31 vtm_data_12month_20141008.txt
-rw-r--r-- 1 nbswmcm advboprd 13551903 Oct  9 02:33 vtm_data_12month_20141009.txt
-rw-r--r-- 1 nbswmcm advboprd 13567484 Oct 10 02:33 vtm_data_12month_20141010.txt
-rw-r--r-- 1 nbswmcm advboprd 13569503 Oct 12 08:40 vtm_data_12month_20141012.txt
-rw-r--r-- 1 nbswmcm advboprd 13567657 Oct 14 02:25 vtm_data_12month_20141014.txt
-rw-r--r-- 1 nbswmcm advboprd 13574132 Oct 15 02:40 vtm_data_12month_20141015.txt
-rw-r--r-- 1 nbswmcm advboprd 13581260 Oct 16 02:50 vtm_data_12month_20141016.txt
-rw-r--r-- 1 nbswmcm advboprd 13585758 Oct 17 02:27 vtm_data_12month_20141017.txt
-rw-r--r-- 1 nbswmcm advboprd 13587851 Oct 19 10:02 vtm_data_12month_20141019.txt
-rw-r--r-- 1 nbswmcm advboprd 13591515 Oct 21 02:43 vtm_data_12month_20141021.txt
-rw-r--r-- 1 nbswmcm advboprd 13602271 Oct 22 02:59 vtm_data_12month_20141022.txt
-rw-r--r-- 1 nbswmcm advboprd 13604358 Oct 23 03:22 vtm_data_12month_20141023.txt
-rw-r--r-- 1 nbswmcm advboprd 13607622 Oct 24 02:52 vtm_data_12month_20141024.txt
-rw-r--r-- 1 nbswmcm advboprd 13605666 Oct 26 10:25 vtm_data_12month_20141026.txt
-rw-r--r-- 1 nbswmcm advboprd 13612303 Oct 28 02:32 vtm_data_12month_20141028.txt
-rw-r--r-- 1 nbswmcm advboprd 13617526 Oct 29 02:49 vtm_data_12month_20141029.txt
-rw-r--r-- 1 nbswmcm advboprd 13627963 Oct 30 02:38 vtm_data_12month_20141030.txt
  • Just make new directory XX and then do mv "* 30 02*" XX, remove the remaining files and move the files back from XX. That hardly warrants a script. – Anthon Nov 5 '14 at 14:04
2

On a GNU system, with that particular file name pattern,

ls -r | uniq -w23

would give you the ones to keep. So you could move them away:

ls -r | uniq -w23 | xargs mv -t ../to-keep/

And remove all the remaining files.

23 is the length of vtm_data_12month_201409. So uniq would return only the first file (in the reversely sorted list of files) among the list of files that have an identical 23 character long prefix.

To do it in one go:

ls -r | awk -F_ 'a[substr($NF,1,6)]++'

That lists the files to remove.

ls -r | awk -F_ 'a[substr($NF,1,6)]++' | xargs rm -f

(that assumes the file names don't contain, blank, quote or backslash characters).

Here, instead of the 23 first characters, we look at the first 6 characters after the last _ character.

  • can you be more clear pls...i couldnt understand...can you post exact command for my files. – Vamsi Krishna Nov 5 '14 at 14:10
  • @VamsiKrishna, those are the exact commands. – Stéphane Chazelas Nov 5 '14 at 14:11
  • sorry it is giving all files including latest file for each month – Vamsi Krishna Nov 5 '14 at 14:29
2

Use the logrotate command. That's the command that handles your log files in /var/log. It can rotate logs based on date or size.

Look at /etc/cron.daily/logrotate and /etc/logrotate.conf. You should just be able to put an entry in /etc/logrotate.conf to handle the file your want to rotate.

Also see this website: http://www.rackspace.com/knowledge_center/article/understanding-logrotate-utility

Let me know if you need anymore help. I will try my best.

OK, so on my system /etc/cron.daily/logrotate calls logrotate once every day. /etc/logrotate.conf tells logrotate how to handle each file. So you only need to worry about modifying /etc/logrotate.conf

You should just have to add an entry like this to your /etc/logrotate.conf file.

/var/log/LOGFILENAME {
    monthly
    create 0664 root root
    rotate 1
}

Then restart your system or cron or however you make the changes take effect. You can also test your config file by calling logrotate directly, thusly:

/usr/sbin/logrotate /etc/logrotate.conf

The line rotate 1 tells logrotate to keep one old copy so you will end up with two months worth of logs: the previous month and this month. Change rotate 1 to rotate 0 if you want to keep only the current month's log.

  • Are you sure it is called rotate? I searched for it and I could find only logrotate. Also, please consider expanding on your answer so that it would be helpful for future readers. – Ramesh Nov 6 '14 at 0:43
  • Logrotate. I'm on my phone so doing it from memory – shrewmouse Nov 6 '14 at 0:44
  • I will show you how to use it in a little while. If you''ll probably see calls to it in crontab. – shrewmouse Nov 6 '14 at 0:45
  • I just scanned though the web page I posted for you. It has all the info that you'll need. This is definitely the way to go. You can tell logrotate to keep 1 copy every month. /etc/cron.monthly/logrotate. – shrewmouse Nov 6 '14 at 1:04
  • Welcome to U & L SE and we expect such answers. +1 for posting a detailed answer. – Ramesh Nov 6 '14 at 1:06
0

Do you need to solve this problem generally or just once? If you just need to solve it once, sometimes a simple solution is best. We can use brace expansion in Bash:

rm vtm_data_12month_2014{09,10}{01..29}.txt

This might work in your case since:

  • The number of arguments that we will send to rm is unlikely to send us over ARG_MAX, and
  • The day you want to stop at is the same in both October and September.

If you need to solve this problem more generally, you would need to account for the fact that different months end on different days (October, for instance, ends on the 31st, but your example only has data up to the 30th).

If the file naming scheme in your example can be relied on, then the following will delete all but the latest file available for every month (based on the file name, not the actual timestamps related to the file's creation or modification):

get_years() {
  find ./ -type f | cut -d'_' -f4 | cut -c1-4 | sort | uniq
}

get_months_for_year() {
  year=$1
  find ./ -iname "vtm_data_12month_${year}*.txt" -type f | cut -d'_' -f4 | cut -c5-6 | sort | uniq
}

get_latest_for_year_month() {
  year=$1
  month=$2
  find ./ -iname "vtm_data_12month_${year}${month}*.txt" -type f | cut -d'_' -f4 | cut -c7-8 | sort | tail -1
}

for year in $(get_years); do
  for month in $(get_months_for_year $year); do
     latest=$(get_latest_for_year_month $year $month)
     end=$(($latest - 1))
     for i in $(seq 1 $end); do 
      day=$(printf '%02d' $i)
      rm vtm_data_12month_${year}${month}${day}.txt
     done
  done
done

A language other than bash that has a DateTime library will likely give you a more parsimonious and reliable solution. Both solutions presented here will emit errors if files for certain days are missing.

  • Thanks Steve. But I'd like to prepare script which has to satisfy above condition for every month. Can you please advice how can i achieve that? – Vamsi Krishna Nov 5 '14 at 13:21
  • Is the "above condition" just the 30th of every month, the last calendar day of the month, or the latest day you happen to have data for in that month? – Steven D Nov 5 '14 at 13:22
  • Steve, we have to search what is the last file for particular month and we have to keep that(whatever it is it could be 29,28,30,31) rest all should be deleted. pls advice. – Vamsi Krishna Nov 5 '14 at 13:32
  • Also it is throwing errros when i ran it. – Vamsi Krishna Nov 5 '14 at 13:35
  • find: vtm_data_12month_2014091.txt: No such file or directory find: vtm_data_12month_2014092.txt: No such file or directory find: vtm_data_12month_2014093.txt: No such file or directory find: vtm_data_12month_2014094.txt: No such file or directory find: vtm_data_12month_2014095.txt: No such file or directory find: vtm_data_12month_2014096.txt: No such file or directory find: vtm_data_12month_2014097.txt: No such file or directory find: vtm_data_12month_2014098.txt: No such file or directory find: vtm_data_12month_2014099.txt: No such file or directory – Vamsi Krishna Nov 5 '14 at 13:36
0

Checking validity of date and generating date range is difficult and error prone using bash script. The best choice would be to use python or perl or any other higher level scripting languages which can do indepth checks. I have updated the script so that it can delete files within a range of date and excluding the range of date.

The options will be like

To delete files within the range

./dfile.py --from 20140929 --to 20141001  --range

To delete files out side of the range

./dfile.py --from 20140929 --to 20141001  --out-range

This program also generates a useful help message is the arguments are given wrongly.

#!/usr/bin/env python3


import os
import sys
import argparse
import datetime

parser = argparse.ArgumentParser()
parser.add_argument('--from', dest="fromd", type=str, required=True, help="From date")
parser.add_argument('--to', dest="tod", type=str, required=True, help="To date")
parser.add_argument('--range', dest='drange', action='store_true', help="Delete files between the given date")
parser.add_argument('--out-range', dest="orange", action='store_true', help="Delete files apart form the given date")

args = parser.parse_args()

if args.drange and args.orange:
    print('Enter either --range or --out-range option')
    sys.exit(1)
elif not (args.drange or args.orange):
    print('You must select either --range or --out-range option')
    sys.exit(1)

from_date = args.fromd
to_date = args.tod
path='/tmp/tmp.54JUy4ZP6x/'


try:
    from_date = datetime.datetime.strptime(from_date, '%Y%m%d').date()
    to_date   = datetime.datetime.strptime(to_date, '%Y%m%d').date()
except ValueError:
    print('Check if the date has been given in YYYYMMDD format')
    sys.exit(1)

files = [ x for x in os.listdir(path) if x.endswith('.txt') ]


for file in files:

    file_date = datetime.datetime.strptime(file[17:25], '%Y%m%d').date()

    if args.orange:
        # Delete files out of date range
        if not from_date < file_date < to_date:
            print('Deleting file {}'.format(file))
            os.remove(path + '/' + file)
    else:
        # Delete files within date range
        if from_date < file_date < to_date:
            print('Deleting file {}'.format(file))
            os.remove(path + '/' + file)

Here all parameters are made configurable except that the path needs to be changed in the program.

Hope it helps you.

  • Thanks Mohan. I'd like to find last file for the each month to keep it and delete all others. can it be done using above script? – Vamsi Krishna Nov 5 '14 at 13:40
  • I have modified the program to support your request – Kannan Mohan Nov 5 '14 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.