3

I've got a directory, containing lots of weekly generated files with names like

 db_20130101_foo.tgz
 db_20130108_foo.tgz
 db_20130115_foo.tgz
 ...

and so on. Over the years, the disks will get pretty full. As the files contain data for several weeks, we can remove older files. I want to remove every file, but always keep the last file of each month. How will i be able to accomplish this, without having to copy & paste filenames manually to rm, which is a lot of work and pretty error-prone?

  • Look into logrotate; it's designed for exactly this purpose. – Wildcard Oct 8 '15 at 17:39
  • logrotate would be alright for the task and i would have used it. But we have to keep the weekly backups for some time and just can remove older backup files. Therefore i'll need a custom solution for this issue to automate the cleaning process. – mru Oct 8 '15 at 18:48
  • Although I haven't had a need to configure logrotate yet, it is quite configurable and is the tool for the job. There are simple options to specify how long to keep old files, and with more advanced options such as postrotate, you can definitely set it up to keep the monthly files forever if that is what you want. If I could say how off the top of my head, I would write an answer; but I at least want it noted for posterity that logrotate will do what you want and is designed for exactly this. Read over man logrotate to get familiar with what is possible. – Wildcard Oct 8 '15 at 18:59
2

This oneliner will give you the files you want to delete:

(ls -1 db_*_foo.tgz; echo) | awk '{prevym=ym; prevfile=file; ym=substr($0,4,6); file=$0; if (ym==prevym)print prevfile}'

The first part just lists ALL the files (and adds an extra line to the end of the list, to simplify the later awk command). The awk part just checks each line to see if the ym (yearmonth) changed from one line to the next.

Test and make sure that the above lists the files you DO want to delete. Then, to delete all of the files, simply pipe the command into:

...ABOVE_COMMAND... | xargs rm

  • This is the step in the right direction that i was looking for. Unfortunately i had too few contacts with awk yet to find out how to do that myself. I'll do some testing tomorrow, but i think with this oneliner as a basis i'll get the task done. – mru Oct 8 '15 at 18:45
1

out-there answer: ruby. it has a nice group_by method, so you can grab the files with the same "key" (first 9 characters of the filename):

files = Dir.glob("*")
to_delete = []
files.group_by {|f| f[0,9]}.each_pair {|k,v| to_delete.push *(v.sort[0..-2])}
puts "removing: #{to_delete}"
File.delete to_delete

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.