0

I require a bash script to keep the most recent 'date_YYYY-MM-DD_HHMMSS' directory, removing others

The directory structure is as follows. The directories in tags are variable and will change, i.e 'ENVIRONMENT', could be production, staging, development. Specifically in advance I will not know 'productname'

/home/'USER'/'ENVIRONMENT'/Content/'productname'/'date_YYYY-MM-DD_HHMMSS'

I've tried

ls -t  .\directory | tail -n +1

and

find .\directory type f -mtime +1 

(this was during a moment of madness where I was happy to only keep the last days worth of content)

and

find . -maxdepth 6 -mindepth 5 -type f -printf '%f\n'

So for example, if i had the following

/home/USER1/production/Content/productname1/date_20151009_120000
/home/USER1/production/Content/productname1/date_20151010_120000
/home/USER1/production/Content/productname2/date_20151009_120000
/home/USER1/production/Content/productname2/date_20151010_120000
/home/USER1/production/Content/productname1/date_20151009_120000
/home/USER1/production/Content/productname1/date_20151010_120000
/home/USER1/staging/Content/productname2/date_20151009_120000
/home/USER1/staging/Content/productname2/date_20151010_120000

I would keep

/home/USER1/production/Content/productname1/date_20151010_120000
/home/USER1/production/Content/productname2/date_20151010_120000
/home/USER1/staging/Content/productname2/date_20151010_120000
  • 1
    Please edit your question and clarify: the most recent directory among which? The most recent /home/USER/XXX? /home/USER/XXX/Content/? The most recent under each productname? Ideally, show us some examples and the result you expect. – terdon Dec 8 '15 at 11:58
  • most recent 'date' directory. first line, right there. I've updated for clarity. you're assumption was correct. – Gaz Dec 8 '15 at 16:40
  • Yes, and most recent among what? Most recent in environment? Most recent in productname? Most recent overall? Should only one directory be kept or do you, for example, need the most recent dirs from each of the different ENVIRONMENT or USER values? Also, should we take the time from the directory's name or from its timestamp? – terdon Dec 8 '15 at 16:43
  • I see what you mean, most recent in their own sub dir. I will update again. thanks. – Gaz Dec 8 '15 at 17:07
1

You can use a combination of awk and xargs to achieve this,

sort -r directories | awk -F/ 'cache[$3, $4, $6]++' | xargs -L1 rm -rf

directories is just a file containing all your directories, sort -r directories could easily be replaced by a pipeline (e.g. printf "%s\n" /home/*/*/Content/*/* | sort -r).

sort -r puts the directories to keep 'toward the top' of the list (it uses the fact that your dates can be compared lexicographically, if they weren't you'd have to parse them properly first). Finally awk does the bookeeping, the first time it sees a (user, environment, product) it doesn't print it, all the following occurrences get printed. This generates a list of directories to delete, which then gets deleted thanks to xargs/rm.

Note: if your filenames/directories contains spaces or 'weird' characters it won't quite work so you'll have to modify the command.


What does awk -F/ 'cache[$3, $4, $6]++' do? First it splits the input lines on /. Then for each input line it checks if the tuple containing the 3rd, 4th, and 6th values are present in the cache associative array. The first time a tuple is seen it's not in the array, so cache[$3, $4, $6] returns 0 hence the line gets dropped, the ++ then sets cache[$3, $4, $6] to 1. The next times a tuple is seen cache[$3, $4, $6] returns something > 0, which means that the line gets printed.

Here's a simple worked example:

$ cat example
a,c,0
a,a,2
c,a,0
a,b,2
b,a,2
a,b,0
c,a,1
b,b,0
a,b,1
a,a,1
a,a,0
b,a,1
b,c,2
b,b,2
b,c,1
a,c,1
c,c,1
$ awk -F, 'cache[$1, $2]++' example
a,b,0
c,a,1
a,b,1
a,a,1
a,a,0
b,a,1
b,b,2
b,c,1
a,c,1
$ fgrep a,b example
a,b,2
a,b,0
a,b,1
$

Notice how the a,b,2 line got dropped while a,b,0 and a,b,1 didn't. Using an associative array and the ++ operator is a pretty common pattern in awk, for instance see 43. Remove duplicate, nonconsecutive lines.

  • please can you explain 'cache[$3, $4, $6]++' thank you – Gaz Dec 8 '15 at 20:36
  • @Gaz: added some explanation about how the awk call works; it might still be confusing if you're not familiar with awk (if so <grymoire.com/Unix/Awk.html> is a good resource to get started). – foo Dec 8 '15 at 21:18
  • Thank you. However the AWK grabs the product directory too. It was getting the /home/environment/ but i got around that by adding the first and second value. – Gaz Dec 8 '15 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.