2

I want to automatically clean up the oldest log files first in an embedded system where the clock may be reset (so file times are not useful). I am looking for a simple shell script to find the latest file so I know what not to delete. They follow this form:

LogFileTypeA.log
LogFileTypeA.log.0
LogFileTypeA.log.1
LogFileTypeB.log     <-- this is the latest of Type B
LogFileTypeA.log.2
LogFileTypeC.log 
LogFileTypeC.log.0   <-- this is the latest of Type C
LogFileTypeA.log.3   <-- this is the latest of Type A

I want to delete everything but the latest of each type. Is there an easy way to find the latest using a shell script?

I'm using Bash version GNU bash, version 4.4.12(1)-release (sparc-buildroot-linux-gnu), and sort --version reveals BusyBox v1.26.2 (2018-05-07 10:50:40 MDT) multi-call binary.

Here is a quick test.... if you run this to set up a scenario:

#!/bin/bash
touch typeA.log
touch typeA.log.0
touch typeA.log.1
touch typeA.log.5
touch typeB.log
touch typeB.log.0
touch typeC.log
touch typeD.log.0
touch typeD.log.1
touch typeD.log.2
touch typeD.log.3
touch typeD.log.4
touch typeD.log.5
touch typeD.log.6
touch typeD.log.7
touch typeD.log.8
touch typeD.log.9
touch typeD.log.10
touch typeD.log.11
touch typeD.log.12
touch typeD.log.99
touch typeD.log.100
touch typeD.log.101
touch typeD.log.215
echo A=5
echo B=0
echo C=log
echo D=215

After running a solution script, you should get out:

typeA.log.5
typeB.log.0
typeC.log
typeD.log.215
  • @JeffSchaller Yes, there are three different "latest" files here. There are three different processes logging. Each can have an independent extension tacked on after ".log". If there is only ".log", then that is the latest for that particular application. But if there is a number tacked on, the highest number is the latest. – kmort Aug 15 '18 at 15:52
  • You mentioned an "embedded system". What shell(s) can you use? Are Bash/ksh/zsh available, or just something like busybox? – ilkkachu Aug 15 '18 at 15:53
  • @ilkkachu The system uses bash, built from busybox. – kmort Aug 15 '18 at 15:55
  • @ilkkachu bash --version leads to GNU bash, version 4.4.12(1)-release (sparc-buildroot-linux-gnu), and sh --version leads to GNU bash, version 4.4.12(1)-release (sparc-buildroot-linux-gnu) – kmort Aug 15 '18 at 16:04
4

With zsh:

typeset -A seen=()
for f (*.log*(nOn)) {((seen[${f%%.*}]++)) && echo rm -f -- $f}

(remove echo if happy with the result.

*.log*(nOn) lists the *.log* files numerically in reverse (where f.log sorts after f.log.0 itself after f.log.9 itself after f.log.10).

typeset -A seen=() declares a $seen associative array. We rm the file if the part before the first . (${f%%.*}) has been seen before.

With any shell and GNU utilities

printf '%s\0' *.log* |
  sort -rzV |
  gawk -v RS='\0' -v ORS='\0' -F. 'seen[$1]++' |
  xargs -r0 echo rm -f --

POSIXly or with busybox utilities

But assuming file names don't contain spacing characters or quotes or backslashes:

printf '%s\n' *.log* |
  sort -rt. -k1,1 -k3,3rn |
  awk -F. 'seen[$1]++' |
  xargs echo rm -f --

All those assume the part of the file name before .log doesn't contain dots.

1

It seems to me you want to keep the file that sorts last in the default lexicographic sort order.

Well, the default sort order doesn't work with numbers that have a variable amount of digits. But if the numbers are consecutive, with none missing between zero and the highest (like it was in the original question), then we can count up from zero until we find the number that isn't there, and remove all but the last one found:

#!/bin/sh
for group in *.log; do
    i=0
    last="$group"
    while [ -f "$group.$i" ]; do
        rm "$last"
        last="$group.$i"
        i=$((i+1))
    done
    echo "did not remove '$last'"
done

After touch foo.log foo.log.{0..13} bar.log asdf.log asdf.log.0 that removes all but foo.log.13, bar.log and asdf.log.0.

  • This works fairly well. The only place it falls down is when there are more than two digits in the new extension. For example, it picked .99 instead of .215. – kmort Aug 15 '18 at 16:01
  • @kmort, ah, yeah, that does break the default sort order. – ilkkachu Aug 15 '18 at 16:02
  • Hmmm. With the new one, I get ./test3.sh: line 3: syntax error near unexpected token do' ./test3.sh: line 3: up in *.log; do' – kmort Aug 15 '18 at 16:30
  • @kmort, hmh. I copy-pasted it here and back-again, and I don't see any issues. Though this won't work either if your file lists can have holes in the number sequence. :) (If you had 0, 1, 2, 5, 6, this would stop at 2 and ignore 5 and 6.) – ilkkachu Aug 15 '18 at 17:07
  • 1
    @kmort, especially since I relied on the non-numbered file existing to find the numbered sets. Your latest example set doesn't have typeD.log, so that would fail miserably. :) Just goes on to show that it's important to know the general rule and how much it can be simplified... – ilkkachu Aug 15 '18 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.