grep end of several files

Question

I've got a number of log files (20-500) from an application that crashes. All the log files are from one run of the application - its heavily multithreaded, with each thread writing to its own file. But the logs are large. (Can be 100s of MB each in some cases)

Now one of the threads is crashing, and when it does it writes a message into the last hundred or so lines of its log file. Sometimes the threads all complete OK and that line is never written.

My wrapper script that runs the crashing application is in bash, and I'd like to detect when the app crashes in this way so I can restart it. Can I do anything nicer than:

# We want to run at least once
CRASHED=1
while [ CRASHED -eq 1 ]
    # Run the app
    run_application
    # Check the end of all the logs for KEY
    CRASHED=0
    for x in logs/* ; do
      if tail -n 100 $x | grep "KEY" ; then
        CRASHED=1
        # We'll only find it once, so may as well bail out now
        break
      fi
    done
done  

I'm primarily interested if I can replace the loop over the log files with something built in. I can't just use

grep "KEY" logs/* 

since the files are too large for this to be efficient.

Answer 1

Use;

tail --follow -n 100 *.log | grep -q key

omit --follow if the app actually exits on crash. https://linux.die.net/man/1/tail

Answer 2

Since you already have the ability to pause until the application exits, you can search just the recently modified log files. You can further restrict this set either by number or by time.

Here are some alternative solutions that assume that the log file names are reasonably sane.

1. Assuming the log file names contain no whitespace and that the crash message is written somewhere in one of the last few (6) files written:

   if grep -q "KEY" $(ls -dt logs/* | head -6)
   then echo FOUND; else echo NOT FOUND
   fi
   

2. Assuming the log file names may contain spaces but that the crash message is written in one of the last few (6) files written

   ls -t logs | head -6 | (
       while IFS= read -r f
       do
           grep -q "KEY" logs/"$f" && break
       done
       exit 1
   )
   if [[ $? -eq 0 ]]; then echo FOUND; else echo NOT FOUND; fi
   

3. Assuming the log file names may contain spaces but that the crash message is written in the last part (400 lines) of one of the last few (6) files written

   ls -t logs | head -6 | (
       while IFS= read -r f
       do
           tail -n400 logs/"$f" | grep -q "KEY" && break
       done
       exit 1
   )
   if [[ $? -eq 0 ]]; then echo FOUND; else echo NOT FOUND; fi
   

4. Assuming the log file names contain no whitespace and that the crash message is written in one of the files written in the last few (5) minutes

   if grep -q "KEY" $(find logs -type f -min -5)
   then echo FOUND; else echo NOT FOUND
   fi
   

5. Assuming the log file names may contain spaces but that the crash message is written in one of the files written in the last few (5) minutes

   find logs -type f -min -5 | (
       while IFS= read -r f
       do
           grep -q "KEY" "$f" && break
       done
       exit 1
   )
   if [[ $? -eq 0 ]]; then echo FOUND; else echo NOT FOUND; fi
   

6. Assuming the log file names may contain spaces but that the crash message is written in the last part (400 lines) of the files written in the last few (5) minutes

   find logs -type f -min -5 | (
       while IFS= read -r f
       do
           tail -n400 "$f" | grep -q "KEY" && break
       done
       exit 1
   )
   if [[ $? -eq 0 ]]; then echo FOUND; else echo NOT FOUND; fi
   

Answer 3

In the end I went with some of the methods from the comments. Getting the tail out of the core loop allowed further simplification of the logic: What I ended up with was something like:

while true
    echo "Starting the application"
    run_application
    echo "Application exited - checking logs for KEY"
    if ! tail -qn 100 logs/* | grep -q "KEY"
    then
      echo "Failed without KEY in the logs - exiting"
      break
    fi
    echo "Failed with KEY in the logs - restarting"
done