0

I've got a number of log files (20-500) from an application that crashes. All the log files are from one run of the application - its heavily multithreaded, with each thread writing to its own file. But the logs are large. (Can be 100s of MB each in some cases)

Now one of the threads is crashing, and when it does it writes a message into the last hundred or so lines of its log file. Sometimes the threads all complete OK and that line is never written.

My wrapper script that runs the crashing application is in bash, and I'd like to detect when the app crashes in this way so I can restart it. Can I do anything nicer than:

# We want to run at least once
CRASHED=1
while [ CRASHED -eq 1 ]
    # Run the app
    run_application
    # Check the end of all the logs for KEY
    CRASHED=0
    for x in logs/* ; do
      if tail -n 100 $x | grep "KEY" ; then
        CRASHED=1
        # We'll only find it once, so may as well bail out now
        break
      fi
    done
done  

I'm primarily interested if I can replace the loop over the log files with something built in. I can't just use

grep "KEY" logs/* 

since the files are too large for this to be efficient.

  • 2
    you can always do tail -n 100 logs/* | grep -q "KEY" if you "only need to find it once" - it should work unless "KEY" matches ==> filenames <== which tail uses to separate the output from each file – don_crissti Nov 8 '16 at 2:39
  • 1
    ...which btw, can be avoided in linux via -q: tail -qn 100 logs/* | grep -q "KEY" – don_crissti Nov 8 '16 at 3:08
  • Does the application exit on this crash? If so, ISTM that it would be far more efficient to wait until the process was no longer running, and only then grep for the magic string within files modified with the last N minutes, restarting if found. – roaima Nov 8 '16 at 9:52
  • @roaima Yes, I only check the logs once the application actually crashes. (run_application in the script above blocks until the application exits) – Michael Anderson Nov 9 '16 at 0:18
0

Use;

tail --follow -n 100 *.log | grep -q key

omit --follow if the app actually exits on crash. https://linux.die.net/man/1/tail

  • Were the exit code ever sane I'd use that. But its not - hence the need to grep the logs. – Michael Anderson Nov 8 '16 at 4:50
  • Why the --follow in this case? Why use head -n 1 over grep -q? – Michael Anderson Nov 8 '16 at 4:52
0

Since you already have the ability to pause until the application exits, you can search just the recently modified log files. You can further restrict this set either by number or by time.

Here are some alternative solutions that assume that the log file names are reasonably sane.

  1. Assuming the log file names contain no whitespace and that the crash message is written somewhere in one of the last few (6) files written:

    if grep -q "KEY" $(ls -dt logs/* | head -6)
    then echo FOUND; else echo NOT FOUND
    fi
    
  2. Assuming the log file names may contain spaces but that the crash message is written in one of the last few (6) files written

    ls -t logs | head -6 | (
        while IFS= read -r f
        do
            grep -q "KEY" logs/"$f" && break
        done
        exit 1
    )
    if [[ $? -eq 0 ]]; then echo FOUND; else echo NOT FOUND; fi
    
  3. Assuming the log file names may contain spaces but that the crash message is written in the last part (400 lines) of one of the last few (6) files written

    ls -t logs | head -6 | (
        while IFS= read -r f
        do
            tail -n400 logs/"$f" | grep -q "KEY" && break
        done
        exit 1
    )
    if [[ $? -eq 0 ]]; then echo FOUND; else echo NOT FOUND; fi
    
  4. Assuming the log file names contain no whitespace and that the crash message is written in one of the files written in the last few (5) minutes

    if grep -q "KEY" $(find logs -type f -min -5)
    then echo FOUND; else echo NOT FOUND
    fi
    
  5. Assuming the log file names may contain spaces but that the crash message is written in one of the files written in the last few (5) minutes

    find logs -type f -min -5 | (
        while IFS= read -r f
        do
            grep -q "KEY" "$f" && break
        done
        exit 1
    )
    if [[ $? -eq 0 ]]; then echo FOUND; else echo NOT FOUND; fi
    
  6. Assuming the log file names may contain spaces but that the crash message is written in the last part (400 lines) of the files written in the last few (5) minutes

    find logs -type f -min -5 | (
        while IFS= read -r f
        do
            tail -n400 "$f" | grep -q "KEY" && break
        done
        exit 1
    )
    if [[ $? -eq 0 ]]; then echo FOUND; else echo NOT FOUND; fi
    
  • All the 100+ log files are large and frequently modified (every couple of seconds) - and all of them get data written into them as the application crashes ( since the core application thread notices one thread has gone bad and tells all the others to shut down.) So while sorting by modification time is a great idea in general, unfortunately it will not help in my situation. – Michael Anderson Nov 15 '16 at 1:32
0

In the end I went with some of the methods from the comments. Getting the tail out of the core loop allowed further simplification of the logic: What I ended up with was something like:

while true
    echo "Starting the application"
    run_application
    echo "Application exited - checking logs for KEY"
    if ! tail -qn 100 logs/* | grep -q "KEY"
    then
      echo "Failed without KEY in the logs - exiting"
      break
    fi
    echo "Failed with KEY in the logs - restarting"
done 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.