I've got a number of log files (20-500) from an application that crashes. All the log files are from one run of the application - its heavily multithreaded, with each thread writing to its own file. But the logs are large. (Can be 100s of MB each in some cases)
Now one of the threads is crashing, and when it does it writes a message into the last hundred or so lines of its log file. Sometimes the threads all complete OK and that line is never written.
My wrapper script that runs the crashing application is in bash, and I'd like to detect when the app crashes in this way so I can restart it. Can I do anything nicer than:
# We want to run at least once
CRASHED=1
while [ CRASHED -eq 1 ]
# Run the app
run_application
# Check the end of all the logs for KEY
CRASHED=0
for x in logs/* ; do
if tail -n 100 $x | grep "KEY" ; then
CRASHED=1
# We'll only find it once, so may as well bail out now
break
fi
done
done
I'm primarily interested if I can replace the loop over the log files with something built in. I can't just use
grep "KEY" logs/*
since the files are too large for this to be efficient.
tail -n 100 logs/* | grep -q "KEY"
if you "only need to find it once" - it should work unless"KEY"
matches==> filenames <==
whichtail
uses to separate the output from each file-q
:tail -qn 100 logs/* | grep -q "KEY"
grep
for the magic string within files modified with the last N minutes, restarting if found.run_application
in the script above blocks until the application exits)