1

How to I continuously tail -f a log, find all files (sed), and display (cat) the found files

example data in audit logs.

tail -f /var/log/httpd/modsec_audit.log | sed 's/[^\/]*/\./;s/].*$//g'

output

./apache/20180508/20180508-1428/20180508-142802-WvH6QgoeANwAAMwsFZ4AAAAF
./apache/20180508/20180508-1428/20180508-142803-WvH6QgoeANwAAMwtFfcAAAAG
./apache/20180508/20180508-1428/20180508-142803-WvH6QwoeANwAAMwuFlUAAAAH
./apache/20180508/20180508-1513/20180508-151357-WvIFBQoeANwAAMwnE@4AAAAA
./apache/20180508/20180508-1513/20180508-151357-WvIFBQoeANwAAMwoFD8AAAAB
./apache/20180508/20180508-1516/20180508-151608-WvIFiAoeANwAAMz1FSwAAAAA
./apache/20180508/20180508-1516/20180508-151609-WvIFiQoeANwAAMz2FYIAAAAB
./apache/20180508/20180508-1516/20180508-151611-WvIFiwoeANwAAMz3FeEAAAAC
./apache/20180508/20180508-1516/20180508-151611-WvIFiwoeANwAAMz4Fj4AAAAD
./apache/20180508/20180508-2112/20180508-211205-WvJY9QoeANwAAM1MFCoAAAAA

works with echo

echo "./apache/20180508/20180508-1428/20180508-142802-WvH6QgoeANwAAMwsFZ4AAAAF" | sed 's/[^\/]*/\./;s/].*$//g' | awk '{print $0}' | xargs cat

works with cat

cat /var/log/httpd/modsec_audit.log | sed 's/[^\/]*/\./;s/].*$//g' | awk '{print $0}' | xargs cat

does not work with tail...

tail -f /var/log/httpd/modsec_audit.log | sed 's/[^\/]*/\./;s/].*$//g' | awk '{print $0}' | xargs cat

I assume the tailing does not work because the script never terminates and sed is still caching the results until termination of the script.

Is there a way to make this work, continuously?

1

Buffering is getting in the way.

Use a while read line from the shell, which should read line-by-line and avoid most problems with buffering:

tail -f /var/log/httpd/modsec_audit.log | while read line; do
    echo "$line" | sed 's/[^\/]*/\./;s/].*$//g' | awk '{print $0}' | xargs cat
done

Though you can probably do better than that, by using shell (bash) to match the filename from the log line too:

tail -f /var/log/httpd/modsec_audit.log | while read line; do
    line=${line/*([^\/])/.}
    line=${line%]*}
    [[ -n "${line}" ]] && cat "${line}"
done
  • 2
    Not only buffering, plain xargs would read as many arguments as possible before executing cat. xargs -n 1 cat may solve that. – Kusalananda May 9 '18 at 6:18
  • I like it! Is there a way to break out of the loop if i put this into a custom method? – Artistan May 9 '18 at 12:56
  • @Artistan You can call break from within the while loop, that will work. If using a language such as Python is an option for you as well, that gives you even more flexibility to process the log file. Google for it, you'll find implementations of tail -f in Python that you can reuse. – filbranden May 9 '18 at 13:05
  • 1
    Thanks again @FilipeBrandenburger. - my result is here, if you have any suggestions for improvements gist.github.com/Artistan/981e4b20e539b092b018fa1ad20e1219 – Artistan May 9 '18 at 15:08
1

try with this:

tail -f /var/log/httpd/modsec_audit.log | stdbuf -oL sed 's/[^\/]*/\./;s/].*$//g' | stdbuf -oL awk '{print $0}' | while IFS='' read -r file; do cat $file ; done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.