2

How to I continuously tail -f a log, find all files (sed), and display (cat) the found files

example data in audit logs.

tail -f /var/log/httpd/modsec_audit.log | sed 's/[^\/]*/\./;s/].*$//g'

output

./apache/20180508/20180508-1428/20180508-142802-WvH6QgoeANwAAMwsFZ4AAAAF
./apache/20180508/20180508-1428/20180508-142803-WvH6QgoeANwAAMwtFfcAAAAG
./apache/20180508/20180508-1428/20180508-142803-WvH6QwoeANwAAMwuFlUAAAAH
./apache/20180508/20180508-1513/20180508-151357-WvIFBQoeANwAAMwnE@4AAAAA
./apache/20180508/20180508-1513/20180508-151357-WvIFBQoeANwAAMwoFD8AAAAB
./apache/20180508/20180508-1516/20180508-151608-WvIFiAoeANwAAMz1FSwAAAAA
./apache/20180508/20180508-1516/20180508-151609-WvIFiQoeANwAAMz2FYIAAAAB
./apache/20180508/20180508-1516/20180508-151611-WvIFiwoeANwAAMz3FeEAAAAC
./apache/20180508/20180508-1516/20180508-151611-WvIFiwoeANwAAMz4Fj4AAAAD
./apache/20180508/20180508-2112/20180508-211205-WvJY9QoeANwAAM1MFCoAAAAA

works with echo

echo "./apache/20180508/20180508-1428/20180508-142802-WvH6QgoeANwAAMwsFZ4AAAAF" | sed 's/[^\/]*/\./;s/].*$//g' | awk '{print $0}' | xargs cat

works with cat

cat /var/log/httpd/modsec_audit.log | sed 's/[^\/]*/\./;s/].*$//g' | awk '{print $0}' | xargs cat

does not work with tail...

tail -f /var/log/httpd/modsec_audit.log | sed 's/[^\/]*/\./;s/].*$//g' | awk '{print $0}' | xargs cat

I assume the tailing does not work because the script never terminates and sed is still caching the results until termination of the script.

Is there a way to make this work, continuously?

0

2 Answers 2

3

Buffering is getting in the way.

Use a while read line from the shell, which should read line-by-line and avoid most problems with buffering:

tail -f /var/log/httpd/modsec_audit.log | while read line; do
    echo "$line" | sed 's/[^\/]*/\./;s/].*$//g' | awk '{print $0}' | xargs cat
done

Though you can probably do better than that, by using shell (bash) to match the filename from the log line too:

tail -f /var/log/httpd/modsec_audit.log | while read line; do
    line=${line/*([^\/])/.}
    line=${line%]*}
    [[ -n "${line}" ]] && cat "${line}"
done
4
  • 2
    Not only buffering, plain xargs would read as many arguments as possible before executing cat. xargs -n 1 cat may solve that.
    – Kusalananda
    Commented May 9, 2018 at 6:18
  • I like it! Is there a way to break out of the loop if i put this into a custom method?
    – Artistan
    Commented May 9, 2018 at 12:56
  • @Artistan You can call break from within the while loop, that will work. If using a language such as Python is an option for you as well, that gives you even more flexibility to process the log file. Google for it, you'll find implementations of tail -f in Python that you can reuse.
    – filbranden
    Commented May 9, 2018 at 13:05
  • 1
    Thanks again @FilipeBrandenburger. - my result is here, if you have any suggestions for improvements gist.github.com/Artistan/981e4b20e539b092b018fa1ad20e1219
    – Artistan
    Commented May 9, 2018 at 15:08
1

try with this:

tail -f /var/log/httpd/modsec_audit.log | stdbuf -oL sed 's/[^\/]*/\./;s/].*$//g' | stdbuf -oL awk '{print $0}' | while IFS='' read -r file; do cat $file ; done

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .