0

I have a script which searches through some log files, all of which are in .gz format. I feed the script a search term, and it zgreps for the term, then grabs the transaction ID from the start of the line, and re-zgrep's the same file in order to pull back all of the lines for that transaction ID, as well as trimming off some noise from the output.

while read -r line
do
        rgx="^(.*?)(\.log\.gz)"
        fn=$(echo $line | grep -oP $rgx)
        rgx="(\d{11})"
        trid=$(echo $line | grep -oP $rgx)
        rgx="(\w{3} \d{2} ).*($trid).*"
        zgrep -ohP "$rgx" $fn
done < <(zgrep -oP $regex $path)

The problem here is it's slow, my understanding is that zgrep is actually just running gzip -cdfq, so it's decompressing the file twice? These files are anywhere from 100MB to 500MB.

Is there a way I can speed this up? Also worth nothing, the log files are split over a bunch of different folders, something to do with Kubernetes, so all of the logs for a day are almost never just in one file - this is why I'm grabbing the filename along with the transaction ID, so I'm not going through every file more than I need to.

It's a company managed server, so I can't install anything extra.

Every line on the log starts with the format below, followed by the message

2022-07-12T17:21:34+00:00 filename log 2022-07-12T17:21:30.490880384+01:00 stdout F Jul 12 17:21:30 *  192.168.0.1 (NAME, 482, 26122393785)

In this instance, the 11 digit number, 26122393785, is the transaction ID. The log messages consist mostly of XML - I'm searching for something in the XML content, usually a URL or similar, which obviously only appears on one line. So I'm grabbing the transaction ID, which is that 11 digit number preceding every linked line of the XML.

So ultimately, I want to find all of the lines containing the transaction ID, where one of those lines contains the initial search term.

So if the search term was google, I'd want to return

2022-07-12T17:21:34+00:00 filename log 2022-07-12T17:21:30.490880384+01:00 stdout F Jul 12 17:21:30 *  192.168.0.1 (NAME, 482, 26122393785) <xml>here
2022-07-12T17:21:34+00:00 filename log 2022-07-12T17:21:30.490880384+01:00 stdout F Jul 12 17:21:30 *  192.168.0.1 (NAME, 482, 26122393785) <url>google.co.uk</url>
2022-07-12T17:21:34+00:00 filename log 2022-07-12T17:21:30.490880384+01:00 stdout F Jul 12 17:21:30 *  192.168.0.1 (NAME, 482, 26122393785) end</xml>
2
  • Will the ID always appear before the respective transactions? Could you show an example excerpt of the (decompressed) file with ID, transaction and irrelevant lines? Then explain what parts of a line you desire (please provide example input and output)
    – FelixJN
    Aug 4, 2022 at 9:52
  • Have edited the post to add an example line
    – Brickscrap
    Aug 4, 2022 at 9:59

1 Answer 1

0

If you really don't have an XML-parser at hand, I'd suggest decorating the XML-blocks with NUL-bytes, then grepping the keyword with -z.

#!/bin/bash
KEYWORD=google

for file in *.log.gz ; do
    zcat "$file"  |\
    sed -e 's|^.*<xml>|\x00&|' -e 's|</xml>$|&\x00|' |\
    grep -z "$KEYWORD" | tr -d '\0'
done

That way each file is decompressed only once. If you want to match multiple keywords - in order not to run the program once per keyword -, use -f in grep and then split the output afterwards. E.g. with awk.

2
  • It's not really about parsing the XML - the transactions aren't necessarily all XML. This is why I'm first grep'ing to find the line with the keyword, then from that line grabbing the transaction ID. However, you may be right that it'd be easier just to zcat each file once
    – Brickscrap
    Aug 4, 2022 at 13:45
  • Please make sure your problem is correctly reflected in your question and that the example properly covers what might be present. It is hard to find a solution, if your problem is not well defined. If there is no proper START or END pattern for a block and an ID appears earlier than the keyword, reading the file twice seems quite unavoidable if you do not want to e.g. index the whole file into an awk array ... which might be another approach.
    – FelixJN
    Aug 4, 2022 at 14:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .