extract 2 strings from a log file

Question

I have a log file that gets continuously populated; let's state its name is logfile.txt. In this log file I want to continuously capture sub strings from a single line that contains other strings on the same line using any bash commands.

This is an example of a line in the log file :

2023-08-31 09:56:39,925 [G_M80T72|utx:0:ffffac17000b:45253299:64ec7753:79cb0e|chnl:LN1_CRYPTO_IN_MQ_REQ_AMP|id:196010175121000000000002134] WARN  transaction.GeneralTransactionFailureException - GeneralTransactionFailureException Gateway.Transaction[196010175121000000000000067v1], StatusReason[ProcessingFailed/PayloadIntegrityCheckFailed]

I want to capture the string1 LN1 and string2 CRYPTO_IN_MQ_REQ_AMP each time the line contains PayloadIntegrityCheckFailed.

String1 is always preceded by chnl:'and string2 is always followed by a pipe. The 2 strings are always separated by an underscore (but string2 can also contains underscore) In my example string1=LN1 but it could be like {any letter characters}LN{any number of digits} String2 can be made of letters and digits

Once string1 and string2 are captured, I would use them as argument in another command like: {another_command} string1 string2

Of course only new entries should then be captured and I don't want to scan the full log continuously, only new lines, so I guess a first start would be to tail -f the file, then use some sed magic along with regular expressions.

Would you have an idea how to proceed?

Answer 1

$ sed -e '/PayloadIntegrityCheckFailed/!d' -e 's/.*chnl:\([^|]*\).*/\1/' -e 's/_/ /' file
LN1 CRYPTO_IN_MQ_REQ_AMP

The sed command above discards all lines that does not contain the string PayloadIntegrityCheckFailed. The remaining lines are substituted with whatever comes between chnl: and the following | character. The first underscore in that string is replaced by a space character, and the result is outputted.

The second sed expression, s/.*chnl:\([^|]*\).*/\1/, could probaly be replaced by two simpler substitutions, s/.*chnl:// and s/|.*// (which is not strictly the same thing, but would have the same effect on the example data where the sub-string chnl: exists and is later followed by a | character).

Instead of a file, the sed command could be made to read from the output of tail -f for continuous operation. In that case, you may also want to use line-buffered operation with sed (the non-standard -u option), if possible.

tail -f /var/log/some.log | sed -u -e ...as before...

To read these strings into variables and use them in some other process, you may read the output of the above pipeline in a while loop:

tail -f ... | sed ... |
while read -r first_string second_string; do
    some-command "$first_string" "$second_string"
done

Answer 2

There are multiple approaches to your issue, each of them using different tools: cat, grep, awk, etc,...

Given your statement "I want to continuously capture sub strings" I guess the right approach would be to use tail -f to show the matches on screen or write them to a file.

To print them on screen:

tail -f /path/to/your/file.log | grep "PayloadIntegrityCheckFailed" | grep -E "LN1.*CRYPTO_IN_MQ_REQ_AMP"

To save results to a file:

tail -f /path/to/your/file.log | grep "PayloadIntegrityCheckFailed" | grep -E "LN1.*CRYPTO_IN_MQ_REQ_AMP" >> /tmp/some_file.txt

Explanation:

• tail -f keeps outputting new lines added to the file.log in real time.
• grep "PayloadIntegrityCheckFailed" filters those new lines and ignores the ones not containing the PayloadIntegrityCheckFailed string
• grep -E puts grep in REGEXP mode
• The search pattern "LN1.*CRYPTO_IN_MQ_REQ_AMP" grabs any lines containing the string LN1 and also CRYPTO_IN_MQ_REQ_AMP somewhere else further ahead in the line.

In your example the LN1 string seems to be concatenated to CRYPTO_IN_MQ_REQ_AMP by a "_" character. Nonetheless as you don't state that clearly in the question I have separated both strings by ".*" any number of characters.