Using awk to find pattern pair and join lines

Question

I have a huge file with some process´s log. The log have lines like "REQUEST (always)/RESPONSE (sometimes)" but RESPONSE is not necessarily the next line after REQUEST. The REQUEST header may occur several times before a RESPONSE if it occurs. I would like to join the REQUEST and RESPONSE (if exists) and then print the line. This is what I tried so far but the output is missing some lines:

awk 'BEGIN {filename = "log1.etb"}
    {line_num++; print "FNR: " FNR " NR: " NR " Counter: " line_num;
    if ($0 ~ /REQUEST.*RPCLIB/)
                        {seqid = $0; sub(/^.*@SeqID/,"SeqID",seqid);
                        line_req = $0; line_resp = ""; ref_resp = 0;
                        ref_req = line_num; tot_req++;
                        print "REQUEST: " $0;
                        for(i=1;i<=line_num+99999;i++1) {getline < "log.etb"; if ($0 ~ /RESPONSE/ && $0 ~ seqid) {ref_resp = +i; line_resp = $0; break;}};
                        print "FNR: " FNR " NR: " NR " REQUEST: " ref_req " RESPONSE: " ref_resp " " seqid;
                        print line_req"+"line_resp > filename;
                        FNR = line_num-1; NR = FNR;
                        }
    }
    END {print "Total REQUEST: " tot_req}
    ' ../EXX/log.etb

Input:

REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9517
RESPONSE 2019-01-16 00:32:07.809@{fields}@SeqID     = 9517 , Partner SeqID = 3393
REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9515
REQUEST  2019-01-16 00:32:07.810@{fields}@SeqID     = 9520
RESPONSE 2019-01-16 00:32:07.810@{fields}@SeqID     = 9520 , Partner SeqID = 3395

Desired output:

REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9517+W02/RESPONSE 2019-01-16 00:32:07.809@{fields}@SeqID     = 9517 , Partner SeqID = 3393
REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9515+
REQUEST  2019-01-16 00:32:07.810@{fields}@SeqID     = 9520+W02/RESPONSE 2019-01-16 00:32:07.810@{fields}@SeqID     = 9520 , Partner SeqID = 3395

SeqID number connects the REQUEST/RESPONSE but it may reoccur in log at some point. Also REQUESTs can occur many times before a RESPONSE and RESPONSE may or may not occur.

Answer 1

Tried with below method and it worked fine

for i in `cat k.txt| awk  -F "=" '{print $2}'| awk -F "," '{print $1}'| sed -r "s/\s+//g"| sort| uniq`; do sed -n '/'$i'/p' k.txt|sed '1s/$/\+/g'| sed "N;s/\n/W02\//g";done

output

for i in `cat k.txt| awk  -F "=" '{print $2}'| awk -F "," '{print $1}'| sed -r "s/\s+//g"| sort| uniq`; do sed -n '/'$i'/p' k.txt|sed '1s/$/\+/g'| sed "N;s/\n/W02\//g";done

REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9515+
REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9517+W02/RESPONSE 2019-01-16 00:32:07.809@{fields}@SeqID     = 9517 , Partner SeqID = 3393
REQUEST  2019-01-16 00:32:07.810@{fields}@SeqID     = 9520+W02/RESPONSE 2019-01-16 00:32:07.810@{fields}@SeqID     = 9520 , Partner SeqID = 3395
praveen@praveen:~$ 

Answer 2

I can't comment, so i post my comment as answer, sorry for that.
You want to join REQUEST and RESPONSE if their seqid matches, right? Why not sort your data by seqid first? It will make sure RESPONSE always follow it's REQUEST.

Answer 3

I can't help you with awk but I made this Bash script that does the thing.

It requires at least Bash v4, but that should be quite widespread..

It expects input from stdin, meaning that you need to invoke it as in:

cat logfile | script.sh

or also:

script.sh < logfile

I did so on purpose thinking that it might be desirable, but it's easy to embed the filename within the script by just e.g. adding it to the cat -n command, before the |.

It handles:

• missing responses
• duplicate requests, joining the response to the latest request found with that id
• takes SeqID fields following the @-sign (see the regex in the first sed command)
• uses REQUEST and RESPONSE literal regexs as criteria for record distinction (see the if-elif-else-fi code block)
• joins matching req/resp pairs with +

HTH

#!/bin/bash

declare -A reqs=()

{
while IFS= read -r line ; do
    read -r seqid rest <<<"${line}"
    line="${line#${seqid} }"
    if [[ "${line}" =~ REQUEST ]] ; then
        [ "${reqs[$seqid]}" ] && printf '%s+\n' "${reqs[$seqid]}"
        reqs[$seqid]="${line}"
    elif [[ "${line}" =~ RESPONSE ]] && [ "${reqs[$seqid]}" ] ; then
        printf '%s+%s\n' "${reqs[$seqid]}" "${line}"
        unset reqs[$seqid]
    else
        printf 'strange record at line no. %s\n' "${line}" >&2
    fi
done < <(cat -n | sed -e 's/\(.*@\)SeqID *= *\([0-9]\+\)\(.*\)/\2 &/') ;
printf '%s+\n' "${reqs[@]}" ;
} | sort -k 1 | sed -e 's/\(^\|+\)[[:blank:]]\+[0-9]\+[[:blank:]]\+/\1/g'

Answer 4

I tried the following code but it takes too long on huge file:

awk '{
    if ($0 ~ /REQUEST/ && $0 ~ /RPCLIB/)
        {seqid = $0; sub(/^.*@SeqID/,"SeqID",seqid);
        line_req = $0; line_resp = "";
        rng_s = NR; rng_e = NR + 99999;
        cmd = "awk '\''/RESPONSE.*" seqid "/ && NR >= " rng_s " && NR <= " rng_e " {print $0;exit}'\'' logfile"
        cmd | getline line_resp;
        close(cmd);
        print line_req"+"line_resp;
        }
    }
    ' logfile