2

I have a huge file with some process´s log. The log have lines like "REQUEST (always)/RESPONSE (sometimes)" but RESPONSE is not necessarily the next line after REQUEST. The REQUEST header may occur several times before a RESPONSE if it occurs. I would like to join the REQUEST and RESPONSE (if exists) and then print the line. This is what I tried so far but the output is missing some lines:

awk 'BEGIN {filename = "log1.etb"}
    {line_num++; print "FNR: " FNR " NR: " NR " Counter: " line_num;
    if ($0 ~ /REQUEST.*RPCLIB/)
                        {seqid = $0; sub(/^.*@SeqID/,"SeqID",seqid);
                        line_req = $0; line_resp = ""; ref_resp = 0;
                        ref_req = line_num; tot_req++;
                        print "REQUEST: " $0;
                        for(i=1;i<=line_num+99999;i++1) {getline < "log.etb"; if ($0 ~ /RESPONSE/ && $0 ~ seqid) {ref_resp = +i; line_resp = $0; break;}};
                        print "FNR: " FNR " NR: " NR " REQUEST: " ref_req " RESPONSE: " ref_resp " " seqid;
                        print line_req"+"line_resp > filename;
                        FNR = line_num-1; NR = FNR;
                        }
    }
    END {print "Total REQUEST: " tot_req}
    ' ../EXX/log.etb

Input:

REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9517
RESPONSE 2019-01-16 00:32:07.809@{fields}@SeqID     = 9517 , Partner SeqID = 3393
REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9515
REQUEST  2019-01-16 00:32:07.810@{fields}@SeqID     = 9520
RESPONSE 2019-01-16 00:32:07.810@{fields}@SeqID     = 9520 , Partner SeqID = 3395

Desired output:

REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9517+W02/RESPONSE 2019-01-16 00:32:07.809@{fields}@SeqID     = 9517 , Partner SeqID = 3393
REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9515+
REQUEST  2019-01-16 00:32:07.810@{fields}@SeqID     = 9520+W02/RESPONSE 2019-01-16 00:32:07.810@{fields}@SeqID     = 9520 , Partner SeqID = 3395

SeqID number connects the REQUEST/RESPONSE but it may reoccur in log at some point. Also REQUESTs can occur many times before a RESPONSE and RESPONSE may or may not occur.

  • It's not explicit in the question, but from tho code I think I can make out that it's the number after SeqID = that connects a request with a response. Is that correct? Could you maybe mention that in the question? – Kusalananda Mar 31 at 7:00
  • That's correct but in some point the SeqID may repeat that's why I put a counter to search the next 99999 lines after the REQUEST header. – Marcelo Miranda Mar 31 at 10:48
  • 1
    KIndly post input and expected output – Praveen Kumar BS Mar 31 at 13:22
  • Edited the post to insert new info. Thanks @PraveenKumarBS – Marcelo Miranda Mar 31 at 16:48
  • Does the Unique-ID uniquely identify a request and it's associated response ? – iruvar Apr 1 at 0:09
0

Tried with below method and it worked fine

for i in `cat k.txt| awk  -F "=" '{print $2}'| awk -F "," '{print $1}'| sed -r "s/\s+//g"| sort| uniq`; do sed -n '/'$i'/p' k.txt|sed '1s/$/\+/g'| sed "N;s/\n/W02\//g";done

output

for i in `cat k.txt| awk  -F "=" '{print $2}'| awk -F "," '{print $1}'| sed -r "s/\s+//g"| sort| uniq`; do sed -n '/'$i'/p' k.txt|sed '1s/$/\+/g'| sed "N;s/\n/W02\//g";done

REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9515+
REQUEST  2019-01-16 00:32:07.809@{fields}@SeqID     = 9517+W02/RESPONSE 2019-01-16 00:32:07.809@{fields}@SeqID     = 9517 , Partner SeqID = 3393
REQUEST  2019-01-16 00:32:07.810@{fields}@SeqID     = 9520+W02/RESPONSE 2019-01-16 00:32:07.810@{fields}@SeqID     = 9520 , Partner SeqID = 3395
praveen@praveen:~$ 
1

I can't comment, so i post my comment as answer, sorry for that.
You want to join REQUEST and RESPONSE if their seqid matches, right? Why not sort your data by seqid first? It will make sure RESPONSE always follow it's REQUEST.

  • I tried that but in some point the seqid repeats also if I put in date+time order the next line could not be the RESPONSE, could be another REQUEST. – Marcelo Miranda Apr 1 at 15:44
1

I can't help you with awk but I made this Bash script that does the thing.

It requires at least Bash v4, but that should be quite widespread..

It expects input from stdin, meaning that you need to invoke it as in:

cat logfile | script.sh

or also:

script.sh < logfile

I did so on purpose thinking that it might be desirable, but it's easy to embed the filename within the script by just e.g. adding it to the cat -n command, before the |.

It handles:

  • missing responses
  • duplicate requests, joining the response to the latest request found with that id
  • takes SeqID fields following the @-sign (see the regex in the first sed command)
  • uses REQUEST and RESPONSE literal regexs as criteria for record distinction (see the if-elif-else-fi code block)
  • joins matching req/resp pairs with +

HTH

#!/bin/bash

declare -A reqs=()

{
while IFS= read -r line ; do
    read -r seqid rest <<<"${line}"
    line="${line#${seqid} }"
    if [[ "${line}" =~ REQUEST ]] ; then
        [ "${reqs[$seqid]}" ] && printf '%s+\n' "${reqs[$seqid]}"
        reqs[$seqid]="${line}"
    elif [[ "${line}" =~ RESPONSE ]] && [ "${reqs[$seqid]}" ] ; then
        printf '%s+%s\n' "${reqs[$seqid]}" "${line}"
        unset reqs[$seqid]
    else
        printf 'strange record at line no. %s\n' "${line}" >&2
    fi
done < <(cat -n | sed -e 's/\(.*@\)SeqID *= *\([0-9]\+\)\(.*\)/\2 &/') ;
printf '%s+\n' "${reqs[@]}" ;
} | sort -k 1 | sed -e 's/\(^\|+\)[[:blank:]]\+[0-9]\+[[:blank:]]\+/\1/g'
0

I tried the following code but it takes too long on huge file:

awk '{
    if ($0 ~ /REQUEST/ && $0 ~ /RPCLIB/)
        {seqid = $0; sub(/^.*@SeqID/,"SeqID",seqid);
        line_req = $0; line_resp = "";
        rng_s = NR; rng_e = NR + 99999;
        cmd = "awk '\''/RESPONSE.*" seqid "/ && NR >= " rng_s " && NR <= " rng_e " {print $0;exit}'\'' logfile"
        cmd | getline line_resp;
        close(cmd);
        print line_req"+"line_resp;
        }
    }
    ' logfile

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.