1

I'm parsing a log file from a webserver in an attempt to associate the IP address on one line of a log entry with the username which appears on another line of the same log entry.

I've got log entries like this below so there is that big chunk of JSON in the way. The first part is the user id on the first line, always followed by the same format of extra logging info, and then the line at the end with the IP address, and then more (not included).

Can awk deal with multiple lines, or should I be looking at putting the first item into a memory variable and outputting it when the 2nd field passes through?

2017-08-11 17:12:53.763 TRACE 25528 --- [http-nio-10040-exec-16] c.b.g.s.SpnegoAuthenticationFilter       : Principal [pqr-xyz-coredev@ABC1.COM] user ID =='pqr-xyz-coredev'
2017-08-11 17:12:53.764 TRACE 25528 --- [http-nio-10040-exec-16] c.b.g.s.SpnegoAuthenticationFilter       : No GoP header [Gop-User] for pqr-xyz-coredev@ABC1.COM
2017-08-11 17:12:53.764 TRACE 25528 --- [http-nio-10040-exec-16] c.b.g.s.SpnegoAuthenticationFilter       : User [pqr-xyz-coredev] details obtained.
 {
    "username": "pqr-xyz-coredev",
    "password": "[PROTECTED]",
    "country": null,
    "department": null,
    "title": null,
    "email": null,
    "enabled": true,
    "accountNonExpired": true,
    "credentialsNonExpired": true,
    "accountNonLocked": true,
    "grantedAuthorities": [
        "GBSUD101 SUPPORT",
        "RG OAG WE1-T1-01234 ADM"
    ]
}
2017-08-11 17:12:53.764 DEBUG 25528 --- [http-nio-10040-exec-16] c.b.g.u.s.RequestResponseLoggingFilter   : 936 URI: /data/load/REF.SWE.RUN.ALL.M.KBD/2017-08-11 GET
2017-08-11 17:12:53.764 DEBUG 25528 --- [http-nio-10040-exec-16] c.b.g.u.s.RequestResponseLoggingFilter   : 936 HEADER 'host': rsadffs199.abc1.com:10040
2017-08-11 17:12:53.764 DEBUG 25528 --- [http-nio-10040-exec-16] c.b.g.u.s.RequestResponseLoggingFilter   : 936 HEADER 'user-agent': PycURL/7.43.0 libcurl/7.52.1 WinSSL zlib/1.2.8
2017-08-11 17:12:53.764 DEBUG 25528 --- [http-nio-10040-exec-16] c.b.g.u.s.RequestResponseLoggingFilter   : 936 HEADER 'accept': */*
2017-08-11 17:12:53.764 DEBUG 25528 --- [http-nio-10040-exec-16] c.b.g.u.s.RequestResponseLoggingFilter   : 936 AUTHORIZATION: hash not displayed
2017-08-11 17:12:53.764 DEBUG 25528 --- [http-nio-10040-exec-16] c.b.g.u.s.RequestResponseLoggingFilter   : 936 PARAM 'endDate': 2099-12-31
2017-08-11 17:12:53.764 DEBUG 25528 --- [http-nio-10040-exec-16] c.b.g.u.s.RequestResponseLoggingFilter   : 936 PARAM 'startDate': 1900-01-01
2017-08-11 17:12:53.764 DEBUG 25528 --- [http-nio-10040-exec-16] c.b.g.u.s.RequestResponseLoggingFilter   : 936 REMOTE ADDRESS: 101.16.75.75
1

Try below awk:

awk -F"[=':]" '/user ID/{userID=$(NF-1)} /REMOTE ADDRESS/{print userID"," $NF}' `infile.txt

The output:

pqr-xyz-coredev, 101.16.75.75
0
perl -F"\h+|'" -lane '
   /\huser\h+ID\h/ and $a = $F[-1];
   /\hREMOTE\h+ADDRESS:\h/ and print "$a $F[-1]";
' log.file
  • Perl options:
    • -F"\h+|'" This splits the current line on either a run of horizontal whitespaces or the single-quote character. This ensures that the info we are looking for is always to be found in the last field.
    • -l This sets the ORS to a newline and also the IRS to a newline.
    • -a This will split the input records based on the field separator provided by the -F option or it defaults to a white space. The fields carved out of the current record are stored in the array @F.
    • -n This will setup an implicit input file read-in loop + autoprinting of records to stdout is disabled.
  • when we encounter a line with user ID and bounded on both sides by horizontal whitespace, we go ahead and remember the user's ID stored in the last field, viz., $F[-1].
  • when we encounter the line with REMOTE ADDRESS: and bounded on both sides by horiz. whitespace, we go ahead and print the user's ID stored earlier and the last field of this line, which shall be holding the IP address.

Results:

pqr-xyz-coredev 101.16.75.75

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.