3

I have a log file, whose formatting is very consistent and which I control. It produces pipe-delimited fields of constant length.

Two fields are relevant to the filtering process I wish to create, call them primary and secondary.

Using Grep I can first filter on primary. This will produce an incomplete list of relevant lines. In this list will show several lines, and these will have one of two distinct values in the secondary field. There will also be other lines that do not match the primary field, but whose primary field is blank, that match on one or the other of the secondary field values. All of these rows are relevant. And I want them in the final output, but I don't know them until I have gone through the initial pass.

All entries where the secondary field matches will either have the initial primary field or a blank primary field. In no case will a secondary field be blank.

My strategy is to 1. awk the logfile once, pulling out all rows where the primary field matches (this will be the input for the script). For each such row, examine the secondary field until both possible matching secondary field values have been found. 2. awk the logfile again, pulling out all rows where the primary field matches or the secondary field matches one of two values learned in pass one.

How can I store the two secondary field values learned in pass 1, and then use them in writing the conditions for pass 2?

I've been asked to provide samples so here is a simplified version of the data illustrating the important points. "Primary" is field 2 and "Secondary" is field 3.

This is the first pull (input value is 05478900172)

2015-03-10 09:13:40,598|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,617|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,617|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,626|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,626|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:14:16,686|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,695|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,705|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,705|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,714|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,714|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:23,838|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,878|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,878|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|   0
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,888|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,888|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:15:01,915|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,936|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,936|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,945|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,946|05478900172|4792761 | 17|s|D|S|----

From this we can see that the secondary field ($3) contains two possible values for this primary field (4792761 or 4792964).

We want to have our script pull the following dataset, which contains all records with 05478900172 in $2 and either (4792761 or 4792964) in $3. I don't know these two values until I've done the initial scan, so I need to pass these values as variables that somehow get shared between the first pass and the second.

2015-03-10 09:13:40,598|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,608|       null|4792761 |---|-|K|-|----
2015-03-10 09:13:40,608|       null|4792761 |---|-|K|-|----
2015-03-10 09:13:40,617|       null|4792761 |---|r|D|S|----
2015-03-10 09:13:40,617|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,617|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,626|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,626|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:14:16,686|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,695|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,696|       null|4792964 |---|-|K|-|----
2015-03-10 09:14:16,696|       null|4792964 |---|-|K|-|----
2015-03-10 09:14:16,704|       null|4792964 |---|r|D|S|----
2015-03-10 09:14:16,705|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,705|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,714|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,714|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,760|       null|4792964 |---|r|D|S|----
2015-03-10 09:14:16,760|       null|4792964 |---|r|D|S|----
2015-03-10 09:14:23,817|       null|4792964 | 42|-|D|S|----
2015-03-10 09:14:23,817|       null|4792964 | 42|-|D|S|----
2015-03-10 09:14:23,817|       null|4792964 | 42|-|D|S|7057
2015-03-10 09:14:23,817|       null|4792964 | 42|-|D|S|----
2015-03-10 09:14:23,818|       null|4792964 | 42|-|D|S|----
2015-03-10 09:14:23,818|       null|4792964 | 42|-|D|S|----
2015-03-10 09:14:23,838|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,876|       null|4792964 |---|-|K|-|----
2015-03-10 09:14:23,876|       null|4792964 |---|-|K|-|----
2015-03-10 09:14:23,878|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,878|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,878|       null|4792964 |---|r|D|S|----
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,879|       null|4792964 |---|r|D|S|----
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|   0
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,888|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,888|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:15:01,915|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,925|       null|4792761 |---|-|K|-|----
2015-03-10 09:15:01,925|       null|4792761 |---|-|K|-|----
2015-03-10 09:15:01,936|       null|4792761 |---|r|D|S|----
2015-03-10 09:15:01,936|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,936|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,945|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,946|05478900172|4792761 | 17|s|D|S|----

closed as unclear what you're asking by jasonwryan, jimmij, Ramesh, Karlson, John WH Smith Mar 11 '15 at 18:00

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • You need to come up with a small example dataset that illustrates your requests, and show your desired output. A picture's worth a thousand words, and all that. – glenn jackman Mar 10 '15 at 19:25
  • OK, will edit post to proice this info now. – Steve Cohen Mar 10 '15 at 19:54
  • For $2 in the input file, you want the unique values in $3 and then search the entire input for rows containing those $3s? Do you know values of $2 ahead of time? – KM. Mar 10 '15 at 20:24
  • yes. And thanks, you helped me realize I no longer need to include $2 on the second pass. – Steve Cohen Mar 10 '15 at 20:58
  • @SteveCohen Does this work: awk -F"|" '$2 ~ 05478900172 {print $3}' input.log | sort -u | xargs -I {} grep {} input.log ... assuming you know $2s ahead of time? This will also return rows that match the $3s where $2s are missing. – KM. Mar 11 '15 at 0:36
1

As requested by @KM, here is my answer.

#! /bin/sh

# this script pulls all rows from a log that are directly or 
# indirectly related to a given session id.  Session IDs are stored
# in $2 of each row.  This field may be null.  Directly related
# rows are those with $2 matching the supplied parameter.  Indirectly
# related rows are those with $3 (aka xid) matching $3 in some other 
# row where $2 matches the supplied parameter.
# It may be assumed that for any rows with the same $3, 
# the $2 field will be identical or null.


SESS_SRCH="$1"
if [ -z $2 ]
then 
 LOGFILE=/path/to/default/log
else 
 LOGFILE=$2
fi

# pass 1:
# read the logfile once to find all unique XIDs associated
# with the supplied session ID ($SESS_SRCH)

XIDS=$(awk -F\| -v sessid="$1" '$2 ~ sessid { xids[$3]=0 } 
END{ 
    for (xid in xids) { 
        print xid 
    } 
}' < ${LOGFILE}
)

XID_SRCH=""

#build a search string from these xids to form a new search string.
for XID in $XIDS
do
 XID_SRCH="${XID_SRCH}|${XID}" 
done

#strip off the leading "|"
XID_SRCH=${XID_SRCH:1}

# pass 2
# read the logfile again, this time seaching on $3, for any of the
# xids found in pass 1.
awk -F\| -v search="$XID_SRCH" '$3 ~ search { print }' < ${LOGFILE}
0

This is a bit of code snippit that should do what your asking, though it seems to me there a logical problem is the question, since the output of this loop is identical with or without the second test since the match will occur either time. I'm guessing you need looking to do a more complex test in the second run of awk than you described.

What this snippet does is first extract all lines in datafile that match field 2 with 'PATTERN' extracting field 3, then by use of sort and uniq eliminate duplicates of field 3. Which is then run over a while loop for each uniq field 3 values (4792761 or 4792964) but this time testing for both field 2 against PATTERN and field 3 against the looped values.

PATTERN="05478900172"

awk -F\| -v matchpat="$PATTERN" '$2 ~ matchpat {print $3}' | sort | uniq | while read field 
do 
   awk -F\| -v matchpat=$PATTERN -v secondpat="$field" '$2 ~ matchpat { if ( $3 ~ secondpat ) {print $0}}' datafile
done

Now I'm guessing you really want to do something more complex than you described because you could simplify this and eliminate the while loop by just sorting the output from the first awk command using field 3 as a sort key.

  • What I THINK you're missing is that $2 ~ matchpat should not be a condition for the inner awk since $2 could be null as well. – Steve Cohen Mar 10 '15 at 21:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.