How to Save variables in a script that can be shared between two runs of awk against the same input file in the script? [closed]

Question

Closed. This question needs details or clarity. It is not currently accepting answers.

Want to improve this question? Add details and clarify the problem by editing this post.

Closed 9 years ago.

I have a log file, whose formatting is very consistent and which I control. It produces pipe-delimited fields of constant length.

Two fields are relevant to the filtering process I wish to create, call them primary and secondary.

Using Grep I can first filter on primary. This will produce an incomplete list of relevant lines. In this list will show several lines, and these will have one of two distinct values in the secondary field. There will also be other lines that do not match the primary field, but whose primary field is blank, that match on one or the other of the secondary field values. All of these rows are relevant. And I want them in the final output, but I don't know them until I have gone through the initial pass.

All entries where the secondary field matches will either have the initial primary field or a blank primary field. In no case will a secondary field be blank.

My strategy is to 1. awk the logfile once, pulling out all rows where the primary field matches (this will be the input for the script). For each such row, examine the secondary field until both possible matching secondary field values have been found. 2. awk the logfile again, pulling out all rows where the primary field matches or the secondary field matches one of two values learned in pass one.

How can I store the two secondary field values learned in pass 1, and then use them in writing the conditions for pass 2?

I've been asked to provide samples so here is a simplified version of the data illustrating the important points. "Primary" is field 2 and "Secondary" is field 3.

This is the first pull (input value is 05478900172)

2015-03-10 09:13:40,598|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,617|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,617|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,626|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,626|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:14:16,686|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,695|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,705|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,705|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,714|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,714|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:23,838|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,878|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,878|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|   0
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,888|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,888|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:15:01,915|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,936|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,936|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,945|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,946|05478900172|4792761 | 17|s|D|S|----

From this we can see that the secondary field ($3) contains two possible values for this primary field (4792761 or 4792964).

We want to have our script pull the following dataset, which contains all records with 05478900172 in $2 and either (4792761 or 4792964) in $3. I don't know these two values until I've done the initial scan, so I need to pass these values as variables that somehow get shared between the first pass and the second.

2015-03-10 09:13:40,598|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,601|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,608|       null|4792761 |---|-|K|-|----
2015-03-10 09:13:40,608|       null|4792761 |---|-|K|-|----
2015-03-10 09:13:40,617|       null|4792761 |---|r|D|S|----
2015-03-10 09:13:40,617|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,617|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,626|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:13:40,626|05478900172|4792761 | 15|s|D|S|----
2015-03-10 09:14:16,686|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,694|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,695|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,696|       null|4792964 |---|-|K|-|----
2015-03-10 09:14:16,696|       null|4792964 |---|-|K|-|----
2015-03-10 09:14:16,704|       null|4792964 |---|r|D|S|----
2015-03-10 09:14:16,705|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,705|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,714|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,714|05478900172|4792964 | 41|s|D|S|----
2015-03-10 09:14:16,760|       null|4792964 |---|r|D|S|----
2015-03-10 09:14:16,760|       null|4792964 |---|r|D|S|----
2015-03-10 09:14:23,817|       null|4792964 | 42|-|D|S|----
2015-03-10 09:14:23,817|       null|4792964 | 42|-|D|S|----
2015-03-10 09:14:23,817|       null|4792964 | 42|-|D|S|7057
2015-03-10 09:14:23,817|       null|4792964 | 42|-|D|S|----
2015-03-10 09:14:23,818|       null|4792964 | 42|-|D|S|----
2015-03-10 09:14:23,818|       null|4792964 | 42|-|D|S|----
2015-03-10 09:14:23,838|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,876|       null|4792964 |---|-|K|-|----
2015-03-10 09:14:23,876|       null|4792964 |---|-|K|-|----
2015-03-10 09:14:23,878|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,878|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,878|       null|4792964 |---|r|D|S|----
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,879|       null|4792964 |---|r|D|S|----
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|   0
2015-03-10 09:14:23,879|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,888|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:14:23,888|05478900172|4792964 | 43|s|D|S|----
2015-03-10 09:15:01,915|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,917|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,925|       null|4792761 |---|-|K|-|----
2015-03-10 09:15:01,925|       null|4792761 |---|-|K|-|----
2015-03-10 09:15:01,936|       null|4792761 |---|r|D|S|----
2015-03-10 09:15:01,936|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,936|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,945|05478900172|4792761 | 17|s|D|S|----
2015-03-10 09:15:01,946|05478900172|4792761 | 17|s|D|S|----

You need to come up with a small example dataset that illustrates your requests, and show your desired output. A picture's worth a thousand words, and all that. — glenn jackman, Commented Mar 10, 2015 at 19:25
For $2 in the input file, you want the unique values in $3 and then search the entire input for rows containing those $3s? Do you know values of $2 ahead of time? — KM., Commented Mar 10, 2015 at 20:24
yes. And thanks, you helped me realize I no longer need to include $2 on the second pass. — Steve Cohen, Commented Mar 10, 2015 at 20:58
@SteveCohen Does this work: awk -F"|" '$2 ~ 05478900172 {print $3}' input.log | sort -u | xargs -I {} grep {} input.log ... assuming you know $2s ahead of time? This will also return rows that match the $3s where $2s are missing. — KM., Commented Mar 11, 2015 at 0:36

Steve Cohen · Accepted Answer · 2015-03-11 15:07:40Z

As requested by @KM, here is my answer.

#! /bin/sh

# this script pulls all rows from a log that are directly or 
# indirectly related to a given session id.  Session IDs are stored
# in $2 of each row.  This field may be null.  Directly related
# rows are those with $2 matching the supplied parameter.  Indirectly
# related rows are those with $3 (aka xid) matching $3 in some other 
# row where $2 matches the supplied parameter.
# It may be assumed that for any rows with the same $3, 
# the $2 field will be identical or null.


SESS_SRCH="$1"
if [ -z $2 ]
then 
 LOGFILE=/path/to/default/log
else 
 LOGFILE=$2
fi

# pass 1:
# read the logfile once to find all unique XIDs associated
# with the supplied session ID ($SESS_SRCH)

XIDS=$(awk -F\| -v sessid="$1" '$2 ~ sessid { xids[$3]=0 } 
END{ 
    for (xid in xids) { 
        print xid 
    } 
}' < ${LOGFILE}
)

XID_SRCH=""

#build a search string from these xids to form a new search string.
for XID in $XIDS
do
 XID_SRCH="${XID_SRCH}|${XID}" 
done

#strip off the leading "|"
XID_SRCH=${XID_SRCH:1}

# pass 2
# read the logfile again, this time seaching on $3, for any of the
# xids found in pass 1.
awk -F\| -v search="$XID_SRCH" '$3 ~ search { print }' < ${LOGFILE}

Cosmo F · Accepted Answer · 2015-03-10 20:56:16Z

This is a bit of code snippit that should do what your asking, though it seems to me there a logical problem is the question, since the output of this loop is identical with or without the second test since the match will occur either time. I'm guessing you need looking to do a more complex test in the second run of awk than you described.

What this snippet does is first extract all lines in datafile that match field 2 with 'PATTERN' extracting field 3, then by use of sort and uniq eliminate duplicates of field 3. Which is then run over a while loop for each uniq field 3 values (4792761 or 4792964) but this time testing for both field 2 against PATTERN and field 3 against the looped values.

PATTERN="05478900172"

awk -F\| -v matchpat="$PATTERN" '$2 ~ matchpat {print $3}' | sort | uniq | while read field 
do 
   awk -F\| -v matchpat=$PATTERN -v secondpat="$field" '$2 ~ matchpat { if ( $3 ~ secondpat ) {print $0}}' datafile
done

Now I'm guessing you really want to do something more complex than you described because you could simplify this and eliminate the while loop by just sorting the output from the first awk command using field 3 as a sort key.

What I THINK you're missing is that $2 ~ matchpat should not be a condition for the inner awk since $2 could be null as well. — Steve Cohen, Commented Mar 10, 2015 at 21:42

Stack Exchange Network

How to Save variables in a script that can be shared between two runs of awk against the same input file in the script? [closed]

2 Answers 2

Not the answer you're looking for? Browse other questions tagged
shell-script
awk
.

Hot Network Questions

How to Save variables in a script that can be shared between two runs of awk against the same input file in the script? [closed]

2 Answers 2

Not the answer you're looking for? Browse other questions tagged shell-scriptawk.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
shell-script
awk
.