awk not working on piped output from sed

Question

I have a log file which roughly looks like this:

Sep 23 10:28:26 node kernel: em0: device is going DOWN
Sep 23 10:28:26 node kernel: em0: device is going UP
Sep 23 10:29:14 node cdsmon: /tmp/instance0 ; core dumped
Sep 23 10:29:14 node cdsmon: /tmp/instance0 ; core dumped
Sep 23 10:28:26 node kernel: em0: device is going DOWN
Sep 23 10:29:14 node cdsmon: /tmp/instance1 ; core dumped
Sep 23 10:28:26 node kernel: em0: device is going UP
Sep 23 10:29:14 node cdsmon: /tmp/instance2 ; core dumped

I want to detect the lines with cdsmon and then split the line by ; (to get the /tmp/instance0 and the event like core dumped).

For this I used sed as:

sed -u -n -e "s/^.*cdsmon: //p" /tmp/dev.log

which gives output as:

/tmp/instance0 ; core dumped
/tmp/instance0 ; core dumped
/tmp/instance1 ; core dumped
/tmp/instance2 ; core dumped

But upon piping this output to awk as shown below, it gives the same output as above:

sed -u -n -e "s/^.*cdsmon: //p" /tmp/dev.log | awk -F ";" "{print $1}"

The same is observed despite removing the -u option from sed.

Can anyone please point out if I'm missing anything? I'm using a FreeBSD box with regular awk/sed and unfortunately cannot install any new package.

Answer 1

The reason for the behavior of awk is that you have enclosed the program in double quotes, which leaves the string open to variable expansion by the shell. That means the shell from which you are running the program will first expand $1, and since that is likely undefined, it expands to the empty string.

So, your program amounts to

awk -F ";" "{print}"

and this is why the entire line is printed. This is one of the reasons you should always include your awk (and sed) programs in single quotes.

Note that in most cases you don't need to pipe the output from sed into awk or vice versa. In your example, if you want to get the first field after the "event label", you could do the following:

sed -E -n 's/^.*cdsmon: ([^;]*).*$/\1/p' /tmp/dev.log 

This will define a capture group around the string after cdsmon: and up to the first ;, and replace the entire line with the content of that capture group.

If you want to print a summary of the events logged by cdsmon, you can expand the sed approach above as:

sed -E -n 's/^.*cdsmon: ([^;]*) ; (.*)$/\1 : \2/p' dev.log 

Alternatively, here is another awk-only approach:

awk -F'(cdsmon: | ; )' 'NF==3{printf "%s : %s\n",$2,$3}' dev.log 

For your example, both will print

/tmp/instance0 : core dumped
/tmp/instance0 : core dumped
/tmp/instance1 : core dumped
/tmp/instance2 : core dumped

but be aware that the awk approach can stumble on edge cases. It takes the patterns cdsmon: and ; as field separators. When there are three fields (in your example, it can only happen for the cdsmon: entries), it prints the second and third field, corresponding to the instance name after cdsmon: and the reason after ;.

Answer 2

I'd use awk for the entire operation. Here, I'm splitting on colons, so after the date/time has been considered, the host matching must be applied to the third field (14 node cdsmon for example):

awk -F: '
    $3 ~ / cdsmon$/ {
        split($4, text, / *; */);    # Split field at semicolon
        sub(/^ */, "", text[1]);     # Remove leading space
        printf "instance %s, reason %s\n", text[1], text[2]
    }
' /tmp/dev.log

Here is an alternate and simpler solution that was suggested in a comment, where we split on either colon or semicolon so the necessary fields are already directly in awk variables:

awk -F': | *; *' '
    $1 ~ / cdsmon$/ { printf "instance %s, reason %s\n", $2, $3 }
' /tmp/dev.log

You didn't say how you wanted to extract the instance and reason (or if you did, I missed it), so I've just printed them out in a string, demonstrating that they have been correctly extracted.

Answer 3

According to the manual:

Double quotes protect most things between the opening and closing quotes. The shell does at least variable and command substitution on the quoted text. Different shells may do additional kinds of processing on double-quoted text.

Because certain characters within double-quoted text are processed by the shell, they must be escaped within the text. Of note are the characters ‘$’, ‘`’, ‘\’, and ‘"’, all of which must be preceded by a backslash within double-quoted text if they are to be passed on literally to the program.

So in your case you can escape the dollar sign $:

sed -u -n -e "s/^.*cdsmon: //p" /tmp/dev.log | awk -F ";" "{print \$1}"

But is easier just to use single quotes:

sed -u -n -e "s/^.*cdsmon: //p" /tmp/dev.log | awk -F ' ; ' '{ print $1 }'

Also you can leave spaces between the separator ' ; ' so you won't end with an invisible space after each line.

You can also just use awk:

$ awk -F': | ; ' '/cdsmon/ { print $2 }' /tmp/dev.log
/tmp/instance0
/tmp/instance0
/tmp/instance1
/tmp/instance2

Answer 4

awk '{for(i=1;i<=NF;i++){if($i ~ /cdsmon/){print $(i+1),$(i+3),$(i+4)}}}' filename

output

/tmp/instance0 core dumped
/tmp/instance0 core dumped
/tmp/instance1 core dumped
/tmp/instance2 core dumped





awk '{for(i=1;i<=NF;i++){if($i ~ /cdsmon/){print $(i+1)}}}' filename
/tmp/instance0
/tmp/instance0
/tmp/instance1
/tmp/instance2