5

I have a log file which roughly looks like this:

Sep 23 10:28:26 node kernel: em0: device is going DOWN
Sep 23 10:28:26 node kernel: em0: device is going UP
Sep 23 10:29:14 node cdsmon: /tmp/instance0 ; core dumped
Sep 23 10:29:14 node cdsmon: /tmp/instance0 ; core dumped
Sep 23 10:28:26 node kernel: em0: device is going DOWN
Sep 23 10:29:14 node cdsmon: /tmp/instance1 ; core dumped
Sep 23 10:28:26 node kernel: em0: device is going UP
Sep 23 10:29:14 node cdsmon: /tmp/instance2 ; core dumped

I want to detect the lines with cdsmon and then split the line by ; (to get the /tmp/instance0 and the event like core dumped).

For this I used sed as:

sed -u -n -e "s/^.*cdsmon: //p" /tmp/dev.log

which gives output as:

/tmp/instance0 ; core dumped
/tmp/instance0 ; core dumped
/tmp/instance1 ; core dumped
/tmp/instance2 ; core dumped

But upon piping this output to awk as shown below, it gives the same output as above:

sed -u -n -e "s/^.*cdsmon: //p" /tmp/dev.log | awk -F ";" "{print $1}"

The same is observed despite removing the -u option from sed.

Can anyone please point out if I'm missing anything? I'm using a FreeBSD box with regular awk/sed and unfortunately cannot install any new package.

3
  • 9
    You never need sed when you're using awk. Any time you find yourself piping any combination of sed, grep, and/or awk to each other you can be sure you could do the whole thing with just awk.
    – Ed Morton
    Sep 23, 2021 at 12:54
  • 1
    @EdMorton: In most cases, sure. But that can't beat sed oneliners.
    – s.ouchene
    Sep 24, 2021 at 9:49
  • 1
    @s.ouchene if you have sed piped to grep, etc. as I described then it's not a sed oneliner. Also, in my 40+ years of experience using sed and as one of the top answerers of sed questions on SO (see stackoverflow.com/tags/sed/topusers), in many cases other than a simple s/old/new/ with no variables in the search or replacement sections you can easily beat a sed oneliner for robustness, portability, efficiency, clarity, etc. so YMMV.
    – Ed Morton
    Sep 24, 2021 at 13:24

4 Answers 4

20

The reason for the behavior of awk is that you have enclosed the program in double quotes, which leaves the string open to variable expansion by the shell. That means the shell from which you are running the program will first expand $1, and since that is likely undefined, it expands to the empty string.

So, your program amounts to

awk -F ";" "{print}"

and this is why the entire line is printed. This is one of the reasons you should always include your awk (and sed) programs in single quotes.

Note that in most cases you don't need to pipe the output from sed into awk or vice versa. In your example, if you want to get the first field after the "event label", you could do the following:

sed -E -n 's/^.*cdsmon: ([^;]*).*$/\1/p' /tmp/dev.log 

This will define a capture group around the string after cdsmon: and up to the first ;, and replace the entire line with the content of that capture group.

If you want to print a summary of the events logged by cdsmon, you can expand the sed approach above as:

sed -E -n 's/^.*cdsmon: ([^;]*) ; (.*)$/\1 : \2/p' dev.log 

Alternatively, here is another awk-only approach:

awk -F'(cdsmon: | ; )' 'NF==3{printf "%s : %s\n",$2,$3}' dev.log 

For your example, both will print

/tmp/instance0 : core dumped
/tmp/instance0 : core dumped
/tmp/instance1 : core dumped
/tmp/instance2 : core dumped

but be aware that the awk approach can stumble on edge cases. It takes the patterns cdsmon: and ; as field separators. When there are three fields (in your example, it can only happen for the cdsmon: entries), it prints the second and third field, corresponding to the instance name after cdsmon: and the reason after ;.

0
8

I'd use awk for the entire operation. Here, I'm splitting on colons, so after the date/time has been considered, the host matching must be applied to the third field (14 node cdsmon for example):

awk -F: '
    $3 ~ / cdsmon$/ {
        split($4, text, / *; */);    # Split field at semicolon
        sub(/^ */, "", text[1]);     # Remove leading space
        printf "instance %s, reason %s\n", text[1], text[2]
    }
' /tmp/dev.log

Here is an alternate and simpler solution that was suggested in a comment, where we split on either colon or semicolon so the necessary fields are already directly in awk variables:

awk -F': | *; *' '
    $1 ~ / cdsmon$/ { printf "instance %s, reason %s\n", $2, $3 }
' /tmp/dev.log

You didn't say how you wanted to extract the instance and reason (or if you did, I missed it), so I've just printed them out in a string, demonstrating that they have been correctly extracted.

5
  • Thank you for this awk only example. Splitting by : might not be reliable for my case. I believe using /cdsmon/ and rest of your program will help. Sep 23, 2021 at 11:36
  • 2
    Isn't it easier to use -F': | ; ' as field separator? Sep 23, 2021 at 11:56
  • 1
    @schrodigerscatcuriosity that could work too. I tend to forget that FS can be a multicharacter ERE
    – roaima
    Sep 23, 2021 at 12:00
  • Could even use ' cdsmon: ' as a separator (discard any line that doesn't split). Sep 24, 2021 at 7:52
  • @TobySpeight there are lots of possibilities. I think that one would end up with me having to revert to the complexity of the first option
    – roaima
    Sep 24, 2021 at 7:55
8

According to the manual:

Double quotes protect most things between the opening and closing quotes. The shell does at least variable and command substitution on the quoted text. Different shells may do additional kinds of processing on double-quoted text.

Because certain characters within double-quoted text are processed by the shell, they must be escaped within the text. Of note are the characters ‘$’, ‘`’, ‘\’, and ‘"’, all of which must be preceded by a backslash within double-quoted text if they are to be passed on literally to the program.

So in your case you can escape the dollar sign $:

sed -u -n -e "s/^.*cdsmon: //p" /tmp/dev.log | awk -F ";" "{print \$1}"

But is easier just to use single quotes:

sed -u -n -e "s/^.*cdsmon: //p" /tmp/dev.log | awk -F ' ; ' '{ print $1 }'

Also you can leave spaces between the separator ' ; ' so you won't end with an invisible space after each line.

You can also just use awk:

$ awk -F': | ; ' '/cdsmon/ { print $2 }' /tmp/dev.log
/tmp/instance0
/tmp/instance0
/tmp/instance1
/tmp/instance2
1
  • Thank you! I was using zsh and it was still treating $1 as an input, this suggestion helps. Sep 23, 2021 at 11:32
1
awk '{for(i=1;i<=NF;i++){if($i ~ /cdsmon/){print $(i+1),$(i+3),$(i+4)}}}' filename

output

/tmp/instance0 core dumped
/tmp/instance0 core dumped
/tmp/instance1 core dumped
/tmp/instance2 core dumped





awk '{for(i=1;i<=NF;i++){if($i ~ /cdsmon/){print $(i+1)}}}' filename
/tmp/instance0
/tmp/instance0
/tmp/instance1
/tmp/instance2
1
  • Please note that your first approach only works if the "reason" part of the cdsmon log message consists of exactly two words.
    – AdminBee
    Sep 24, 2021 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.