Shell script to filter string from a data stream and output to screen

Question

I am trying to make it easy to tail a log file and print to the screen only the output of a single field.

I have the log file that looks like this on a Linux machine:

2022-10-21 16:00:08;areq_in=0;areq_qavg=0;areq_qmax=0;areq_sent=0;ares_out=0;ares_out_err=0;ares_out_ok=0;ares_qavg=0;ares_qmax=0;ares
_recv=0;ares_tavg=0;ares_tmax=0;creq_out=0;creq_qavg=0;creq_qmax=0;creq_recv=0;cres_in=0;cres_qavg=0;cres_qmax=0;cres_sent=0;lic_rej=0
;lic_use=0;mbr_from=0;mbr_to=0;mreject=0;mreject_conn=0;msys_recv=0;msys_sent=0;

Each line in the log consists of ;-separated fields, most of which are in the "key=value" form.

So what I am trying to do is only print the field whose "key" name is creq_recv on the screen.

I took this script from someone's fix but I think I am missing something.

#!/bin/bash
creq_recv=$1
my_variable=`tail -f stats_2022102116.log | awk -F: -v creq_recv="$1" '$1 == creq_recv {$1=$1; print}' stats_2022102116.log`
echo "$my_variable"

Because the output is empty

[root@priti ]$ ./chk_conn_log.sh
    
[root@priti ]$ 

The output should print on screen whatever value creq_recv is. For example, like below

creq_recv=12
creq_recv=34
creq_recv=65

Answer 1

You can do what you want just with grep with the -o option to only print the matching portion of the line and -P or -E for PCRE or extended regular expressions respectively, which let us use [^;]+ for "one or more non-; characters":

tail -f stats_2022102116.log | grep -oP '(?<=^|;)creq_recv=[^;]+'

Or, safer in case we can have other field names that end with creq_recv:

tail -f stats_2022102116.log | grep -oP '(^|;)\Kcreq_recv=[^;]+'

Your awk was failing because -v creq_recv="$1" means "set the value of the awk variable creq_recv to the value of the first argument passed to the script". But since you are launching the script with no arguments (./chk_conn_log.sh), $1 is empty, so creq_recv is also empty.

In any case, that wouldn't have worked, even if the variable were being set. Both because the awk script was wrong and because tail -f will never exit unless you stop it, so the echo wouldn't be executed. You wanted something like this:

#!/bin/bash

tail -f stats_2022102116.log | 
    awk -F';'  '{ for(i=1; i<=NF; i++){ if ($i ~ /^creq_recv=/) { print $i } } }'

No echo, nothing but the tail and the corrected awk script. But just use the grep command above, it's much simpler.

Answer 2

The solution you tried contains several problems that lead to the undesired behavior you experience.

1. Your script expects a command-line argument ($1 at the "bash" level) that you don't seem to specify when calling it as ./chk_conn_log.sh
2. You script copies that (unspecified = empty) value to a shell variable creq_recv. So, creq_recv as far as your shell is concerned, is an empty variable.
3. You want to filter the ongoing output of a process via tail, but assign the result to a shell variable that you later print with echo. This will not work, because the variable will only ever be filled when both processes terminate. However, if the logfile is not written to anymore, tail will simply wait for new input - so it will never terminate, and your script would literally be stuck there, however,
4. Although you want to pipe the output of tail to awk, you also explicitly state the log file as argument, so awk will simply run over the content of the file as it looks like at the moment the script is called, where it likely doesn't yet contain any lines, and then terminate. So your variable my_variable is empty, and your script's output, too.

Even so,

5. You say (and the example shows) that the log file is ;-separated, but you instruct awk to use : as separator.
6. You set a awk variable creq_recv to the (unspecified) command-line argument, and inside the awk program check whether the first :-separated field is equal to the content of the awk variable creq_recv. However, that variable is empty (see above), and the "field" will contain the string 2022-10-21 16 (the date/time up to the first :), so for many reasons the condition will never be met, meaning that even if the log file already contained lines when the script was called, the output would still be empty.
7. You would print the entire line, not only the field that says creq_recv=xx.

---

That said, what I think you want to achieve can be done as follows:

tail -f stats_2022102116.log | awk -F';' '{for (i=1;i<=NF;i++) {if (index($i,"creq_recv=")==1) print $i}}'

• This will set the field separator to ;.
• It will then, for every (incoming) input line, iterate over all such fields, and if the field starts with the string creq_recv=, print that field.

Note that I chose to use a literal string comparison instead of a regular expression match, because (in the general case) that allows you to look for strings that contain characters with special meaning in regular expressions without the hassle of escaping them. If you need regular expression matching (which is not the case here, according to your post), you can change

if (index($i,"creq_recv=")==1)

to

if ($i ~ /your regular expression/)

Answer 3

Using Raku (formerly known as Perl_6)

#recover all key/value pairs:

~$ cat file | raku -e 'my @a = lines>>.split(";", :skip-empty)>>.[1..*].flat;  \
                       my %hash;  for @a>>.split("=") {%hash.=append: $_};  \
                      .say for %hash.sort;'  

#recover values for specific key:

~$ cat file | raku -e 'my @a = lines>>.split(";", :skip-empty)>>.[1..*].flat; \
                       my %hash;  for @a>>.split("=") {%hash.=append: $_}; \
                       say %hash<creq_recv>;' 

Above is an answer coded in Raku, a Perl-family programming language. The question practically begs for a key-value solution, and Raku doesn't disappoint.

For the above two code examples, they are identical except for the last statement. Briefly, lines of the file are read in, which has the effect of chomping off trailing \n newlines by default. Then this input is split on ; semicolons to produce element that are [1..*] indexed to remove the first (time/date) element, flattened, and stored in the @a array. Then the elements of @a are are >> individually split on = equal-sign. These resulting elements are appended en mass to the %hash. Raku when given input in this manner makes a key/value pair out of every successive two elements.

In the first code example, all keys/value pairs are returned. In the second code example, only the values associated with the creq_recv key are returned.

Sample Input:

2022-10-21 16:00:08;areq_in=0;areq_qavg=0;areq_qmax=0;areq_sent=0;ares_out=0;ares_out_err=0;ares_out_ok=0;ares_qavg=0;ares_qmax=0;ares_recv=0;ares_tavg=0;ares_tmax=0;creq_out=0;creq_qavg=0;creq_qmax=0;creq_recv=0;cres_in=0;cres_qavg=0;cres_qmax=0;cres_sent=0;lic_rej=0;lic_use=0;mbr_from=0;mbr_to=0;mreject=0;mreject_conn=0;msys_recv=0;msys_sent=0;

Sample Output for first code example (second code example returns 0):

areq_in => 0
areq_qavg => 0
areq_qmax => 0
areq_sent => 0
ares_out => 0
ares_out_err => 0
ares_out_ok => 0
ares_qavg => 0
ares_qmax => 0
ares_recv => 0
ares_tavg => 0
ares_tmax => 0
creq_out => 0
creq_qavg => 0
creq_qmax => 0
creq_recv => 0
cres_in => 0
cres_qavg => 0
cres_qmax => 0
cres_sent => 0
lic_rej => 0
lic_use => 0
mbr_from => 0
mbr_to => 0
mreject => 0
mreject_conn => 0
msys_recv => 0
msys_sent => 0

For a more informative return, use Raku's => pair-constructor, changing the last statement to:

$_ = "creq_recv" andthen say $_ => %hash{$_};

...which returns:

creq_recv => 0

Finally, Raku implements react blocks, which can designed to receive a Supply (lets say, for example, $*IN.lines lines-received-via-StdIN), and whenever received, run code. Like so:

~$ tail -f test.log | raku -e 'react {whenever Supply( $*IN.lines ) {   \
                               my @a = $_.map( *.split(";", :skip-empty).[1..*] ).flat;  \
                               my %hash; for @a.map: *.split("=") { %hash.=append: $_ };  \
                               $_ = "creq_recv" andthen say $_ => %hash{$_} // Nil; } };'
creq_recv => 0
creq_recv => 0
^C

https://docs.raku.org/language/concurrency#react
https://docs.raku.org/language/hashmap
https://docs.raku.org/routine/=%3E
https://raku.org

PS: There's an absolutely stellar chapter on "Associatives" (i.e. Hashes and Maps) in brian d foy's book Learning Perl6.