1

I am parsing a log file. My goal is gather avg, min, max of time from it.

I am able to parse the milliseconds from each line. When I try to do math on it, I get the error: 17607 + 0 : syntax error: invalid arithmetic operator (error token is " + 0 ")

Lines from the file look like this:

./test_1_iws_1.txt:[info] 1   - 12-05 22:18:13.436 :    Finished createWalletRandom, total time 17607ms 
./test_1_iws_1.txt:[info] 1   - 12-05 22:18:19.504 :    Finished createWalletRandom, total time 21978ms 
./test_1_iws_1.txt:[info] 1   - 12-05 22:18:23.419 :    Finished createWalletRandom, total time 25815ms

and my code is this:

#!/usr/bin/env bash
LINE_COUNT=0
FILE_SLOWEST=1
FILE_LONGEST=1
FILE_SUM=0
while read ONE_LINE;
do
    RAW_MILLI_SECONDS="$(cut -d' ' -f13 <<<"$ONE_LINE")"
    echo "raw = |${RAW_MILLI_SECONDS}|"
    MILLI_SECONDS=${RAW_MILLI_SECONDS/ms/}
    LINE_COUNT=$(( LINE_COUNT + 1 ))
    echo "finished value = .${MILLI_SECONDS}."
    echo "line count = ${LINE_COUNT}"
    FILE_SUM=$(( $MILLI_SECONDS + $FILE_SUM ))
done < logfile.txt

output is this:

raw = |17607ms|
finished value = .17607.
line count = 1
get_wallet_times.sh: line 14: 17607 + 0 : syntax error: invalid arithmetic operator (error token is " + 0 ")

From the output, it sure looks like I have the milliseconds correct. So I do not understand why I get that error (above).

One additional bit of information. I am running this on MacOS, in case that makes any difference in the bash syntax.

Thnx for you help. Please let me know if there is anything else I need to add here.

EDIT:

head -1 logfile.txt | od -c
0000000    .   /   t   e   s   t   _   1   _   i   w   s   _   1   .   t
0000020    x   t   :   [ 033   [   3   2   m   i   n   f   o 033   [   3
0000040    9   m   ]       1               -       1   2   -   0   5
0000060    2   2   :   1   8   :   1   3   .   4   3   6       :      \t
0000100  033   [   3   2   m   F   i   n   i   s   h   e   d       c   r
0000120    e   a   t   e   W   a   l   l   e   t   R   a   n   d   o   m
0000140    ,       t   o   t   a   l       t   i   m   e       1   7   6
0000160    0   7   m   s 033   [   3   9   m  \n
0000172
11
  • 1
    The script runs well here. Are there any CR's in your files? – Isaac Dec 6 '19 at 0:06
  • or more generally any other invisible characters? Can you add the output of head -1 logfile.txt | od -c to the question? – icarus Dec 6 '19 at 4:49
  • After modifying one line to RAW_MILLI_SECONDS="$(<<< "$ONE_LINE" tr -s ' ' ' ' | cut -d' ' -f11)" it works well for me in Lubuntu 18.04.x LTS. Are there invisible characters in the file, that are not transmitted via your snippet in the edited question? MacOS is not linux, and some command may act in a different way from what we (linux users) expect. – sudodus Dec 6 '19 at 6:11
  • 1
    The script posted has 15 lines but the error is being reported in the line number 26. So, either try the script as posted to find the error or post the full script you are using. – Isaac Dec 6 '19 at 15:01
  • @Isaac edited output using script above. – just_a_programmer Dec 6 '19 at 15:53
1

Without solving the several special character issues:
Awk is able to process the whole file a lot faster:

#!/bin/bash
LINE_COUNT=0
FILE_SLOWEST=1
FILE_LONGEST=1

awk '{ val=$NF+0;
       if(NR==1)  {min=val;max=val}
       if(max<val){max=val}
       if(min>val){min=val}
       sum+=val
     } END {
          avg=sum/NR
          print(sum,avg,max,min)
     }' logfile.txt  | {

        read SUM AVG MAX MIN
        echo "The Sum of values           is = $SUM"
        echo "The averaged value of times is = $AVG"
        echo "The maximum of times        is = $MAX"
        echo "The minimum of times        is = $MIN"

    }

Understand that to keep the read values after the pipe has ended you need something like:

read SUM AVG MAX MIN < <(awk ...)
5
  • That would work for computing average. Would I end up with the same problem though when I want to show min/max? Your solution to use awk on one line (in the comments above) would be the solution I will use. – just_a_programmer Dec 6 '19 at 17:01
  • Are you saying that the awk script above works with your logfile? If so, awk is perfectly able to calculate max/min/avg and std if needed also. I'll edit to include those. – Isaac Dec 6 '19 at 17:04
  • yes. that solves it for me. thank you :) – just_a_programmer Dec 6 '19 at 17:05
  • Do that solve your problem? @just_a_programmer – Isaac Dec 6 '19 at 17:24
  • yes. thank you. I am going to have to learn about awk. very powerful. much appreciated. – just_a_programmer Dec 6 '19 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.