0

Got some valuable help here earlier with grep so hopefully I can get this sorted out too.

This is from Rclone log

Transferred:      577.080M / 577.080 MBytes, 100%, 12.660 MBytes/s, ETA 0s
Errors:                 0
Checks:                 2 / 2, 100%
Transferred:            2 / 2, 100%
Elapsed time:       45.5s

What I am trying to do is create a email notification with custom text. Something like "Transferred 577 MBytes, 2 files with 0 errors in 45.5 seconds @ 12,660 MBytes/s"

So for this to work I need to print the values. I tried the same way I did before without any luck. Transferred it two times in the log, how to split them to get TRF= 577.080Mbytes and TRS= 12.660 MBytes/s

TRF=$(grep -o 'Transferred:.*' $logfile| cut -d\   -f4)
ERR=$(grep -o 'Errors:.*' $logfile | cut -d\   -f4)
TIM=$(grep -o 'Elapsed time:.*' $logfile | cut -d\   -f3-)
TRS=$(grep -o 'Transferred:.*' $logfile | cut -d\   -f4)
2
  • Do you really have two lines that both start with Transferred: but give different information in different formats?
    – John1024
    Jul 23 '19 at 1:18
  • Yeah, unfortunately Jul 23 '19 at 7:56
0

TRF:

$ grep -Pom1 'Transferred:.*/ \K[^,]*' "$logfile"
577.080 MBytes
  • -P enable perl-compatible regular expressions and \K to not include the things before \K in the matched output
  • -o only output matched parts of the line
  • -m1 quit after the first matching line to exclude the second Transferred: line

TRS:

$ grep -Po 'Transferred:[^,]*,[^,]*, \K[^,]*' "$logfile"
12.660 MBytes/s
  • similar to TRF: get everything following the second comma + space which is not a comma
  • we don't need -m1 here as the pattern contains more than one comma

ERR:

$ grep -Po 'Errors:[[:space:]]*\K.*' "$logfile"
0
  • only get the digits at the end of the line, strip space characters

TIM:

$ grep -Po 'Elapsed time:[[:space:]]*\K.*' "$logfile"
45.5s

Number of transferred files: (it's the first of the two numbers)

$ grep -Po 'Transferred:[[:space:]]*\K[0-9]+(?= /)' "$logfile"
2
  • (?= /) is a positive lookahead (PCRE) where we're looking for a space and /. This part is like \K not included in the matched output and needed to not match the first Transferred line.
1
  • PERFECT! Thank you so much, just what I needed Jul 23 '19 at 9:59
0

This approach creates the whole output line with only a single process. Consequently, it should be much faster than using a series of grep and cut. Also, awk is quite powerful (it does floating point math, for example) which might be useful if you later do enhancements to the output.

Try:

$ awk '/ETA/{mb=$2; speed=$7" "$8} /Errors/{err=$2} /Transferred/{n=$2} /Elapsed/{printf "Transferred %s, %s files with %s errors in %s @ %s\n",mb,n,err,$3,speed}' rclone.log 
Transferred 577.080M, 2 files with 0 errors in 45.5s @ 12.660 MBytes/s,

How it works

  • /ETA/{mb=$2; speed=$7" "$8}

    If the current line contains ETA (meaning that it is the first Transferred line, then save the second field in mb and the seventh and eighth fields in speed.

  • /Errors/{err=$2}

    If this line contains Errors, then save the second field in err.

  • /Transferred/{n=$2}

    If this line contains Transferred, save the second field in variable n.

  • /Elapsed/{printf "Transferred %s, %s files with %s errors in %s @ %s\n",mb,n,err,$3,speed}'

    Finally, when we reach a line containing Elapsed, print out our message.

Multiline Version

For those who prefer their code spread over multiple lines:

awk '
/ETA/{
    mb=$2
    speed=$7" "$8
}
/Errors/{
    err=$2
}
/Transferred/{
    n=$2
}
/Elapsed/{
    printf "Transferred %s, %s files with %s errors in %s @ %s\n",mb,n,err,$3,speed
}' rclone.log 
1
  • Thank you, will test this in a script I am trying to create :) Jul 24 '19 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.