2

I have a script with a pipeline where the data is throttled by a program. Every 20 minutes, a throughput status is sent on stderr, and I simply redirect this to a file:

command | cstream -t -512k -T 1200 -B 64m 2>>/home/user/totals.log | command

I could also send a SIGUSR1 signal to cstream (kill -s 10 PID) and it would produce exactly the same output as when I set an interval like I did, and a line is added to the file. The idea is that you can set the interval and also query a status on demand. The output is formatted like this with a single whitespace character all over and terminated with a newline:

...
1931255808 B 1.8 GB 3683.6 s (1:01 h) 524289 B/s 512.00 KB/s
2560401408 B 2.4 GB 4883.6 s (1:21 h) 524289 B/s 512.00 KB/s
3189547008 B 3.0 GB 6083.6 s (1:41 h) 524289 B/s 512.00 KB/s
3818692608 B 3.6 GB 7283.6 s (2:01 h) 524289 B/s 512.00 KB/s
4447838208 B 4.1 GB 8483.6 s (2:21 h) 524289 B/s 512.00 KB/s
10829824 B 10.3 MB 20.65 s 524487 B/s 512.19 KB/s

You can see the data progress but you couldn't just sum(or average) each column. There are just 2 events in this sample(lines 1-5, 6) i.e. I used the script twice. It just so happens here that we could add the two last lines - because I restarted it and we have a snapshot of that moment but that's arbitrary. Which leads to what happens when I exit the command and restart it, as you can see happening on the last line - the counter is of course reset.

The objective is to have a cumulative total of the status output for all the instances of the script I launch over time. I usually never launch more than one instance of it at a time. So I'm thinking of adding to my script("sane" for one instance of the script only):

  • First delete all but the last line of my log
  • Last, when the script exits, make it write its status to the log
    • Sum/avg each column of the first and last line
    • Delete all intermediary lines

I'm afraid I can't connect the dots. What is the simplest and best way to design and implement what I want? Does the complexity overwhelm the benefit and should I just focus on manipulating the log data with a simple command as it fills up?

1

Well, based on what you show, I did manage to split it up pretty reliably, but there's a serious problem with this data: it's not normal. You've got human-friendly values here - that's no good. For instance the MB and GB differences between the first and last lines - handling that is a lot of extra work you shouldn't have to do - why not just byte counts? And what's up with the ([h]:[mm]) thing - why is it in the first line and not the last line and why not Unix time?

Honestly, this isn't the kind of data you should be logging at all - it isn't very useful. Sure, it's easier for you to read, but are you going to read 10,000 lines of it? I think you'd rather not and that's why you've asked this question. You need to alter that output - get no letters at all, only byte counts and seconds since the epoch. Do that, and this will be a lot easier on you.

Now, that said, here's what I did do:

set -- $(
sed '$bl;1H;d;:l;x;G
     s/([1-9][^)]*) //;h
     s/\n/First:&       /
     s/[^:]\(\n\)/&Last:\1      /
     w /dev/fd/2
     g' <<\DATA
1931255808 B 1.8 GB 3683.6 s (1:01 h) 524289 B/s 512.00 KB/s
2560401408 B 2.4 GB 4883.6 s (1:21 h) 524289 B/s 512.00 KB/s
3189547008 B 3.0 GB 6083.6 s (1:41 h) 524289 B/s 512.00 KB/s
3818692608 B 3.6 GB 7283.6 s (2:01 h) 524289 B/s 512.00 KB/s
4447838208 B 4.1 GB 8483.6 s (2:21 h) 524289 B/s 512.00 KB/s
10829824 B 10.3 MB 20.65 s 524487 B/s 512.19 KB/s
DATA
)

That first sed line is all it takes to get just your first and last lines and put them in the same pattern space for sed editing, each of them preceded by a \newline character. This statement does all of that:

$bl;1H;d;:l;x;G

The next line clears that funky time thing out the data - which is part of the problem - then stores an extra copy of the results in hold space:

s/([1-9][^)]*) //;h

The next three lines insert the words First: and Last: then \newline and <tab> characters before their respective lines and write the results to stderr:

 s/\n/First:&       /
 s/[^:]\(\n\)/&Last:\1      /
 w /dev/fd/2

The last sed line just gets that second copy out of hold space and overwrites the current pattern space with it, which sed then follows up with its default action of printing the final pattern space followed by a \newline. The results, at this point, are, admittedly, not that impressive. Running the above bit of script will only output the following:

First:
        1931255808 B 1.8 GB 3683.6 s 524289 B/s 512.00 KB/s
Last:
        10829824 B 10.3 MB 20.65 s 524487 B/s 512.19 KB/s

But I intentionally set the results into the shell array and kept both lines accessible in sed's pattern space for a reason. For instance, following that last g line in sed - if you wanted to - you could work with a pattern space that looks like this:

\n1931255808 B 1.8 GB 3683.6 s 524289 B/s 512.00 KB/s\n10829824 B 10.3 MB 20.65 s 524487 B/s 512.19 KB/s$ 

Or, if you left it as is and only appended the following to what's already there...

printf '%s LINE, FIELDs 1 and 2: %s and %s' \
    FIRST "$1" "$2" LAST "${11}" "${12}"

Your output should look like:

FIRST LINE, FIELDs 1 and 2: 1931255808 and B
LAST LINE, FIELDs 1 and 2: 10829824 and B

That's in addition to the stderr output it already provides.

  • Thank you! This is really insightful. It is very good advice to really analyze what that output is made of. I have to rethink this maybe. For instance removing my timer and always querying with kill. I could put this in a little script or alias, call this and it would send the kill and sanitize the log and update it. It's that sort of thinking that I miss as I'm very much a beginner. Then later on as I will try to build totals I will carefully analyze your sed processing but your general explanation is quite clear and helpful! +1 – user44370 Jun 9 '14 at 4:13
  • @illuminÉ - Absolutely - if you can get me a little different input, I can show you how to easily get all of your averages, totals, whatever with some very simple 3 or 4 character functions and either dc or bc. – mikeserv Jun 9 '14 at 4:16
  • @illuminÉ - oh, but the kill thing is an excellent idea, by the way. Try to mimic what dd does with SIGUSR1 in a trap in your function - or even filter your output through dd for exact byte counts and kill SIGUSR1 it from cron in addition to some date output. – mikeserv Jun 9 '14 at 4:25
  • Thank you! Give it a bit of time. Your answer stands. The output frequency etc. will depend on how I refactor how I trigger the writes. Maybe 1-when the script launches, 2- when the script exits, 3- when the kill alias/script is manually triggered. You've put me on the right track! fyi this is the heirloom script - just cstream is in between the random generator and tr. – user44370 Jun 9 '14 at 4:35
  • @illuminÉ - I'm glad I could help. And thank you for the feedback - good luck. – mikeserv Jun 9 '14 at 4:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy