I have a script with a pipeline where the data is throttled by a program. Every 20 minutes, a throughput status is sent on stderr, and I simply redirect this to a file:
command | cstream -t -512k -T 1200 -B 64m 2>>/home/user/totals.log | command
I could also send a SIGUSR1 signal to cstream (
kill -s 10 PID) and it would produce exactly the same output as when I set an interval like I did, and a line is added to the file. The idea is that you can set the interval and also query a status on demand. The output is formatted like this with a single whitespace character all over and terminated with a newline:
... 1931255808 B 1.8 GB 3683.6 s (1:01 h) 524289 B/s 512.00 KB/s 2560401408 B 2.4 GB 4883.6 s (1:21 h) 524289 B/s 512.00 KB/s 3189547008 B 3.0 GB 6083.6 s (1:41 h) 524289 B/s 512.00 KB/s 3818692608 B 3.6 GB 7283.6 s (2:01 h) 524289 B/s 512.00 KB/s 4447838208 B 4.1 GB 8483.6 s (2:21 h) 524289 B/s 512.00 KB/s 10829824 B 10.3 MB 20.65 s 524487 B/s 512.19 KB/s
You can see the data progress but you couldn't just sum(or average) each column. There are just 2 events in this sample(lines 1-5, 6) i.e. I used the script twice. It just so happens here that we could add the two last lines - because I restarted it and we have a snapshot of that moment but that's arbitrary. Which leads to what happens when I exit the command and restart it, as you can see happening on the last line - the counter is of course reset.
The objective is to have a cumulative total of the status output for all the instances of the script I launch over time. I usually never launch more than one instance of it at a time. So I'm thinking of adding to my script("sane" for one instance of the script only):
- First delete all but the last line of my log
- Last, when the script exits, make it write its status to the log
- Sum/avg each column of the first and last line
- Delete all intermediary lines
I'm afraid I can't connect the dots. What is the simplest and best way to design and implement what I want? Does the complexity overwhelm the benefit and should I just focus on manipulating the log data with a simple command as it fills up?