3

The headline a generalization of what I want. Specific problem: given some command that outputs multiple timestamps, e.g:

$ cat timestamps | sort -n
1508349271820
1508349271821
1508349425222
1508349425223
1508349454218
1508349476419
1508349500018
1508349500020
1508349698820
1508349698822
1508350047721
1508350047724
1508351635621
1508351635623
1508351699618
1508351699620
1508351699621
1508351699622
1508351699623
1508352230120
1508352230123
1508352230124
1508352230125
1508352232219
1508352232220
1508352364919
1508352364920
1508352387618
1508352387619

I want to compute each 2 diff. I ended up doing something like:

$ wc -l timestamps
29

cat <(sort -n timestamps | head -28) <(sort -n timestamps | tail -28) | sort -n | xargs -n 2 sh 'calc $2 - $1' sh
1
153401
1
28995
22201
23599
2
198800
2
348899
3
1587897
2
63995
2
1
1
1
530497
3
1
1
2094
1
132699
1
22698
1

So I managed to get by, but there's must be an easier way. The generalization is: given an output with multiple lines, how can I compute a sliding window of x args at once, with y args step size?

  • 1
    The specific example is clear, but the generalization is not. Please clarify what a "sliding window" is. – agc Mar 7 '18 at 6:07
7

Awk is well suited for this:

awk 'NR>1{print $1-last} {last=$1}' timestamps

In the above, for each line after the first (NR>1), we print the value on the current value, $1, minus the value on the previous line, last. Next, we update the value of last.

Example

$ awk 'NR>1{print $1-last} {last=$1}' timestamps
1
153401
1
28995
22201
23599
2
198800
2
348899
3
1587897
2
63995
2
1
1
1
530497
3
1
1
2094
1
132699
1
22698
1

More complex calculation

The code below starts with the number in the current line, adds twice the number in the preceding line, and then subtracts three times the number on the line five lines previous:

awk '{a[NR]=$1} NR>5{print a[NR]+2*a[NR-1]-3*a[NR-5]}' timestamps
  • Thanks. I'm not an awk expert, so how do I use an external command, like what I did with calc using xargs? Moreover, this seems like something that is hard to generalize for every number of args and step size. What if I wanted 3 args, with 1 argument shared each time. Like 1,2,3 and then 3,4,5 etc'... Its great for my specific usecase, but I am interested in a more general solution – gilad hoch Mar 7 '18 at 5:39
  • 4
    @giladhoch As best I can tell from your description (I am not, for example, at all sure what "1 argument shared" really means), awk would be good for all those things. Separately, if the question that you posed here is not the one that you want answered, you might ask another question that is aimed more directly at what you want. – John1024 Mar 7 '18 at 5:48
  • 1
    also, why would you want to use a long, complicated, and fragile pipeline of commands when you can do the same task far easier in a scripting language like awk (or perl or python, etc). – cas Mar 7 '18 at 5:49
  • 1
    I think what you are meaning by "sliding window" can be achieved using the modulo operator (%) instead of just subtracting the previous line from the current line. e.g. use NR % 3 to check for every third line. I'm not sure exactly what you want to do with every group of 3 lines so I can't give a more useful example. – cas Mar 7 '18 at 6:07
  • 1
    Thanks @John1024 & cas . This does seem to answer my needs :) – gilad hoch Mar 7 '18 at 6:13
5

Install num-utils, then pipe the file or data to numinterval:

sort -n timestamps | numinterval
  • Thats a nifty util. thanks for the pointer :) – gilad hoch Mar 7 '18 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.