2

Not being a perl programmer I'm thinking this would be fairly easy with --tagstring, but basically I'd like to time-stamp each line of output from each job individually from Parallel. Something like, with the right replacement for “STUFF”, the output may look like, assuming millisecond resolution (although nanosecond resolution would be nice too):

$ seq 8 | parallel --tags 'sequence {} {=STUFF=}' -j2 'sleep=$((1 + RANDOM % 2)); echo sleeping $sleep; sleep $sleep; echo done; echo $sleep {#} {%} {}'
sequence 1 0.001   sleeping 1
sequence 1 1.001   done
sequence 1 1.002   1 1 1 1
sequence 2 0.001   sleeping 2
sequence 2 2.001   done
sequence 2 2.002   2 2 2 2
sequence 3 0.001   sleeping 2
sequence 3 2.001   done
sequence 3 2.002   2 3 1 3
sequence 5 0.001   sleeping 1
sequence 5 1.001   done
sequence 5 1.002   1 5 1 5
sequence 4 0.001   sleeping 2
sequence 4 2.001   done
sequence 4 2.002   2 4 2 4
sequence 6 0.001   sleeping 1
sequence 6 1.001   done
sequence 6 1.002   1 6 1 6
sequence 7 0.001   sleeping 2
sequence 7 2.001   done
sequence 7 2.002   2 7 2 7
sequence 8 0.001   sleeping 2
sequence 8 2.001   done
sequence 8 2.002   2 8 1 8

1 Answer 1

2

You would deceptively think that is easy, and I can't blame you for that.

It is, however, not possible for normal output.

This is because the tagstring is only computed twice, and it is only added after the job is completed.

GNU Parallel runs:

job1 > tmpout1 2> tmperr1
job2 > tmpout2 2> tmperr2
job3 > tmpout3 2> tmperr3

(This is of course not 100% true, but it is close enough).

When the job is done, GNU Parallel reads the tmp* files in nice big chunks, prepends --tagstring and outputs the job.

The important part here is: Tagging is not done while running. And --tagstring is only computed twice: before the job starts and after the job finishes (and it is the final result that will be added).

This design is selected because it is CPU consuming to compute the tagstring, and if your output is 3600000 lines, then even a delay of 1 ms/line would be 1 hour(!) of waiting.

There is, however, one exception: --line-buffer.

--line-buffer does compute the tagstring for every line of output. This design is a chosen because --line-buffer already takes up more CPU time (having to poll for new data from each running job and not being able to only deal with large chunks of data).

So this works:

$ seq 8 | parallel --lb --tagstring 'sequence {} {= $start{$job}||=::now(); $_=sprintf"%06.3f",::now()-$start{$job} =}' -j2 'sleep 1; echo Begin {}; sleep 0.{}; echo End {}'|sort
sequence 1 01.027       Begin 1
sequence 1 01.116       End 1
sequence 2 01.024       Begin 2
sequence 2 01.216       End 2
sequence 3 01.098       Begin 3
sequence 3 01.312       End 3
sequence 4 01.049       Begin 4
sequence 4 01.411       End 4
sequence 5 01.031       Begin 5
sequence 5 01.509       End 5
sequence 6 01.039       Begin 6
sequence 6 01.613       End 6
sequence 7 01.048       Begin 7
sequence 7 01.711       End 7
sequence 8 01.071       Begin 8
sequence 8 01.811       End 8
1
  • Aha! That's very useful to know about tagstring, and line-buffer. Thanks Ole!
    – Larry
    May 26, 2020 at 17:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .