6

I know that GNU Parallel buffers std/stderr because it doesn't want jobs output to be mangled, but if I run my jobs with parallel do_something ::: task_1 task_2 task_3, is there anyway for task_1's output to be displayed immediately, then after task_1 finishes, task_2's up to its current output, etc.

If Parallel cannot solve this problem, is there any other similar program that could?

8

From version 20160422 you can do:

parallel -k --lb do_something ::: task_1 task_2 task_3
  • I've just installed it and it works exactly how I wanted. Thank you very much! – Hai Luong Dong Apr 22 '16 at 19:04
  • There are --ungroup as well if you don't care about the order at all. – zyxue Sep 21 '17 at 16:53
  • From version 20170822 --lb is not much slower than --ungroup. – Ole Tange Sep 22 '17 at 6:20
2

parallel actually makes no reservations about making output come in order. It just so happens that jobs are typically small enough and evenly split in cpu time to come out in order. You notice it more when you have a lot of jobs running or their task duration differs a lot.

By default parallel will only run the same amount of jobs as your CPU has cores. On most laptops and desktops that's 2-4 which means it's only running a couple of jobs at a time. You can increase that with -j.

Here's an example to demonstrate job order does not output in the order it was submitted.

seq 20 | parallel -j 20 'sleep $[RANDOM % 20]; echo '

output on my system was (yours will likely be different)

7
3
13
20
8
16
2
4
18
17
1
5
9
14
12
6
10
19
11
15

seq 20 is a command that will output numbers 1-20. I pipe that to parallel and then tell it to run 20 simultaneous jobs to make sure they all start at the same time. 'sleep $[\[RANDOM][1] % 20]; is using sleep plus a zsh parameter that will return a random number between 1 and 20. Each job will sleep this random amount and then echo. Once the job echos you'll immediately get the output from parallel.

You could also do something similar with parallel --shuf which will shuffle the job order.

  • 1
    Actually, it can with -k/--keep-order (I used parallel version 20141022) seq 20 | parallel -k -j2000% 'sleep $[RANDOM % 20]; echo ' 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 – Hai Luong Dong Apr 21 '16 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.