1

I am trying to download multiple files parallelly in bash and I came across GNU parallel. It looks very simple and straight forward. But I am having a hard time getting GNU parallel working. What am I doing wrong? Any pointers are appreciated. As you can see the output is very sequential and I expect output to be different each time. I saw a similar question in SO (GNU parallel not working at all) but that solutions mention there did not work for me.

svarkey@svarkey-Precision-5510:~$ seq 1 3 | xargs -I{} -n 1 -P 4 kubectl version --short=true --context cs-prod{} --v=6
I0904 11:33:10.635636   24861 loader.go:375] Config loaded from file:  /home/svarkey/.kube/config
I0904 11:33:10.640718   24863 loader.go:375] Config loaded from file:  /home/svarkey/.kube/config
I0904 11:33:10.640806   24862 loader.go:375] Config loaded from file:  /home/svarkey/.kube/config
I0904 11:33:11.727974   24863 round_trippers.go:443] GET https://kube-api.awsw3.cld.dtvops.net/version?timeout=32s 200 OK in 1086 milliseconds
Client Version: v1.18.7
Server Version: v1.14.6
I0904 11:33:11.741985   24861 round_trippers.go:443] GET https://kube-api.awsw1.cld.dtvops.net/version?timeout=32s 200 OK in 1105 milliseconds
Client Version: v1.18.7
Server Version: v1.14.6
I0904 11:33:11.859882   24862 round_trippers.go:443] GET https://kube-api.awsw2.cld.dtvops.net/version?timeout=32s 200 OK in 1218 milliseconds
Client Version: v1.18.7
Server Version: v1.14.6
svarkey@svarkey-Precision-5510:~$ seq 1 3 | parallel -j 4 -I{} kubectl version --short=true --context cs-prod{} --v=6
Client Version: v1.18.7
Server Version: v1.14.6
I0904 11:33:18.584076   24923 loader.go:375] Config loaded from file:  /home/svarkey/.kube/config
I0904 11:33:19.662197   24923 round_trippers.go:443] GET https://kube-api.awsw1.cld.dtvops.net/version?timeout=32s 200 OK in 1077 milliseconds
Client Version: v1.18.7
Server Version: v1.14.6
I0904 11:33:18.591033   24928 loader.go:375] Config loaded from file:  /home/svarkey/.kube/config
I0904 11:33:19.691343   24928 round_trippers.go:443] GET https://kube-api.awsw3.cld.dtvops.net/version?timeout=32s 200 OK in 1099 milliseconds
Client Version: v1.18.7
Server Version: v1.14.6
I0904 11:33:18.591033   24924 loader.go:375] Config loaded from file:  /home/svarkey/.kube/config
I0904 11:33:19.775152   24924 round_trippers.go:443] GET https://kube-api.awsw2.cld.dtvops.net/version?timeout=32s 200 OK in 1183 milliseconds
svarkey@svarkey-Precision-5510:/tmp/parallel-20200822$ parallel --version
GNU parallel 20200822
Copyright (C) 2007-2020 Ole Tange, http://ole.tange.dk and Free Software
Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: https://www.gnu.org/software/parallel
6
  • Seems okay to me.... what do you expect, and especially ... what is your real world problem ? echo calls are probably just pretty fast ...
    – pLumo
    Sep 4, 2020 at 10:17
  • @pLumo I think I am doing something wrong but cant figure out I updated my question and I expect similar behavior from those2 commands Sep 4, 2020 at 10:57
  • parallel output is sequential because it captures the processes output and prints it only when that process is finished, unlike xargs which let the processes print output immediately.
    – pLumo
    Sep 4, 2020 at 11:00
  • ok ok I see but it is actually doing stuff in parallel. right? Sep 4, 2020 at 11:01
  • sure it does, you can check that using processes you know that take some time, e.g. sleep.
    – pLumo
    Sep 4, 2020 at 11:13

1 Answer 1

3

parallel output is sequential because it captures the processes output and prints it only when that process is finished, unlike xargs which let the processes print the output immediately.

From man parallel

   GNU parallel makes sure output from the commands is the same output as
   you would get had you run the commands sequentially. This makes it
   possible to use output from GNU parallel as input for other programs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.