1

I'm learning GNU parallel and tried the following:

$ for i in {1.txt,2.txt}; do time wc -l $i; done
100 1.txt

real    0m0.010s
user    0m0.000s
sys     0m0.010s
10000012 2.txt

real    0m0.069s
user    0m0.050s
sys     0m0.018s

I then reran the above command with parallel, but it slowed things down. Why?

$ for i in {1.txt,2.txt}; do time parallel --nonall wc -l $i; done
100 1.txt

real    0m0.325s
user    0m0.192s
sys     0m0.042s
10000012 2.txt

real    0m0.305s
user    0m0.220s
sys     0m0.043s
3
  • Walk through the tutorial, then you will have a much better understanding of what GNU Parallel can do: gnu.org/software/parallel/parallel_tutorial.html
    – Ole Tange
    Commented Jan 19, 2014 at 12:39
  • 2
    Aside from your improper usage as noted by others, running jobs like this in parallel via any method is not likely to produce results you want. For starters you need jobs that take longer to run than the overhead introduced by parallelizing them.
    – casey
    Commented Jan 19, 2014 at 14:28
  • GNU Parallel is a 7820 line perl script. 0.3 seconds of startup overhead sounds reasonable. Commented Aug 28, 2015 at 3:58

2 Answers 2

5

In your case you're calling it from a for loop so you aren't really running anything in parallel. All you're doing is adding the overhead of calling parallel in the second example, but it's still only running the files through in a single fashion.

Example

This might help you see what's going on.

without parallel

$ time for i in {1..2}; do sleep 2;done

real    0m4.004s
user    0m0.001s
sys 0m0.002s

with parallel

$ time for i in {1..2}; do parallel "sleep 2" < /dev/null;done

real    0m4.574s
user    0m0.245s
sys 0m0.089s

An alternative

You could call parallel like this instead.

$ time parallel --gnu time wc -l ::: 1.txt 2.txt 

real    0m0.007s
user    0m0.001s
sys 0m0.000s
1000 1.txt

real    0m0.003s
user    0m0.000s
sys 0m0.001s
1000 2.txt

real    0m0.207s
user    0m0.120s
sys 0m0.052s

Here we can see there is overhead in having to call `parallel with the 3rd time grouping showing the "overall" amount of time taken to run the entire parallel command.

References

2
  • @user13107 You should pose questions as questions and not as comments, not many people apart from are going to read your comment-question.
    – Timo
    Commented Jan 19, 2014 at 8:34
  • @Timo corrected
    – user13107
    Commented Jan 19, 2014 at 9:02
1

Thanks to this post, I understood why I was loosing time with parallel on a similar job. I hope my findings help more people!

In this example, I use both for and parallel to: - do 100000 operations to the power of 10 - do 10 operations to the power of 1000000

Results clearly confirm what Casey mentions: "you need jobs that take longer to run than the overhead introduced by parallelizing them". The for loop has the upper hand for the 100000 small operations, while parallel has the upperhand for the ^1000000

$ time for i in $(seq 100000); do echo "$i^10"|bc>/dev/null; done
real    1m19.859s
user    0m4.788s
sys     0m24.204s

$ time seq 100000|parallel "echo {}^10|bc>/dev/null"
real    2m31.269s
user    1m43.833s
sys     1m40.089s

$ time for i in $(seq 10); do echo "$i^1000000"|bc>/dev/null; done
real    1m54.729s
user    1m54.690s                                                                                                                                                                                                                                                                           
sys     0m0.023s                                                                                                                                                                                                                                                                            

$ time seq 10|parallel "echo {}^1000000|bc>/dev/null"
real    0m27.950s                                                                                                                                                                                                                                                                           
user    2m28.476s                                                                                                                                                                                                                                                                           
sys     0m0.047s 

As you can see, the user time is much higher using parallel (overhead)... but the real time gets much better, effectively getting the results faster (by a factor of 4)

These are not the results of extensive testing, but might help you understand the kind of operations where the use of parallel will be beneficial.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .