Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm running A file | B --params > file.txt. Since I want to accelerate the processing speed, I used the parallel -j+0< a.txt to run 20 jobs concurrently. a.txt contains all the commands:

A file1 | B --params > file1.txt 

A file2 | B --params > file2.txt

A fileN | B --params > fileN.txt

Is this way safe? Will the stdout from different programs be messed when running in parallel?

share|improve this question
    
Safe from what? What kind of mess-up do you fear? Since all of these commands are independent, why would they interact? –  Gilles Jun 10 at 23:16

3 Answers 3

Will the stdout from different programs be messed when running in parallel?

Not if they are all independent processes writing to separate files, which they appear to be -- each instance of B is distinct, and outputs to its own place.

share|improve this answer

As long as each instance of B writes to different files, you will be OK. However, the safety of this approach ultimately depends on B.

For example, I have been bitten by poorly-coded programs in the past, when attempting to do something similar to your solution. In my case, the program used srand(time(NULL)) and then wrote to a rand()-based temporary file. This broke horribly when 10 instances of the program were started at the same time, since they all wrote to the same temporary file.

As long as your program does not contain such a bug, and does not write to some fixed path, you should be OK.

share|improve this answer

The only situation where this is not safe is if A or B write to a resource that is hard coded in the program (e.g. /etc/passwd).

But for the vast majority of programs that is not the case. So it will be safe to run them in parallel.

share|improve this answer
    
And even that is a bug in A or B, not the script. (Should have locked the file) –  MSalters Jun 10 at 21:25
    
@MSalters OP is interested in whether it works, not who to blame if it doesn't. –  immibis Jun 11 at 6:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.