5

I am trying to write a script that has the purpose to parallelize an execution (a program that creates some files) running the processes in background and, when all commands in the for loop are done, will perform an extra command (namely move all produced files in another folder). This is what I came out with for the moment:

#!/bin/bash

for f in ./lqns/*.lqn
do
    java -jar DiffLQN.jar $f &
done
mv ./lqns/*.csv csvs

The parallelism works, but they never reach the mv line and the terminal waits and doesn't return. Why is it not returning? How do I fix this?

Maybe the problem is the & of the final for instance? Because it waits for another command but there's nothing more? Even if adding the mv line I thought would have solved it...

4
  • 1
    Does it not run, or does it run immediately, before the (presumably) output csv files are generated?
    – terdon
    Nov 25 '21 at 11:08
  • The csvs files are created correctly in the lqns folder. I wish the execution to reach the mv line and then return but it simply stays on hold without moving the files in the csvs folder.
    – Robb1
    Nov 25 '21 at 11:17
  • 1
    Yes, but are the csv files created by the java command? The thing is that your mv will run almost immediately: since all your processes are sent to the background, the loop will exit in milliseconds and the mv will be run immediately. So if the csv files are generated by the java commands, the mv won't find anything to copy. Is that what is happening?
    – terdon
    Nov 25 '21 at 11:22
  • Oh you are right! I didn't think about that. Yes, the java command generates the csv files but the mv command doesn't do anything... So that is definitely the explanation. How do I ensure that the mv command is run only after all java commands finished the execution?
    – Robb1
    Nov 25 '21 at 11:33
8

If the csv files are generated by the java command, this will fail because the mv will run before any files have been generated. Since all java processes are sent to the background, the loop will finish almost immediately, so the script continues to the mv which finds no files to move and so does nothing.

A simple solution is to use wait. From help wait (in a bash shell):

$ help wait 
wait: wait [-fn] [-p var] [id ...]
    Wait for job completion and return exit status.
    
    Waits for each process identified by an ID, which may be a process ID or a
    job specification, and reports its termination status.  If ID is not
    given, waits for all currently active child processes, and the return
    status is zero.  If ID is a job specification, waits for all processes
    in that job's pipeline.

The relevant bit here is "If ID is not given, waits for all currently active child processes". This means that you can just add wait after your loop and that will make the script wait until all child processes are finished before continuing:

#!/bin/bash

for f in ./lqns/*.lqn
do
    java -jar DiffLQN.jar "$f" &
done

wait

mv ./lqns/*.csv csvs

Alternatively, you can combine the java and mv commands:

#!/bin/bash

for f in ./lqns/*.lqn
do
    java -jar DiffLQN.jar "$f" && mv /lqns/*.csv csvs &
done

Another, possibly better, option is to use GNU parallel (which should be in the repositories of whatever operating system you are running), a tool designed for precisely this sort of thing. With it, you could do:

parallel java -jar DiffLQN.jar ::: ./lqns/*.lqn
mv ./lqns/*.csv csvs
2
  • Adding wait worked perfectly! Thank you very much. Just a curiosity: why do you think the implementation with parallel could be better?
    – Robb1
    Nov 25 '21 at 11:57
  • 3
    @Robb1 because it is an incredibly powerful and versatile tool. You can tell it to only run N processes at a time, and even pass it a file with the number of processes to run in parallel and changing the number in the file lets you change the number running without stopping and restarting. It has great tools for collecting the files you want as input (note how your whole script became just two commands), it can spread load across multiple computers on the same network, it's just a much better and more powerful tool.
    – terdon
    Nov 25 '21 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.